I want to be clear on my bias here: I firmly believe that open source would not be a ‘thing’ if it weren’t for Red Hat. Linus Torvalds himself once said (albeit 10 years ago) that the shares he received from Red Hat before their IPO was ‘his only big Linux payout’. I don’t think anyone would disagree with the statement that Red Hat has had a major significant positive impact on Open Source across the world.
This morning I listened to an excellent podcast called “Ask Noah” where he interviewed Red Hat’s Mike McGrath who has been active on the linux subreddit and other social media. It seems that Mike has been involved in the decision to restrict Red Hat’s sources on git.centos.org:
https://podcast.asknoahshow.com/343 (listen at ~20 mins)
It’s really worth a listen. Mike clearly lays out the work that Red Hat (I was surprised to find out that it is NOT the Rebuilders) does to debrand the Red Hat sources, why they’re pulling that back on those unbranded sources, and that they understand the ramifications of doing so. It’s also interesting that Mike is of the opinion that there is nothing wrong with doing a Rebuild, and he defends them by stating “that’s the cost of doing business”. Noah and Mike go into many of the nuances of the decision and again, it’s really worth listening to. Mike also talks about “bad faith” when dealing with the Rebuilders at 40:30, which I think explains Red Hat’s decision. I got the distinct feeling he’s bound by some ethical code so he won’t/can’t say too much though.
There’s also this discussion about Rocky Linux securing a contract with NASA:
https://news.ycombinator.com/item?id=36417968
that had a lot of internal discussion at my company this week, which given what’s just happened may shed some more light on Red Hat’s decision.
There are always two sides to every story but in this case there are three sides to this story.
On one side, you have Red Hat, a long time champion of open source software, that has poured billions of dollars into open source development, and which has 1000s of employees who not only on ‘company’ time but in their own time manage, develop, contribute, and create open source code. They have funded countless successful and unsuccessful projects that we all use.
Against Red Hat are two largely distinct groups. The first is the Rebuilders themselves, who Red Hat has claimed ‘don’t offer anything of value back to the community’. This is not meant to be a statement on the usefulness of the rebuilds (Rocky, Alma, Oracle, etc.) but rather a very directed statement on whether or not the rebuilders are providing bug report, feedback, and contributions to the packages that Red Hat has included in RHEL.
The second group, which stands somewhat behind the Rebuilders, are the Rebuild users. One could argue that the users are caught in the middle of Red Hat and the Rebuilders, however, I think it is better to look at them as being an equal ‘side’ in this discussion.
The Rebuild users are in a very unfortunate position: they’re about to lose access to a free product that they’ve come to depend on. They are, as expected, unhappy about Red Hat’s decision to stop providing access to RHEL sources. My next statement is callous, and I expect it to be read as such: You get what you paid for. That is not meant to indicate anyone is cheap, it’s just that you shouldn’t have expectations when you are using something for free.
Here’s the interesting part for me. As far as I can see, none of the users are jumping to the Rebuilder’s defence of Red Hat’s accusation that the Rebuilders provide nothing back to the community. And, as far as I can tell across various social media and news platforms’ comments sections, largely the user community AGREES with Red Hat’s position. Informed users – not all users – are using a RHEL Rebuild knowing that there is no benefit in doing so for the community.
I have yet to read a reply from the Rebuilders where they categorically deny that this is the case. And to me, that’s glaring and damning of the Rebuilders’ position. Even the ‘defenders’ (for lack of a better word) of the Rebuilders have yet to provide a response.
deleted by creator
Accusation that the Rebuilders provide nothing back to the community.
Actually, what Redhat are saying about rebuilders is that they “don’t add value” - and that’s for Redhat, NOT to the community which they patently do. That’s quite a badly twisted misquote there, friend.
Also, Redhat didn’t create open source software. They’re a big player, sure, but I remember writing and releasing my code back in the 80s and 90s when it was called Freeware and Public Domain and distributed on cassette tape.
One other thing I want to add: I’ve read a bunch of comments about how the Rebuilds were used in educational and scientific settings, and that there is a prohibitive cost for RHEL in those environments. After reading so many comments about it, I have to believe that Red Hat is going to make some modification to their Developer License program to allow more than 16 ‘seats’ for those use cases.
I don’t have any expectations of them doing this (but I also have no expectations to the contrary), but I think it would be a good move from Red Hat to make the official RHEL more available, as you suggest.
In another thread I compared the RHEL rebuilds to piracy, and in that vein one could quote Gabe Newell and say that piracy is a service problem – part of the reason Alma/Rocky/etc. exist is because there is a group of users who want to use RHEL but cannot afford it. Red Hat seems to believe that these users should be satisfied with CentOS Stream, and maybe most of them would be, if they only gave it a try. But making RHEL more widely accessible, both to paying users and developers, would probably be good too.
Yes, from what I’ve heard they are raising to just over 200 (iirc there was already an agreement for this but the caveat being the type of services the systems ran) which still doesn’t cover many educational and academic research scenarios. We’d only be covered about 30% and we operate a comparatively small environment.
I have not listened to the podcast unfortunately.
Rebuilders are fine, and RedHat is fine to not spend the effort to debrand their source rpms. The problem is one of value. The value RedHat provides for some people is probably worth more than RedHat charges. The value RedHat provides to others is less than the effort it takes to renew a developer license once a year for 16 installs. The problem is that there are several who are ending support for RHEL because they fall into the latter group (notably Jeff Geerling for ansible roles). RHEL losing out on that support might be huge, might not, only time will tell.
My company runs thousands of centos VMs. We cannot exist if we have to license rhel. We’ve been working on switching to Alma. We may have to look elsewhere for a free distro that has robust SeLinux support.
Can’t you keep using CentOS stream? Isn’t it still a very stable distribution? Just slightly upstream of RHEL instead of downstream.
Nice post, and a good overview over why RedHat is doing what it’s doing.
Before reading this I wasn’t really feeling good about redhat and the stuff happening rn but now i’m able to understand the decision making and there’s still hope for me that redhat won’t turn into a shitshow in a couple years haha
Also working with RedHat in the past has been quite nice so it’s good that i don’t feel a slight hate against the company anymore.
Quite hard to solve the problem when everyone is so emotional
Thanks again for the very informative post!
I have one major quibble with your analysis. It is this: Redhat no longer exists as an organization. Redhat is merely a trademark of IBM. You can’t defend IBM’s actions based on Redhat’s history. That was a different company, and a different era.
What’s the harm in doing a rebuild? Serious question. I simply don’t understand where the harm comes from. I would appreciate any insight. Thanks.
I don’t think there is anyone arguing that a Rebuild by itself is a problem. Given Mike’s comments in the podcast linked above, the problem Red Hat has is when one of those (or many of those) competed directly against Red Hat for a contract.
From the general feeling I get from reading many threads on this issue, the general consensus is that the community agrees that, specifically, this behavior by the Rebuilders is wrong.
Oh, I see. But what do you think of this translation:
“Company Foo makes TVs and is always working to make them better. They give them out for free with the hopes of making money installing them and providing guidance on how to use them, but someone starts Company Bar and installs them for cheaper and starts taking on installation jobs.”
Is this wrong? Isn’t this just the definition of an open market? Please let me know if I’m missing some kind of context. I hope that we can continue to discuss this respectfully.
I should say that I want any open source project with the motivation to write good software to have all of the funding they need to make that happen. I just don’t see how it can be justified in this instance when compared to any other market.
There is no problem with your scenario, and it’s spot on to the issue that Red Hat has raised.
However, the piece you’re missing is that the TVs come from Foo. They don’t have to give company Bar TVs to install. If company Bar doesn’t have TVs then what should they do? They have some choices: work with Foo or develop their own TV.
I don’t see how Company Foo can dictate that all other entities (customers, for example) can receive a free TV on their doorstep (since the code is open source) except for Company Bar. To make it map better to the situation, Company Bar would receive a shipment of free TVs, rebrand them, ship them out to customers, and install them.
“They don’t have to give Company Bar TVs to install.” So the GPL doesn’t require that Company Foo permit free access to the TVs? They could decide to not give out their TVs to anyone?
Also, what if I wanted to get my cousin a free TV but charge him a few bucks to install it? Is this only a problem at scale?
Here’s where your analogy falls apart. The TV isn’t being shipped to everyone. It’s being shipped (“rebuilt”) by Bar, and then installed by them. They’re free to do that but Foo is under no obligation to help them do it.
Within the analogy (as it compares to Redhat and the Rebuilders), how is Foo helping Bar? Isn’t Foo simply leaving the TVs outside the factory for people to come and pickup? A bunch of trucks branded “Bar” come by, pick some of them up, rebrand them, and take jobs to install them, jobs that Foo thought they were going to get? Isn’t Foo now requiring individual people to walk through a lockable door, sign their name, verify that they don’t work for Bar, and grab a TV instead of just leaving them outside in a pile?
Yes, that kind of makes sense, but Foo was leaving the TVs outside because they thought that was the most expedient thing to do. It takes effort to move them outside, and Foo doesn’t want to do that anymore. So now Foo, as you point out, has moved the TVs inside where only paying customers can get them.
It seems that he is bother by how they rebuild it and then do not add or contribute any code and then sell support to the customer on REHL work which in my opinion its not okay and I will agree with RedHat.
Thanks for a solid assessment of the situation and providing some sources 👍
I might be wrong but AFAIK the only other big company that’s contributing to Fedora is Facebook/Meta. If Oracle etc. were also contributing to Fedora, my gut feeling is that Red Hat would not be so pissed about rebuilding RHEL because RHEL would also be benefiting from Oracle’s Fedora contributions.
So joke aside, I don’t see anything in that video that is a defence of the Rebuilders against the accusations made by Red Hat. Is there something I was supposed to get out of watching it?
So i guess ultimately, no, it doesnt counter the claim that rebuilders don’t contribute upstream. Ultimately though, I see this as a diversion. It is counter to the way the opensource community has traditionally operated.
So no, it is not a defense of the rebuilders except to say that the rebuilders have done what has been the norm for decades. Opensource is full of forks and derivitave projects.
If I listen to that video will I, in fact, get a laptop for free? Inquiring minds wanna know.
Was Almalinux and Alpine charging customers for support in their builds?
Alpine is completely separate by RHEL by a country mile (hell, it doesn’t even use glibc). You’re probably thinking of Rocky
I don’t think Mike McGrath called out any specific company but if you look at that ycombinator link it looks like the ‘offender’ was Rocky Linux. That is purely speculation on my part.
Well if thats the case thats really bad in my opinion , I might side with Redhat on this one.
