20 Years of Gentoo

Prehistory: 1980’s-2001

I grew up using DOS in the 80’s and 90’s. Even after Windows 95 came out, I continued to boot to the DOS command prompt. One did, after all, need to play games and in those days Windows consumed too many resources to make some games playable on my 486.

Microsoft eventually forced my hand and I was forced to live with Windows. While useful for web browsing, I missed writing emails in text mode, and I really missed Norton Commander. No file manager on Windows made me as efficient as Norton Commander did. [1]

Compounding those headaches was the proliferation of adware/spyware on Windows. It was routine to install software just to flush these out of your system. And we all remember the pain of “It’s been a year since I installed Windows and is now much slower than when I installed it. Let me reinstall it”.

Mandrake Linux: 2001

In 2001, I bought a second hard drive for my PC. Armed with more space, experimenting with another operating system became less risky. I could install Linux on the other drive without worrying about any harm coming to my Windows OS. [2]

Which Linux to install? I had heard of Red Hat, but the Internet suggested Mandrake. It was supposedly “compatible” with Red Hat [3], and a lot more user friendly, without compromising on power. And of course, it was free.

Being on dialup, downloading the ISOs for the CDs was a non-option. A kind grad student friend of mine had an office with a CD burner. He created the CD for me. I also bought an O’Reilly book on Linux.

The installation was a breeze. And I was astounded at the result. Whereas Windows came with very little software, Mandrake came packed with a ton. Not just one web browser, but several. Support for several languages and compilers. Multiple text editors. Multiple file managers. Even multiple “office” suites. And LaTeX! And Gimp! And a decent MATLAB alternative! [4]. And a good music player! And, and, and…

What’s more: There were no strings attached! These were not trial versions. They were not handicapped versions. I did not have to pay anyone to get the full version. I did not have to watch ads to get them to work.

Once again I could live in text mode for emails and other tasks. Instead of Norton Commander, they had Midnight Commander.

And package management! What a concept! No more hunting the web to find software, and worrying if you’re getting the official one or an ad-laden version. Just tell Mandrake what you’d like to install, and it would download and install for you!

What more could one want?

Discontent: 2001-2003

After installing Mandrake, I alternated between Windows and Linux - spending a few weeks at a time in each. Life was good - for a while. But alas, little frustrations began to bubble up.

Occasionally a package would not function well. The Internet told me the solution would be to download an rpm and manually install it. But many rpm’s did not work - they expected a different directory structure from the one Mandrake provided. I lost a lot of time hunting for a compatible rpm. Isn’t this the problem package managers were supposed to solve?

Or I would install the package from source. I chanted the mantra of ./configure && make && make install. A bit of a pain, but manageable. However, I now had to “manage” these installations manually. I learned what “dependency hell” meant. Over and over again. If I installed something manually, then the package manager would not know about it. It would complain the library I had installed didn’t exist. And would try to install what it thought was the right one - clobbering my work. All. Too. Often.

There was a more serious problem: Remember Windows getting slow after a year or so? I was paranoid that Mandrake was doing the same. There were so many packages installed on my system. And so many services running all the time. Were they all needed? Were they eating up precious CPU power? I was too scared to uninstall or shut down services.

Once again, I did not feel in control of what was on my computer!

So I searched for solutions online. Could I not get a bare minimum distribution, and install just what I need? A friend suggested Debian. It seemed too hard core and had a reputation for being beginner hostile. Anything else?

Why yes! Linux From Scratch! Everything is installed from the very bare minimum. You have to compile all the sources for every little thing you want. This way you can configure your system to your needs, and no more! I removed Mandrake from my system and got to work on LFS.

LFS is not a trivial install. I needed to dedicate a few days for it. But the sales pitch was that one will learn a lot about how Linux works. So I put in the time in 2002 and got a bootable system.

The system was really bare. OK - now for the job of getting a graphical server working, building the Mozilla browser, and everything else I wanted. They had a guide for that called Beyond Linux From Scratch.

It wasn’t long before I decided this was not sustainable. There was no package management. You were the package manager. You have to resolve the dependencies manually. It was good for learning, but figuring out the dependencies every time you want to upgrade a package would be too time consuming. Can’t someone automate all this?

Gentoo: 2003-current

During the beginning of the summer break in 2003, I got Gentoo and did a Stage 0 install on a Pentium 4 2.53GHz machine. [5] I did not even have a high speed Internet connection. It worked like a charm! I kept the emerge.log file from the first machine, so I can tell you how long things took to compile in those days, if anyone is interested!

So what is Gentoo? Like LFS, it compiles everything from source. Unlike LFS, it comes with a pretty good package manager which will automatically calculate dependencies, download, and compile for you. It is very maintainable compared to LFS, which is why I still use it.

You still ended up with a bare minimal install. You still had to configure your network, your graphics server, etc. But fortunately, you did not have to deal with dependency Hell. What’s more, Gentoo had (and still has) fantastic documentation.

One other thing that struck me about Gentoo: Its rolling releases and the lack of versions. People in the Windows/MacOS world think in terms of versions all the time: Windows XP, Vista, 7, 8, 10 and so on. With Gentoo, you never upgrade to a newer version. You merely keep upgrading packages on your system as they become available. That’s why I went 7 years without having to reinstall any OS, and why my emerge.log goes that far back. Rolling releases were not the norm in those days.

Source Based

As far as I know, it is still the only viable distribution that is source based. If you are into pseudo-minimalism, building from source is a good approach.

I say psuedo-minimalism because I get the sense that people will read this and think my PC environment is a very austere one. In reality, you will not be able to distinguish it from any other distribution. I have a fully graphical environment with all the bells and whistles. The important thing is it has only the bells and whistles I want. [6]

Furthermore, having things source based really helps with custom installs. I still occasionally need to Google for a solution to some problem that requires me to rebuild my package with a patch that has not made it into the Gentoo repository. While I’m ashamed to admit I never learned how to write ebuilds from scratch, it is easy to take an existing one and modify it to include the patch. The bonus is the new modified install is fully recognized by the package manager. I have no idea how binary based distributions fare on this metric.

Learning

It is still a great distribution for learning about Linux. You still have to set things up and configure them. You still have to compile the kernel for features some of your packages may need. You still get the “joy” of configuring the bootloader.

If you have time on your hand and want to learn, this may still be the best distribution for you. Unlike LFS, you will have no need or desire to replace it with something else once you have learned it. I think it is ideal for students in STEM fields.

USE Flags

This is the killer feature of Gentoo. USE flags are a convenient way to specify what features you want in a package. Consider a somewhat contrived example: I do not own an iPhone, and my PC has no Bluetooth capability. I can configure my system not to install iPhone/Bluetooth related features when installing packages. Suppose I’m installing a music player. It may have options to sync/connect with iTunes. With my setting, it will install without those features.

No bloat!

You can do this systemwide or per-package.

I used to make it a point to understand all the various USE flags out there. Now, to be honest, I mostly stick to defaults, making modifications only as needed. I’m not as obsessed on being lean as I used to be.

Again, I do not know if any binary based distribution handles this feature well (or at all). I cannot imagine life without it.

Init System

One thing I am forever grateful for: You don’t need systemd.

The Gentoo Community

In the early days, RTFM was the norm in the Linux world, giving it a reputation for harshness. The Gentoo forums, in contrast, was an incredibly friendly place for beginners.

Debian, on the other hand, had a reputation for being nasty to beginner questions.

Somehow, all this lead to a long thread of concern on the Debian mailing list: “Are we losing users to Gentoo?” You can tell from the original post that they did not realize the reason wasn’t just “cool”, but also “friendly”. I mean, consider this response:

Hell yes, and it’s great. The number of morons using Debian has noticably decreased since gentoo came on the scene; they now have something that will give them the stupid things they asked for, so they stop asking us for them.

Someone finally got part of it:

the problem is, the docs are better and niftier and have more colors, that’s, according to a friend, one of the things why he used gentoo.

If you’re interested, here is the thread on the Gentoo forums discussing the same thing.

Antireason: Performance

In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.

In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.

Slow Package Manager

Portage, the Gentoo package manager, is s l o w. It is written in Python, and the dependency graph must be much bigger than in the early days. I am surprised Gentoo has not built an official faster replacement.

Dated Repository

Packages in the official repository are not updated as often as I’d like. For popular packages, you can find them in the tree soon enough, marked as “unstable”. However, it can take a long time to get to stable. As of this writing the latest version in the tree for TeX Live is 2021 - both for stable and unstable. That’s 2 years old.

The latest stable version of GHC is 9.0.2 - released on 25th December 2021. Over a year old.

In the early days, Gentoo was known for being very fast at stabilizing new releases. You can even find posts about it in that Debian thread I link to later. Now it is probably one of the slower distributions in that regard. I don’t think this will ever get better without more people actively using Gentoo and contributing.

In the old days I would take the risk of installing unstable packages, but that comes with dependency problems and a higher maintenance burden. I do it only as needed.

Dependency Hell

Wait, wasn’t not having dependency hell supposed to be one of the perks of Gentoo?!

For the most part, yes. But Gentoo is also one of the most flexible distributions around. And with great flexibility comes great headaches. Portage manages most of those headaches well, but things do fall through the cracks.

If you have a modern desktop system, with lots and lots of packages installed, you simply cannot avoid some dependency pains. On my previous computer, any time I upgraded QT to a new major version, there was hell to deal with. Too many circular dependencies that Portage could not resolve. The solution would usually be to uninstall all qt related packages, and then upgrade.

I update packages once a month. I can easily say that over half of the months I need to deal with a nontrivial dependency issue manually - Portage just doesn’t handle them. Some of this may be due to my liberal use of USE flags, which I’m minimizing on my most recent PC. But some of it is unavoidable.

Broken System

Every once in a while you upgrade a major package, and you misconfigure the files, and the system breaks. Perhaps network capability is lost. Or the XOrg server won’t load. Or you can’t even log in.

These are not fun. You cannot use your PC until you resolve this problem. You have a life to live. How much of your time is debugging this going to eat up?

The worst example of this was when I had to do a nontrivial upgrade to udev. After the upgrade and reboot I could not even get a shell prompt. Unfortunately, this happened just as I was moving to another city for a new job. I simply could not spend time debugging this.

Great: A major move coming up, and I don’t even have a computer! I did not have a smartphone either. Thank God (and taxpayers) for Internet access in libraries!

I think about 6 weeks went by before I fixed it. Debugging wasn’t easy. I knew nothing of udev, and did not find people on the Internet who had the same problem. Ultimately, it was a simple fix. I strongly recommend everyone to have a copy of SystemRescueCD. That was the first time I used it, and have occasionally needed it since.

These kinds of breakages are not that common. Once every 1.5-2 years or so. Most of the time I resolve it within a day or two. Still, I would never use Gentoo for professional work. Imagine trying to explain to your boss that you can’t do any work because you broke a udev upgrade.

Pathological Followers?

I wonder if Gentoo is more prone to attracting unhinged folks?

The only person I have ever converted to Gentoo is now spending a 30+ year sentence in federal prison.

Here’s a mass shooter who was also a Gentoo user:

An Oklahoma resident and software engineer Ariadne Conill filed complaints against Smith with the FBI after receiving online death threats from him starting in October 2006 and lasting through March 2007, Conill alleged Tuesday.

Smith had lashed out at Conill and other software engineers after he discovered that the makers of Gentoo, a computer operating system he was using, removed a software package that he used to play music on his computer and had switched to a different system, Conill told The Oregonian/OregonLive.

Smith started to make random demands that the old system be restored and then started issuing direct threats and graphic death threats online, according to Conill. He wrote that he was going to go on a road trip to Oklahoma and “when you step outside I’m going to stab you” or he would send “pictures of guns and knives and stuff and say he’s going to come to our houses,” Conill recalled.

Conill said the FBI never responded other than noting that the complaints had been received after they were filed online with the FBI’s Internet Crime Complaint Center.

Gentoo Documentation

In the early years, Gentoo was known for having superb documentation. Often when I would tell people I ran Gentoo, they would relate a time they were stuck in their non-Gentoo distribution, but found the solution to their problems in the Gentoo docs.

The documentation is still good, but at some point Ubuntu became the resource with the best documentation. I suspect Arch Linux probably holds the title now.

Statistics

A user asked for some statistics. They are below. I think the columns are compilation counts, cumulative hours, average compile time (in minutes). And then repeated for uninstalls.

Compilation times on the first computer (P4 2.53GHz - single core):

2003 Total  1060   76:27:40  4:19   363    25:14  4
2004 Total  1295  117:37:11  5:26   976  1:15:25  4
2005 Total  2149  167:00:38  4:39  1247  3:05:21  8
2006 Total  2438  207:15:20  5:06  1155  2:47:45  8
2007 Total  1668  178:35:20  6:25  1151  1:58:42  6
2008 Total  3114  256:59:51  4:57  1061  1:52:08  6
2009 Total  2359  215:12:56  5:28  1755  2:31:28  5
2010 Total   575   58:57:37  6:09   549    34:52  3

And specifically for gcc:

2003 Total  7  3:42:36    31:48  3  1:03    21
2004 Total  4  2:00:11    30:02  3    31    10
2005 Total  7  2:51:31    24:30  3  3:46  1:15
2006 Total  8  5:22:26    40:18  5  3:02    36
2007 Total  5  4:56:54    59:22  5  2:22    28
2008 Total  1    58:26    58:26  1    48    48
2009 Total  2  2:43:44  1:21:52  1    53    53

For the second computer (quad core Intel - i7-870)

2010 Total  3524  37:39:00    38  2321    37:48  0
2011 Total  2128  37:07:22  1:02  2024    38:14  1
2012 Total  1279  31:56:32  1:29  1153    33:33  1
2013 Total  1913  41:36:12  1:18  1612  1:14:04  2
2014 Total  1667  37:35:29  1:21  1575  1:19:06  3
2015 Total  1973  48:33:21  1:28  1886  1:30:05  2
2016 Total  1025  15:38:31    54    63     2:42  2
2017 Total  3579  55:47:37    56  3000  2:29:06  2
2018 Total   692  16:33:28  1:26   529    27:38  3

And specifically for gcc:

2010 Total  7  1:39:30  14:12  7  16   2
2011 Total  9  2:56:15  19:35  8  19   2
2012 Total  2    44:48  22:24  2   9   4
2013 Total  2    56:00  28:00  0   0   ?
2014 Total  5  2:18:52  27:46  4  15   3
2015 Total  3  1:31:56  30:38  5  58  11
2017 Total  8  3:28:53  26:06  4  12   3
2018 Total  2    54:08  27:04  2   6   3

For my current computer (22 core Xeon(R) CPU E5-2696 v4 @ 2.20GHz):

2017 Total   182     45:33    15     9        7  0
2018 Total  2952  29:41:07    36  1817    51:33  1
2019 Total  1217  27:44:57  1:22  1081    54:22  3
2020 Total  2372  46:54:31  1:11  2554  1:53:59  2
2021 Total  2795  52:49:48  1:08  2793  1:52:58  2
2022 Total  2454  48:06:54  1:10  2287  1:20:19  2
2023 Total  1097  16:00:16    52  1022    32:07  1

Not sure where the 2017 numbers come from, given that I didn’t build it till 2018 - I would ignore 2017.

And specifically for gcc:

2018 Total   5  2:00:38  24:07   5  20  4
2019 Total   7  3:10:32  27:13   5  38  7
2020 Total   9  3:45:21  25:02  10  52  5
2021 Total  11  5:20:21  29:07   9  25  2
2022 Total   5  2:52:23  34:28   5  19  3
2023 Total   6  2:19:33  23:15   4   8  2

If I combine all the emerge.log files, I get:

2003 Total  1060   76:27:40  4:19   363    25:14  4
2004 Total  1295  117:37:11  5:26   976  1:15:25  4
2005 Total  2149  167:00:38  4:39  1247  3:05:21  8
2006 Total  2438  207:15:20  5:06  1155  2:47:45  8
2007 Total  1668  178:35:20  6:25  1151  1:58:42  6
2008 Total  3114  256:59:51  4:57  1061  1:52:08  6
2009 Total  2359  215:12:56  5:28  1755  2:31:28  5
2010 Total  4099   96:36:37  1:24  2870  1:12:40  1
2011 Total  2128   37:07:22  1:02  2024    38:14  1
2012 Total  1279   31:56:32  1:29  1153    33:33  1
2013 Total  1913   41:36:12  1:18  1612  1:14:04  2
2014 Total  1667   37:35:29  1:21  1575  1:19:06  3
2015 Total  1973   48:33:21  1:28  1886  1:30:05  2
2016 Total  1025   15:38:31    54    63     2:42  2
2017 Total  3579   55:47:37    56  3000  2:29:06  2
2018 Total  3826   47:00:08    44  2355  1:19:18  2
2019 Total  1217   27:44:57  1:22  1081    54:22  3
2020 Total  2372   46:54:31  1:11  2554  1:53:59  2
2021 Total  2795   52:49:48  1:08  2793  1:52:58  2
2022 Total  2454   48:06:54  1:10  2287  1:20:19  2
2023 Total  1097   16:00:16    52  1022    32:07  1

And specifically for gcc:

2003 Total   7  3:42:36    31:48   3  1:03    21
2004 Total   4  2:00:11    30:02   3    31    10
2005 Total   7  2:51:31    24:30   3  3:46  1:15
2006 Total   8  5:22:26    40:18   5  3:02    36
2007 Total   5  4:56:54    59:22   5  2:22    28
2008 Total   1    58:26    58:26   1    48    48
2009 Total   2  2:43:44  1:21:52   1    53    53
2010 Total   7  1:39:30    14:12   7    16     2
2011 Total   9  2:56:15    19:35   8    19     2
2012 Total   2    44:48    22:24   2     9     4
2013 Total   2    56:00    28:00   0     0     ?
2014 Total   5  2:18:52    27:46   4    15     3
2015 Total   3  1:31:56    30:38   5    58    11
2017 Total   8  3:28:53    26:06   4    12     3
2018 Total   7  2:54:46    24:58   7    26     3
2019 Total   7  3:10:32    27:13   5    38     7
2020 Total   9  3:45:21    25:02  10    52     5
2021 Total  11  5:20:21    29:07   9    25     2
2022 Total   5  2:52:23    34:28   5    19     3
2023 Total   6  2:19:33    23:15   4     8     2