Robert Elder's Guide To GNU Coreutils

2024-07-21 - By Robert Elder

The GNU Coreutils Package

     Over the last 18 months, I've taken the time to create 108 short YouTube videos that summarize all of the commands that are found in the Coreutils Package.  These videos focus mostly on Coreutils v8.3, but also included are a few commands from v9.0 as well.  For each of the blog posts, I've adapted and modified the script slightly to make it more appropriate for written format.  These videos and articles are intended to provide a quick reference and at least one useful example of why you'd want to use each command, rather than a detailed encyclopedic reference.  You can find all of the videos/blog posts linked below:

What Is This?

     At the top of this blog post, you'll find links to a blog post and video for all 108 Coreutils commands.  In most cases, the blog post contains approximately the same information as the video, but in some cases I did add a bit more detail or extra references in the article.  For the first 50 commands, I released one video every day, but for the remaining 50 or so, I did one per week so the later ones are a bit higher quality.

     The purpose behind each video is to provide a quick introduction and at least one practical example of why you would even want to use this command in the first place. My expectation is that once you learn that a command exists and is useful, you'll be in a position to do your own research to find more detailed information.

What is GNU Coreutils?

     The 'GNU Coreutils' package is an extremely important set of software tools that are found in basically every GNU/Linux system.  In other words, the commands that are present in 'Coreutils' are the 'core utilities' that required to perform many of basic the tasks on your system, like creating files, or changing permissions.  Many of the GNU Coreutils commands are part of the POSIX specification.  POSIX stands for 'Portable Operating System Interface', which means that many of the commands that belong to the POSIX specification are also present on other systems such as macOS and in some cases Windows (although, often with subtle differences).  Therefore, learning about the commands in 'GNU Coreutils' is likely to provide knowledge that is mostly transferable to many other platforms as well.

     The Coreutils package also includes a number of other commands that are not included in the POSIX specification.  A few examples of this are the 'factor' command which finds the prime factors of large numbers, or the 'ptx' command for creating permuted indexes. Many of the commands that belong to Coreutils are not necessarily included because they're particularly well designed or efficient, but rather because they are extremely backward-compatible and likely have historical significance. In some cases this history can date back as much as 60 years.

     The GNU Coreutils package is one of the most widely used implementations of 'Coreutils', however there are several other implementations, such as busybox, a single-executable minimalist implementation that's designed for use in embedded systems, bsdutils a collection of coreutils-like utilities from the FreeBSD source code tree, and also uutils coreutils, a Rust-based reimplementation of GNU Coreutils.

History Of Coreutils

     Many of the commands in Coreutils can be traced back to the first version of unix:

     To name a few them this includes cat, chmod, chown, cp, date, df, du, ln, ls, mkdir, mv, od, pr, rm, rmdir, stat, su, sum, tty, wc, and who.  If you want to see a more detailed comparison, you can watch a video where I review how many of the original V1 Unix commands are still present in modern Ubuntu:

     This close connection between modern Linux and early Unix shows up in more places than you might realize. For example, in man pages:

     Quite a few of the commands that are part of Coreutils have a surprisingly rich history that isn't well documented, such as the printenv command which was written by Bill Joy in 1979 as a part of the Berkeley Software Distribution.

     Another surprisingly interesting command is the pinky command.  On the surface, reading the documentation on what the 'pinky' command does makes it seem to have little value on a modern Linux machine. However, upon digging into the history of 'pinky', you'll find that it's based on the historically very important 'finger' command.  The 'finger' command was written around 1971 by Les Earnest, who is himself quite an interesting character.  In 1988, the finger protocol was subject to the infamous Morris Worm, which was the first computer worm to receive mainstream attention.

     The 'chroot' command is yet another example of a historically significant command that is still used today.  It could be argued that 'chroot' is effectively the earliest form of software 'containerization' that is ubiquitous today in other forms (such as Docker or LXD containers).

     A few other commands that have a surprisingly rich history are the link, and unlink commands.  The 'link' command can be used to create a hard link to a file, and the 'unlink' command can be used to delete a single file (or a hard link).  You might wonder why these two commands exist at all, since their functions can already be done by the 'ln' or 'rm' commands respectively.  The reason for their existence can be explained by exploring their surprisingly long history.

     In fact, references to the 'link' and 'unlink' commands can be traced back as early as 1964 in the Multics Design Notebook Documents, which was 5 years before development of the Unix operating system had even started:

Before GNU Coreutils

     If you check the git repo for GNU Coreutils, you can see that the first commit is from October 31, 1992:

git log 14fd34b78818660e05806b6eda178e3f846c5c21

commit 14fd34b78818660e05806b6eda178e3f846c5c21
Author: Jim Meyering <jim@meyering.net>
Date:   Sat Oct 31 20:42:48 1992 +0000

    Initial revision

     According to this page on GNU shell utils:

     "Fileutils, Shellutils, and Textutils have been combined into the GNU Coreutils package. All further development and discussion is now taking place as Coreutils. The last separate versions were fileutils-4.1.11, textutils-2.1, and sh-utils-2.0.15. The first major release of coreutils-5.0 was announced on Fri, 4 April 2003."

     Therefore, any historical research into the many core utilities should also consider a review of the pre-2004 archives for file utils, shell utils and text utils.

Core Utilities & Shell Built-ins

     Since some of these 'core' utilities are so important, a few of them are not only found in Coreutils, but some are built directly into the shell itself. This can become a source of confusion.  For example, the behaviour of the built-in 'printf' command may not always be identical to the behaviour the executable '/usr/bin/printf':

Is This AI?

     A few people have made comments or asked questions implying that these shorts videos are all AI generated. Most of these people are probably joking, but I think that of few of them are serious. Rest assured, that all of these videos and blog posts are 100% human-made by me, and I've used 0% AI in making these. Out of curiosity, I did test out using ChatGPT to see how well it would do in writing a script for one of these videos, and I was very unimpressed by the performance. For well-known commands that have tons of existing documentation (ls, mv, rm etc.), ChatGPT provides a very bland 'average' kind of script that is slightly worse than the average google search result. For less common commands that don't have a lot of existing documentation (like the 'ptx' command), the generated scripts are completely wrong non-sense. In the case of 'ptx', the script that ChatGPT provided me seemed to confuse 'ptx' with the 'ffmpeg' command and overall consisted of pure hallucinated gibberish.

Why did I Work On This?

     I mainly did this as an experiment to test out YouTube shorts and see if it could grow my channel. In hindsight, releasing so many of these videos definitely did get me a lot of new subscribers, but these videos also took quite a bit of time to produce and they generate almost no revenue. Financially speaking, I think I would have better off to release them as non-shorts, although I do think that the shorts probably do bring a wider and younger demographic.

     Additionally, this was also an opportunity for me to deepen my own knowledge of the 'core' utilities that exist in Linux. In particular, I found the mkfifo and mknod commands to be interesting new ones for me.

Which One Is Your Favourite?

     I actually talked about that in my most recent YouTube video.