TOPIC: RM
Grouping directories first in output from ls commands executed in terminal sessions on macOS and Linux
This enquiry began with my seeing directories and files being sorted by alphabetical order without regard for type in macOS Finder. In Windows and Linux file managers, I am accustomed to directories and files being listed in distinct blocks, albeit within the same listings, with the former preceding the latter. What I had missed was that the ls command and its aliases did what I was seeing in macOS Finder, which perhaps is why the operating system and its default apps work like that.
Over to Linux
On the zsh implementation that macOS uses, there is no way to order the output so that directories are listed before files. However, the situation is different on Linux because of the use of GNU tooling. Here, the --group-directories-first switch is available, and I have started to use this on my own Linux systems, web servers as well as workstations. This can be set up in .bashrc or .bash_aliases like the following:
alias ls='ls --color=auto --group-directories-first'
alias ll='ls -lh --color=auto --group-directories-first'
Above, the --color=auto switch adds colour to the output too. Issuing the following command makes the updates available in a terminal session (~ is the shorthand for the home directory below):
source ~/.bashrc
Back to macOS
While that works well on Linux, additional tweaks are needed to implement the same on macOS. Firstly, you have to install Homebrew using this command (you may be asked for your system password to let the process proceed):
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
To make it work, this should be added to the .zshrc file in your home folder:
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
Then, you need to install coreutils for GNU commands like gls (a different name is used to distinguish it from what comes with macOS) and adding dircolors gives you coloured text output as well:
brew install coreutils
brew install dircolors
Once those were in place, I found that adding these lines to the .zshrc file was all that was needed (note those extra g's):
alias ls='gls --color=auto --group-directories-first'
alias ll='gls -lh --color=auto --group-directories-first'
If your experience differs, they may need to be preceded with this line in the same configuration file:
eval "$(dircolors -b)"
The final additions then look like this:
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
eval "$(dircolors -b)"
alias ls='gls --color=auto --group-directories-first'
alias ll='gls -lh --color=auto --group-directories-first'
Following those, issuing this command will make the settings available in your terminal session:
source ~/.zshrc
Closing Remarks
In summary, you have learned how to list directories before files, and not intermingled as is the default situation. For me, this discovery was educational and adds some extra user-friendliness that was not there before the tweaks. While we may be considering two operating systems and two different shells (bash and zsh), there is enough crossover to make terminal directory and file listing operations function consistently regardless of where you are working.
Streamlining text case conversion across multiple apps with a reusable macOS terminal command
Changing text from mixed case to lower case is something that I often do. Much of the time until recently, this has been accomplished manually, but I started to wonder if a quicker way could be found. Thus, here is one that I use when working in macOS. It involves using a command that works in the terminal, and I have added a short alias for it to my .zshrc file. Here is the full pipeline:
pbpaste | tr '[:upper:]' '[:lower:]' | pbcopy
In the above, pbpaste reads from the paste buffer (or clipboard) while pbcopy writes the final output to the clipboard, replacing what was there before. In between those, tr '[:upper:]' '[:lower:]' changes any lower case letters to lower case ones.
With that, the process becomes this: copy the text into the paste buffer, run the command, paste output where it is wanted. While there may be a few steps, it is quicker than doing everything manually or opening another app to do the job. This suffices for now, and I. may get to look at something similar for Linux in time.
Keeping a file or directory out of a Git or GitHub repository
Recently, I have begun to do more version control of files with Git and GitHub. However, GitHub is not a place to keep files with log in credentials. Thus, I wanted to keep these locally but avoid having them being tracked in either Git or GitHub.
Adding the names to a .gitignore file will avoid their inclusion prospectively, but what can you do if they get added in error before you do? The answer that I found is to execute a command like the following:
git rm -r --cached [path to file or directory with its name]
That takes it out of the staging area and allows the .gitignore functionality to do its job. The -r switch makes the command recursive, should you be working with the contents of a directory. Then, the --cached flag is what does the removal from the staging area.
While the aforementioned worked for me when I had an oversight, the following is also suggested:
git update-index --assume-unchanged [path to file or directory with its name]
That may be working without a .gitignore file, which was not how I was doing things. Nevertheless, it may have its uses for someone else, so that is why I include it above.
Using a BASH command to count the files in a directory
As part of my backup workflow, I maintain a machine running OpenMediaVault that I only power up when backups are to be performed. Typically, this often happens when I have new photography images to load, and I have a NAS that acts as an online backup system. The OpenMediaVault machine is a near-offline counterpart to the NAS for added safety.
Recently, I needed to check on the number of image files in a directory from an SSH session because of a need to create a new repository for 2024. Some files from this year had ended up in the 2023 one, and I needed to be sure that nothing from last year ended in the 2024 folder, or vice versa. Getting a file count from a trusted source was a quick way of doing exactly this.
Due to clumsiness with the NAS, I had to do this using the OpenMediaVault machine. While I could go mounting drives on an interim basis, it was quicker to work from a BASH session. The trick was to use the wc command for counting the lines output by an invocation of the ls command. An example follows:
ls -l | wc -l
The -l (as in l for Lima) switch forces wc to count lines, while the counterpart (same letter) for ls forces it to list the contents in long form, one item per line. Thus, counting the number of lines gets you the count of the number of files. The call to the ls command can be customised to add other things life the number of dot files, but the above was enough for my purposes. When the files in both 2023 directories matched, I was satisfied that all was in order.
Changing the number of lines produced by the tail command in a Linux or UNIX session
Since I often use the tail command to look at the end of a log file and occasionally in combination with the watch command for constant updates, I got to wondering if the number of lines issued by the tail command could be changed. That is a simple thing to do with the -n switch. All you need is something like the following:
tail -n 20 logfile.log
Here the value of 20 is the number of lines produced when it would be 10 by default, and logfile.log gets replaced by the path and name of what you are examining. One thing to watch is that your terminal emulator can show all the lines being displayed. If you find that you are not seeing all the lines that you expect, then that might be the cause.
While you could find this by looking through the documentation, things do not always register with you during dry reading of something laden with lists of parameters or switches. That is an affliction with tools that do a lot and/or allow a lot of customisation.
Avoiding permissions, times or ownership failure messages when using rsync
The rsync command is one that I use heavily for doing backups and web publishing. The latter means that it is part of how I update websites built using Hugo because new and/or updated files need uploading. The command also sees usage when uploading files onto other websites as well. During one of these operations, and I am unsure now as to which type is relevant, I encountered errors about being unable to set permissions.
The cause was the encompassing -a option. This is a shorthand for -rltpgoD, and the individual options perform the following:
-r: recursive transfer, copying all contents within a directory hierarchy
-l: symbolic links copied as symbolic links
-t: preserve times
-p: preserve permissions
-g: preserve groups
-o: preserve owners
-D: preserve device and special files
The solution is to some of the options if they are inappropriate. The minimum is to omit the option for permissions preservation, but others may not apply between different servers either, especially when operating systems differ. Removing the options for preserving permissions, groups and owners results in something like this:
rsync -rltD [rest of command]
While it can be good to have a more powerful command with the setting of a single option, it can mean trying to do too much. Another way to avoid permissions and similar errors is to have consistency between source and destination files systems, but that is not always possible.
Removing duplicate characters from strings using BASH scripting
Recently, I wanted to extract some text from the Linux command by word number only for multiple spaces to make things less predictable. The solution was to remove the duplicate spaces. This can be done using sed, but you add the complexity of regular expressions if you opt for that solution. Instead, the tr command offers a neater approach. For removing duplicate spaces, the command takes the following form:
echo "test test" | tr -s " "
Since I was piping some text to the command, that is what I have above. The tr command is intended to replace or delete characters, and the -s switch is a shorthand for --squeeze-repeats. The actual character to be deduplicated is passed in quotes at the end; here, it is a space, but it could be anything that is duplicated. The resulting text in this example becomes:
test test
After the processing, there is now only one space separating the two words, which is the solution that I sought. It certainly cut out any variability that I was encountering in my usage.
Generating PNG files in SAS using ODS Graphics
Recently, I had someone ask me how to create PNG files in SAS using ODS Graphics, so I sought out the answer for them. Normally, the suggestion would have been to create RTF or PDF files instead, but there was a specific need that needed a different approach. Adding something like the following lines before an SGPLOT, SGPANEL or SGRENDER procedure should do the needful:
ods listing gpath='E:\';
ods graphics / imagename="test" imagefmt=png;
Here, the ODS LISTING statement declares the destination for the desired graphics file, while the ODS GRAPHICS statement defines the file name and type. In the above example, the file test.png would be created in the root of the E drive of a Windows machine. However, this also works with Linux or UNIX directory paths.
Copying a directory tree on a Windows system using XCOPY and ROBOCOPY
My usual method for copying a directory tree without any of the files in there involves the use of the Windows command line tool XCOPY and the command takes the following form:
xcopy /t /e <source> <destination>
The /t switch tells XCOPY to copy only the directory structure, while the /e one tells it to include empty directories too. Substituting /s for /e would ensure that only non-empty directories are copied. <source> and <destination> are the directory paths that you want to use and need to be enclosed in quotes if you have a space in a directory name.
There is one drawback to this approach that I have discovered. When you have long directory paths, messages about there being insufficient memory are issued and the command fails. The limitation has nothing to do with the machine that you are using, but is a limitation of XCOPY itself.
After discovering that, I got to check if ROBOCOPY can do the same thing without the same file path length limitation because I did not have the liberty of shortening folder names to get the whole path within the length expected by XCOPY. The following is the form of the command that I found did what I needed:
robocopy <source1> <destination1> /e /xf *.* /r:0 /w:0 /fft
Here, <source1> and <destination1> are the directory paths that you want to use and need to be enclosed in quotes if you have a space in a directory name. The /e switch copies all subdirectories and not just non-empty ones. Then, the xf *.* portion excludes all files from the copying process. The remaining options are added to help with getting around access issues and to try to copy only those directories that do not exist in the destination location. The /ftt switch was added to address the latter by causing ROBOCOPY to assume FAT file times. To get around the folder permission delays, the /r:0 switch was added to stop any operation being retried, with /w:0 setting wait times to 0 seconds. All this was enough to achieve what I wanted, and I am keeping it on file for my future reference, as well as sharing it with you.
Creating soft and hard symbolic links using the Windows command line
In the world of UNIX and Linux, symbolic links are shortcuts, but they do not work like normal Windows shortcuts because you do not jump from one location to another with the file manager's address bar changing what it shows. Instead, it is as if you see the contents of the directory at another quicker to access location in the file system, and the same sort of thinking applies to files too. In some ways, it is like giving files and directories alternative aliases. There are soft links that point to the name of a given directory or file, and hard links that point to actual files or directories.
For a long time, I was under the mistaken impression that such things did not exist on Windows until I came across the mklink command, which came with the launch of Windows Vista at the start of 2007. While this feature might not be widely known, it demonstrates that Windows did adopt some UNIX and Linux capability long before other UNIX-like features, such as virtual desktops, were introduced in Windows 10.
By default, the aforementioned command sets up symbolic links to files and the /D switch allows the same to be done for directories too. The /H switch makes a hard link instead of a soft link, so we get much of the functionality of the ln command in UNIX and Linux. Here is an example that creates a soft symbolic link for a directory:
mklink /D shortcut target_directory
Above, shortcut is the name of the symbolic link file and target_directory is the destination to which it links. In my experience, it works best for destinations beyond your home folder and, from what I have read, hard links may not be possible across different disks either.