TOPIC: FIND
Grouping directories first in output from ls commands executed in terminal sessions on macOS and Linux
This enquiry began with my seeing directories and files being sorted by alphabetical order without regard for type in macOS Finder. In Windows and Linux file managers, I am accustomed to directories and files being listed in distinct blocks, albeit within the same listings, with the former preceding the latter. What I had missed was that the ls command and its aliases did what I was seeing in macOS Finder, which perhaps is why the operating system and its default apps work like that.
Over to Linux
On the zsh implementation that macOS uses, there is no way to order the output so that directories are listed before files. However, the situation is different on Linux because of the use of GNU tooling. Here, the --group-directories-first switch is available, and I have started to use this on my own Linux systems, web servers as well as workstations. This can be set up in .bashrc or .bash_aliases like the following:
alias ls='ls --color=auto --group-directories-first'
alias ll='ls -lh --color=auto --group-directories-first'
Above, the --color=auto switch adds colour to the output too. Issuing the following command makes the updates available in a terminal session (~ is the shorthand for the home directory below):
source ~/.bashrc
Back to macOS
While that works well on Linux, additional tweaks are needed to implement the same on macOS. Firstly, you have to install Homebrew using this command (you may be asked for your system password to let the process proceed):
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
To make it work, this should be added to the .zshrc file in your home folder:
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
Then, you need to install coreutils for GNU commands like gls (a different name is used to distinguish it from what comes with macOS) and adding dircolors gives you coloured text output as well:
brew install coreutils
brew install dircolors
Once those were in place, I found that adding these lines to the .zshrc file was all that was needed (note those extra g's):
alias ls='gls --color=auto --group-directories-first'
alias ll='gls -lh --color=auto --group-directories-first'
If your experience differs, they may need to be preceded with this line in the same configuration file:
eval "$(dircolors -b)"
The final additions then look like this:
export PATH="/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH"
eval "$(dircolors -b)"
alias ls='gls --color=auto --group-directories-first'
alias ll='gls -lh --color=auto --group-directories-first'
Following those, issuing this command will make the settings available in your terminal session:
source ~/.zshrc
Closing Remarks
In summary, you have learned how to list directories before files, and not intermingled as is the default situation. For me, this discovery was educational and adds some extra user-friendliness that was not there before the tweaks. While we may be considering two operating systems and two different shells (bash and zsh), there is enough crossover to make terminal directory and file listing operations function consistently regardless of where you are working.
Searching file contents using PowerShell
Having made plenty of use of grep on the Linux/UNIX command and findstr on the legacy Windows command line, I wondered if PowerShell could be used to search the contents of files for a text string. Usefully, this turns out to be the case, but I found that the native functionality does not use what I have used before. The form of the command is given below:
Select-String -Path <filename search expression> -Pattern "<search expression>" > <output file>
While you can have the output appear on the screen, it always seems easier to send it to a file for subsequent use, and that is what I am doing above. The input to the -Path switch can be a filename or a wildcard expression, while that to the -Pattern can be a text string enclosed in quotes or a regular expression. Given that it works well once you know what to do, here is an example:
Select-String -Path *.sas -Pattern "proc report" > c:\temp\search.txt
The search.txt file then includes both the file information and the text that has been found for the sake of checking that you have what you want. What you do next is up to you.
Batch conversion of DNG files to other file types with the Linux command line
At the time of writing, Google Drive is unable to accept DNG files, the Adobe file type for RAW images from digital cameras. While the uploads themselves work fine, the additional processing at the end that, I believe, is needed for Google Photos appears to be failing. Because of this, I thought of other possibilities like uploading them to Dropbox or enclosing them in ZIP archives instead; of these, it is the first that I have been doing and with nothing but success so far. Another idea is to convert the files into an image format that Google Drive can handle, and TIFF came to mind because it keeps all the detail from the original image. In contrast, JPEG files lose some information because of the nature of the compression.
Handily, a one line command does the conversion for all files in a directory once you have all the required software installed:
find -type f | grep -i "DNG" | parallel mogrify -format tiff {}
The find and grep commands are standard, with the first getting you a list of all the files in the current directory and sending (piping) these to the grep command, so the list only retains the names of all DNG files. The last part uses two commands for which I found installation was needed on my Linux Mint machine. The parallel package is the first of these and distributes the heavy workload across all the cores in your processor, and this command will add it to your system:
sudo apt-get install parallel
The mogrify command is part of the ImageMagick suite along with others like convert and this is how you add that to your system:
sudo apt-get install imagemagick
In the command at the top, the parallel command works through all the files in the list provided to it and feeds them to mogrify for conversion. Without the use of parallel, the basic command is like this:
mogrify -format tiff *.DNG
In both cases, the -format switch specifies the output file type, with the tiff portion triggering the creation of TIFF files. The *.DNG portion itself captures all DNG files in a directory, but {} does this in the main command at the top of this post. If you wanted JPEG ones, you would replace tiff with jpg. Should you ever need them, a full list of what file types are supported is produced using the identify command (also part of ImageMagick) as follows:
identify -list format
Preventing PROC SGPLOT PNG file clutter by changing the working directory in a SAS session
It appears that PROC SGPLOT along with other statistical graphics procedures creates image files, even if you are creating RTF or PDF files. By default, these are PNG files, but there are other possibilities. When working with PC SAS, I have seen them written to the current working directory and that could clutter up your folder structure, especially if they are unwanted.
Being unable to track down a setting that controls this behaviour, I resolved to find a way around it by sending the files to the SAS work directory so they are removed when a SAS session is ended. One option is to set the session's working directory to be the SAS work one, which can be done in SAS code without needing to use the user interface. As a result, you get some automation.
The method is implicit, though, in that you need to use an X statement to tell the operating system to change the folder for you. Here is the line of code that I have used:
x "cd %sysfunc(pathname(work))";
The X statement passes commands to an operating system's command line, and they are enclosed in quotes. %sysfunc then is a macro command that allows certain data step functions or call routines as well as some SCL functions to be executed. An example of the latter is pathname and this resolves library or file references, and it is interrogating the location of the SAS work library here so it can be passed to the operating systems cd (change directory) command for processing. Since this method works on Windows and UNIX, Linux should be covered too, offering a certain amount of automation since you don't have to specify the location of the SAS work library in every session due to the folder name changing all the while.
Of course, if someone were to tell me of another way to declare the location of the generated PNG files that works with RTF and PDF ODS destinations, then I would be all ears. Even direct output without image file creation would be even better. Until then, though, the above will do nicely.
Command line file comparison in Windows
While UNIX and Linux both have the diff command for comparing the contents of text files, the Windows counterpart was unknown to me until recently. Its name is fc, and it looks as if the f is for file and c is for comparison, though I cannot confirm that as of now. The usage of that command is not dissimilar to the way that things work with diff. Here is an example command:
fc file1.txt file2.txt > file3.txt
This compares file1.txt with file2.txt and sends the output to file3.txt. Any differences between the two files being compared appear to be more clearly labelled than in the diff output's < and > labels. That verbosity could have its uses, but the existence of the fc command is stopping envious glances at the diff one for now, just as findstr is doing the same in comparison with grep.
Checking existence of files and directories on UNIX using shell scripting
Having had a UNIX shell script attempt to copy a non-existent file, I decided to take another look for ways to test the existence of a file. For directory existence checking, I was testing for the return code from the cd command, and I suppose that the ls command might help for files. However, I did find a better way:
if [ -f $filename ]
then
echo "This filename [$filename] exists"
elif [ -d $dirname ]
then
echo "This dirname [$dirname] exists"
else
echo "Neither [$dirname] or [$filename] exist"
fi
The -d and -f flags within the evaluation expressions test for the existence of directories and files, respectively. One gotcha is that those spaces within the brackets are important too, but it is a very way of doing what I wanted.