Technology Tales

Something to try when you get a message like this caused by a filename with a leading hyphen: "mv: illegal option -- u"

Recently, I downloaded some WEBP files from Ideogram and attempted to move them to another folder. That is when I got the message that you see in the title of this entry. Because I had not looked at the filenames, I baffled when I got this from a simple command that I had been using with some success until then. Because I was using an iMac, I tried the suggestion of installing coreutils to get GNU mv and cp to see if that would help:

brew install coreutils

The above command gave me gmv and gcp for the GNU versions of mv and cp that comes with macOS. Trying gmv only got me the following message:

gmv: cannot combine --backup with --exchange, -n, or --update=none-fail

The ls command could list all files, but not the WEBP ones. Thus, I executed the following to show what I wanted:

ls | grep -i webp

That got around the problem by doing a subset of the directory listing. It was then that I spotted the leading hyphen. To avoid the problem tripping me up again, I renamed the offending file using this command:

mv -- -iunS9U4RFevWpaju6ArIQ.webp iunS9U4RFevWpaju6ArIQ.webp

Here, the -- switch tells the mv command not to look for any more options and only to expect filenames. When I tried enclosing the filename in quotes, I still got problems, even that might have because I was using double quotes instead of single quotes. Another option is to escape the leading hyphen like this:

mv ./-iunS9U4RFevWpaju6ArIQ.webp iunS9U4RFevWpaju6ArIQ.webp

Once the offending file was renamed, I could move the files to their final location. That could have used the -- option too, saving me an extra command, only for my wanting this not to trip me up again. Naturally, working in Finder might have avoided all this as much as not having a file with a leading hyphen in its name, but there would have been nothing to learn then.

Using a BASH command to count the files in a directory

As part of my backup workflow, I maintain a machine running OpenMediaVault that I only power up when backups are to be performed. Typically, this often happens when I have new photography images to load, and I have a NAS that acts as an online backup system. The OpenMediaVault machine is a near-offline counterpart to the NAS for added safety.

Recently, I needed to check on the number of image files in a directory from an SSH session because of a need to create a new repository for 2024. Some files from this year had ended up in the 2023 one, and I needed to be sure that nothing from last year ended in the 2024 folder, or vice versa. Getting a file count from a trusted source was a quick way of doing exactly this.

Due to clumsiness with the NAS, I had to do this using the OpenMediaVault machine. While I could go mounting drives on an interim basis, it was quicker to work from a BASH session. The trick was to use the wc command for counting the lines output by an invocation of the ls command. An example follows:

ls -l | wc -l

The -l (as in l for Lima) switch forces wc to count lines, while the counterpart (same letter) for ls forces it to list the contents in long form, one item per line. Thus, counting the number of lines gets you the count of the number of files. The call to the ls command can be customised to add other things life the number of dot files, but the above was enough for my purposes. When the files in both 2023 directories matched, I was satisfied that all was in order.

Avoiding permissions, times or ownership failure messages when using rsync

The rsync command is one that I use heavily for doing backups and web publishing. The latter means that it is part of how I update websites built using Hugo because new and/or updated files need uploading. The command also sees usage when uploading files onto other websites as well. During one of these operations, and I am unsure now as to which type is relevant, I encountered errors about being unable to set permissions.

The cause was the encompassing -a option. This is a shorthand for -rltpgoD, and the individual options perform the following:

-r: recursive transfer, copying all contents within a directory hierarchy

-l: symbolic links copied as symbolic links

-t: preserve times

-p: preserve permissions

-g: preserve groups

-o: preserve owners

-D: preserve device and special files

The solution is to some of the options if they are inappropriate. The minimum is to omit the option for permissions preservation, but others may not apply between different servers either, especially when operating systems differ. Removing the options for preserving permissions, groups and owners results in something like this:

rsync -rltD [rest of command]

While it can be good to have a more powerful command with the setting of a single option, it can mean trying to do too much. Another way to avoid permissions and similar errors is to have consistency between source and destination files systems, but that is not always possible.

Creating soft and hard symbolic links using the Windows command line

In the world of UNIX and Linux, symbolic links are shortcuts, but they do not work like normal Windows shortcuts because you do not jump from one location to another with the file manager's address bar changing what it shows. Instead, it is as if you see the contents of the directory at another quicker to access location in the file system, and the same sort of thinking applies to files too. In some ways, it is like giving files and directories alternative aliases. There are soft links that point to the name of a given directory or file, and hard links that point to actual files or directories.

For a long time, I was under the mistaken impression that such things did not exist on Windows until I came across the mklink command, which came with the launch of Windows Vista at the start of 2007. While this feature might not be widely known, it demonstrates that Windows did adopt some UNIX and Linux capability long before other UNIX-like features, such as virtual desktops, were introduced in Windows 10.

By default, the aforementioned command sets up symbolic links to files and the /D switch allows the same to be done for directories too. The /H switch makes a hard link instead of a soft link, so we get much of the functionality of the ln command in UNIX and Linux. Here is an example that creates a soft symbolic link for a directory:

mklink /D shortcut target_directory

Above, shortcut is the name of the symbolic link file and target_directory is the destination to which it links. In my experience, it works best for destinations beyond your home folder and, from what I have read, hard links may not be possible across different disks either.

Numeric for loops in Korn shell scripting: from ksh88 to ksh93

The time-honoured syntax for a for loop in a UNIX script is what you see below, and that is what works with the default shell in Sun's Solaris UNIX operating system, ksh88.

for i in 1 2 3 4 5 6 7 8 9 10
do
    if [[ -d dir$i ]]
    then
        :
    else
        mkdir dir$i
    fi
done

However, there is a much nicer syntax supported since the advent of ksh93. It follows C language conventions found in all sorts of places like Java, Perl, PHP and so on. Here is an example:

for (( i=1; i<11; i++ ))
do
    if [[ -d dir$i ]]
    then
        :
    else
        mkdir dir$i
    fi
done

The power of pipes

One of the great features of the UNIX shell is that you can send the output from one command to another for further processing. Take the following example for instance:

ls -l | grep "Jul 12"

This takes the long directory file listing output and sends it to grep for subsetting (all files created today in this example) before it is returned to the screen. The | character is the pipe trigger, and you can have as many pipes in your command as you want, though readability may dictate how far you want to go.

Using SAS FILENAME statement to extract directory file listings into SAS

The filename statement's pipe option allows you to direct the output of operating system commands into SAS for further processing. Usefully, the Windows dir command (with its /s switch) and the UNIX and Linux equivalent ls allow you to get a file listing into SAS. For example, here's how you extract the list of files in your UNIX or Linux home directory into SAS:

filename DIRLIST pipe 'ls ~';
data dirlist;
    length filename $200;
    infile dirlist length=reclen;
    input buffer $varying200. reclen;
run;

Using the ftp option on the filename statement allows you to get a list of the files in a directory on a remote server, even one with a different operating system to that used on the client (PC or server), very useful for cases where cross-platform systems are involved. Here's some example code:

filename dirlist ftp ' ' ls user='user' host='host' prompt;
data _null_;
    length filename $200;
    infile dirlist length=reclen;
    input buffer $varying200. reclen;
run;

The PROMPT option will cause SAS to ask you for a password, and the null string is where you would otherwise specify the name of a file.

Checking existence of files and directories on UNIX using shell scripting

Having had a UNIX shell script attempt to copy a non-existent file, I decided to take another look for ways to test the existence of a file. For directory existence checking, I was testing for the return code from the cd command, and I suppose that the ls command might help for files. However, I did find a better way:

if [ -f $filename ]
then
    echo "This filename [$filename] exists"
elif [ -d $dirname ]
then
    echo "This dirname [$dirname] exists"
else
    echo "Neither [$dirname] or [$filename] exist"
fi

The -d and -f flags within the evaluation expressions test for the existence of directories and files, respectively. One gotcha is that those spaces within the brackets are important too, but it is a very way of doing what I wanted.