Technology Tales

File comparison using PowerShell

In the past, I have compared files on the Linux/UNIX command line as well as the legacy Windows command line. Recently, I decided to try it using PowerShell. Here is the command structure:

Compare-Object (Get-Content ".\[name of one text file]") (Get-Content ".\[name of another text file]") > [path and name of output file]

Admittedly, this is more verbose than the others that I have mentioned above. Nevertheless, it does the job and sends everything to a text file for review. The Compare-Object piece does the comparison once the Get-Content portions have read in the content.

Getting rsync to resolve symbolic links

Given how Dropbox changed its handling of symbolic links in 2019 such that internal links within a Dropbox file hierarchy got fixed and links leading outside from the Dropbox area no longer worked. Thankfully, the rsync utility found in many Linux and UNIX settings does not do that, as long as you have called it correctly.

By default, symbolic links are synchronised like any other file. That is what Dropbox does now. To get rsync to resolve the links as shortcuts to either a single file or more likely a folder containing more than one file, it needs the -L switch or option in the command. When that is present, the linked file or files will get synchronised and honours the point of having these links in the first place: allowing more flexibility with folder structures and avoiding any duplication of files and folders.

Generating PNG files in SAS using ODS Graphics

Recently, I had someone ask me how to create PNG files in SAS using ODS Graphics, so I sought out the answer for them. Normally, the suggestion would have been to create RTF or PDF files instead, but there was a specific need that needed a different approach. Adding something like the following lines before an SGPLOT, SGPANEL or SGRENDER procedure should do the needful:

ods listing gpath='E:\';
ods graphics / imagename="test" imagefmt=png;

Here, the ODS LISTING statement declares the destination for the desired graphics file, while the ODS GRAPHICS statement defines the file name and type. In the above example, the file test.png would be created in the root of the E drive of a Windows machine. However, this also works with Linux or UNIX directory paths.

Updating Piwik using the Linux Command Line

Because updating Piwik using its web interface has proved tempestuous, I have decided to update the self-hosted analytics application on an SSH session. The production web servers that I use are hosted on Linux systems, so that is why any commands apply to the Linux or UNIX command line only. What is needed for Windows servers may differ.

The first step is to down the required ZIP file with this command:

wget https://builds.piwik.org/piwik.zip

Once the download is complete, the contents of the ZIP archive are extracted into a new subfolder. This is a process that I carry out in a separate folder to that where the website files are kept before copying everything from the extraction folder in there. Here is the unzip command, and the -o switch turns on overwriting of any previously existing files:

unzip -o piwik.zip

Without the required folder in the web server area to be updated, the next step is to do the actual system update that includes any updates to the Piwik database that you are using. There are two commands that you can use once you have specified the location of your Piwik installation. The second is needed when the first option cannot find where the PHP executable is stored. My systems had something more specific than these because both PHP 5.6 and PHP 7.0 are installed. Looking in /usr/bin was enough to find what I needed to execute in place of PHP below. Otherwise, the command was the same.

./[path to piwik]/console core:update

php [path to piwik]/console core:update

While the upgrade is ongoing, it prompts you to permit it to continue before it goes and modifies the database. This did not take long on my systems, but that depends on how much data there is. Once, the process has completed, you can delete any extraneous files using the rm command.

Creating soft and hard symbolic links using the Windows command line

In the world of UNIX and Linux, symbolic links are shortcuts, but they do not work like normal Windows shortcuts because you do not jump from one location to another with the file manager's address bar changing what it shows. Instead, it is as if you see the contents of the directory at another quicker to access location in the file system, and the same sort of thinking applies to files too. In some ways, it is like giving files and directories alternative aliases. There are soft links that point to the name of a given directory or file, and hard links that point to actual files or directories.

For a long time, I was under the mistaken impression that such things did not exist on Windows until I came across the mklink command, which came with the launch of Windows Vista at the start of 2007. While this feature might not be widely known, it demonstrates that Windows did adopt some UNIX and Linux capability long before other UNIX-like features, such as virtual desktops, were introduced in Windows 10.

By default, the aforementioned command sets up symbolic links to files and the /D switch allows the same to be done for directories too. The /H switch makes a hard link instead of a soft link, so we get much of the functionality of the ln command in UNIX and Linux. Here is an example that creates a soft symbolic link for a directory:

mklink /D shortcut target_directory

Above, shortcut is the name of the symbolic link file and target_directory is the destination to which it links. In my experience, it works best for destinations beyond your home folder and, from what I have read, hard links may not be possible across different disks either.

Windows commands for setting default applications for opening certain file types

On Friday, I was working on a system where a session is instantiated from a stored virtual machine that produces a fresh session every time, meaning that all previous changes get lost. What I have is a batch script that I run to reinstate what I need, and I encountered another task that I wanted it to do.

Part of my work involves the creation of plain text files with the extension lst and this is getting associated with SAS instead of Notepad. While you can reassign such associations using the GUI, it would be a bonus to do it via the command line too, so the assoc and ftype commands caught my interest. The first of these associates a file with a given extension with a desired file type, while the second shows the available file types together with the associated applications that open them. The assoc command also shows all the associations that are in place when it is executed with no parameters, and the ftype command does the same for file types.

Once you have picked out a file type with the ftype command, then the assoc can be used like the following:

assoc .lst=txtfile

The above associates an extension with a file type. In this, .lst files are going to get opened by Notepad because of the txtfile association. Though it did not do what I wanted on Friday due to system lockdown, it is good to know that this is possible and that even the Windows command line supports goodies like these.

Copying only updated new or updated files by command line in Linux or Windows

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don't want to keep copying the same files over and over again, so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is on Linux, it is the bash shell that I use, so the switches may not apply with others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always, so it needs adding. The r switch is useful because the copy then in recursive, so only a directory needs to be specified as the source and the destination needs to be one level up from a folder with the same name there to avoid file duplication. It is the u switch that makes the file copy incremental, and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session, and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental, and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files, while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried, so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening, and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell on Linux.

Renaming multiple files in Linux

The Linux and UNIX command mv has a number of limitations, such as not overwriting destination files and not renaming multiple files using wildcards. The only solution to the first that I can find is one that involves combining the cp and rm commands. For the second, there's another command: rename. Here's an example like what I used recently:

rename s/fedora/fedora2/ fedora.*

The first argument in the above command is a regular expression much like what Perl is famous for implementing; in fact, it is Perl-compatible ones (PCRE) that are used. The s before the first slash stands for substitute, with fedora being the string that needs to be replaced and fedora2 being what replaces it. The third command is the file name glob that you want to use, fedora.* in this case. Therefore, all files in a directory named fedora will be renamed fedora2 regardless of the file type. The same sort of operation can be performed for all files with the same extension when it needs to be changed, htm to html, for instance. Of course, there are other uses, but these are handy ones to know.

Command line setting of Windows file attributes

Aside from permissions that can be set using the cacls command, Windows files have properties like read only, archive and hidden. Of course, these are not the same or as robust as access permissions, but they may have a use in stopping accidental updates to files when you don't have access to use of the cacls command. While you could set these attributes using the properties page of any file, executing the attrib command on the Windows command is more convenient. Here are some possible usage options:

Set the read-only flag on a file:

attrib +r test.txt

Remove the read-only flag from a file (found a use for this one recently):

attrib -r test.txt

Set the archive flag on a file:

attrib +a test.txt

Remove the archive flag from a file:

attrib -a test.txt

Set the hidden only flag on a file:

attrib +h test.txt

Remove the hidden flag from a file:

attrib -h test.txt

Using the /s option and wildcards processes a number of files at a time and /d applies the command to directories. They could come in handy when removing read only attributes (also called bits in places) from files copied from read only optical media, such as CD's and DVD's.

Anquet and VirtualBox Shared Folders

For a while now, I have had Anquet installed in a virtual machine instance of Windows XP, but it has been throwing errors continuously on start up. Perhaps surprisingly, it only dawned upon me recently what might have been the cause. A bit of fiddling revealed that my storing the mapping data Linux side and sharing it into the VM wasn't helping, and copying it to a VM hard drive set things to rights. This type of thing can also cause problems when it comes to getting Photoshop to save files using VirtualBox's Shared Folders feature too. Untangling the situation is a multi-layered exercise. On the Linux side, permissions need to be in order and that involves some work with chmod (775 is my usual remedy) and chgrp to open things up to the vboxusers group. Adding in Windows' foibles when it comes to networked drives and their mapping to drive letters brings extra complexity; any shared folders are made visible to Windows in a path of this form: \\vboxsvr\shared_folder_name\. The solution is either a lot of rebooting, extensive use of the net use command, or both. It induces the sort of going and coming that causes copying things over and back as needed to look less involved and more sensible, if a little more manual than might be liked.