Batch conversion of DNG files to other file types with the Linux command line

At the time of writing, Google Drive is unable to accept DNG files, the Adobe file type for RAW images from digital cameras. The uploads themselves work fine but the additional processing at the end that I believe is needed for Google Photos appears to be failing. Because of this, I thought of other possibilities like uploading them to Dropbox or enclosing them in ZIP archives instead; of these, it is the first that I have been doing and with nothing but success so far. Another idea is to convert the files into an image format that Google Drive can handle and TIFF came to mind because it keeps all the detail from the original image. In contrast, JPEG files lose some information because of the nature of the compression.

Handily, a one line command does the conversion for all files in a directory once you have all the required software installed:

find -type f | grep -i “DNG” | parallel mogrify -format tiff {}

The find and grep commands are standard with the first getting you a list of all the files in the current directory and sending (piping) these to the grep command so the list only retains the names of all DNG files. The last part uses two commands for which I found installation was needed on my Linux Mint machine. The parallel package is the first of these and distributes the heavy workload across all the cores in your processor and this command will add it to your system:

sudo apt-get install parallel

The mogrify command is part of the ImageMagick suite along with others like convert and this is how you add that to your system:

sudo apt-get install imagemagick

In the command at the top, the parallel command works through all the files in the list provided to it and feeds them to mogrify for conversion. Without the use of parallel, the basic command is like this:

mogrify -format tiff *.DNG

In both cases, the -format switch specifies the output file type with tiff triggering the creation of TIFF files. The *.DNG portion itself captures all DNG files in a directory but {} does this in the main command at the top of this post. If you wanted JPEG ones, you would replace tiff with jpg. Shoudl you ever need them, a full list of what file types are supported is produced using the identify command (also part of ImageMagick) as follows:

identify -list format

Migrating a virtual machine from VirtualBox to VMware Player on Linux

The progress of Windows 10 is something that I have been watching. Early signs have been promising and the most recent live event certainly contained its share of excitement. The subsequent build that was released was another step in the journey though the new Start Menu appears more of a work in progress than it did in previous builds. Keeping up with these advances sometimes steps ahead of VirtualBox support for them and I discovered that again in the last few days. VMware Player seems unaffected so I thought that I’d try a migration of the VirtualBox VM with Windows 10 onto there. In the past, I did something similar with a 32-bit instance of Windows 7 that subsequently got upgraded all the the way up to 8.1 but that may not have been as slick as the latest effort so I thought that I would share it here.

The first step was to export the virtual machine as an OVF appliance and I used File > Export Appliance… only to make a foolish choice regarding the version of OVF. The one that I picked was 2.0 and I subsequently discovered that 1.0 was the better option. The equivalent command line would look like the following (there are two dashes before the ovf10 option below):

vboxmanage export [name of VM] -o [name of file].ova --ovf10

VMware have a tool for extracting virtual machines from OVF files that will generate a set of files that will work with Player and other similar products of theirs. It goes under the unsurprising name of OVF Tool and it usefully works from a command line session. When I first tried it with an OVF 2.0 files, I got the following error and it stopped doing anything as a result:

Line 2: Incorrect namespace http://schemas.dmtf.org/ovf/envelope/2 found.

The only solution was to create a version 1.0 file and use a command like the following (it’s a single line though it wraps over two here and there are two dashes before the lax switch):

ovftool --lax [name of file].ova [directory location of VM files]/[name of file].vmx

The --lax option is needed to ensure successful execution even with an OVF 1.0 file as the input. Once I had done this on my Ubuntu GNOME system, the virtual machine could be opened up on VMware Player and I could use the latest build of Windows 10 at full screen, something that was not possible with VirtualBox. This may be how I survey the various builds of the operating that appear before its final edition is launched later this year.

Smarter file renaming using PowerShell

It seems that the Rename-Item commandlet in Powershell is a very useful tool when it comes to smarter renaming of files. Even text substitution is a possibility and what follows is an example that takes the output of the Dir command for listing the files in a directory and replaces hyphens with underscores in each one.

Dir | Rename-Item –NewName { $_.name –replace “-“,”_” }

The result is that something like the-file.txt becomes the_file.txt. This behaviour is reminiscent of the rename command found on Linux and UNIX systems, where regular expressions can be used like in the following example that has the same result as the above:

rename ‘s/-/_/g’ *

In both cases, you do need to be careful as to what files are in a directory for this though the wildcard syntax on Linux or UNIX will be more familiar who has worked with files via almost any command line. Another thing to watch in the UNIX world is that ** parses the whole directory structure and that could be something that is not wanted for much of the time.

All of this is a far cry from the capabilities of the ren or rename command used in the days of MS-DOS and what has become the legacy Windows command line. Apart from simple renaming, any attempt at tweaking a filename through substitution ended up with the extra string getting appended to filenames when I tried it. Thus, the Powershell option looks better in comparison.

Changing the working directory in a SAS session

It appears that PROC SGPLOT and other statistical graphics procedures create image files even if you are creating RTF or PDF files. By default, these are PNG files but there are other possibilities. When working with PC SAS , I have seen them written to the current working directory and that could clutter up your folder structure, especially if they are unwanted.

Being unable to track down a setting that controls this behaviour, I resolved to find a way around it by sending the files to the SAS work directory so they are removed when a SAS session is ended. One option is to set the session’s working directory to be the SAS work one and that can be done in SAS code without needing to use the user interface. As a result, you get some automation.

The method is implicit though in that you need to use an X statement to tell the operating system to change folder for you. Here is the line of code that I have used:

x “cd %sysfunc(pathname(work))”;

The X statement passes commands to an operating system’s command line and they are enclosed in quotes. %sysfunc then is a macro command that allows certain data step functions or call routines as well as some SCL functions to be executed. An example of the latter is pathname and this resolves library or file references and it is interrogating the location of the SAS work library here so it can be passed to the operating systems cd (change directory) command for processing. This method works on Windows and UNIX so Linux should be covered too, offering a certain amount of automation since you don’t have to specify the location of the SAS work library in every session due to the folder name changing all the while.

Of course, if someone were to tell me of another way to declare the location of the generated PNG files that works with RTF and PDF ODS destinations, then I would be all ears. Even direct output without image file creation would be even better. Until then though, the above will do nicely.

Copying only updated new or updated files by command line in Linux or Windows

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don’t want to keep copying the same files over and over again so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is in Linux, it the bash shell that I use so the switches may not apply to others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always so it needs adding. The r switch is useful because the copy then in recursive so only a directory needs to specified as the source and the destination needs to be one level up from a folder with the same name there so as to avoid file duplication. It is the u switch that makes the file copy incremental and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell in Linux.