Technology Tales

Batch conversion of DNG files to other file types with the Linux command line

At the time of writing, Google Drive is unable to accept DNG files, the Adobe file type for RAW images from digital cameras. While the uploads themselves work fine, the additional processing at the end that, I believe, is needed for Google Photos appears to be failing. Because of this, I thought of other possibilities like uploading them to Dropbox or enclosing them in ZIP archives instead; of these, it is the first that I have been doing and with nothing but success so far. Another idea is to convert the files into an image format that Google Drive can handle, and TIFF came to mind because it keeps all the detail from the original image. In contrast, JPEG files lose some information because of the nature of the compression.

Handily, a one line command does the conversion for all files in a directory once you have all the required software installed:

find -type f | grep -i "DNG" | parallel mogrify -format tiff {}

The find and grep commands are standard, with the first getting you a list of all the files in the current directory and sending (piping) these to the grep command, so the list only retains the names of all DNG files. The last part uses two commands for which I found installation was needed on my Linux Mint machine. The parallel package is the first of these and distributes the heavy workload across all the cores in your processor, and this command will add it to your system:

sudo apt-get install parallel

The mogrify command is part of the ImageMagick suite along with others like convert and this is how you add that to your system:

sudo apt-get install imagemagick

In the command at the top, the parallel command works through all the files in the list provided to it and feeds them to mogrify for conversion. Without the use of parallel, the basic command is like this:

mogrify -format tiff *.DNG

In both cases, the -format switch specifies the output file type, with the tiff portion triggering the creation of TIFF files. The *.DNG portion itself captures all DNG files in a directory, but {} does this in the main command at the top of this post. If you wanted JPEG ones, you would replace tiff with jpg. Should you ever need them, a full list of what file types are supported is produced using the identify command (also part of ImageMagick) as follows:

identify -list format

A display of brand loyalty

Since 2007, I've used a Pentax K10D DSLR as my main camera. It has travelled with me extensively, capturing over 15,000 images and serving reliably. Though its autofocus isn't the fastest, this doesn't matter for my typical subjects: landscapes, architecture, flora and transport (stationary buses and trains, for instance). Any slight underexposure in photos can be easily fixed since I shoot in DNG format, which preserves all raw image data. The camera has performed so well that I've found it difficult to justify buying a new SLR.

Lately, I have been looking at processed photos and asking myself if time has moved along for what is not far from being a six-year-old camera. At various times, I have been looking at higher members of the Pentax while wondering if an upgrade would be a good idea. First, there was the K7 and then the K5 before the K5 II got launched. Even though its predecessor is still to be found on sale, it was the newer model that became my choice.

My move to Pentax in 2007 was a case of brand disloyalty, since I had been a Canon user from when I acquired my first SLR, an EOS 300. Even now, I still have a PowerShot G11 that finds itself slipped into a pocket on many a time. Nevertheless, I find that Canon images feel a little washed out before post-processing, and that hasn't been the case with the K10D. In fact, I have been hearing good things about Nikon cameras delivering punchy results, so one of them would be a contender were it not for how well the Pentax performed.

So, what has my new K5 II body gained me that I didn't have before? For one thing, the autofocus is a major improvement on that in the K10D. It may not stop me persevering with manual focusing for most of the time, but there are occasions the option of solid autofocus is good to have. Other advances include a 16.3 megapixel sensor with a much larger ISO range. The advances in sensor technology since when the K10D appeared may give me better quality photos, and noise is something that my eyes may have begun to detect in K10D photos even at my usual ISO of 400.

Some features will not get used, though. While I rely on Live View with my PowerShot G11 due to its poor optical viewfinder, I rarely use it on the K5 II with its bright, sharp viewfinder, especially after noticing unreliable autofocus when using it (though manual focusing should work fine). By default, the screen stays on constantly, which irritated me as an optical viewfinder user, so I consulted the manual to turn it off. Similarly, I quickly abandoned the image level display. However, I've kept the horizon autocorrection feature enabled, as it helps fix the crooked horizons that often result from handheld shooting.

The K5 II may have a 3" screen on its back, but it has done nothing to increase the size of the camera. If anything, it is smaller than the K10D, and that usefully means that I am not on the lookout for a new camera holster. Not having a bigger body also means there is little change in how the much camera feels in the hand compared with the older one.

In many ways, the K5 II works very like the K10D once I took control over settings that didn't suit me. Both have Shake Reduction in their camera bodies, though the setting has been moved into the settings menu in the new camera, when the older one had a separate switch on its body. Since I'd be inclined to leave it on all the time and prefer not to have it knocked off accidentally, this is not an issue.  Otherwise, many of the various switches are in the same places, so it's not that difficult to find my way around them.

The K5 II includes other changes, like a mode dial lock, but I'm familiar with this feature from Canon EOS cameras, so it doesn't feel like a downgrade. The exposure compensation button has been relocated to the top of the camera, making it easier to find and use; I'm using it more than I did on the K10D. Since I also use this feature on my G11, I'm applying similar experimentation to the K5 II. Next to it is a new ISO button, which I plan to test to see how it performs.

My main criticism of the K5 II is its cluttered menu system. The K10D's long scrollable lists have been replaced with multiple tabs that eliminate on-screen scrolling. However, I think this fragments the interface too much and makes navigating settings more intimidating for less technical users. Essential settings remain accessible; I continue using RAW DNG files as usual, though JPEG and Pentax's proprietary RAW format are also available. After initially forgetting to set the date, I quickly fixed this. Similarly, I disabled the default setting that stores files in date-based folders, instead directing everything to a single PENTX directory to match my workflow. Recently, I discovered the option to add photographer and copyright information to image file metadata. This seems particularly useful given proposed U.K. legislation that would weaken automatic copyright protections, even though most photographers oppose these legal changes.

A camera's true value lies in the quality of its images, and I'm pleased with what the K5 II produces. The larger file sizes mean fewer images fit on a memory card, though increased SDHC capacities help offset this, even as I avoid excessive shooting. While using the camera, I was surprised to find apertures like F/14 and F/18, as I was used to values like F/11, F/13, F/16, F/19, and F/22. Most traditional values remain available, so there's not a complete departure from convention. Similarly with shutter speeds, I noticed 1/100 and 1/160 where I expected only 1/90, 1/125, and 1/250. These additional options provide more flexibility and potentially make achieving correct exposures easier, though what constitutes "correct" should be determined by the photographer, not the camera's algorithms. So far, I've mostly used ISO 400, except for some evening woodland testing shortly after getting the camera.

While I've been meaning to share my thoughts on the K5 II since acquiring it a few months ago, I needed time to organize my ideas. Initially, I felt overwhelmed by how much there was to say, even more than what you've read here. There are still features to explore, like white balance adjustments and their effect on image quality. After discovering its shake reduction was switched off, I'm also reconsidering my K10D, especially since this might explain the quality issues I mentioned earlier. Using my tripod more consistently would be another improvement. Clearly, I have many more photographic explorations ahead.

Command Line Processing of EXIF Image Metadata

There is a bill making its way through the U.K. parliament at this time that could reduce the power of copyright when it comes to images placed on the web. The current situation is that anyone who creates an image automatically holds the copyright for it. However, the new legislation will remove that if it becomes law as it stands. As it happens, the Royal Photographic Society is doing what it can to avoid any changes to what we have now. Though there may be the barrier of due diligence, how many of us take steps to mark our own intellectual property? For one, I have been less that attentive to this and now wonder if there is anything more that I should be doing. While others may copyleft their images instead, I don't want to find myself unable to share my own photos because another party is claiming rights over them. There's watermarking as an object, yet I also want to add something to the image metadata too.

That got me wondering about adding metadata to any images that I post online that assert my status as the copyright holder. It may not be perfect, but any action is better than doing nothing at all. Given that I don't post photos where EXIF metadata is stripped as part of the uploading process, it should be there to see for anyone who bothers to check, and there may not be many who do.

Because I also wanted to batch process images, I looked for a command line tool to do the needful and found ExifTool. Being a Perl library, it is cross-platform so you can use it on Linux, Windows and even OS X. To install it on a Debian or Ubuntu-based Linux distro, just use the following command:

sudo apt-get install libimage-exiftool-perl

The form of the command that I found useful for adding the actual copyright information is below:

exiftool -p "-copyright=(c) John ..." -ext jpg -overwrite_original

The -p switch preserves the timestamp of the image file, while the -overwrite_original one ensures that you don't end up with unwanted backup files. The copyright message goes within the quotes along with the -copyright option. With a little shell scripting, you can traverse a directory structure and change the metadata for any image files contained in different sub-folders. If you wish to do more than this, there's always the user documentation to be consulted.

Why the manual step?

One of the consequences of buying a new camera is that your current photo processing software may not be fully equipped for the job of handling the images that it creates. This especially manifests itself with raw image files; Adobe Photoshop Elements 5 was unable to completely handle DNG files made with my Pentax K10D until I upgraded to version 7.

As things stood, Elements 7 was unable to import CR2 files from my Canon PowerShot G11 into the Organiser, so it was off to the appropriate page on the Adobe website for a Camera Raw updater. Thus, I picked up the latest release of Camera Raw (5.6 at the time of writing) even though it was found in the Elements 8 category (don't be put by this because release notes address the version compatibility question more extensively).

Strangely, the updater doesn't complete everything because you still need to copy Camera Raw.8bi from the zip archive and backup the original. Quite why this couldn't have been more automated, even with user prompts for file names and locations, is beyond me, yet that is how it is. However, once all was in place, CR2 files were handled by Elements without missing a beat.

Converting from CGM to Postscript

One thing that I recently had to investigate was the possibility of converting CGM vector graphics files into Postscript and from there into PDF. Having used ImageMagick for converting images before, that was an obvious option. However, that cannot process CGM files on its own and needs a delegate or helper application as well. This is the case with raw digital camera files too, with UFRaw being the program chosen. For CGM images, the more obscure RALCGM is what's needed, and tracking it down is a bit of an art. Though the history is that it was developed at the U.K.'s Rutherford Appleton Laboratory, it appears that it was left to go off into the wilderness rather than someone keeping an eye on things. With that in mind, here are the installation packages for Windows and Linux (RPM):

Windows Installer

Linux RPM

RALCGM is a handy command line tool that can covert from CGM to Postscript on its own without any need for ImageMagick at all. From what I have seen, fonts on graphical output may look greyer than black, but it otherwise does its job well. However, considering that it is a freely available tool, one cannot complain too much. There are other packages for doing vector to raster conversion and the ones that I have seen do have GUI's but the freedom to look at for cost software wasn't mine to have. The required command looks something like the following:

ralcgm -d PS -oL test.cgm test.ps

The switch -d PS uses the software's Postscript driver and -oL specifies landscape orientation. If you like to find out more, here's a PDF rendition of the help file that comes with the thing:

RALCGM Documentation

A certain lack of speed

A little while, I encountered a problem with ImageMagick processing DNG files in Ubuntu 9.04. Not realising that I could solve me own problem by editing a file named delegates.xml, I took to getting a Debian VM to do the legwork for me. That's where you'll find all the commands used when helper software is used by ImageMagick to help it on its way. On its own, ImageMagick cannot deal with DNG files, so the command line variant of UFRaw (itself a front end for DCRaw) is used to create a PNM file that ImageMagick can handle. The problem a few months back was that the command in delegates.xml wasn't appropriate for a newer version of UFRaw and I got it into my head that things like this were hard-wired into ImageMagick. Now, I know better and admit my error.

With 9.10, it appears that the command in delegates.xml has been corrected, only for another issue to raise its head. UFRaw 0.15, it seems, isn't the speediest when it comes to creating PNM files and, while my raw file processing script works after a spot of modification to deal with changes in output from the identify command used, it takes far too long to run. Since GIMP also uses UFRaw, I wonder if the same problem has surfaced there too. However, it has been noticed by the Debian team, with the outcome being that they have a package for version 0.16 of the software in their unstable branch that looks as if it has sorted the speed issue. However, I am seeing that 0.15 is in the testing branch, which tempts me to stick with Lenny (5.x) if any successor turns out to have slower DNG file handling with ImageMagick and UFRaw. In my estimation, 0.13 does what I need, so why go for a newer release if it turns out to be slower?

What? No DNG?

Since Google's Picasa is now available for Linux, I decided to give it a spin. The availability of downloadable DEB packages made installation a piece of cake. It has been ported using WINE, which means that the look and feel is very much of the Windows world. The functionality is similar too, which means that it can seek out image files on your PC. However, it didn't find any DNG files for me, a surprise when DNG could become the standard raw camera format for digital imaging. In contrast, the lack of support for proprietary formats like Canon's CRW might be understandable. Because I like to review the raw image files before committing to editing them, Picasa will not suffice for this purpose, leading me to stick with what I already use.