Copying only updated new or updated files by command line in Linux or Windows

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don’t want to keep copying the same files over and over again so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is in Linux, it the bash shell that I use so the switches may not apply to others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always so it needs adding. The r switch is useful because the copy then in recursive so only a directory needs to specified as the source and the destination needs to be one level up from a folder with the same name there so as to avoid file duplication. It is the u switch that makes the file copy incremental and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell in Linux.

Recursive FTP with the command line

Here’s a piece of Linux/UNIX shell scripting code that will do a recursive FTP refresh of a website for you:

lftp <<~/Tmp/log_file.tmp 2>>~/Tmp/log_file.tmp

open ${HOSTNAME}

user ${USER} ${PSSWD}

mirror -R -vvv “${REP_SRC}” “${REP_DEST}”

EndFTP

When my normal FTP scripting approach left me with a broken WordPress installation and an invalid ticket in the project’s TRAC system that I had to close, I turned to looking for a more robust way of achieving the website updates and that’s what led me to seek out the options available for FTP transfers that explicitly involve directory recursion. The key pieces in the code above are the use of lftp in place of ftp, my more usual tool for the job, and the invocation of the mirror command that comes with lftp. The -R switch ensures that file transfer is from local to remote (vice versa is the default) and -vvv turns on maximum verbosity, a very useful thing when you find that it takes longer than more usual means. It’s all much slicker than writing your own script to do the back-work of ploughing through the directory structure and ensuring that the recursive transfers take place. Saying that, it is possible to have a one line variant of the above but the way that I have set things up might be more familiar to users of ftp.

iPod, identified

Plug in an iPod to a PC running Ubuntu and it will recognise what it has got. That act mounts the player as a hard drive and fires up the Rhythmbox Music Player. The usual file transfer capabilities are available and it does something that was thwarted partially by iTunes when I last tried it: transferring files from your iPod to your PC. Only music bought from the iTunes store can copied from the player back to the PC. Unsurprisingly, you cannot update the iPod’s firmware or anything like that. To do such things, you need the iTunes player and that means having either Windows or OS X. While I do wonder if it can’t be that hard to port the OS X version to Linux since they both share UNIX roots, it’s over to the Windows VM for me on this one for now.

Connecting to Host USB Devices from VMware

However, while VMware on Windows will happily pick up USB devices as they are connected so long as the VM is in focus, the behaviour on Linux seems to be different. As shown above, you have to go to the VM menu and potter down the chain (Removable Devices > USB Devices) to make the device of interest accessible. Dialogue boxes asking you if you want to disconnect the device from the host operating system will appear and the process may be unsubtle as you progress with it. In fact, Ubuntu was delivering warning messages about how its iPod connection got lost; it would have been wise to unmount the thing in the first place. Accessing USB devices like this opens up other possibilities: using Windows for scanning and for printing digital images.

Returning to the iPod story, Windows will see it once it has been made available and iTunes can access it accordingly. Then, you are free to update the gadget’s firmware or manage the music stored on it, if you prefer.