Linux | Technology Tales

Copying only updated new or updated files by command line in Linux or Windows

2^nd August 2014

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don't want to keep copying the same files over and over again, so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is on Linux, it is the bash shell that I use, so the switches may not apply with others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always, so it needs adding. The r switch is useful because the copy then in recursive, so only a directory needs to be specified as the source and the destination needs to be one level up from a folder with the same name there to avoid file duplication. It is the u switch that makes the file copy incremental, and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session, and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental, and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files, while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried, so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening, and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell on Linux.

Removing advertisements from uTorrent

12^th July 2014

BitTorrent may have got some bad press due to its use for downloading copyrighted material such as music and movies, but it does have its legitimate uses too. In my case, many a Linux distro has been downloaded in this way, and it does take the weight off servers by distributing the load across users instead.

Speaking of Linux, my general choice of client has been Transmission and there are others. In the Windows world, there is a selection that includes BitTorrent, Inc. themselves. However, many favour uTorrent (or μTorrent) so that's the one that I tried and there are free and subscription-based options. To me, the latter feels like overkill when an eternal licence could be made available as an easy way to dispatch the advertisements on display in the free version.

As much as I appreciate the need for ads to provide revenue to a provider of otherwise free software, they do need to be tasteful and those in uTorrent often were for dating websites that had no scruples about exposing folk to images that were unsuitable for a work setting. Those for gaming websites were more tolerable in comparison. With the non-availability of an eternal licence option, I was left pondering alternatives like qBittorrent instead. That is Free Software too, so it does have that added advantage.

However, I uncovered an article on Lifehacker that sorted my problem with uTorrent. The trick is to go into Options > Preferences via the menus and then go to the Advanced section in the dialogue box that appears. In there, go looking for each of the following options and set each one to false in turn:

offers.left_rail_offer_enabled/left_rail_offer
gui.show_plus_upsell
offers.sponsored_torrent_offer_enabled/sponsored_torrent_offer_enabled
bt.enable_pulse
gui.show_notorrents_node
offers.content_offer_autoexec

In practice, I found some of the above already set to false and another missing, though setting those that remained from true to false cleaned up the interface, so I hope never to glimpse those unsuitable ads again. The maker of uTorrent needs to look at the issue or revenue could get lost, and prospective users could see the operation as being cheapened by what is displayed. As for me, I am happy to have gained something in the way of control.

Dropping back to a full screen terminal session from a desktop one in Linux

29^th May 2014

There are times when you might need to access a full screen terminal from a Linux graphical desktop. For example, I have needed this when installing Nvidia's graphics drivers on Ubuntu or Linux Mint. Another instance occurred on Arch Linux when a Cinnamon desktop update prevented me from opening a terminal window. The full screen command let me install an alternative terminal emulator, with Tech Drive-in's list proving helpful. Similar issues might need fixing on FreeBSD installations. These latter examples happened within VirtualBox, which has special requirements for accessing full screen command line sessions, which I'll explain later.

When running Linux on a physical PC, press CTRL + ALT + F1 to enter a full screen terminal and CTRL + ALT + F7 to return to the graphical desktop. In a Linux VirtualBox guest with a Linux host, these shortcuts affect the host instead. For the guest OS, use [Host Key] + F1 to enter a full screen terminal and [Host Key] + F7 to return to the graphical desktop. The default Host Key is the right CTRL key, unless you've changed it.

X sessions in GNOME and Cinnamon desktop environments support this functionality, but I can't confirm it works with alternatives like Wayland. Hopefully, this feature extends to other setups, as terminal sessions are occasionally needed for system recovery. Such mishaps are thankfully rare and should be virtually non-existent for most users.

Installing FreeBSD in a VirtualBox Virtual Machine

2^nd March 2014

With UNIX being the basis of Linux, I have a soft spot for trying out any UNIX that can be installed on a PC. For a while, I had OpenSolaris on the go and even vaguely recall having a look at one of the BSD's. However, any recent attempt to install one of the latter, and there are quite a few around now, got stymied by some sort of kernel panic caused by using AMD CPU's. With the return to the Intel fold arising from the upgrade of my main home PC last year, it perhaps was time to try again.

The recent release of FreeBSD 10.0 was the cue and I downloaded a DVD image for a test installation in a VirtualBox virtual machine with 4 GB of memory and a 32 GB virtual hard drive attached (expanding storage was chosen so not all the allocated space has been taken so far). The variant of FreeBSD chosen was the 64-bit x86 one, and I set to installing it in there. Though not as pretty in appearance as those in various Linux distros, the installer was not that user unfriendly to me. Mind you, I have experience of installing Arch Linux, which might have acclimatised me somewhat.

Those installation screens ask about the keyboard mapping that you want, and I successfully chose one of the UK options. There was limited opportunity for adding extras, though there was a short list of a few from which I made some selections. Given that user account set up also was on offer, I would have been better off knowing what groups to assign for my personal user account to have to avoid needing to log in as root so often following system start up later. Otherwise, all the default options were sufficient.

When the installation process was complete, it was time to boot into the new system and all that was on offer was a command line log in session. After logging in as root, it was time to press pkg into service to get a desktop environment in place. The first step was to install X:

pkg install xorg

Then, it was time to install a desktop environment. While using XFCE or KDE were alternatives, I chose GNOME 2 due to familiarity and more extensive instructions on the corresponding FreeBSD handbook page. Issuing the following command added GNOME and all its helper applications:

pkg install gnome2

So that GNOME starts up at the next reboot, some extra steps are needed. The first of these is to add the following line into /etc/fstab:

proc /proc procfs rw 0 0

Then, two lines were needed in /etc/rc.conf:

gdm_enable="YES" gnome_enable="YES"

The first enables the GNOME display manager, while the second activates other GNOME programs that are needed for a desktop session to start. With each of these in place, I got a graphical login screen at the next boot time.

With FreeBSD being a VirtualBox Guest, it was time to consult the relevant FreeBSD manual page. Here, there are sections for a number of virtual machine tools, so a search was needed to find the one for VirtualBox. VirtualBox support for FreeBSD is incomplete in that there is no installation media for BSD systems, while Linux and Solaris are supported along with Windows. Therefore, it is over to the FreeBSD repositories for the required software:

pkg install virtualbox-ose-additions

Aside from the virtual machine session not capturing and releasing the mouse pointer automatically, that did everything that was needed, even if it was the open source edition of the drivers and their proprietary equivalents. To resolve the mouse pointer issue, I needed to temporarily disable the GNOME desktop session in /etc/rc.conf to drop to a console only session where xorg.conf could be generated using the following commands:

Xorg -configure cp xorg.conf.new /etc/xorg.conf

In the new xorg.conf file, the mouse section needs to be as follows:

Section "InputDevice" Identifier "Mouse0" Driver "vboxmouse" EndSection

If it doesn't look like the above, and it wasn't the case for me, then it needs changing. Also, any extra lines from the default set up also need removing, or the mouse will not function as it should. The ALT+F1 (for accessing GNOME menus) and ALT+F2 (for running commands) keyboard shortcuts then become crucial when your mouse is not working as it should and could avert a panic too; knowing that adjusting a single configuration file will resolve a problem when doing so is less accessible is not a good feeling as I discovered to my own cost. The graphics settings were fine by default, but here's what you should have in case it isn't for you:

Section "Device" ### Available Driver options are:- ### Values: <i>: integer, <f>: float, <bool>: "True"/"False", ### <string>: "String", <freq>: "<f> Hz/kHz/MHz" ### [arg]: arg optional Identifier "Card0" Driver "vboxvideo" VendorName "InnoTek Systemberatung GmbH" BoardName "VirtualBox Graphics Adapter" BusID "PCI:0:2:0" EndSection

The next step is to ensure that your HAL settings are as they should. I needed to create a file in /usr/local/etc/hal/fdi/policy called 90-vboxguest.fdi that contains the following:

<?xml version="1.0" encoding="utf-8"?>  <deviceinfo version="0.2"> <device> <match key="info.subsystem" string="pci"> <match key="info.product" string="VirtualBox guest Service"> <append key="info.capabilities" type="strlist">input</append> <append key="info.capabilities" type="strlist">input.mouse</append> <merge key="input.x11_driver" type="string">vboxmouse</merge> <merge key="input.device" type="string">/dev/vboxguest</merge> </match> </match> </device> </deviceinfo>

With all that set, it is time to ensure that the custom user account is added to the wheel and operator groups using this command:

pw user mod [user name] -G wheel operator

Executing the above as root means that the custom account can run the su command so that logging in as root at the start of a desktop session no longer is needed. That is what being in the wheel group allows, so anyone in the operator group can shut down or restart the system. Since both are facilities readily available on Linux, so I fancied having them in FreeBSD too.

Being able to switch to root in a terminal session meant that I could go on to add software like Firefox, LibreOffice, GIMP, EMACS, Geany, NetBeans, Banshee and so on. Though there may be a line of opinion that FreeBSD is a server operating system, all of these make it more than passable for serving as a desktop one too. There may be no package management GUI as such and the ones that come with GNOME do not work either, yet anyone familiar with command line working will get around that.

While FreeBSD may be conservative, that has its place too, and being able to build up a system one item at a time teaches far more than getting everything already sorted in one hit. So far, there is enough documentation to get me going, leaving me to see where else things go too. So far, the OS hasn't been that intimidating, which is good to see.

The bold gamble behind Linux Voice magazine

1^st March 2014

During the latter part of last year, the magazine Linux Format suffered a staff clear-out, and I was left wondering why. It was as if a load of folk left at once and, even if I have seen that sort of thing happening at my current place of work, I was asking if something went wrong at Future Publishing. Instead, I had missed the fact that the former Linux Format staff were starting their own magazine. They crowdfunded it on Indiegogo. It took the appearance of Linux Voice on a shelf in the Macclesfield WHSmith's for me to become enlightened about this.

It seems risky for a whole team from one publisher's magazine to leave and create their own similar publication, especially given the current instability of magazine publishing in the digital age. The mention of a non-compete agreement reminded me of my own workplace. Their former employers' reaction would be interesting to know, as mine might consider legal action if I did something similar, assuming it were possible; I too would be bound by a six-month non-compete clause after leaving.

Regarding the magazine's content, it is appropriately good. While occasional misspellings might occur, articles on OwnCloud and Arch Linux installation, along with reviews of Mageia 4 and FreeBSD 10, would interest me. Many familiar names from Linux Format are also present, creating a sense of continuity. The new magazine's design is less extravagant than its established competitor, and their coexistence will be worth observing.

New ideas take time to develop, and I wish the new magazine success. Its goals are positive: half of its profits will support open-source software, and articles will be openly accessible via a Creative Commons licence. However, its immediate financial stability is crucial, making the next few months significant. The experienced team behind the magazine is a strong asset and could prevent it from becoming like Walking World Ireland and Cycling World, which appear irregularly in stores. The support of an enthusiastic community is also beneficial. I might eventually have to choose between Linux Voice and Linux Format, similar to my choice between Linux Magazine and Linux User & Developer. Despite being a niche operating system, Linux users have a good selection of magazines.

Setting up a WD My Book Live NAS on Ubuntu GNOME 13.10

1^st December 2013

The official line from Western Digital is this: they do not support the use of their My Book Live NAS drives with Linux or UNIX. However, what that means is that they only develop tools for accessing their products for Windows and maybe OS X. It still doesn't mean that you cannot access the drive's configuration settings by pointing your web browser at http://mybooklive.local/. In fact, not having those extra tools is no drawback at all since the drive can be accessed through your file manager of choice under the Network section and the default name is MyBookLive too, so you easily can find the thing once it is connected to a router, or switch anyway.

Once you are in the server's web configuration area, you can do things like changing its name, updating its firmware, finding out what network has been assigned to it, creating and deleting file shares, password protecting file shares and other things. These are the kinds of things that come in handy if you are going to have a more permanent connection to the NAS from a PC that runs Linux. The steps that I describe have worked on Ubuntu 12.04 and 13.10 with the GNOME desktop environment.

What I was surprised to discover that you cannot just set up a symbolic link that points to a file share. Instead, it needs to be mounted and this can be done from the command line using mount or at start-up with /etc/fstab. For this to happen, you need the Common Internet File System utilities and these are added as follows if you need them (check in the Software Centre or in Synaptic):

sudo apt-get install cifs-utils

Once these are added, you can add a line like the following to /etc/fstab:

//[NAS IP address]/[file share name] /[file system mount point] cifs credentials=[full file location]/.creds, iocharset=utf8, sec=ntlm, gid=1000, uid=1000, file_mode=0775, dir_mode=0775 0 0

Though I have broken it over several lines above, this is one unwrapped line in /etc/fstab with all the fields in square brackets populated for your system and with no brackets around these. Though there are other ways to specify the server, using its IP address is what has given me the most success; this is found under Settings > Network on the web console. Next up is the actual file share name on the NAS; I have used a custom term instead of the default of Public. The NAS file share needs to be mounted to an actual directory in your file system like /media/nas or whatever you like; however, you will need to create this beforehand. After that, you have to specify the file system, and it is cifs instead of more conventional alternatives like ext4 or swap. After this and before the final two space delimited zeroes in the line comes the chunk that deals with the security of the mount point.

What I have done in my case is to have a password-protected file share and the user ID and password have been placed in a file in my home area with only the owner having read and write permissions for it (600 in chmod-speak). Preceding the filename with a "." also affords extra invisibility. That file then is populated with the user ID and password like the following. Of course, the bracketed values have to be replaced with what you have in your case.

username=[NAS file share user ID] password=[NAS file share password]

With the credentials file created, its options have to be set. First, there is the character set of the file (usually UTF-8 and I got error code 79 when I mistyped this) and the security that is to be applied to the credentials (ntlm in this case). To save having no write access to the mounted file share, the uid and gid for your user needs specification, with 1000 being the values for the first non-root user created on a Linux system. After that, it does no harm to set the file and directory permissions because they only can be set at mount time; using chmod, chown and chgrp afterwards, has no effect whatsoever. Here, I have set permissions to read, write and execute for the owner and the user group while only allowing read and execute access for everyone else (that's 775 in the world of chmod).

All of what I have described here worked for me and had to be gleaned from disparate sources like Mount Windows Shares Permanently from the Ubuntu Wiki, another blog entry regarding the permissions settings for a CIFS mount point and an Ubuntu forum posting on mounting CIFS with UTF-8 support. Because of the scattering of information, I just felt that it needed to all together in one place for others to use, and I hope that fulfils someone else's needs similarly to mine.

A way to get Rigo working again in Sabayon

23^rd October 2013

After having Sabayon running on a PC until it came to pieces after an attempted version upgrade, I went away from the Linux distro for a while and Linux Mint now runs on the aforementioned machine. It only was a certain curiosity that got me installing it into a virtual machine on VirtualBox to see if my command line method of keeping the system up to date was the cause or whether rolling or partially rolling distros have a certain fragility that is not seen in their discrete release counterparts.

Recently, that ran into a hitch, the Sabayon package manager Rigo failed to start up for me. After waiting to see if it sorted itself on its own, I looked into returning to those command line ways, and that line of enquiry led me to a method of restoring Rigo's functionality from Sabayon's wiki page on the underlying Entropy. The first step was to issue a command to become root:

su

That needed the appropriate password, and the next command issued updated Sabayon's repositories:

equo update

Once that had done its thing, it was time to install new versions of Entropy and Rigo:

equo install entropy rigo

With that complete, it was time to exit the root session with the exit command. Then, it was time to try running Rigo and it worked as expected. Any thoughts of adding in the superseded Sulfur (Rigo's predecessor) were banished on seeing that success.

Command Line Processing of EXIF Image Metadata

8^th July 2013

There is a bill making its way through the U.K. parliament at this time that could reduce the power of copyright when it comes to images placed on the web. The current situation is that anyone who creates an image automatically holds the copyright for it. However, the new legislation will remove that if it becomes law as it stands. As it happens, the Royal Photographic Society is doing what it can to avoid any changes to what we have now. Though there may be the barrier of due diligence, how many of us take steps to mark our own intellectual property? For one, I have been less that attentive to this and now wonder if there is anything more that I should be doing. While others may copyleft their images instead, I don't want to find myself unable to share my own photos because another party is claiming rights over them. There's watermarking as an object, yet I also want to add something to the image metadata too.

That got me wondering about adding metadata to any images that I post online that assert my status as the copyright holder. It may not be perfect, but any action is better than doing nothing at all. Given that I don't post photos where EXIF metadata is stripped as part of the uploading process, it should be there to see for anyone who bothers to check, and there may not be many who do.

Because I also wanted to batch process images, I looked for a command line tool to do the needful and found ExifTool. Being a Perl library, it is cross-platform so you can use it on Linux, Windows and even OS X. To install it on a Debian or Ubuntu-based Linux distro, just use the following command:

sudo apt-get install libimage-exiftool-perl

The form of the command that I found useful for adding the actual copyright information is below:

exiftool -p "-copyright=(c) John ..." -ext jpg -overwrite_original

The -p switch preserves the timestamp of the image file, while the -overwrite_original one ensures that you don't end up with unwanted backup files. The copyright message goes within the quotes along with the -copyright option. With a little shell scripting, you can traverse a directory structure and change the metadata for any image files contained in different sub-folders. If you wish to do more than this, there's always the user documentation to be consulted.

Turning on autocompletion for the bash shell in terminal sessions

26^th June 2013

At some point, I managed to lose the ability to have tab-key-based autocompletion on terminal sessions on my Ubuntu GNOME machine. Wanting it caused had me to turn to the web for an answer, and I found it on a Linux Mint forum; the bash shell is so pervasive in the UNIX and Linux worlds that you can look anywhere for a fix like this.

The problem centred around the .bashrc file in my home area. It does have quite a few handy custom aliases, and I must have done a foolish spring-clean on the file sometime. That is the only way that I can explain how the following lines got removed:

if [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
fi

What they do is look to see if /etc/bash_completion can be found on your system and to use it for tab-based autocompletion. With the lines not in .bashrc, it couldn't happen. Others may replace bash_completion with bash.bashrc to get a fuller complement of features, but I'll stick with what I have for now.

A little look at Debian 7.0

12^th May 2013

Having a virtual machine with Debian 6 on there, I was interested to hear that Debian 7.0 is out. In another VM, I decided to give it a go. Installing it on there using the Net Install CD image took a little while but proved fairly standard with my choice of the GUI-based option. GNOME was the desktop environment with which I went and all started up without any real fuss after the installation was complete; it even disconnected the CD image from the VM before rebooting, a common failing in many Linux operating installations that lands into the installation cycle again unless you kill the virtual machine.

Though the GNOME desktop looked familiar, a certain amount of conservatism reigned too, since the version was 3.4.2. That was no bad thing, since raiding the GNOME Extension site for a set of mature extensions was made easier. In fact, a certain number of these were included in the standard installation anyway and the omission of a power off entry on the user menu was corrected as a matter of course without needing any intervention from this user. Adding to what already was there made for a more friendly desktop experience in a short period of time.

Debian's variant of Firefox, IceWeasel, is version 10 so a bit of tweaking is needed to get the latest version. LibreOffice is there now too, and it's version 3.5 rather than 4. Shotwell too is the older 0.12 and not the 0.14 that is found in the likes of Ubuntu 13.04. As it happens, GIMP is about the only software with a current version and that is 2.8; a slower release cycle may be the cause of that, though. All in all, the general sense is that older versions of current software are being included for the sake of stability and that is sensible too, so I am not complaining very much about this at all.

The reason for not complaining is that the very reason for having a virtual machine with Debian 6 on there is to have Zinio and Dropbox available too. Adobe's curtailment of support for Linux means that any application needing Adobe Air may not work on a more current Linux distribution. That affects Zinio, so I'll be retaining a Debian 6 instance for a while yet, unless a bout of testing reveals that a move to the newer version is possible. As for Dropbox, I am sure that I can recall why I moved it onto Debian, but it's working well on there so I am in no hurry to move it over either. There are times when slower software development cycles are better...

« Older Entries « » Newer Entries »