Technology Tales

Copying only updated new or updated files by command line in Linux or Windows

2^nd August 2014

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don't want to keep copying the same files over and over again, so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is on Linux, it is the bash shell that I use, so the switches may not apply with others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always, so it needs adding. The r switch is useful because the copy then in recursive, so only a directory needs to be specified as the source and the destination needs to be one level up from a folder with the same name there to avoid file duplication. It is the u switch that makes the file copy incremental, and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session, and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental, and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files, while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried, so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening, and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell on Linux.

Suppressing Update Notifier messages in Ubuntu GNOME 14.04

18^th July 2014

Even though I use only the command line for system updates, I still received system restart messages after kernel updates. Though these can be helpful, I prefer to restart my system on my own schedule, especially when running processes I don't want to interrupt.

In Ubuntu GNOME 13.10 and earlier, these messages didn't appear. I wanted to return to a notification-free experience. The Update Notifier application causes these messages, but removing it significantly impacts the system, making that approach impractical.

The application starts automatically at boot but doesn't appear in Startup Applications Preferences (launched via gnome-session-properties). This is because the NoDisplay flag in its autostart shortcut located in /etc/xdg/autostart/ is set to true. To make it visible, you need to set this flag to false.

The following command will reveal all hidden startup applications (note that the command is split across two lines for display purposes, and you may need to adjust quotes when executing it):

sudo sed -i 's/NoDisplay=true/NoDisplay=false/g' /etc/xdg/autostart/*.desktop

The sed command changes NoDisplay=true to NoDisplay=false in all desktop files, revealing hidden entries in Startup Applications Preferences. This allows you to disable the Update Notifier. While deleting the desktop file would be more permanent, this solution works well as it stops restart notifications. Since I regularly update my system and shut down daily, updates are applied automatically, and everything functions properly.

Removing advertisements from uTorrent

12^th July 2014

BitTorrent may have got some bad press due to its use for downloading copyrighted material such as music and movies, but it does have its legitimate uses too. In my case, many a Linux distro has been downloaded in this way, and it does take the weight off servers by distributing the load across users instead.

Speaking of Linux, my general choice of client has been Transmission and there are others. In the Windows world, there is a selection that includes BitTorrent, Inc. themselves. However, many favour uTorrent (or μTorrent) so that's the one that I tried and there are free and subscription-based options. To me, the latter feels like overkill when an eternal licence could be made available as an easy way to dispatch the advertisements on display in the free version.

As much as I appreciate the need for ads to provide revenue to a provider of otherwise free software, they do need to be tasteful and those in uTorrent often were for dating websites that had no scruples about exposing folk to images that were unsuitable for a work setting. Those for gaming websites were more tolerable in comparison. With the non-availability of an eternal licence option, I was left pondering alternatives like qBittorrent instead. That is Free Software too, so it does have that added advantage.

However, I uncovered an article on Lifehacker that sorted my problem with uTorrent. The trick is to go into Options > Preferences via the menus and then go to the Advanced section in the dialogue box that appears. In there, go looking for each of the following options and set each one to false in turn:

offers.left_rail_offer_enabled/left_rail_offer
gui.show_plus_upsell
offers.sponsored_torrent_offer_enabled/sponsored_torrent_offer_enabled
bt.enable_pulse
gui.show_notorrents_node
offers.content_offer_autoexec

In practice, I found some of the above already set to false and another missing, though setting those that remained from true to false cleaned up the interface, so I hope never to glimpse those unsuitable ads again. The maker of uTorrent needs to look at the issue or revenue could get lost, and prospective users could see the operation as being cheapened by what is displayed. As for me, I am happy to have gained something in the way of control.

Dropping back to a full screen terminal session from a desktop one in Linux

29^th May 2014

There are times when you might need to access a full screen terminal from a Linux graphical desktop. For example, I have needed this when installing Nvidia's graphics drivers on Ubuntu or Linux Mint. Another instance occurred on Arch Linux when a Cinnamon desktop update prevented me from opening a terminal window. The full screen command let me install an alternative terminal emulator, with Tech Drive-in's list proving helpful. Similar issues might need fixing on FreeBSD installations. These latter examples happened within VirtualBox, which has special requirements for accessing full screen command line sessions, which I'll explain later.

When running Linux on a physical PC, press CTRL + ALT + F1 to enter a full screen terminal and CTRL + ALT + F7 to return to the graphical desktop. In a Linux VirtualBox guest with a Linux host, these shortcuts affect the host instead. For the guest OS, use [Host Key] + F1 to enter a full screen terminal and [Host Key] + F7 to return to the graphical desktop. The default Host Key is the right CTRL key, unless you've changed it.

X sessions in GNOME and Cinnamon desktop environments support this functionality, but I can't confirm it works with alternatives like Wayland. Hopefully, this feature extends to other setups, as terminal sessions are occasionally needed for system recovery. Such mishaps are thankfully rare and should be virtually non-existent for most users.

Fixing Background Image Display in GNOME Shell 3.10

2^nd May 2014

On upgrading from Ubuntu GNOME 13.10 to Ubuntu GNOME 14.04, a few rough edges were to be noticed. One was the display of my chosen background image: it was garbled. Later, I discovered that there is a maximum width of 2560 px for background images in GNOME Shell these days and that things get messy beyond that.

In my case, the image width was around 6000 px, and I was used to it getting resized in GNOME Shell 3.8 and its predecessors. It appears that the functionality got removed after that though, so the workaround of manual image resizing in the GIMP needed to be employed. Though having big images open in memory creates an additional overhead, not handling them very well at all looks like a bug caused by setting 2560 px as a maximum screen width for the GNOME Shell panel and the complete removal of Nautilus from desktop rendering duties. Let's hope that sense is seen with ever larger screen sizes and resolutions coming our way.

It's the sort of thing that did get me looking at adding on Cinnamon 2.2 for a while before setting background image scaling using the indispensable GNOME Tweak Tool was discovered. LinuxG.net has a useful tutorial on this for anyone with such an adventurous streak in them. For now though, I am OK with my set-up but the GNOME project's focus on minimalism could affect us in other ways, so I can see why Clem Lefebvre started the Cinnamon one primarily for Linux Mint and the desktop environment is appearing elsewhere too. After all, Gedit lost its menu bar in GNOME 3.12 so it's just as well that we have alternatives.

Update 2014-05-06: It appears that the desktop image bug that afflicts GNOME Shell 3.10 got sorted for GNOME Shell 3.12. At least, that is the impression that an Antergos instance in a VirtualBox virtual machine gives me.

ERROR: Can't find the archive-keyring

10^th April 2014

When I recently did my usual system update for the stable version Ubuntu GNOME, there were some updates pertaining to apt and the process failed when I executed the following command:

sudo apt-get upgrade

Usefully, some messages were issued and here's a flavour:

Setting up apt (0.9.9.1~ubuntu3.1) ... ERROR: Can't find the archive-keyring Is the ubuntu-keyring package installed? dpkg: error processing apt (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: apt E: Sub-process /usr/bin/dpkg returned an error code (1)

Web searches indicated the issue was missing files in /usr/share/keyring, which I didn't delete. Since apt was disabled due to the missing keyring files, installing software for fixes was impossible. My solution was to copy the /usr/share/keyring files from an Ubuntu GNOME 14.04 virtual machine to the same location on my Ubuntu GNOME 13.10 host. For others without this setup, I've included these files in a zip file below. While other solutions like Y PPA were mentioned, they required prior installation, making them useless when tools like Synaptic were unavailable. I'd appreciate information on other fixes that don't involve reinstalling the operating system, potential causes for the file loss, and how to prevent it.

Ubuntu Keyrings

Sorting out a system update failure for FreeBSD

3^rd April 2014

With my tendency to apply Linux updates using the command, I was happy to see that something similar was possible in FreeBSD too. The first step is to fire up a terminal session and drop into root using the su command. That needs the root superuser password to continue, and the next step is to update the local repositories using the following command:

pkg update

After that, it is time to download updated packages and install these by issuing this command:

pkg upgrade

Most of the time, that is sufficient, but I discovered that there are times when the above fails and additional interventions are needed. What I had uncovered were dependency error messages, and I set to looking around the web for remedies to this. One forum question that was similar to what I had met with the suggestion of consulting the file called UPDATING in /usr/ports/. An answer like that looks unhelpful, but for the inclusion of advice where extra actions were needed. Also, there is a useful article on updating FreeBSD ports that gives more in the way of background knowledge so you understand the more about what needs doing.

Following both that and the UPDATING file resulted in my taking the following sequence of steps. The first act was to download and initialise the Ports Collection, a set of build instructions.

portsnap fetch extract

The above is a one time only action, so future updates are done as follows:

portsnap fetch update

With an up to date Ports Collection in place, it was time to install portman:

pkg install portman

A look through /usr/ports/UPDATING revealed the commands I needed for updating Python and Perl to address the dependency problem that I was having:

portmaster -o devel/py-setuptools27 devel/py-setuptools portmaster -r py\*setuptools

With those completed, I re-ran pkg update again and all was well. The extra actions needed to get that result will not get forgotten, and I am sharing them on here so I know where they are. If anyone else has use for them, that would be even better.

Installing FreeBSD in a VirtualBox Virtual Machine

2^nd March 2014

With UNIX being the basis of Linux, I have a soft spot for trying out any UNIX that can be installed on a PC. For a while, I had OpenSolaris on the go and even vaguely recall having a look at one of the BSD's. However, any recent attempt to install one of the latter, and there are quite a few around now, got stymied by some sort of kernel panic caused by using AMD CPU's. With the return to the Intel fold arising from the upgrade of my main home PC last year, it perhaps was time to try again.

The recent release of FreeBSD 10.0 was the cue and I downloaded a DVD image for a test installation in a VirtualBox virtual machine with 4 GB of memory and a 32 GB virtual hard drive attached (expanding storage was chosen so not all the allocated space has been taken so far). The variant of FreeBSD chosen was the 64-bit x86 one, and I set to installing it in there. Though not as pretty in appearance as those in various Linux distros, the installer was not that user unfriendly to me. Mind you, I have experience of installing Arch Linux, which might have acclimatised me somewhat.

Those installation screens ask about the keyboard mapping that you want, and I successfully chose one of the UK options. There was limited opportunity for adding extras, though there was a short list of a few from which I made some selections. Given that user account set up also was on offer, I would have been better off knowing what groups to assign for my personal user account to have to avoid needing to log in as root so often following system start up later. Otherwise, all the default options were sufficient.

When the installation process was complete, it was time to boot into the new system and all that was on offer was a command line log in session. After logging in as root, it was time to press pkg into service to get a desktop environment in place. The first step was to install X:

pkg install xorg

Then, it was time to install a desktop environment. While using XFCE or KDE were alternatives, I chose GNOME 2 due to familiarity and more extensive instructions on the corresponding FreeBSD handbook page. Issuing the following command added GNOME and all its helper applications:

pkg install gnome2

So that GNOME starts up at the next reboot, some extra steps are needed. The first of these is to add the following line into /etc/fstab:

proc /proc procfs rw 0 0

Then, two lines were needed in /etc/rc.conf:

gdm_enable="YES" gnome_enable="YES"

The first enables the GNOME display manager, while the second activates other GNOME programs that are needed for a desktop session to start. With each of these in place, I got a graphical login screen at the next boot time.

With FreeBSD being a VirtualBox Guest, it was time to consult the relevant FreeBSD manual page. Here, there are sections for a number of virtual machine tools, so a search was needed to find the one for VirtualBox. VirtualBox support for FreeBSD is incomplete in that there is no installation media for BSD systems, while Linux and Solaris are supported along with Windows. Therefore, it is over to the FreeBSD repositories for the required software:

pkg install virtualbox-ose-additions

Aside from the virtual machine session not capturing and releasing the mouse pointer automatically, that did everything that was needed, even if it was the open source edition of the drivers and their proprietary equivalents. To resolve the mouse pointer issue, I needed to temporarily disable the GNOME desktop session in /etc/rc.conf to drop to a console only session where xorg.conf could be generated using the following commands:

Xorg -configure cp xorg.conf.new /etc/xorg.conf

In the new xorg.conf file, the mouse section needs to be as follows:

Section "InputDevice" Identifier "Mouse0" Driver "vboxmouse" EndSection

If it doesn't look like the above, and it wasn't the case for me, then it needs changing. Also, any extra lines from the default set up also need removing, or the mouse will not function as it should. The ALT+F1 (for accessing GNOME menus) and ALT+F2 (for running commands) keyboard shortcuts then become crucial when your mouse is not working as it should and could avert a panic too; knowing that adjusting a single configuration file will resolve a problem when doing so is less accessible is not a good feeling as I discovered to my own cost. The graphics settings were fine by default, but here's what you should have in case it isn't for you:

Section "Device" ### Available Driver options are:- ### Values: <i>: integer, <f>: float, <bool>: "True"/"False", ### <string>: "String", <freq>: "<f> Hz/kHz/MHz" ### [arg]: arg optional Identifier "Card0" Driver "vboxvideo" VendorName "InnoTek Systemberatung GmbH" BoardName "VirtualBox Graphics Adapter" BusID "PCI:0:2:0" EndSection

The next step is to ensure that your HAL settings are as they should. I needed to create a file in /usr/local/etc/hal/fdi/policy called 90-vboxguest.fdi that contains the following:

<?xml version="1.0" encoding="utf-8"?>  <deviceinfo version="0.2"> <device> <match key="info.subsystem" string="pci"> <match key="info.product" string="VirtualBox guest Service"> <append key="info.capabilities" type="strlist">input</append> <append key="info.capabilities" type="strlist">input.mouse</append> <merge key="input.x11_driver" type="string">vboxmouse</merge> <merge key="input.device" type="string">/dev/vboxguest</merge> </match> </match> </device> </deviceinfo>

With all that set, it is time to ensure that the custom user account is added to the wheel and operator groups using this command:

pw user mod [user name] -G wheel operator

Executing the above as root means that the custom account can run the su command so that logging in as root at the start of a desktop session no longer is needed. That is what being in the wheel group allows, so anyone in the operator group can shut down or restart the system. Since both are facilities readily available on Linux, so I fancied having them in FreeBSD too.

Being able to switch to root in a terminal session meant that I could go on to add software like Firefox, LibreOffice, GIMP, EMACS, Geany, NetBeans, Banshee and so on. Though there may be a line of opinion that FreeBSD is a server operating system, all of these make it more than passable for serving as a desktop one too. There may be no package management GUI as such and the ones that come with GNOME do not work either, yet anyone familiar with command line working will get around that.

While FreeBSD may be conservative, that has its place too, and being able to build up a system one item at a time teaches far more than getting everything already sorted in one hit. So far, there is enough documentation to get me going, leaving me to see where else things go too. So far, the OS hasn't been that intimidating, which is good to see.

The bold gamble behind Linux Voice magazine

1^st March 2014

During the latter part of last year, the magazine Linux Format suffered a staff clear-out, and I was left wondering why. It was as if a load of folk left at once and, even if I have seen that sort of thing happening at my current place of work, I was asking if something went wrong at Future Publishing. Instead, I had missed the fact that the former Linux Format staff were starting their own magazine. They crowdfunded it on Indiegogo. It took the appearance of Linux Voice on a shelf in the Macclesfield WHSmith's for me to become enlightened about this.

It seems risky for a whole team from one publisher's magazine to leave and create their own similar publication, especially given the current instability of magazine publishing in the digital age. The mention of a non-compete agreement reminded me of my own workplace. Their former employers' reaction would be interesting to know, as mine might consider legal action if I did something similar, assuming it were possible; I too would be bound by a six-month non-compete clause after leaving.

Regarding the magazine's content, it is appropriately good. While occasional misspellings might occur, articles on OwnCloud and Arch Linux installation, along with reviews of Mageia 4 and FreeBSD 10, would interest me. Many familiar names from Linux Format are also present, creating a sense of continuity. The new magazine's design is less extravagant than its established competitor, and their coexistence will be worth observing.

New ideas take time to develop, and I wish the new magazine success. Its goals are positive: half of its profits will support open-source software, and articles will be openly accessible via a Creative Commons licence. However, its immediate financial stability is crucial, making the next few months significant. The experienced team behind the magazine is a strong asset and could prevent it from becoming like Walking World Ireland and Cycling World, which appear irregularly in stores. The support of an enthusiastic community is also beneficial. I might eventually have to choose between Linux Voice and Linux Format, similar to my choice between Linux Magazine and Linux User & Developer. Despite being a niche operating system, Linux users have a good selection of magazines.

Some SAS Macro code for detecting the presence or absence of a variable in a dataset

4^th December 2013

Recently, I needed to put in place some code to detect the presence or absence of a variable in a dataset, and I chose SAS Macro programming as the way to do what I wanted. The logic was based on a SAS sample that achieved the same result in a data step and some code that I had for detecting the presence or absence of a dataset. Mixing the two together gave me something like the following:

%macro testvar(ds=,var=);

%let dsid=%sysfunc(open(&ds,in)); %let varexist=%sysfunc(varnum(&dsid,&var)); %if &dsid > 0 %then %let rc=%sysfunc(close(&dsid));

%if &varexist gt 0 %then %put Info: Variable &var is in the &ds dataset; %else %put Info: Variable &var is not in the &ds dataset;

%mend testvar;

%testvar(ds=dataset,var=var);

What this does is open up a dataset and look for the variable number in the dataset. In datasets, variables are numbered from left to right with 1 for the first one, 2 for the second and so on. If the variable is not in the dataset, the result is 0 so you know that it is not there. All of this is what the VARNUM SCL function within the SYSFUNC macro function does. In the example, this resolves to %sysfunc(varnum(&dsid,var)) with no quotes around the variable name, akin to what you would do in data step programming. Once you have the variable number or 0, then you can put in place some conditional logic that makes use of the information, like what you see in the above simple example. Of course, that would be expanded to something more useful in real life, but I hope it helps to show you the possibilities here.

« Older Entries « » Newer Entries »