Technology Tales

Updating Piwik using the Linux Command Line

Because updating Piwik using its web interface has proved tempestuous, I have decided to update the self-hosted analytics application on an SSH session. The production web servers that I use are hosted on Linux systems, so that is why any commands apply to the Linux or UNIX command line only. What is needed for Windows servers may differ.

The first step is to down the required ZIP file with this command:

wget https://builds.piwik.org/piwik.zip

Once the download is complete, the contents of the ZIP archive are extracted into a new subfolder. This is a process that I carry out in a separate folder to that where the website files are kept before copying everything from the extraction folder in there. Here is the unzip command, and the -o switch turns on overwriting of any previously existing files:

unzip -o piwik.zip

Without the required folder in the web server area to be updated, the next step is to do the actual system update that includes any updates to the Piwik database that you are using. There are two commands that you can use once you have specified the location of your Piwik installation. The second is needed when the first option cannot find where the PHP executable is stored. My systems had something more specific than these because both PHP 5.6 and PHP 7.0 are installed. Looking in /usr/bin was enough to find what I needed to execute in place of PHP below. Otherwise, the command was the same.

./[path to piwik]/console core:update

php [path to piwik]/console core:update

While the upgrade is ongoing, it prompts you to permit it to continue before it goes and modifies the database. This did not take long on my systems, but that depends on how much data there is. Once, the process has completed, you can delete any extraneous files using the rm command.

Reloading .bashrc within a BASH terminal session

BASH is a command-line interpreter that is commonly used by Linux and UNIX operating systems. Chances are that you will find yourself in a BASH session if you start up a terminal emulator in many of these, though there are others like KSH and SSH too.

BASH comes with its own configuration files and one of these is located in your own home directory, .bashrc. Among other things, it can become a place to store command shortcuts or aliases. Here is an example:

alias us='sudo apt-get update && sudo apt-get upgrade'

Such a definition needs there to be no spaces around the equals sign, and the actual command to be declared in single quotes. Doing anything other than this will not work, as I have found. Also, there are times when you want to update or add one of these and use it without shutting down a terminal emulator and restarting it.

To reload the .bashrc file to use the updates contained in there, one of the following commands can be issued:

source ~/.bashrc

. ~/.bashrc

Both will read the file and execute its contents so you get those updates made available so you can continue what you are doing. There appears to be a tendency for this kind of thing in the world of Linux and UNIX because it also applies to remounting drives after a change to /etc/fstab and restarting system services like Apache, MySQL or Nginx. The command for the former is below:

sudo mount -a

Often, the means for applying the sorts of in-situ changes that you make are simple ones too, and anything that avoids system reboots has to be good since you have less work interruptions.

Compressing a VirtualBox VDI file for a Linux guest

In a previous posting, I talked about compressing a virtual hard disk for a Windows guest system running in VirtualBox on a Linux system. Since then, I have needed to do the same for a Linux guest following some housekeeping. Because the Linux distribution used is Debian, the instructions are relevant to that and maybe its derivatives such as Ubuntu, Linux Mint and their like.

While there are other alternatives like dd, I am going to stick with a utility named zerofree to overwrite the newly freed up disk space with zeroes to aid compression later on in the process for this and the first step is to install it using the following command:

apt-get install zerofree

Once that has been completed, the next step is to unmount the relevant disk partition. Luckily for me, what I needed to compress was an area that I reserved for synchronisation with Dropbox. If it was the root area where the operating system files are kept, a live distro would be needed instead. In any event, the required command takes the following form, with the mount point being whatever it is on your system (/home, for instance):

sudo umount [mount point]

With the disk partition unmounted, zerofree can be run by issuing a command that looks like this:

zerofree -v /dev/sdxN

Above, the -v switch tells zerofree to display its progress and a continually updating percentage count tells you how it is going. The /dev/sdxN piece is generic with the x corresponding to the letter assigned to the disk on which the partition resides (a, b, c or whatever) and the N is the partition number (1, 2, 3 or whatever; before GPT, the maximum was 4). Putting all this together, we get an example like /dev/sdb2.

Once, that had completed, the next step is to shut down the VM and execute a command like the following on the host Linux system ([file location/file name] needs to be replaced with whatever applies on your system):

VBoxManage modifyhd [file location/file name].vdi --compact

With the zero filling in place, there was a lot of space released when I tried this. While it would be nice for dynamic virtual disks to reduce in size automatically, I accept that there may be data integrity risks with those, so the manual process will suffice for now. It has not been needed that often anyway.

Killing Windows processes from the command line

During my days at work, I often hear about the need to restart a server because something has gone awry with it. This makes me wonder if you can kill processes from the command line, like you do in Linux and UNIX. A recent need to reset Windows Update on a Windows 10 machine gave me enough reason to answer the question.

Because I already knew the names of the services, I had no need to look at the Services tab in the Task Manager like you otherwise would. Then, it was a matter of opening up a command line session with Administrator privileges and issuing a command like the following (replacing [service name] with the name of the service):

sc queryex [service name]

From the output of the above command, you can find the process identifier, or PID. With that information, you can execute a command like the following in the same command line session (replacing [PID] with the actual numeric value of the PID):

taskkill /f /pid [PID]

After the above, the process no longer exists and the service can be restarted. With any system, you need to find the service that is stuck to kill it, but that would be the subject of another posting. What I have not got to testing is whether these work in PowerShell, since I used them with the legacy command line instead. Along with processes belonging to software applications (think Word, Excel, Firefox, etc.), that may be something else to try should the occasion arise.

A few more shell commands

Here are some Linux commands that I encountered in a feature article in the current issue of Linux User & Developer that I had not met before:

cd -

This returns you to the previous directory where you were before with having to go back through the folder hierarchy to get there and is handy if you are jumping around a file system and any other means is far from speedy.

lsb_release -a

It can be useful to uncover what version of a distro you have from the command line and the above works for distros as diverse as Linux Mint, Debian, Fedora (it automatically installs in Fedora 22 if it is not installed already, a more advanced approach than showing you the command like in Linux Mint or Ubuntu), openSUSE and Manjaro. These days, the version may not change too often, but it still is good to uncover what you have.

yum install fedora-upgrade

This one can be run either with sudo or in a root session started with su and it is specific to Fedora. The command performs an upgrade of the Fedora distro itself, and I wonder if the functionality has been ported to the dnf command that has taken over from yum. My experiences with that in Fedora 22 so far suggest that it should be the case, though I need to check that further with the VirtualBox VM that I have created.

Controlling clearance of /tmp on Linux systems

While some may view the behaviour in a less favourable, I always have liked the way that Linux can clear its /tmp directory every time the system is restarted. The setting for this is in /etc/default/rcS and the associated line looks something like:

TMPTIME=0

The value of 0 means that the directory is flushed completely every time the system is restarted, but there are other options. A setting of -1 makes the directory behave like any other one on the system, where any file deletions are manual affairs. Using other positive integer values like 7 will specify the number of days that a file can stay in /tmp before it is removed.

What brought me to this topic was the observation that my main Linux Mint system was accumulating files in /tmp and the cause was the commenting out of the TMPTIME=0 line in /etc/default/rcS. This is not the case on Ubuntu, and using that is how I got accustomed to automatic file removal from /tmp in the first place.

All of this discussion so far has pertained to PC's where systems are turned off or restarted regularly. Things are different for servers of course and I have seen tools like tmpreaper and tmpwatch being given a mention. As if to prove that there is more than one way to do anything on Linux, shell scripting and cron remain an ever present fallback.

Restoring GRUB for dual booting of Linux and Windows

Once you end up with Windows overwriting your master boot record (MBR), you have lost the ability to use GRUB. Therefore, it would be handy to get it back if you want to start up Linux again. Though the loss of GRUB from the MBR was a deliberate act of mine, I knew that I'd have to restore GRUB to get Linux working again. So, I have been addressing the situation with a Live DVD for the likes of Ubuntu or Linux Mint. Once one of those had loaded its copy of the distribution, issuing the following command in a terminal session gets things back again:

sudo grub-install --root-directory=/media/0d104aff-ec8c-44c8-b811-92b993823444 /dev/sda

When there were error messages, I tried this one to see if I could get additional information:

sudo grub-install --root-directory=/media/0d104aff-ec8c-44c8-b811-92b993823444 /dev/sda --recheck

Also, it is possible to mount a partition on the boot drive and use that in the command to restore GRUB. Here is the required combination:

sudo mount /dev/sda1 /mnt
sudo grub-install --root-directory=/mnt /dev/sda

Either of these will get GRUB working without a hitch, and they are far more snappy than downloading Boot-Repair and using that; I was doing that for a while until a feature on triple booting appeared in an issue of Linux User & Developer that reminded me of the more readily available option. Once, there was a need to manually add an entry for Windows 7 to the GRUB menu too and, with that instated, I was able to dual-boot Ubuntu and Windows using GRUB to select which one was to start for me. Since then, I have been able to dual boot Linux Mint and Windows 8.1, with GRUB finding the latter all by itself. Since your experiences too may show this variation, it's worth bearing in mind.

Copying only updated new or updated files by command line in Linux or Windows

With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don't want to keep copying the same files over and over again, so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:

cp -pruv [source] [destination]

Because this is on Linux, it is the bash shell that I use, so the switches may not apply with others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always, so it needs adding. The r switch is useful because the copy then in recursive, so only a directory needs to be specified as the source and the destination needs to be one level up from a folder with the same name there to avoid file duplication. It is the u switch that makes the file copy incremental, and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.

What inspired this post though is my need to do the same in a Windows session, and issuing xcopy commands will achieve the same end. Here are two that will do the needful:

xcopy [source] [destination] /d /s

xcopy [source] [destination] /d /e

In both cases, it is the d switch that ensures that the copy is incremental, and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files, while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried, so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening, and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell on Linux.

Removing advertisements from uTorrent

BitTorrent may have got some bad press due to its use for downloading copyrighted material such as music and movies, but it does have its legitimate uses too. In my case, many a Linux distro has been downloaded in this way, and it does take the weight off servers by distributing the load across users instead.

Speaking of Linux, my general choice of client has been Transmission and there are others. In the Windows world, there is a selection that includes BitTorrent, Inc. themselves. However, many favour uTorrent (or μTorrent) so that's the one that I tried and there are free and subscription-based options. To me, the latter feels like overkill when an eternal licence could be made available as an easy way to dispatch the advertisements on display in the free version.

As much as I appreciate the need for ads to provide revenue to a provider of otherwise free software, they do need to be tasteful and those in uTorrent often were for dating websites that had no scruples about exposing folk to images that were unsuitable for a work setting. Those for gaming websites were more tolerable in comparison. With the non-availability of an eternal licence option, I was left pondering alternatives like qBittorrent instead. That is Free Software too, so it does have that added advantage.

However, I uncovered an article on Lifehacker that sorted my problem with uTorrent. The trick is to go into Options > Preferences via the menus and then go to the Advanced section in the dialogue box that appears. In there, go looking for each of the following options and set each one to false in turn:

• offers.left_rail_offer_enabled/left_rail_offer
• gui.show_plus_upsell
• offers.sponsored_torrent_offer_enabled/sponsored_torrent_offer_enabled
• bt.enable_pulse
• gui.show_notorrents_node
• offers.content_offer_autoexec

In practice, I found some of the above already set to false and another missing, though setting those that remained from true to false cleaned up the interface, so I hope never to glimpse those unsuitable ads again. The maker of uTorrent needs to look at the issue or revenue could get lost, and prospective users could see the operation as being cheapened by what is displayed. As for me, I am happy to have gained something in the way of control.

Dropping back to a full screen terminal session from a desktop one in Linux

There are times when you might need to access a full screen terminal from a Linux graphical desktop. For example, I have needed this when installing Nvidia's graphics drivers on Ubuntu or Linux Mint. Another instance occurred on Arch Linux when a Cinnamon desktop update prevented me from opening a terminal window. The full screen command let me install an alternative terminal emulator, with Tech Drive-in's list proving helpful. Similar issues might need fixing on FreeBSD installations. These latter examples happened within VirtualBox, which has special requirements for accessing full screen command line sessions, which I'll explain later.

When running Linux on a physical PC, press CTRL + ALT + F1 to enter a full screen terminal and CTRL + ALT + F7 to return to the graphical desktop. In a Linux VirtualBox guest with a Linux host, these shortcuts affect the host instead. For the guest OS, use [Host Key] + F1 to enter a full screen terminal and [Host Key] + F7 to return to the graphical desktop. The default Host Key is the right CTRL key, unless you've changed it.

X sessions in GNOME and Cinnamon desktop environments support this functionality, but I can't confirm it works with alternatives like Wayland. Hopefully, this feature extends to other setups, as terminal sessions are occasionally needed for system recovery. Such mishaps are thankfully rare and should be virtually non-existent for most users.