TOPIC: CROSS-PLATFORM SOFTWARE
Copying only updated new or updated files by command line in Linux or Windows
2nd August 2014
With a growing collection of photographic images, I often find myself making backups of files using copy commands and the data volumes are such that I don't want to keep copying the same files over and over again, so incremental file transfers are what I need. So commands like the following often get issued from a Linux command line:
cp -pruv [source] [destination]
Because this is on Linux, it is the bash shell that I use, so the switches may not apply with others like ssh, fish or ksh. For my case, p preserves file properties such as its time and date and the cp command does not do this always, so it needs adding. The r switch is useful because the copy then in recursive, so only a directory needs to be specified as the source and the destination needs to be one level up from a folder with the same name there to avoid file duplication. It is the u switch that makes the file copy incremental, and the v one issues messages to the shell that show how the copying is going. Seeing a file name issued by the latter does tell you how much more needs to be copied and that the files are going where they should.
What inspired this post though is my need to do the same in a Windows session, and issuing xcopy commands will achieve the same end. Here are two that will do the needful:
xcopy [source] [destination] /d /s
xcopy [source] [destination] /d /e
In both cases, it is the d switch that ensures that the copy is incremental, and you can add a date too, with a colon between it and the /d, if you see fit. The s switch copies only directories that contain files, while the e one copies even empty directories. Using the d switch without either of those did not trigger any copying action when I tried, so I reckon that you cannot do without either of them. By default, both of these commands issue output to the command line so you can keep an eye on what is happening, and this especially is useful when ensuring that files are going to the right destination because the behaviour differs from that of the bash shell on Linux.
Removing advertisements from uTorrent
12th July 2014
BitTorrent may have got some bad press due to its use for downloading copyrighted material such as music and movies, but it does have its legitimate uses too. In my case, many a Linux distro has been downloaded in this way, and it does take the weight off servers by distributing the load across users instead.
Speaking of Linux, my general choice of client has been Transmission and there are others. In the Windows world, there is a selection that includes BitTorrent, Inc. themselves. However, many favour uTorrent (or μTorrent) so that's the one that I tried and there are free and subscription-based options. To me, the latter feels like overkill when an eternal licence could be made available as an easy way to dispatch the advertisements on display in the free version.
As much as I appreciate the need for ads to provide revenue to a provider of otherwise free software, they do need to be tasteful and those in uTorrent often were for dating websites that had no scruples about exposing folk to images that were unsuitable for a work setting. Those for gaming websites were more tolerable in comparison. With the non-availability of an eternal licence option, I was left pondering alternatives like qBittorrent instead. That is Free Software too, so it does have that added advantage.
However, I uncovered an article on Lifehacker that sorted my problem with uTorrent. The trick is to go into Options > Preferences via the menus and then go to the Advanced section in the dialogue box that appears. In there, go looking for each of the following options and set each one to false in turn:
offers.left_rail_offer_enabled/left_rail_offergui.show_plus_upselloffers.sponsored_torrent_offer_enabled/sponsored_torrent_offer_enabledbt.enable_pulsegui.show_notorrents_nodeoffers.content_offer_autoexec
In practice, I found some of the above already set to false and another missing, though setting those that remained from true to false cleaned up the interface, so I hope never to glimpse those unsuitable ads again. The maker of uTorrent needs to look at the issue or revenue could get lost, and prospective users could see the operation as being cheapened by what is displayed. As for me, I am happy to have gained something in the way of control.
Turning off seccomp sandbox in vsftpd
21st September 2013
Within the last week, I set up a virtual web server using Arch Linux to satisfy my own curiosity, since the DIY nature of Arch means that you can build up exactly what you need without having any real constraints put upon you. Something that didn't surprise me about this was that it took me more work than the virtual server that I created using Ubuntu Server, yet I didn't expect Proftpd to be missing from the main repositories. Though the package can be found in the AUR, I didn't fancy the prospect of dragging more work on myself, so I went with vsftpd (Very Secure FTP Daemon) instead. In contrast to Proftpd, this is available in the standard repositories and there is a guide to its use in the Arch user documentation.
However, while vsftpd worked well just after installation, connections to the virtual FTP soon failed with FileZilla began issuing uninformative messages. In fact, it was the standard command line FTP client on my Ubuntu machine that was more revealing. It issued the following message that let me to the cause after my engaging the services of Google:
500 OOPS: priv_sock_get_cmd
With version 3.0 of vsftpd, a new feature was introduced, and it appears that this has caused problems for a few people. That feature is seccomp_sandbox and it can be turned off by adding the following line in /etc/vsftpd.conf:
seccomp_sandbox=NO
That solved my problem, and version 3.0.2 of vsftpd should address the issue with seccomp sandboxing anyway. In case, this solution isn't as robust as it should be because seccomp is not supported in the Linux kernel that you are using, turning off the new feature still needs to be an option, though.
Protecting your photos with copyright metadata using ExifTool
8th July 2013
There is a bill making its way through the U.K. parliament at this time that could reduce the power of copyright when it comes to images placed on the web. The current situation is that anyone who creates an image automatically holds the copyright for it. However, the new legislation will remove that if it becomes law as it stands. As it happens, the Royal Photographic Society is doing what it can to avoid any changes to what we have now.
Though there may be the barrier of due diligence, how many of us take steps to mark our own intellectual property? For one, I have been less that attentive to this and now wonder if there is anything more that I should be doing. While others may copyleft their images instead, I don't want to find myself unable to share my own photos because another party is claiming rights over them. There's watermarking as an object, yet I also want to add something to the image metadata too.
That got me wondering about adding metadata to any images that I post online that assert my status as the copyright holder. It may not be perfect, but any action is better than doing nothing at all. Given that I don't post photos where EXIF metadata is stripped as part of the uploading process, it should be there to see for anyone who bothers to check, and there may not be many who do.
Because I also wanted to batch process images, I looked for a command line tool to do the needful and found ExifTool. Being a Perl library, it is cross-platform so you can use it on Linux, Windows and even OS X. To install it on a Debian or Ubuntu-based Linux distro, just use the following command:
sudo apt-get install libimage-exiftool-perl
The form of the command that I found useful for adding the actual copyright information is below:
exiftool -p "-copyright=(c) John ..." -ext jpg -overwrite_original
The -p switch preserves the timestamp of the image file, while the -overwrite_original one ensures that you don't end up with unwanted backup files. The copyright message goes within the quotes along with the -copyright option. With a little shell scripting, you can traverse a directory structure and change the metadata for any image files contained in different sub-folders. If you wish to do more than this, there's always the user documentation to be consulted.
Using the IN operator in SAS Macro programming
8th October 2012
This useful addition came in SAS 9.2, and I am amazed that it isn’t enabled by default. To accomplish that, you need to set the MINOPERATOR option, unless someone has done it for you in the SAS AUTOEXEC or another configuration program. Thus, the safety first approach is to have code like the following:
options minoperator;
%macro inop(x);
%if &x in (a b c) %then %do;
%put Value is included;
%end;
%else %do;
%put Value not included;
%end;
%mend inop;%inop(a);
Also, the default delimiter is the space, so if you need to change that, then the MINDELIMITER option needs setting. Adjusting the above code so that the delimiter now is the comma character gives us the following:
options minoperator mindelimiter=",";
%macro inop(x);
%if &x in (a b c) %then %do;
%put Value is included;
%end;
%else %do;
%put Value not included;
%end;
%mend inop;%inop(a);
Without any of the above, the only approach is to have the following, and that is what we had to do for SAS versions before 9.2:
%macro inop(x);
%if &x=a or &x=b or &x=c %then %do;
%put Value is included;
%end;
%else %do;
%put Value not included;
%end;
%mend inop;%inop(a);
While it may be clunky, it does work and remains a fallback in newer versions of SAS. Saying that, having the IN operator available makes writing SAS Macro code that little bit more swish, so it's a good thing to know.
Setting up MySQL on Sabayon Linux
27th September 2012
For quite a while now, I have offline web servers for doing a spot of tweaking and tinkering away from the gaze of web users that visit what I have on there. Therefore, one of the tests that I apply to any prospective main Linux distro is the ability to set up a web server on there. This is more straightforward for some than for others. For Ubuntu and Linux Mint, it is a matter of installing the required software and doing a small bit of configuration. My experience with Sabayon is that it needs a little more effort than this, so I am sharing it here for the installation of MySQL.
The first step is to install the software using the commands that you find below. The first pops the software onto the system while the second completes the set-up. The --basedir option is need with the latter because it won't find things without it. It specifies the base location on the system, and it's /usr in my case.
sudo equo install dev-db/mysql
sudo /usr/bin/mysql_install_db --basedir=/usr
With the above complete, it's time to start the database server and set the password for the root user. That's what the two following commands achieve. Once your root password is set, you can go about creating databases and adding other users using the MySQL command line
sudo /etc/init.d/mysql start
mysqladmin -u root password 'password'
The last step is to set the database server to start every time you start your Sabayon system. The first command adds an entry for MySQL to the default run level so that this happens. The purpose of the second command is to check that this happened before restarting your computer to discover if it really happens. This procedure also is necessary for having an Apache web server behave in the same way, so the commands are worth having and even may have a use for other services on your system. ProFTP is another that comes to mind, for instance.
sudo rc-update add mysql default
sudo rc-update show | grep mysql
Renaming multiple files in Linux
19th August 2012
The Linux and UNIX command mv has a number of limitations, such as not overwriting destination files and not renaming multiple files using wildcards. The only solution to the first that I can find is one that involves combining the cp and rm commands. For the second, there's another command: rename. Here's an example like what I used recently:
rename s/fedora/fedora2/ fedora.*
The first argument in the above command is a regular expression much like what Perl is famous for implementing; in fact, it is Perl-compatible ones (PCRE) that are used. The s before the first slash stands for substitute, with fedora being the string that needs to be replaced and fedora2 being what replaces it. The third command is the file name glob that you want to use, fedora.* in this case. Therefore, all files in a directory named fedora will be renamed fedora2 regardless of the file type. The same sort of operation can be performed for all files with the same extension when it needs to be changed, htm to html, for instance. Of course, there are other uses, but these are handy ones to know.
Synchronising package selections between Linux Mint and Linux Mint Debian Edition
18th April 2012
To generate the package list on the GNOME version of Linux Mint, I used the Backup Tool. It simply was a matter of using the Backup Software Selection button and telling it where to put the file that it generates. Alternatively, dpkg can be used from the command line like this:
sudo dpkg --get-selections > /backup/installed-software.txt
After transferring the file to the machine with Linux Mint Debian Edition, I tried using the Backup Tool on there too. However, using the Restore Software Selection button and loading the required only produced an irrecoverable error. Therefore, I set to looking around the web and found a command line approach that did the job for me.
The first step is to load the software selection using dpkg by issuing this command (it didn't matter that the file wasn't made using the dpkg command, though I suspect that's what the Linux Mint Backup Tool was doing that behind the scenes):
sudo dpkg --set-selections < /backup/installed-software.txt
Then, I started dselect and chose the installation option from the menu that appeared. The first time around, it fell over but trying again was enough to complete the job. Packages available to the vanilla variant of Linux Mint but not found in the LMDE repositories were overlooked as I had hoped, and installation of the extra packages had no impact on system stability either.
sudo dselect
Apparently, there is an alternative to using dselect that is based on the much used apt-get command, but I didn't make use of it so cannot say more:
sudo apt-get dselect-upgrade
All that I can say is that the dpkg/dselect combination did what I wanted, so I'll keep them in mind if ever need to synchronise software selections between two Debian-based distributions in the future again. While the standard edition of Linux Mint may be based on Ubuntu rather than Debian, Ubuntu is itself based on Debian. Thus, the description holds here.
Sorting out MySQL on Arch Linux
5th November 2011
Seeing Arch Linux running so solidly in a VirtualBox virtual box has me contemplating whether I should have it installed on a real PC. Saying that, recent announcements regarding the implementation of GNOME 3 in Linux Mint have caught my interest, even if the idea of using a rolling distribution as my main home operating system still has a lot of appeal for me. Having an upheaval come my way every six months when a new version of Linux Mint is released is the main cause of that.
While remaining undecided, I continue to evaluate the idea of Arch Linux acting as my main OS for day-to-day home computing. Towards that end, I have set up a working web server instance on there using the usual combination of Apache, Perl, PHP and MySQL. Of these, it was MySQL that went the least smoothly of all because the daemon wouldn't start for me.
It was then that I started to turn to Google for inspiration, and a range of actions resulted that combined to give the result that I wanted. One problem was a lack of disk space caused by months of software upgrades. Since tools like it in other Linux distros allow you to clear some disk space of obsolete installation files, I decided to see if it was possible to do the same with pacman, the Arch Linux command line package manager. The following command, executed as root, cleared about 2 GB of cruft for me:
pacman -Sc
The S in the switch tells pacman to perform package database synchronization, while the c instructs it to clear its cache of obsolete packages. In fact, using the following command as root every time an update is performed both updates software and removes redundant or outmoded packages:
pacman -Syuc
So I don't forget the needful housekeeping, this will be what I use from this point forward, with y being the switch for a refresh and u triggering a system upgrade. It's nice to have everything happen together without too much effort.
To do the required debugging that led me to the above along with other things, I issued the following command:
mysqld_safe --datadir=/var/lib/mysql/ &
This starts up the MySQL daemon in safe mode if all is working properly, and it wasn't in my case. Nevertheless, it creates a useful log file called myhost.err in /var/lib/mysql/. This gave me the messages that allowed the debugging of what was happening. It led me to installing net-tools and inettools using pacman; it was the latter of these that put hostname on my system and got the MySQL server startup a little further along. Other actions included unlocking the ibdata1 data file and removing the ib_logfile0 and ib_logfile1 files to gain something of a clean sheet. The kill command was used to shut down any lingering mysqld sessions too. To ensure that the ibdata1 file was unlocked, I executed the following commands:
mv ibdata1 ibdata1.bad
cp -a ibdata1.bad ibdata1
These renamed the original and then crafted a new duplicate of it, with the -a switch on the cp command forcing copying with greater integrity than normal. Along with the various file operations, I also created a link to my.cnf, the MySQL configuration file on Linux systems, in /etc using the following command executed by root:
ln -s /etc/mysql/ my.cnf /etc/my.cnf
While I am unsure if this made a real difference, uncommenting the lines in the same file that pertained to InnoDB tables. What directed me to these were complaints from mysqld_safe in the myhost.err log file. All I did was to uncomment the lines beginning with innodb and these were 116-118, 121-122 and 124-127 in my configuration file, but it may be different in yours.
After all the above, the MySQL daemon ran happily and, more importantly, started when I rebooted the virtual machine. Thinking about it now, I believe that it was a lack of disk space, the locking of a data file and the lack of InnoDB support that was stopping the MySQL service from running. Running commands like mysqld start weren't yielding useful messages, so a lot of digging was needed to get the result that I needed. In fact, that's one of the reasons why I am sharing my experiences here.
In the end, creating databases and loading them with data were all that was needed for me to start to see functioning websites on my (virtual) Arch Linux system. It turned out to be another step on the way to making it workable as a potential replacement for the Linux distributions that I use most often (Linux Mint, Fedora and Ubuntu).
Turning off the admin bar in WordPress 3.1
25th October 2010
Work on WordPress 3.1 is in full swing at the moment, though I initially thought that they were taking a little break after 3.0. From what I can see, many refinements are being made to the multi-blog functionality and behind-the-scenes work is ongoing on the administration screens too.
Another under-the-bonnet change has been to make WordPress less tied to MySQL, since the possibility of dropping in support for an alternative such as PostgreSQL is now a reality even if it isn't part of the default package. For now, it looks as if this will be plugin territory rather than default multi-database support, though that may become a sensible development in the light of Oracle's acquisition of MySQL and its sabre-rattling regarding Java patents. So far, the change to WordPress has affected my use of its database engine to power an offline version of my online photo gallery, but a quick spot of code editing sorted that issue.
One more obvious alteration will be the addition of a WordPress.com style administration bar to the top of all content and administration screens for a user who is logged into the system. Though it will be turned on by default, there will be the option of turning it off for those among who prefer things that way. All that will be needed for this is to add the following line near the top of wp-config.php:
define( "WP_SHOW_ADMIN_BAR", false);
The chance to see new additions like those above and be ready for is my main reason for following WordPress development. It's best to be ready instead of being surprised, though it has to be said that the blogging or CMS platform is a very polished one these days.