Technology Tales

The wonders of mod_rewrite

When I wrote about tidying dynamic URL’s a little while back, I had no inkling that that would be a second part to the tale. My discovery of mod_rewrite, an Apache module that facilitates URL translation. The effect is that one URL is presented to the user in the browser address bar, and the exact same URL is also seen by search engines, while another is passed to the server for processing. It might sound like subterfuge but it works very well once you manage to get it set up properly. The web host for my hillwalking blog/photo gallery has everything configured such it is ready to go but the same did not apply to the offline Apache 2.2.x server that I have going on my own Windows XP box. There were two parts to getting it working there:

1. Activating mod-rewrite on the server: this is as easy as uncommenting a line in the httpd.conf file for the site (the line in question is: LoadModule rewrite_module modules/mod_rewrite.so).
2. Ensuring that the .htaccess file in the root of the web server directory is active. You need to set the values of the AllowOverride directives for the server root and CGI directories to All so that .htaccess is active. Not doing it for the latter will result in the an error beginning with the following: Options FollowSymLinks or SymLinksIfOwnerMatch is off which implies that RewriteRule directive is forbidden. Having Allow from All set for the required directories is another option to consider when you see errors like that.

Once you have got the above sorted, adding this line to .htaccess: RewriteEngine On. Preceding it with an Options directive to ensure that FollowSymLinks and SymLinksIfOwnerMatch are switched on does no harm at all and may even be needed to get things running. That done, you can set about putting mod_write to work with lines like this:

RewriteRule ^pages/(.*)/?$ pages.php?query=$1

The effect of this is to take http://www.website.com/pages/input and convert it into a form for action by the server; in this case, that is http://www.website.com/pages.php?query=input. Anything contained by a bracket is assigned to the value of a system-named variable. If you have several bracketed sections, they are assigned to sequentially numbered variables as follows: $1 for the first, $2 for the second and so on. It’s all good stuff when you get it going and not only does it make things look much neater but it also possesses an advantage when it comes to future-proofing too. Web addresses can be kept constant over time, even if things change behind the scenes. It means that returning visitors will find what they saw the last time that they visited and surely must ensure good karma in eyes of those all important search engines.

Why are there no savings on buying software using electronic distribution, Adobe?

If you ever potter over to Adobe’s online software store, a curious anomaly awaits you: electronic download editions of their software are never cheaper than the equivalent boxed versions. In fact, there are cases where the electronic version costs more than the boxed one. One would have thought that ditching the box, the disc(s) and whatever accompanies them would save Adobe money and they would pass this onto you but it does not seem to make its way into the pricing for some reason. Another thing is that selling direct should allow Adobe to undercut retailers and make more money from their software but it is the likes of Amazon that have the better prices. Whatever way you look at it, you have to admit that this pricing model doesn’t make a lot of sense.

Adding a new domain or subdomain to an SSL certificate using Certbot

On checking the Site Health page of a WordPress blog, I saw errors that pointed to a problem with its SSL set up. The www subdomain was not included in the site’s certificate and was causing PHP errors as a result though they had no major effect on what visitors saw. Still, it was best to get rid of them so I needed to update the certificate as needed. Execution of a command like the following did the job:

sudo certbot --expand -d existing.com,www.example.com

Using a Let’s Encrypt certificate meant that I could use the certbot command since that already was installed on the server. The --expand and -d switches ensured that the listed domains were added to the certificate to sort out the observed problem. In the above, a dummy domain name is used but this was replaced by the real one to produce the desired effect and make things as they should have been.

Moving a website from shared hosting to a virtual private server

This year has seen some optimisation being applied to my web presences guided by the results of GTMetrix scans. It was then that I realised how slow things were, so server loads were reduced. Anything that slowed response times, such as WordPress plugins, got removed. Usage of Matomo also was curtailed in favour of Google Analytics while HTML, CSS and JS minification followed. What had yet to happen was a search for a faster server. Now, another website has been moved onto a virtual private server (VPS) to see how that would go.

Speed was not the only consideration since security was a factor too. After all, a VPS is more locked away from other users than a folder on a shared server. There also is the added sense of control, so Let’s Encrypt SSL certificates can be added using the Electronic Frontier Foundation’s Certbot. That avoids the expense of using an SSL certificate provided through my shared hosting provider and a successful transition for my travel website may mean that this one undergoes the same move.

For the VPS, I chose Ubuntu 18.04 as its operating system, and it came with the LAMP stack already in place. Have offload development websites, the mix of Apache, MySQL and PHP is more familiar to me than anything using Nginx or Python. It also means that .htaccess files become more useful than they were on my previous Nginx-based platform. Having full access to the operating system using SSH helps too and should mean that I have fewer calls on technical support since I can do more for myself. Any extra tinkering should not affect others either, since this type of setup is well known to me and having an offline counterpart means that anything riskier is tried there beforehand.

Naturally, there were niggles to overcome with the move. The first to fix was to make the MySQL instance accept calls from outside the server so that I could migrate data there from elsewhere, and I even got my shared hosting setup to start using the new database to see what performance boost it might give. To make all this happen, I first found the location of the relevant my.cnf configuration file using the following command:

find / -name my.cnf

Once I had the right file, I commented out the following line that it contained and restarted the database service afterwards using another command to stop the appearance of any error 111 messages:

bind-address 127.0.0.1
service mysql restart

After that, things worked as required and I moved onto another matter: uploading the requisite files. That meant installing an FTP server, so I chose proftpd since I knew that well from previous tinkering. Once that was in place, file transfer commenced.

When that was done, I could do some testing to see if I had an active web server that loaded the website. Along the way, I also instated some Apache modules like mod-rewrite using the a2enmod command, restarting Apache each time I enabled another module.

Then, I discovered that Textpattern needed php-7.2-xml installed, so the following command was executed to do this:

apt install php7.2-xml

Then, the following line was uncommented in the correct php.ini configuration file that I found using the same method as that described already for the my.cnf configuration and that was followed by yet another Apache restart:

extension=php_xmlrpc.dll

Addressing the above issues yielded enough success for me to change the IP address in my Cloudflare dashboard so it pointed at the VPS and not the shared server. The changeover happened seamlessly without having to await DNS updates as once would have been the case. It had the added advantage of making both WordPress and Textpattern work fully.

With everything working to my satisfaction, I then followed the instructions on Certbot to set up my new Let’s Encrypt SSL certificate. Aside from a tweak to a configuration file and another Apache restart, the process was more automated than I had expected, so I was ready to embark on some fine-tuning to embed the new security arrangements. That meant updating .htaccess files and Textpattern has its own, so the following addition was needed there:

RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

This complemented what was already in the main .htaccess file and WordPress allows you to include http(s) in the address it uses, so that was another task completed. The general .htaccess only needed the following lines to be added:

RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.assortedexplorations.com/$1 [R,L]

What all these achieve is to redirect insecure connections to secure ones for every visitor to the website. After that, internal hyperlinks without https needed updating along with any forms so that a padlock sign could be shown for all pages.

With the main work completed, it was time to sort out a lingering niggle regarding the appearance of an FTP login page every time a WordPress installation or update was requested. The main solution was to make the web server account the owner of the files and directories, but the following line was added to wp-config.php as part of the fix even if it probably is not necessary:

define('FS_METHOD', 'direct');

There also was the non-operation of WP Cron and that was addressed using WP-CLI and a script from Bjorn Johansen. To make double sure of its effectiveness, the following was added to wp-config.php to turn off the usual WP-Cron behaviour:

define('DISABLE_WP_CRON', true);

Intriguingly, WP-CLI offers a long list of possible commands that are worth investigating. A few have been examined, but more await attention.

Before those, I still need to get my new VPS to send emails. So far, sendmail has been installed, the hostname changed from localhost and the server restarted. More investigations are needed, but what I have not is faster than what was there before, so the effort has been rewarded already.

Using .htaccess to control hotlinking

There are times when blogs cease to exist and the only place to find the content is on the Wayback Machine. Even then, it is in danger of being lost completely. One such example is the subject of this post.

Though this website makes use of the facilities of Cloudflare for various functions that include the blocking of image hotlinking, the same outcome can be achieved using .htaccess files on Apache web servers. It may work on Nginx to a point too but there are other configuration files that ought to be updated instead of using a .htaccess when some frown upon the approach. In any case, the lines that need adding to .htaccess are listed below though the web address needs to include your own domain in place of the dummy example provided:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpe?g|png|bmp)$ [F,NC]

The first line turns on the mod_rewrite engine and you may have that done anyway. Of course, the module needs enabling in your Apache configuration for this to work and you have to be allowed to perform the required action as well. This means changing the Apache configuration files. The next pair of lines look at the HTTP referer strings and the third one only allows images to be served from your own web domain and not others. To add more, you need to copy the third line and change the web address accordingly. Any new lines need to precede the last line that defines the file extensions that are to be blocked to other web addresses.

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule \.(gif|jpe?g|png|bmp)$ /images/image.gif [L,NC]

Another variant of the previous code involves changing the last line to display a default image showing others what is happening. That may not reduce the bandwidth usage as much as complete blocking but it may be useful for telling others what is happening.

It isn’t all iTunes in the UK

The iTunes store was a breath of fresh air following an experience of several OD2 offerings; broken downloads were a regular nuisance but that may have been down to my not having a broadband connection at the time. Its seamless mix of purchasing, downloading and playing impressed me so much that I used no other media player for my music in the days when I almost exclusively used Windows.

Now that i have jumped ship to Linux, having to fire up a Windows VM to hear my music is starting to feel a little over the top. The result is that I am keen to use DRM-free digital music when I can. Because I like to stay legal, it means that I would like to buy DRM-free files on the web. Here, iTunes leaves me down a little since most of what they offer is locked down and I have to burn a CD and extract from it to release music from its iTunes-only shackles.

So when I saw on an article on Tech.co.uk that made mention of 7Digital and that they purveyed unlocked music, my interest perked up. The file formats on offer are WMA, MP3 and AAC and there are high quality 320K variants of the latter two of these about too. Only the WMA files have any DRM associated with them. Previewing whole albums is a simple matter of clicking on a single button, a trick that iTunes would do well to learn. Payment using PayPal augments the usual credit card options and any purchases seem to be available for download more than once; pottering over to the My Locker part of your account gives you access to your purchases, another of its trump cards over iTunes. Downloading is on a file by file basis though and it is here that I notice an area usually addressed by a player like iTunes: the ability to download whole albums at once and background directory creation. Not having to have player has one advantage though: platform independence. Anyway, spot of shell scripting would resolve any file management gaps. Overall, there’s a lot to commend 7Digital and I wouldn’t be surprised if I were to return some time again. it might even usurp iTunes as my digital music store of choice…

Free & Open-Source Software

The terms Free Software and Open-Source Software often are used interchangeably though a certain Robert Stallman could have a thing or two to say about that. The first of these refers to freedom in the sense that you can do whatever you want with the software. That even means reprogramming it if it doesn’t do what you need. The whole concept began in the mid-1980’s and has grown since then.

The term Open-Source means that you can look at the source code though that apparently doesn’t mean that you always can do what you want with it. For that, it needs to be in the software licence and that’s where GPL, the GNU Public Licence, comes, though there also are competing licences such as those from BSD, which are far more permissive and business-friendly.

What GPL and its counterparts do not restrict is the ability to earn money from the software. You only have to look at what Red Hat earns to see that it can be done. It also means that open-source software (even the copylefted GPL-licensed variety) need not be free of charge.

In practice though, it is amazing what you get without paying for it. Whole Linux distributions with a wide selection of software coming with the operating system are a big example and there are many different ones too. When you see what Microsoft offers for a fee, it could come as a fair shock.

With that in mind, I thought it to be useful to offer an insight into the world of open-source software, especially give how much choice there is. It’s good to have options though that they can confound when they’re so many. However, recent instances of new software releases not being to users’ tastes make it a more important attribute than ever before.

There used to be a single list of what I thought worth highlighting, but that’s been divided now it got every, very long. Here are the categories that I have used for dividing up things so that they might be more useful:

Operating Systems

While Ubuntu or Linux Mint are among the most prominent of the Linux bunch, there are a multitude of others. Then, there are UNIX counterparts and the ones that I have found largely are based on BSD UNIX though there are OpenSolaris forks out there too. As if that weren’t all, some Linux distros are looking at using BSD UNIX kernels in addition to the ones that they usually have so that hybridisation cannot be ignored either.

Assorted Desktop Packages

This selection is all desktop software and then only a little sample of what there is to be found. Some make their way onto installation discs, but others have to be sought. All should do the work for which they’re designed, though.

Desktop Environments

While other operating systems typical offer you one interface at a time, Linux and UNIX have a tendency to offer plenty of choice. Sometimes, it’s the cause of controversy too, although major changes made to the two main players have meant fewer arguments between any advocates for either. In their stead, we have had moaning about what an open-source project has done on its users. Hopefully, that will subside though a meeting of minds may be needed for that in one case.

Web Servers

Websites would not exist without web servers and it was a choice between Apache’s open source HTTPD and Microsoft’s proprietary IIS until the upstart Nginx made its appearance. It, too, is open source and has been popping up in all sorts of places so it was time to make a short list for the sake of reference.

Databases & Programming

UNIX and Linux tend to attract those with an interest in technical computing, so programming and scripting languages remain an integral part of those operating systems, as can databases too. Here are a few of each.

Temptations, temptations…

The last time that I went out and bought a new camera was over two years ago and I am minded not to make another purchase for a while. Apart from damage to the battery cover arising from a fall, my Pentax K10D has survived so far without a problem and I admit to being satisfied with the photos that it makes. Following a professional sensor clean, my Canon EOS 10D has been pressed into service over the past few months too. In the meantime, 6 and 10 megapixel sensors generate nowhere near the attention that might have been case a few years back but that’s by the by. In fact, the megapixel race seems to have stalled with features like video being added to stills cameras over the last year and live screens coming to prominence also. Neither would make me rush out to buy a new DSLR anyway, perhaps because having things the old way suits me just fine and megapixel counts never ever moved me in the first place either.

That’s never to say that the likes of Pentax’s K-7 or Canon’s EOS 50D and 5D Mark II don’t capture my attention with their promises of better quality. However, with things the way that they are in the world, I am more likely to hold onto my cash or maybe invest in new photo processing software for making the most of what I already have. Ideas for photography projects creep into my head when I get to looking over my online photo gallery and realise that not have my tastes changed but my photographic eye has developed too. That seeing of things in a new light may mean that old subjects get revisited and I don’t need a new camera to do that.

High end compact cameras such as Canon’s G11 and Ricoh’s GR Digital III do detain attention for a while but a quick look at their prices proves that you really got to need the portability and I never can justify the outlay when a DSLR will do all that I want from it, and perhaps even for less money. While I admit to pondering the purchase of a GR Digital to cover for the EOS 10D while it was away for cleaning, the Pentax came to be acquired when I realised that the versatility of a DSLR was too much to lose, even for a while. Olympus’ E-P1 may have bridged the gap but the old question of going miniature for the price of a full sized article recurs.

All in all, I am going to stick with what I have right now. We are coming to a time of year when things appear more golden and that combination of lighting and colour are what really matters, not how many megapixels is in you camera sensor unless you are making large prints or supplying stock libraries. As long as my cameras continue to deliver pleasing results, I’ll stick with improving my skills and taking my time over that task, even with all the announcements of new cameras at various exhibitions and shows.

Moves to Hugo

What amazes me is how things can become more complicated over time. As long as you knew HTML, CSS and JavaScript, building a website was not as onerous as long as web browsers played ball with it. Since then, things have got easier to use but more complex at the same time. One example is WordPress: in the early days, themes were much simpler than they are now. The web also has got more insecure over time, and that adds to complexity as well. It sometimes feels as if there is a choice to make between ease of use and simplicity.

It is against that background that I reassessed the technology that I was using on my public transport and Irish history websites. The former used WordPress, while the latter used Drupal. The irony was that the simpler website was using the more complex platform, so the act of going simpler probably was not before time. Alternatives to WordPress were being surveyed for the first of the pair, but none had quite the flexibility, pervasiveness and ease of use that WordPress offers.

There is another approach that has been gaining notice recently. One part of this is the use of Markdown for web publishing. This is a simple and distraction-free plain text format that can be transformed into something more readable. It sees usage in blogs hosted on GitHub, but also facilitates the generation of static websites. The clutter is absent for those who have no need of the Gutenberg Editor on WordPress.

With the content written in Markdown, it can be fed to a static website generator like Hugo. Using defined templates and fixed assets like CSS together with images and other static files, it can slot the content into HTML files very speedily since it is written in the Go programming language. Once you get acclimatised, there are no folder structures that cannot be used, so you get full flexibility in how you build out your website. Sitemaps and RSS feeds can be built at the same time, both using the same input as the HTML files.

In a nutshell, it automates what once needed manual effort used a code editor or a visual web page editor. The use of HTML snippets and layouts means that there is no necessity for hand-coding content, like there was at the start of the web. It also helps that Bootstrap can be built in using Node, so that gives a basis for any styling. Then, SCSS can take care of things, giving even more automation.

Given that there is no database involved in any of this, the required information has to be stored somewhere, and neither the Markdown content nor the layout files contain all that is needed. The main site configuration is defined in a single TOML file, and you can have a single one of these for every publishing destination; I have development and production servers, which makes this a very handy feature. Otherwise, every Markdown file needs a YAML header where titles, template references, publishing status and other similar information gets defined. The layouts then are linked to their components, and control logic and other advanced functionality can be added too.

Because static files are being created, it does mean that site searching and commenting, or contact pages cannot work like they would on a dynamic web platform. Often, external services are plugged in using JavaScript. One that I use for contact forms is Getform.io. Then, Zapier has had its uses in using the RSS feed to tweet site updates on Twitter when new content gets added. Though I made different choices, Disqus can be used for comments and Algolia for site searching. Generally, though, you can find yourself needing to pay, particularly if you need to remove advertising or gain advanced features.

Some comments service providers offer open source self-hosted options, but I found these difficult to set up and ended up not offering commenting at all. That was after I tried out Cactus Comments only to find that it was not discriminating between pages, so it showed the same comments everywhere. There are numerous alternatives like Remark42, Hyvor Talk, Commento, FastComments, Utterances, Isso, Mouthful, Muut and HyperComments but trying them all out was too time-consuming for what commenting was worth to me. It also explains why some static websites even send readers to Twitter if they have something to say, though I have not followed this way of working.

For searching, I added a JavaScript/JSON self-hosted component to the transport website, and it works well. However, it adds to the size of what a browser needs to download. That is not a major issue for desktop browsers, but the situation with mobile browsers is such that it has a sizeable effect. Testing with PageSpeed and Lighthouse highlighted this, even if I left things as they are. The solution works well in any case.

One thing that I have yet to work out is how to edit or add content while away from home. Editing files using an SSH connection is as much a possibility as setting up a Hugo publishing setup on a laptop. After that, there is the question of using a tablet or phone, since content management systems make everything web based. These are points that I have yet to explore.

As is natural with a code-based solution, there is a learning curve with Hugo. Reading a book provided some orientation, and looking on the web resolved many conundrums. There is good documentation on the project website, while forum discussions turn up on many a web search. Following any research, there was next to nothing that could not be done in some way.

Migration of content takes some forethought and took quite a bit of time, though there was an opportunity to carry some housekeeping as well. The history website was small, so copying and pasting sufficed. For the transport website, I used Python to convert what was on the database into Markdown files before refining the result. That provided some automation, but left a lot of work to be done afterwards.

The results were satisfactory, and I like the associated simplicity and efficiency. That Hugo works so fast means that it can handle large websites, so it is scalable. The new Markdown method for content production is not problematical so far apart from the need to make it more portable, and it helps that I found a setup that works for me. This also avoids any potential dealbreakers that continued development of publishing platforms like WordPress or Drupal could bring. For the former, I hope to remain with the Classic Editor indefinitely, but now have another option in case things go too far.

Sorting out a system update failure for FreeBSD

With my tendency to apply Linux updates using the command, I was happy to see that something similar was possible in FreeBSD too. The first step is to fire up a terminal session and drop into root using the su command. That needs the root superuser password in order to continue and the next step is to update the local repositories using the following command:

pkg update

After that, it is time download updated packages and install these by issuing this command:

pkg upgrade

Most of the time, that is sufficient but I discovered that there are times when the above fails and additional interventions are needed. What I had uncovered were dependency error messages and I set to looking around the web for remedies to this. One forum question that was similar to what I had met with the suggestion of consulting the file called UPDATING in /usr/ports/. An answer like that looks unhelpful but for the inclusion of advice where extra actions were needed. Also, there is a useful article on updating FreeBSD ports that gives more in the way of background knowledge so you understand the more about what needs doing.

Following both that and the UPDATING  file resulted in my taking the following sequence of steps. The first act was to download and initialise the Ports Collection, a set of build instructions.

portsnap fetch extract

The above is a one time only action so future updates are done as follows:

portsnap fetch update

With an up to date Ports Collection in place, it was time to install portman:

pkg install portman

A look through /usr/ports/UPDTAING revealed the commands I needed for updating Python and Perl to address the dependency problem that I was having:

portmaster -o devel/py-setuptools27 devel/py-setuptools
portmaster -r py\*setuptools

With those completed, I re-ran pkg update again and all was well. The extra actions needed to get that result will not get forgotten and I am sharing them on here so I know where they are. If anyone else has use for them, that would be even better.