Technology Tales

Self-hosted web analytics tracking

It amazes me now to think how little tracking I used to do on my various web "experiments" only a few short years ago. However, there was a time when a mere web counter, perhaps displayed on web pages themselves, was enough to yield some level of satisfaction, or dissatisfaction in many a case. Things have come a long way since then, and we now seem to have analytics packages all around us. In fact, we don't even have to dig into our pockets to get our hands on the means to peruse this sort of information, either.

At this point, I need to admit that I am known to make use of a few simultaneously but thoughts about reducing their number are coming to mind, but there'll be more on that later. Given that this site is hosted using WordPress software, it should come as no surprise that Automattic's own plugin has been set into action to see how things are going. The main focus is on the total number of visits by day, week and month, with a breakdown showing what pages are doing well, together with an indication of how people came to the site and what links they followed while there. Don't go expecting details of your visitors like the software that they are using and the country where they are accessing the site with this minimalist option and satisfaction should head your way.

There is next to no way of discussing the subject of website analytics without mentioning Google's comprehensive offering in the area. You have to admit that it's comprehensive, with perhaps the only bugbear being the lack of live tracking. That need has been addressed very effectively by Woopra, even if its WordPress plugin will not work with IE6. Otherwise, you need the desktop application (being written in Java, it's a cross-platform affair and I have had it going in both Windows and Linux) but that works well too. Apart maybe from the lack of campaigns, Woopra supplies as good as all the information that its main competitor provides. It certainly does what I would need from it.

However, while they can be free as in beer, there are some costs associated with using external services like Google Analytics and Woopra. Their means of tracking your web pages for you is by executing a piece of JavaScript that needs to be added to every page. If you have everything set to use a common header or footer page, that shouldn't be too laborious, especially when there are plugins for publishing platforms like WordPress too. This way of working means that if anyone has JavaScript disabled or decides not to enable JavaScript for the requisite hosts while using the NoScript extension with Firefox, then your numbers are scuppered. Saying that, the same concerns probably any JavaScript code that you may want to execute, but there's another cost again: the calls to external websites can, even with the best attention in the world, slow down the loading of your own pages. When you add in latency caused by servers having to communicate across the web, it is not all about executing JavaScript code.

A self-hosted analytics package would avoid the latter, and I found one recently through Lifehacker: Piwik, formerly known as PHPMyVisites. Usefully, it turns out that it does next to everything that Google Analytics does. While I'd prefer that it used PHP for this, JavaScript is its means of tracking web pages too. Nevertheless, page loading is still faster than with Google Analytics and/or Woopra and Firefox/NoScript users would only have to allow JavaScript for one site too. If you have had experience with installing PHP/MySQL-powered publishing platforms like WordPress, Textpattern and such like, then putting Piwik in place is no ordeal. Though, you may find yourself changing folder access, uploading of the required files, the specification of database credentials and adding an administration user is all fairly standard stuff. After all that, I have the thing tracking this edifice as well as my outdoor activities (hillwalking/cycling/photography) web presence and I cannot say that I have any complaints, so we'll see how it goes from here.

Investigating Textpattern

With the profusion of Content Management Systems out there, open source and otherwise, my curiosity has been aroused for a while now. In fact, Automattic's aspirations for WordPress (the engine powering this blog) now seem to go beyond blogging and include wider CMS-style usage. Though some may even have put the thing to those kinds of uses, I believe that it has a way to go before it can put itself on a par with the likes of Drupal and Joomla!.

Speaking of Drupal, I decided to give it a go a while back and came away with the impression that it's a platform for an entire website. At the time, I was attracted by the idea of having one part of a website on Drupal and another using WordPress, but the complexity of the CSS in the Drupal template thwarted my efforts and I desisted. The heavy connection between template and back end cut down on the level of flexibility too. Though that mix of different platforms might seem odd in architectural terms, my main website also had a custom PHP/MySQL-driven photo gallery too and migrating everything into Drupal wasn't going to be something that I was planning. Since I might have been trying to get Drupal to perform a role for which it was never meant, I am not holding its non-fulfillment of my requirements against it. While Drupal may have changed since I last looked at it, I decided to give an alternative a go regardless.

Towards the end of last year, I began to look at Textpattern (otherwise known as Txp) in the same vein, and it worked well enough after a little effort that I was able to replace what was once a visitor dossier with a set of travel jottings. In some respects, Though Textpattern might feel less polished when you start to compare it with alternatives like WordPress or Drupal, the inherent flexibility of its design leaves a positive impression. In short, I was happy to see that it allowed me to achieve what I wanted to do.

If I recall correctly, Textpattern's default configuration is that of a blog, which means that it can be used for that purpose. So, I got in some content and started to morph the thing into what I had in mind. Because my ideas weren't entirely developed, some of that was going on while I went about bending Txp to my will. Most of that involved tinkering in the Presentation part of the Txp interface, though. It differs from WordPress in that the design information like (X)HTML templates and CSS are stored in the database rather than in the file system à la WP. Txp also has its own tag language called Textile and, though it contains conditional tags, I find that encasing PHP in <txp:php></txp:php> tags is a more succinct way of doing things; only pure PHP code can be used in this way and not a mixture of such in <?php ?> tags and (X)HTML. A look at the tool's documentation together with a perusal of Apress' Textpattern Solutions got me going in this new world (it was thus for me, anyway). The mainstay of the template system is the Page, while each Section can use a different Page. Each Page can share components and, in Txp, these get called Forms. These are included in a Page using Textile tags of the form <txp:output_form form="form1" />. Style information is edited in another section and you can have several style sheets too.

The Txp Presentation system is made up of Sections, Pages, Forms and Styles. The first of these might appear in the wrong place when being under the Content tab would seem more appropriate but the ability to attach different page templates to different sections places their configuration where you find it in Textpattern and the ability to show or hide sections might have something to do with it too. As it happens, I have used the same template for all bar the front page of the site and got it to display single or multiple articles as appropriate using the Category system. Though it may be a hack, it appears to work well in practice. Being able to make a page template work in the way that you require really offers a considerable amount of flexibility; it allowed me to go with one sidebar rather than two as found in the default set up.

Txp also has the facility to add plugins (look in the Admin section of the UI) and this is very different from WordPress in that installation involves the loading of an encoded text file, probably for the sake of maintaining the security and integrity of your installation. I added the navigation facility for my sidebar and breadcrumb links in this manner, with back end stuff like Tiny MCE editor and Akismet coming as plugins too. While there may not be as many of these for Textpattern, the ones that I found were enough to fulfil my needs. If there are plugin configuration pages in the administration interface, you will find these under the Extensions tab.

To get the content in, I went with the more laborious copy, paste and amend route. Given that I was coming from the plain PHP/XHTML way of doing things, the import functionality was never going to do much for me with its focus on Movable Type, WordPress, Blogger and b2. The fact that you only import content into a particular section may displease some, too. Peculiarly, there is no easy facility for performing a Textpattern to Textpattern migration apart from doing a MySQL database copy. While some alternatives to this were suggested, none seemed to work as well as the basic MySQL route. Tiny MCE made editing easier once I went and turned off Textile processing of the article text. This was done on a case by case basis because I didn't want to have to deal with any unintended consequences arising from turning it off at a global level.

While on the subject of content, this is also the part of the interface where you manage files and graphics along with administering things like comments, categories and links (like a blog roll on WordPress). Of these, it is the comment or link facilities that I don't use, allowing me to turn comments off in the Txp preferences. So far, I am using categories to bundle together similar articles for appearance on the same page, while also getting to use the image and file management side of things as time goes on.

All in all, it seems to work well, even if I wouldn't recommend it to the sort of audience to whom WordPress might be geared. My reason for saying that is because it is a technical tool that is used best if you are prepared to your hands dirtier from code cutting than other alternatives. I, for one, don't mind that at all because working in that manner might actually suit me. Nevertheless, not all users of the system need to have the same level of knowledge or access; usefully, it is possible to set up users with different permissions to limit their exposure to the innards of the administration. In line with Textpattern's being a publishing tool, you get roles such as Publisher (administrator in other platforms), Managing Editor, Copy Editor, Staff Writer, Freelancer, Designer and None. Those names may mean more to others, but I have yet to check out what those access levels entail because I use it on a single user basis.

Txp may lack certain features, such as graphical visitor statistics instead of the current text listings. While, its administration interface could benefit from refinement, it fulfils my core requirements. The more streamlined approach makes it more practical in my opinion, which is demonstrated by my creation of A Wanderer's Miscellany. Consider exploring it yourself to see what's possible, since I am certain that I have only scratched the surface of Textpattern's capabilities.

Episodes of poor performance

Over the last few days, I have been noticing from various that this blog isn't performing as I would want it. The first hint was a comment on a tuxmachines.org mention for a recent entry (thanks for the support, by the way) that the link wasn't working as it should have been. Add to this various emails from Are My Sites Up? saying that the site seemed to be down. By all accounts, this free service that I found through Lifehacker would appear to be doing its job and without the annoying advertising emails that Internetseer used to send me in addition to its weekly report when I used its free service. In fact, that you get alert emails several times a day is a factor in favour of the newcomer.

With one exception, these problems would appear to be intermittent. The exception was when I went using the WP Super Cache WordPress plugin. When it seemed to result in breakage of the site, it got disabled, even if it is meant to be helpful during episodes of heavy load. Otherwise, the outages would seem to be general flakiness of the service provided by my hosting provider. I have a site with them on an older server, and that seems to fare far better than the one playing host to this blog. This sort of thing does make me wonder if we are getting real progress, or whether it's a case of one step forward and two steps back. Nevertheless, I'll continue keeping an eye on things and, if there is too much deterioration, a move might be in order, but that's a good bit away yet.

Remove Revisions 2.2

There is already a post on here devoted to version 1.0 of this plugin, which clearly tells you what it does. The new version will work with the forthcoming WordPress 2.7 (itself a release that's had a development cycle with such upheavals that it would make you want to watch from the relative safety of the sidelines) and has been made to be a little more user-friendly in its actions; in fact, it behaves more like any other plugin now.

Download Remove Revisions 2.2

Dean's FCKEditor for WordPress plugin, a wish list

Admittedly, I have a liking for FCKEditor over and above what comes as standard with WordPress. Here, the FCKEditor for WordPress plugin has been addressing my preference for a while now. However, its most recent release dates from last April and its integration with WordPress 2.6.x has been leaving a lot to be desired. In that vein, I have decided to collect a few of them here:

1. Automatic saves: the idea behind this feature of WordPress is that you aren't hitting the save button that often at all. In fact, given that hitting save creates a revision and an extra record in your database, it really isn't something that you should be doing very often anyway. Unless, you don't mind a bloated database, it's probably best to avoid that habit of saving every few minutes like I do when using Word.
2. Word count: this doesn't update without saving a revision, while it should update periodically in a manner akin to the automatic saves.
3. Insertion of media such as images: this is just broken, and it takes away the possibility of having galleries and captions without manual work.

What I have above are the major inconsistencies, but there have always been annoyances like the adding unwanted &nbsp; entities allover the place, probably a habit of FCKEditor itself anyway. Nevertheless, it's the integration work that really shows the lack of attention. Maybe, it's time to move Dean Lee's labour of love over to a fully community-maintained course of development. While I know that it's difficult to see your "baby" leave you and take flight, I am inclined that it's the best way forward when you consider how rapidly WordPress has been changing over the last year. Though some moves have been made towards this, they really do need to go further.

A radical new look on the way for WordPress' administration area

One thing that you can never say about Automattic is that it stands still for very long. That may generate adverse commentary from some, yet it's bad to leave things stagnate, too. In fact, resting on laurels also generates flak, so you can't please everyone all the time.

Earlier on this year, the WordPress administration screens went through something of an overall for version 2.5. In hindsight, it wasn't terribly dramatic, but the prospect had me checking out what was happening with the development version and contributing to the project in my own small way. Now, it appears that a bigger upheaval is in prospect with wireframes and what not being brought into play on the design side.

The first change that anyone will notice is that the navigation has moved from the top to the side, and that some things have been moved around and renamed. Another thing that you'll see soon enough is that there is a QuickPress section added to the dashboard screen for those quick and short postings. Sticking with the subject of content creation, alterations to post and page editing screens and the link creation screen are obvious too. The thing that really comes to mind here is the level of customisation on offer, so you can make yourself feel right at home: some of the screen furniture can be moved about, and you can remove what you feel is nothing but useless clutter. Content generates comments, so the addition of keyboard shortcuts for comment moderation should be a boon for those with very active readers.

Some functionality currently added by plugins is getting incorporated in the main application. An example is automatic upgrades of WordPress itself. In the development version, it installs the latest nightly build, but I am certain that it can be made to point to the latest stable release. That makes it more convenient for keeping a backseat eye on things rather than getting stuck into the hurly-burly of checking on what's in Subversion.

There are some pieces that remain incompletely functional at present, such as the Inbox and Media added to this post pieces, but I have seen nothing that used to work to be broken. So, while the development version of WordPress is more of a work in progress than I have seen for a while, it will do what you need it to do. Since you always need to be cautious, I'd advise you to keep away until it's ready for the big time, unless you have the knowledge to put things back should the undesirable befall your blog. I had an automatic WordPress update kick in maintenance mode without turning it off again. Nothing happened to the database, so a manual re-installation was all that was required to restore order. Otherwise, the development installation remains relatively stable, though there are display problems in IE6 that do not afflict either Firefox, Opera or even IE8 Beta2.

All in all, these changes will make the next major WordPress release a substantial one, and I have little doubt that the new administration interface will make for many comments. While I must admit that I wasn't too taken with it when I first glimpsed it in the crazyhorse branch, it now seems to be growing on me. The most significant change on the aesthetic front is that there's a lot less whitespace, about which may or may not float your boat. In any event, you can always change the colours like I have to make things more amenable.

Even so, I reckon that WordPress 2.7 will be a major step forward when it comes, one that I might like. Saying that, I reckon that the release date is a good while away for now, and the fact that 2.6.2 is being readied for release is telling in itself. Leaving plenty of time to remove any roughness is never a bad thing, especially so when you see the changes that are being made. Meanwhile, I am certain that most people can wait.

Another way to look through WordPress code

At times in the past, I have complained about there not being sufficient documentation of the actual code used on WordPress itself. True, there is the venerable Codex, but that can be incomplete, and any extra help is always useful. Peter Westwood (a.k.a. westi) has generated documentation from the code itself and put it up on the web for all to peruse. While we are on the subject of information on the more advanced aspects of WordPress, blogs by Mark Jaquith and Andrew Ozz are also worth a look. As the cliché goes, every little helps...

An alternative use for Woopra

While the once a day reporting cycle of Google Analytics is all very fine, the availability of real time data does have its advantages. WordPress.com's Stats plugin goes some way to serving the need, only for Woopra to trump it in every way apart from a possible overkill in the amount of information that it makes available. The software remains extremely useful despite being in the beta phase and occasionally crashing.

One of its uses is seeing if there are people visiting your website at a time when you might be thinking of making a change like upgrading WordPress. Timing such activities to avoid a clash is a win-win situation: a better experience from your visitors and more reliable updates for you. After all, it's effortless to make a poor impression and an unreliable site will do that faster than anything else, so it's paramount that your visitors do not get on the receiving end of updates, even if they are all for the better.

WordPress plugin for removing post revisions from database

WordPress 2.6 added post revisions as a new feature that is turned on by default. In an earlier post, I described how you could control this by editing wp-config.php and there are a number of plugins that purport to provide the same level of control through the administration screens. Even so, I decided to look at things from the housekeeping side of things and create my own plugin for clearing the database of revisions at one swoop. Currently, it takes out all revisions, but I am thinking of adding the facility for selecting which revision to keep and which to delete. It goes without saying that you should back up your database first in case anything might go wrong.

Download Remove Revisions 1.0

Controlling the post revision feature in WordPress 2.6

While this may seem esoteric for some, I like to be in charge of the technology that I use. So, when Automattic included post revision retention to WordPress 2.6, I had my reservations about how much it would clutter my database with things that I didn't need. Thankfully, there is a way to control the feature, but you won't find the option in the administration screens (they seem to view this as an advanced setting and so don't want to be adding clutter to the interface for the sake of something that only a few might ever use); you have to edit wp-config.php yourself to add it. Here are the lines that can be added and the effects that they have:

Code: define('WP_POST_REVISIONS','0');

Effect: turns off post revision retention

Code: define('WP_POST_REVISIONS','-1');

Effect: turns it on (the default setting)

Code: define('WP_POST_REVISIONS','2');

Effect: only retains two previous versions of a post (the number can be whatever you want, so long as it's an integer with a value more than zero).

Update (2008-07-23):

There is now a plugin from Dion Hulse that does the above for you and more.