Technology Tales

Getting Grav Admin2 to write files with the correct permissions

19^th September 2026

When I found that the find and replace functionality in the article editor offered by Grav's Admin2 plugin was next to unusable, I opted to do what I needed on the server using nano. However, I then discovered that was not possible without the sudo facility even though my user account was in the www-data group. Here is the command that discovered that 644 (owner with read and write permissions, the rest as read only) was being used instead of 664 (owner and group with read and write permissions, the rest as read only):

stat -c '%a %U %G %n' filename.md

Using -c switch to define a custom format, the output is like this for what I needed it to be:

664 www-data www-data filename.md

Unpacking the codes, %a provides the permissions in their octal or numeric form, %U gives the owner, %G is the associated group and %n is the file name. Though I manually set the permissions, I found that they kept changing whenever a file was updated from the user interface. It appears that the file is copied to a different area and then moved back when edits have been completed.

If I had thought that this was going to be resolved by finding the right setting in the Grav configuration I was about to be disappointed. The actual solution turned out to be in very different place: the web server systemd service settings.

PHP-FPM

If the PHP-FPM service was being used, then things might have been simpler. The first step is to add a systemd override by opening up the service configuration:

sudo systemctl edit php8.3-fpm

Near the top of the file, add these lines (make sure that they are above any line that says everything gets deleted below it). Then, save the file.

[Service]
UMask=0002

Following that, restart the service (this is for version 8.3; clearly, the version number will need changing for any future release):

sudo systemctl restart php8.3-fpm

Next, it is time to check that the override is in place by using this command:

systemctl show php8.3-fpm -p UMask

When you see a result like the one below, Grav should be producing files with rw-rw-r-- or 664 permissions through the Admin2 editor. That is how it is with a Nginx set up that I have with PHP-FPM in action on there. What is happening is that it takes the likes of 777 and makes it 775; if the setting were 0022 (the default, it would appear to me), you would then get 755.

UMask=0002

Apache MOD-PHP

However, the issue initially surfaced on an Apache web server, and it took a while for the solution to be found. Confusingly, both PHP-FPM and MOD-PHP were on there. Then, it was a question of finding which one was in action because the latter is part of Apache and the former is a separate service.

Trying the PHP-FPM override described above did not work, which hinted that something else was needed. That was to follow a similar process to that for PHP-FPM, only to do it for Apache this time around. The first step to add the override after opening the relevant file is as follows:

sudo systemctl edit apache2

As before, these lines need to go near the top of the file, above any line warning of the loss of any changes made below it, before it is saved:

[Service]
UMask=0002

Then, it is time to restart the service:

sudo systemctl restart apache2

Next, the override status needs to be probed:

systemctl show apache2 -p UMask

Getting this result from the above command again demonstrates that all should be in order, which it was in my case:

UMask=0002

After all that, it may be time to switch from MOD-PHP to PHP-FPM anyway. At least, it guarantees configuration consistency and separates PHP interpretation from normal web server activities as well.

Structuring a Grav Microblog using Twig and YAML

6^th July 2026

This year has seen me move to using Grav on multiple websites. The first one has two sections on Grav: Travel Jottings and Varied Surroundings, both based around single articles with the sort of HTML output post-processing that I have described in another entry on here. In this one, I move onto how to display things in a blog structure. This approach is in use on the parts of this website that are hosted using Grav: AI & Data Science, Coding Notebook and Collected Snippets. HTML output is post-processing here too, even if the story is much bigger than all that.

Building a Paginated Blog Template with Twig

First, we work through the blog page template that shows all the entries with pagination to ensure that not everything appears on the same page. In this file, the first step is to refer to a base template:

{% extends 'partials/base.html.twig' %}

Then, you need to define the content block, which encases the rest of the code:

{% block content %}...{% endblock %}

The above overrides any matching block of the same name in the parent template, meaning that everything between these tags replaces whatever the base template had in its content block. That includes the page collection declaration that is defined next:

  {% set collection = page.collection %}

This fetches the page's collection, defined in the page's front matter. In other words, this is the YAML at the top of the Markdown file, and we will examine that later, which should make what is happening here clearer. For a fuller discussion, I have another entry describing how to set up an RSS feed that describes collections in even more detail. For added organisation, I enclosed everything with section HML tags:

  <section class="blog-collation">...</section>

Before displaying the individual microblog entries, I also added a section with a hyperlinked title that allows a visitor to easily reach the website section front page from anywhere within that part of the site. Here is the code for that:

    <article class="the-post">
      <h2><a href="{{ base_url }}" title="Go to first page">{{ page.title }}</a></h2>
    </article>

After that, there is a loop to display a number of posts for each page with some conditional logic to handle when there are no Markdown files available in the expected directory:

    {% for item in collection %}...{% else %}<p>No entries found.</p>{% endfor %}

When there are Markdown files in the required directory, this is the code that adjusts links in the HTML rendered for each Markdown file so they open securely in a new browser tab:

       {% set content = item.content
         |replace({'<a href="http': '<a target="_blank" rel="noopener noreferrer" href="http'})
       %}

The above reads in the page content (item.content) because this is a loop with `item` as the iterator. Then, it replaces <a href="http with <a target="_blank" rel="noopener noreferrer" href="http. The input was solely Markdown in this case, which meant that the rendering and text replacement was streamlined as expected. Beyond, the main article display code follows:

      <article class="the-post">
        <p class="mt-3 mb-4"><strong>
        {% if item.date %}
          {{ item.date|date("H:i, j") }}<sup>{{ item.date|date("S") }}</sup> {{ item.date|date("F Y") }}
        {% endif %}
        </strong></p>
        <div class="entry-body">
          {{ content|raw }}
        </div>
      </article>

The mix of HTML tags above should not surprise anyone who has worked with web content production before. What really needs more comment are these lines from the above:

        {% if item.date %}
          {{ item.date|date("H:i, j") }}<sup>{{ item.date|date("S") }}</sup> {{ item.date|date("F Y") }}
        {% endif %}

In a block guarded by {% if item.date %} to ensure nothing is output if a page has no date set, the publishing time and date for an entry is rendered in three separate filter calls, which combine to produce a single formatted output that looks like this: 14:30, 3^rd April 2026

	Filter	Output	Example
	date("H:i, j")	24-hour time and day number	14:30, 3
	date("S")	Ordinal suffix, in <sup> tags	rd
	date("F Y")	Full month and year	April 2026

Enclosed within the nested tags, it is {{ content|raw }} that displays the paragraph output underneath the date and time for each entry. At the bottom of the page, there is an extra code block needed to add pagination and Previous/Next buttons for navigation:

  {% if config.plugins.pagination.enabled and collection.params.pagination %}
    {% include 'partials/pagination.html.twig' with {'base_url': page.url, 'pagination': collection.params.pagination} %}
  {% endif %}

The above is positioned outside the <section class="blog-collation">...</section> block but within the first part of the {% for item in collection %}...{% else %}<p>No entries found.</p>{% endfor %} construct. The {% if config.plugins.pagination.enabled and collection.params.pagination %} condition ensures that noting is displayed unless the Pagination plugin has been installed and enabled.

Decoding the Associated YAML Front Matter

All of the above does make as much sense unless you also look at the front matter YAML on the Markdown file for the main microblog page, blog.md, itself consistent with the name of its template, which is the one we worked through above. This is what now follows.

---
title: 'AI & Data Science Jottings'
content:
    items: '@self.children'
    order:
        by: date
        dir: desc
    pagination: true
    limit: 10
---

In the above, the title field defines the name of the microblog, one of three on this website. Within the content section, the items page collection is defined as all child pages of the current page. In this case, these would be sub-folders sitting inside this page's folder, each containing a single Markdown file called default.md, another name consistent with that of its associated template. The order of appearance of the microblog entries follows next, with the latest posts appearing first because of the descending date order. Pagination gets switched on with a limit of ten entries per page and navigation buttons to go between them.

Tying the Blog Template and Front Matter Together

Here, we have walked through the two components that make a paginated Grav microblog work: the Twig template that renders the collection and the YAML front matter that defines it. The template handles the display logic, from extending the base file through to looping over each item, adjusting links and formatting the date and time output. The front matter tells Grav which pages belong to the collection, how they should be ordered, and how many should appear per page.

Taken together, this approach has proved flexible enough to serve all three of the Grav-hosted sections on this website. Each relies on the same underlying template and front matter structure, with only the content and folder names differing between them.

In short, anyone setting up a similar microblog structure in Grav should find that these building blocks cover most of what is needed: the base template, the content block, the collection loop and the pagination section. Naturally, there remains plenty of room to adapt the HTML markup and filters to suit a different site's requirements.

Some R packages to explore as you find your feet with the language

24^th March 2026

Here are some commonly used R packages and other tools that are pervasive, along with others that I have encountered while getting started with the language, itself becoming pervasive in my line of business. The collection grew organically as my explorations proceeded, and reflects what I was trying out during my acclimatisation.

General

Here are two general packages to get things started, with one of them being unavoidable in the R world. The other is more advanced, possibly offering more to package developers.

{tidyverse}

You cannot use R without knowing about this collection of packages. In many ways, they form a mini-language of their own, drawing some criticism from those who reckon that base R functionality covers a sufficient gamut anyway. Nevertheless, there is so much here that will get you going with data wrangling and visualisation that it is worth knowing what is possible. The complaints may come from your not needing to use anything else for these purposes.

{plumber}

This R package enables developers to convert existing R functions into web API endpoints by adding roxygen2-like comment annotations to their code. Once annotated, functions can handle HTTP GET and POST requests, accept query string or JSON parameters and return outputs such as plain values or rendered plots. The package is available on CRAN as a stable release, with a development version hosted on GitHub. For deployment, it integrates with DigitalOcean through a companion package called {plumberDeploy}, and also supports Posit Connect, PM2 and Docker as hosting options. Related projects in the same space include OpenCPU, which is designed for hosting R APIs in scientific research contexts, and the now-discontinued jug package, which took a more programmatic approach to API construction.

Data Preparation

You simply cannot avoid working with data during any analysis or reporting work. While there is a learning curve if you are used to other languages, there is little doubt that R is well-endowed when it comes to performing these tasks. Here are some packages that extend base R capabilities and might even add some extra user-friendliness along the way.

{forcats}

The {forcats} package in R provides functions to manage categorical variables by reordering factor levels, collapsing infrequent values and adjusting their sequence based on frequency or other variables. It includes tools such as reordering by another variable, grouping rare categories into 'other' and modifying level order manually, which are useful for data analysis and visualisation workflows. Designed as part of the tidyverse, it integrates with other packages to streamline tasks like counting and plotting categorical data, enhancing clarity and efficiency in handling factors within R.

{tidyr}

Around this time last year, I remember completing a LinkedIn course on a set of good practices known as tidy data, where each variable occupies a column, each observation a row and each value a single cell. This package is designed to help users restructure data so it follows those rules. It provides tools for reshaping data between long and wide formats, handling nested lists, splitting or combining columns, managing missing values and layering or flattening grouped data.

Installation options include the {tidyverse} collection, standalone installation, or the development version from GitHub. The package succeeds earlier reshaping tools like {reshape2} and {reshape}, offering a focused approach to tidying data rather than general reshaping or aggregation.

{haven}

Having a long track record of working with SAS, {haven} with its abilities to read and write data files from statistical software such as SAS, SPSS and Stata, leveraging the ReadStat library, arouses my interest. Handily, it supports a range of file formats, including SAS transport and data files, SPSS system and older portable files and Stata data files up to version 15, converting these into tibbles with enhanced printing capabilities. Value labels are preserved as a labelled class, allowing conversion to factors, while dates and times are transformed into standard R classes.

{RMariaDB}

While there are other approaches to working with databases using R, {RMariaDB} provides a database interface and driver for MariaDB, designed to fully comply with the DBI specification and serve as a replacement for the older {RMySQL} package. It supports connecting to databases using configuration files, executing queries, reading and writing data tables and managing results in chunks. Installation options include binary packages from CRAN or development versions from GitHub, with additional dependencies such as MariaDB Connector/C or libmysqlclient required for Linux and macOS systems. Configuration is typically handled through a MariaDB-specific file, and the package includes acknowledgments for contributions from various developers and organisations.

COVID-19 Data Hub

For many people, the pandemic may be a fading memory, yet it offered its chances for learning R, not least because there was a use case with more than a hint of personal interest about it. Here is a library making it easier to get hold of the data, with some added pre-processing too. Memories of how I needed to wrangle what was published by various sources make me appreciate just how vital it is to have harmonised data for analysis work.

Table Production

While many appear to graphical presentation of results to their tabular display, R does have its options here too. In recent times, the options have improved, particularly of the pharmaverse initiative. Here is a selection of what I found during my explorations.

{officer}

Part of the {officeverse} along with {officedown}, {Flextable}, {Rvg} and {mschart}, the {officer} R package enables users to create and modify Word and PowerPoint documents directly from R, allowing the insertion of images, tables and formatted content, as well as the import of document content into data frames. It supports the generation of RTF files and integrates with other packages for advanced features such as vector graphics and native office charts. Installation options include CRAN and GitHub, with community resources available for assistance and contributions. The package facilitates the manipulation of document elements like paragraphs, tables and section breaks and provides tools for exporting and importing content between R and office formats, alongside functions for managing slide layouts and embedded objects in presentations.

{pharmaRTF}

If you work in clinical research like I do, the need to produce data tabulations is a non-negotiable requirement. That is how this package came to be developed and the pharmaverse of which it is part has numerous other options, should you need to look at using one of those. The flavour of RTF produced here is the Microsoft Word variety, which did not look as well in LibreOffice Writer when I last looked at the results with that open-source alternative. Otherwise, the results look well to many eyes.

{formattable}

Here is a package that enhances data presentation by applying customisable formatting to vectors and data frames, supporting formats such as percentages, currency and accounting. Available on GitHub and CRAN, it integrates with dynamic document tools like {knitr} and {rmarkdown} to produce visually distinct tables, with features including gradient colour scales, conditional styling and icon-based representations. It automatically converts to {htmlwidgets} in interactive environments and is licensed under MIT, enabling flexible use in both static and interactive data displays.

{reactable}

The {reactable} package for R provides interactive data tables built on the React Table library, offering features such as sorting, filtering, pagination, grouping with aggregation, virtual scrolling for large datasets and support for custom rendering through R or JavaScript. It integrates seamlessly into R Markdown documents and Shiny applications, enabling the use of HTML widgets and conditional styling. Installation options include CRAN and GitHub, with examples demonstrating its application across various datasets and scenarios. The package supports major web browsers and is licensed under MIT, designed for developers seeking dynamic data presentation tools within the R ecosystem.

{DT}

Particularly useful in dynamic web applications like Shiny, the {DT} package in R provides a means of rendering interactive HTML tables by building on the DataTables JavaScript library. It supports features including sorting, searching, pagination and advanced filtering, with numeric, date and time columns using range-based sliders whilst factor and character columns rely on search boxes or dropdowns. Filtering operates on the client side by default, though server-side processing is also available. JavaScript callbacks can be injected after initialisation to manipulate table behaviour, such as enabling automatic page navigation or adding child rows to display additional detail. HTML content is escaped by default as a safeguard against cross-site scripting attacks, with the option to adjust this on a per-column basis. Whilst the package integrates with Shiny applications, attention is needed around scrolling and slider positioning to prevent layout problems. Overall, the package is well suited to exploratory data analysis and the building of interactive dashboards.

{gt}

The {gt} package in R enables users to create well-structured tables with a variety of formatting options, starting from data frames or tibbles and incorporating elements such as headers, footers and customised column labels. It supports output in HTML, LaTeX and RTF formats and includes example datasets for experimentation. The package prioritises simplicity for common tasks while offering advanced functions for detailed customisation, with installation available via CRAN or GitHub. Users can access resources like documentation, community forums and example projects to explore its capabilities, and it is supported by a range of related packages that extend its functionality.

{gtsummary}

Enabling users to produce publication-ready outputs with minimal code, the {gtsummary} package offers a streamlined approach to generating analytical and summary tables in R. It automates the summarisation of data frames, regression models and other datasets, identifying variable types and calculating relevant statistics, including measures of data incompleteness. Customisation options allow for formatting, merging and styling tables to suit specific needs, while integration with packages such as {broom} and {gt} facilitates seamless incorporation into R Markdown workflows. The package supports the creation of side-by-side regression tables and provides tools for exporting results as images, HTML, Word, or LaTeX files, enhancing flexibility for reporting and sharing findings.

{huxtable}

Here is an R package designed to generate LaTeX and HTML tables with a modern, user-friendly interface, offering extensive control over styling, formatting, alignment and layout. It supports features such as custom borders, padding, background colours and cell spanning across rows or columns, with tables modifiable using standard R subsetting or dplyr functions. Examples demonstrate its use for creating simple tables, applying conditional formatting and producing regression output with statistical details. The package also facilitates quick export to formats like PDF, DOCX, HTML and XLSX. Installation options include CRAN, R-Universe and GitHub, while the name reflects its origins as an enhanced version of the {xtable} package. The logo was generated using the package itself, and the background design draws inspiration from Piet Mondrian’s artwork.

Figure Generation

R has such a reputation for graphical presentations that it is cited as a strong reason to explore what the ecosystem has to offer. While base R itself is not shabby when it comes to creating graphs and charts, these packages will extend things by quite a way. In fact, the first on this list is near enough pervasive.

{ggplot2}

Though its default formatting does not appeal to me, the myriad of options makes this a very flexible tool, albeit at the expense of some code verbosity. Multi-panel plots are not among its strengths, which may send you elsewhere for that need.

{ggforce}

Focusing on features not included in the core library, the {ggforce} package extends {ggplot2} by offering additional tools to enhance data visualisation. Designed to complement the primary role of {ggplot2} in exploratory data analysis, it provides a range of geoms, stats and other components that are well-documented and implemented, aiming to support more complex and custom plot compositions. Available for installation via CRAN or GitHub, the package includes a variety of functionalities described in detail on its associated website, though specific examples are not included here.

{cowplot}

Developed by Claus O. Wilke for internal use in his lab, {cowplot} is an R package designed to help with the creation of publication-quality figures built on top of {ggplot2}. It provides a set of themes, tools for aligning and arranging plots into compound figures and functions for annotating plots or combining them with images. The package can be installed directly from CRAN or as a development version via GitHub, and it has seen widespread use in the book Fundamentals of Data Visualisation.

{sjPlot}

The {sjPlot} package provides a range of tools for visualising data and statistical results commonly used in social science research, including frequency tables, histograms, box plots, regression models, mixed effects models, PCA, correlation matrices and cluster analyses. It supports installation via CRAN for stable releases or through GitHub for development versions, with documentation and examples available online. The package is licensed under GPL-3 and developed by Daniel Lüdecke, offering functions to create visualisations such as scatter plots, Likert scales and interaction effect plots, along with tools for constructing index variables and presenting statistical outputs in tabular formats.

{thematic}

By offering a centralised approach to theming and enabling automatic adaptation of plot styles within Shiny applications, the {thematic} package simplifies the styling of R graphics, including {ggplot2}, {lattice} and base R plots, R Markdown documents and RStudio. It allows users to apply consistent visual themes across different plotting systems, with auto-theming in Shiny and R Markdown relying on CSS and {bslib} themes, respectively. Installation requires specific versions of dependent packages such as {shiny} and {rmarkdown}, while custom fonts benefit from {showtext} or {ragg}. Users can set global defaults for background, foreground and accent colours, as well as fonts, which can be overridden with plot-specific theme adjustments. The package also defines default colour scales for qualitative and sequential data and integrates with tools like bslib to import Google Fonts, enhancing visual consistency across different environments and user interfaces.

Publishing Tools

The R ecosystem goes beyond mere graphical and tabular display production to offer means for taking things much further, often offering platforms for publishing your work. These can be used locally too, so there is no need to entrust everything to a third-party provider. The uses are endless for what is available, and it appears that Posit has used this to help with building documentation and training too.

R Markdown

What you have here is one of those distinguishing facilities of the R ecosystem, particularly for those wanting to share their analysis work with more than a hint of reproducibility. The tool combines narrative text and code to generate various outputs, supporting multiple programming languages and formats such as HTML, PDF and dashboards. It enables users to produce reports, presentations and interactive applications, with options for publishing and scheduling through platforms like RStudio Connect, facilitating collaboration and distribution of results in professional settings.

Distill for R Markdown

Distill for R Markdown is a tool designed to streamline the creation of technical documents, offering features such as code folding, syntax highlighting and theming. It builds on existing frameworks like Pandoc, MathJax and D3, enabling the production of dynamic, interactive content. Users can customise the appearance with CSS and incorporate appendices for supplementary information. The tool acknowledges the contributions of developers who created foundational libraries, ensuring accessibility and functionality for a wide audience. Its design prioritises clarity, allowing authors to focus on presenting results rather than underlying code, while maintaining flexibility for those who wish to include detailed explanations.

{shiny}

For a while, this was one of R's unique selling points, and remains as compelling a reason to use the language even when Python has got its own version of the package. Enabling the creation of interactive web applications for data analysis without requiring web development expertise allows users to build interfaces that let others explore data through dynamic visualisations and filters. Here is a simple example: an app that generates scatter plots with adjustable variables, species filters and marginal plots, hosted either on personal servers or through a dedicated hosting service.

{bslib}

The {bslib} R package offers a modern user interface toolkit for Shiny and R Markdown applications, leveraging Bootstrap to enable the creation of customisable dashboards and interactive theming. It supports the use of updated Bootstrap and Bootswatch versions while maintaining compatibility with existing defaults, and provides tools for real-time visual adjustments. Installation is available through CRAN, with example previews demonstrating its capabilities.

{rhandsontable}

Enabling users to manipulate and validate data within a spreadsheet-like interface, the {rhandsontable} package introduces an interactive data grid for R. It supports features such as custom cell rendering, validation rules and integration with Shiny applications. When used in Shiny, the widget requires explicit conversion of data using the hot_to_r function, as updates may not be immediately reflected in reactive contexts. Examples demonstrate its application in various scenarios, including date editing, financial calculations and dynamic visualisations linked to charts. The package also accommodates bookmarks in Shiny apps with specific handling. Users are encouraged to report issues or contribute improvements, with guidance provided for those seeking to expand its functionality. The development team welcomes feedback to refine the tool further, ensuring it aligns with evolving user needs.

{xaringanExtra}

{xaringanExtra} offers a range of enhancements and extensions for creating and presenting slides with xaringan, enabling features such as adding an overview tile view, making slides editable, broadcasting in real time, incorporating animations, embedding live video feeds and applying custom styles. It allows users to selectively activate individual tools or load multiple features simultaneously through a single function call, supporting tasks like adding banners, enabling code copying, fitting slides to screen dimensions and integrating utility toolkits. The package is available for installation via CRAN or GitHub, providing flexibility for developers and presenters seeking to expand the functionality of their slides.

Learning R for Data Analysis: Going from the basics to professional practice

22^nd March 2026

R has grown from a specialist statistical language into one of the most widely recognised tools for working with data. Across tutorials, community sites, training platforms and industry resources, it is presented as both a programming language and a software environment for statistical computing, graphics and reporting. It was created by Ross Ihaka and Robert Gentleman at the University of Auckland in New Zealand, and its name draws on the first letter of their first names while also alluding to the Bell Labs language S. It is freely available under the GNU General Public Licence and runs on Linux, Windows and macOS, which has helped it spread across research, education and industry alike.

What Makes R Distinctive

What makes R notable is its combination of programming features with a strong focus on data analysis. Introductory material, such as the tutorials at Tutorialspoint and Datamentor, repeatedly highlights its support for conditionals, loops, user-defined recursive functions and input and output, but these sit alongside effective data handling, a broad set of operators for arrays, lists, vectors and matrices and strong graphical capabilities. That mixture means R can be used for straightforward scripts and for complex analytical workflows. A beginner may start by printing "Hello, World!" with the print() function, while a more experienced user may move on to regression models, interactive dashboards or automated reporting.

The Learning Progression

Learning materials generally present R in a structured progression. A beginner is first introduced to reserved words, variables and constants, operators and the order in which expressions are evaluated. From there, the path usually moves into flow control through if…else, ifelse(), for, while, repeat and the use of break and next, before functions follow naturally, including return values, environments and scope, recursive functions, infix operators and switch(). Most sources agree that confidence with the syntax and fundamentals is the real starting point, and this early sequence matters because it helps learners become comfortable reading and writing R rather than only copying examples.

After the basics, attention tends to turn to the structures that make R so useful for data work. Vectors, matrices, lists, data frames and factors appear in nearly every introductory course because they are central to how information is stored and manipulated. Object-oriented concepts also emerge quite early in some routes through the language, with classes and objects extending into S3, S4 and reference classes. For someone coming from spreadsheets or point-and-click statistical software, this shift can feel significant, but it also opens the way to more reproducible and flexible analysis.

Visualisation

Visualisation is another recurring theme in R education. Basic chart types such as bar plots, histograms, pie charts, box plots and strip charts are common early examples because they show how quickly data can be turned into graphics. More advanced lessons widen the scope through plot functions, multiple plots, saving graphics, colour selection and the production of 3D plots.

Beyond base plotting, there is extensive evidence of the central role of {ggplot2} in contemporary R practice. In the pharmaceutical and clinical research space, the PSI VIS-SIG blog is published by the PSI Visualisation Special Interest Group and summarises its monthly Wonderful Wednesday webinars, presenting real-world datasets and community-contributed chart improvements alongside news from the group.

Data Wrangling and the Tidyverse

Much of modern R work is built around data wrangling, and here the {tidyverse} has become especially prominent. Claudia A. Engel's openly published guide Data Wrangling with R (last updated 3rd November 2023) sets out a preparation phase that assumes some basic R knowledge, a recent installation of R and RStudio and the installation of the {tidyverse} package with install.packages("tidyverse") followed by library(tidyverse). It also recommends creating a dedicated RStudio project and downloading CSV files into a data subdirectory, reinforcing the importance of organised project structure.

That same guide then moves through data manipulation with {dplyr}, covering selecting columns and filtering rows, pipes, adding new columns, split-apply-combine, tallying and joining two tables, before moving on to {tidyr} topics such as long and wide table formats, pivot_wider, pivot_longer and exporting data. These topics reflect a broader pattern in the R ecosystem because data import and export, reshaping, combining tables and counting by group recur across teaching resources as they mirror common analytical tasks.

Applications and Professional Use

The range of applications attached to R is wide, though data science remains the clearest centre of gravity. Educational sources describe R as valuable for data wrangling, visualisation and analysis, often pointing to packages such as {dplyr}, {tidyr}, {ggplot2} and {Shiny}. Statistical modelling is another major strand, with R offering extensible techniques for descriptive and inferential statistics, regression analysis, time series methods and classical tests. Machine learning appears as a further area of growth, supported by a large and expanding package ecosystem. In more advanced contexts, R is also linked with dashboards, web applications, report generation and publishing systems such as Quarto and R Markdown.

R's place in professional settings is underscored by the breadth of organisations and sectors associated with it. Introductory resources mention companies such as Google, Microsoft, Facebook, ANZ Bank, Ford and The New York Times as examples of organisations using R for modelling, forecasting, analysis and visualisation. The NHS-R Community promotes the use of R and open analytics in health and care, building a community of practice for data analysis and data science using open-source software in the NHS and wider UK health and care system. Its resources include reports, blogs, webinars and workshops, books, videos and R packages, with webinar materials archived in a publicly accessible GitHub repository. The R Validation Hub, supported through the pharmaR initiative, is a collaboration to support the adoption of R within a biopharmaceutical regulatory setting and provides tools including the {riskmetric} package, the {riskassessment} app and the {riskscore} package for assessing package quality and risk.

The Wider Ecosystem

The wider ecosystem around R is unusually rich. The R Consortium promotes the growth and development of the R language and its ecosystem by supporting technical and social infrastructure, fostering community engagement and driving industry adoption. It notes that the R language supports over two million users and has been adopted in industries including biotech, finance, research and high technology. Community growth is visible not only through organisations and conferences but through user groups, scholarships, project working groups and local meetups, which matters because learning a language is easier when there is an active support network around it.

Another sign of maturity is the depth of R's package and publication landscape. rdrr.io provides a comprehensive index of over 29,000 CRAN packages alongside more than 2,100 Bioconductor packages, over 2,200 R-Forge packages and more than 76,000 GitHub packages, making it possible to search for packages, functions, documentation and source code in one place. Rdocumentation, powered by DataCamp, covers 32,130 packages across CRAN and Bioconductor and offers a searchable interface for function-level documentation. The Journal of Statistical Software adds a scholarly dimension, publishing open-access articles on statistical computing software together with source code, with full reproducibility mandatory for publication. R-bloggers aggregates R news and tutorials contributed by hundreds of R bloggers, while R Weekly curates a community digest and an accompanying podcast, both helping users keep pace with the steady flow of tutorials, package releases, blog posts and developments across the R world.

Where to Begin

For beginners, one recurring challenge is knowing where to start, and different learning routes reflect different backgrounds. Datamentor points learners towards step-by-step tutorials covering popular topics such as R operators, if...else statements, data frames, lists and histograms, progressing through to more advanced material. R for the Rest of Us offers a staged path through three core courses, Getting Started With R, Fundamentals of R and Going Deeper with R, and extends into nine topics courses covering Git and GitHub, making beautiful tables, mapping, graphics, data cleaning, inferential statistics, package development, reproducibility and interactive dashboards with {Shiny}. The site is explicitly designed for people who may never have coded before and also offers the structured R in 3 Months programme alongside training and consulting. RStudio Education (now part of Posit) outlines six distinct ways to begin learning R, covering installation, a free introductory webinar on tidy statistics, the book R for Data Science, browser-based primers, and further options suited to different learning styles, along with guidance on R Markdown and good project practices.

Despite the variety, the underlying advice is consistent: start by learning the basics well enough to read and write simple code, practise regularly beginning with straightforward exercises and gradually take on more complex tasks, then build projects that matter to you because projects create context and make concepts stick. There is no suggestion that mastery comes from passively reading documentation alone, as practical engagement is treated as essential throughout. The blog Stats and R exemplifies this philosophy well, with the stated aim of making statistics accessible to everyone by sharing, explaining and illustrating statistical concepts and, where appropriate, applying them in R.

That practical engagement can take many forms. Someone interested in data journalism may focus on visualisation and reproducible reporting, while a researcher may prioritise statistical modelling and publishing workflows, and a health analyst may use R for quality assurance, open health data and clinical reporting. Others may work with {Shiny}, package development, machine learning, Git and GitHub or interactive dashboards. The variety shows that R is not confined to a single use case, even if statistics and data science remain the common thread.

Free Learning Resources for R

It is also worth noting that R learning is supported by a great deal of freely available material. Statistics Globe, founded in 2017 by Joachim Schork and now an education and consulting platform, offers more than 3,000 free tutorials and over 1,000 video tutorials on YouTube, spanning R programming, Python and statistical methodology. STHDA (Statistical Tools for High-Throughput Data Analysis) covers basics, data import and export, reshaping, manipulation and visualisation, with material geared towards practical data analysis at every level. Community sites, webinar repositories and newsletters add further layers of accessibility, and even where paid courses exist, the surrounding free ecosystem is substantial.

Taken together, these sources present R as far more than a niche programming language. It is a mature open-source environment with a strong statistical heritage, a practical orientation towards data work and a well-developed community of learners, teachers, developers and organisations. Its core concepts are approachable enough for beginners, yet its package ecosystem and publishing culture support highly specialised and advanced work. For anyone looking to enter data analysis, statistics, visualisation or related areas, R offers a route that begins with simple code and can extend into large-scale analytical workflows.

Windows 11 virtualisation on Linux using KVM and QEMU

5^th March 2026

Windows 11 arrived in October 2021 with a requirement that posed a challenge to many virtualisation users: TPM 2.0 was mandatory, not optional. For anyone running Windows in a virtual machine, that meant their hypervisor needed to emulate a Trusted Platform Module convincingly enough to satisfy the installer.

VirtualBox, which had been my go-to choice for desktop virtualisation for years, could not do this in its 6.1.x series. Support arrived only with VirtualBox 7.0 in October 2022, meaning anyone who needed Windows 11 in a VM faced roughly a year with no straightforward path through their existing tool.

That gap prompted a look at KVM (Kernel-based Virtual Machine), which could handle the TPM requirement through software emulation. This article documents what that investigation found, what the rough edges were at the time, and how the situation has developed in the years since.

What KVM Actually Is

KVM is not a standalone application. It is a virtualisation infrastructure built directly into the Linux kernel, and has been since the module was merged between 2006 and 2007. Rather than sitting on top of the operating system as a separate layer, it turns the Linux kernel itself into a hypervisor. This makes KVM a type-1 hypervisor in practice, even when running on a desktop machine, which is part of why its performance characteristics compare favourably with hosted solutions.

In use, KVM operates alongside QEMU for hardware emulation, libvirt for virtual machine management and virt-manager as a graphical front end. The distinction matters because problems and improvements tend to originate in different parts of that stack. KVM itself is rarely the issue; QEMU and libvirt are where the day-to-day configuration lives.

To confirm that the host CPU supports hardware virtualisation before beginning, the following command checks for the relevant flags:

egrep -c '(vmx|svm)' /proc/cpuinfo

Any result above zero means the hardware is capable. Intel processors expose the vmx flag and AMD processors expose svm.

Installing the Required Packages

The installation is straightforward on any major distribution.

On Debian and Ubuntu:

sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager

On Fedora:

sudo dnf install @virtualization

On Arch Linux:

sudo pacman -S qemu libvirt virt-manager bridge-utils

After installation, the current user needs to be added to the libvirt and kvm groups before the tools will work without root privileges:

sudo usermod -aG libvirt,kvm $(whoami)

Logging out and back in instates the group membership.

Configuring Network Bridging

The default network configuration in libvirt uses NAT, which is sufficient for most purposes and requires no additional setup. The VM can reach the internet and the host, but the host cannot initiate connections to the VM. For a Windows 11 guest used primarily for application compatibility, NAT works without complaint.

A bridged network, which places the VM on the same network segment as the host, requires a wired Ethernet connection. Wireless interfaces do not support bridging in the standard Linux networking stack due to how 802.11 handles MAC addresses. For those on a wired connection, a bridge can be defined with a file named bridge.xml:

<network>
  <name>br0</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
</network>

The bridge is then activated with:

sudo virsh net-define bridge.xml
sudo virsh net-start br0
sudo virsh net-autostart br0

Installing Windows 11

Windows 11 requires TPM 2.0 and Secure Boot. Neither is present in a default KVM configuration, and both need to be added explicitly.

The swtpm package provides software TPM emulation:

sudo apt install swtpm swtpm-tools   # Debian/Ubuntu
sudo dnf install swtpm swtpm-tools   # Fedora

UEFI firmware is provided by the ovmf package, which supplies the file that virt-manager needs for Secure Boot:

sudo apt install ovmf   # Debian/Ubuntu
sudo dnf install edk2-ovmf   # Fedora

In virt-manager, when creating the VM, the firmware should be set to UEFI x86_64: /usr/share/OVMF/OVMF_CODE.fd rather than the default BIOS option. A TPM 2.0 device should be added in the hardware configuration before the VM is started. With those two elements in place, the Windows 11 installer proceeds without complaint about the hardware requirements.

The VirtIO drivers ISO should be attached as a second virtual CD-ROM drive during installation. The installer will not find the storage device otherwise because the VirtIO disk controller is not a standard device that Windows recognises without a driver. When prompted to select an installation location and no disks appear, clicking "Load driver" and browsing to the VirtIO ISO resolves it.

During the out-of-box experience, Windows 11 requires a Microsoft account and an internet connection by default. To bypass this and create a local account instead, opening a command prompt with Shift+F10 and running the following works on the Home edition:

oobebypassNRO

The machine restarts and presents an option to proceed without internet access.

Performance Considerations

KVM performance for a Windows 11 guest is generally good, but one factor specific to Windows 11 is worth understanding. Memory Integrity, also referred to as Hypervisor-Protected Code Integrity (HVCI), is a Windows security feature that uses virtualisation to protect the kernel. Running it inside a virtual machine creates nested virtualisation overhead because the guest is attempting to run its own virtualisation layer inside the host's. The performance impact is more pronounced on processors predating Intel Kaby Lake or AMD Zen 2, where the hardware support for nested virtualisation is less capable.

The CPU type selection in virt-manager also matters more than it might appear. Setting the CPU model to host-passthrough exposes the actual host CPU flags to the guest, which improves performance compared to emulated CPU models, at the cost of reduced portability if the VM image is ever moved to a different machine.

Host File System Access and Clipboard Sharing

This was where the experience diverged most noticeably from VirtualBox. VirtualBox Guest Additions handle shared folders and clipboard integration as a single installation, and the result works reliably with minimal configuration. KVM requires separate solutions for each, and in 2022 neither was as seamless as it has since become.

Clipboard Sharing via SPICE

Clipboard sharing uses the SPICE display protocol rather than VNC. The VM needs a SPICE display and a virtio-serial controller, which virt-manager adds automatically when SPICE is selected. Within the Windows guest, the installer for SPICE guest tools provides the clipboard agent. Once installed, clipboard text passes between host and guest in both directions.

The critical dependency that caused problems in 2022 was the virtio-serial channel. Without a com.redhat.spice.0 character device present in the VM configuration, the clipboard agent installs successfully but does nothing. Virt-manager now adds this automatically when SPICE is selected, which removes one of the more common failure points.

Host Directory Sharing via Virtiofs

At the time of this investigation, the practical option for sharing files between the Linux host and a Windows guest was WebDAV, which worked but felt like a workaround. The proper solution, virtiofs, existed but was not yet well-supported on Windows guests. The situation has since improved to the point where virtiofs is now the standard recommended approach.

It requires three components: the virtiofsd daemon on the host (included in recent QEMU packages), the virtiofs driver from the VirtIO Windows drivers package and WinFsp, which is the Windows equivalent of FUSE. Once configured through virt-manager's file system hardware settings, the shared directory appears as a mapped drive in Windows Explorer. The virtiofsd daemon was also rewritten in Rust in the intervening period, improving both its reliability and performance.

To configure a shared directory, shared memory must first be enabled in the VM's memory settings, then a file system device added with the driver set to virtiofs, a source path on the host and an arbitrary mount tag. The corresponding libvirt XML looks like this:

<memoryBacking>
  <source type='memfd'/>
  <access mode='shared'/>
</memoryBacking>

<filesystem type='mount' accessmode='passthrough'>
  <driver type='virtiofs' queue='1024'/>
  <source dir='/home/user/shared'/>
  <target dir='host_share'/>
</filesystem>

This was the area where VirtualBox held a clear practical advantage in 2022. The gap has since narrowed considerably.

Migrating from VirtualBox

Moving existing VirtualBox VMs to KVM is possible using qemu-img, which converts between disk image formats. The straightforward conversion from VDI to QCOW2 is:

qemu-img convert -f vdi -O qcow2 windows11.vdi windows11.qcow2

For large images or where reliability is a concern, converting via an intermediate RAW format reduces the risk of issues:

qemu-img convert -f vdi -O raw windows11.vdi windows11.raw
qemu-img convert -f raw -O qcow2 windows11.raw windows11.qcow2

The resulting QCOW2 file can then be used when creating a new VM in virt-manager, selecting "Import existing disk image" rather than creating a new one.

How the Landscape Has Shifted Since

The investigation described here took place during a specific window: VirtualBox 6.1.x was the current release, Windows 11 had just launched, and KVM was the most practical route to TPM emulation on Linux. That context has changed in several ways worth noting for anyone reading this in 2026.

VirtualBox 7.0 arrived in October 2022 with TPM 1.2 and 2.0 support, Secure Boot and a number of additional improvements. The original reason for investigating KVM was resolved, and for those who had moved across during the gap period, returning to VirtualBox for Windows guests made sense given its more straightforward Guest Additions integration.

QEMU reached version 10.0 in April 2025, a significant milestone reflecting years of accumulated improvements to hardware emulation, storage performance and x86 guest support. Libvirt has kept pace, adding reliable internal snapshots for UEFI-based VMs, evdev input device hot plug and improved unprivileged user support. The virtiofs situation for Windows guests has moved from "technically possible but awkward" to "the recommended approach with good documentation and a rewritten daemon", which addresses the most significant practical shortcoming from 2022 directly.

The broader desktop virtualisation landscape shifted when VMware Workstation Pro became free for all users, including commercial ones, in November 2024. VMware Workstation Player was discontinued as a separate product at the same time, having become redundant once Workstation Pro was available at no cost. This gave desktop users a third credible option alongside VirtualBox and KVM, with VMware's historically strong Windows guest integration now accessible without a licence fee, though users of the free version are not entitled to support through the global support team.

The miniature PC market also expanded considerably from 2023 onwards, with Intel N100-based and AMD Ryzen Embedded systems offering enough performance to run Windows natively at modest cost. For many people, that proves a cleaner solution than any hypervisor, eliminating the integration limitations entirely by giving Windows its own dedicated hardware.

Final Assessment

KVM handled Windows 11 competently during a period when the alternatives could not, and the platform has continued to improve in the years since. The two areas that fell short in 2022, host file sharing and clipboard integration, have been addressed by developments in virtiofs and the SPICE tooling, and a new user starting today may find the experience noticeably smoother.

Whether KVM is the right choice in 2026 depends on the use case. For Linux-native workloads and server-style VM management, it remains the strongest option on Linux. For a Windows desktop guest where ease of integration matters most, VirtualBox 7.x and VMware Workstation Pro are both strong alternatives, with the latter now free to use for both commercial and personal purposes. The question that drove this investigation was answered by VirtualBox itself in October 2022. KVM provided a workable solution in the meantime, and the platform has only become more capable since then.

Additional Reading

How To Convert VirtualBox Disk Image (VDI) to Qcow2 format

How to enable TPM and secure boot on KVM?

Windows 11 on KVM – How to Install Step by Step?

Enable Virtualization-based Protection of Code Integrity in Microsoft Windows

Getting to know Jira, its workflows, test management capabilities and the need for governance

3^rd March 2026

Developed by Atlassian and first released in 2002 as a straightforward bug and issue tracker aimed at software developers, Jira has since grown into a platform used for project management across a wide range of industries and disciplines. The name itself is a truncation of Gojira, the Japanese word for Godzilla, originating as an internal nickname used by Atlassian developers for Bugzilla, the bug-tracking tool they had previously relied upon.

A Family of Products, Each With a Purpose

The Jira ecosystem has expanded well beyond its original single offering, and it is worth understanding what each product is designed to do. Jira (formerly marketed as Jira Software, now unified with Jira Work Management) remains the flagship, built around agile project management with Scrum and Kanban boards at its core. Jira Service Management serves IT operations and service desk teams, handling ticketing and customer support workflows; it originated as Jira Service Desk in 2013, following Atlassian's discovery that nearly 40 per cent of their customers had already adapted the base product for service requests, and it was rebranded in 2020. At the enterprise level, Jira Align connects team delivery to strategic business goals, while Jira Product Discovery helps product teams capture feedback, prioritise ideas and build roadmaps. Together, these products span the full organisational hierarchy, from individual contributors up to executive portfolio management.

Core Features

Agile Boards and Backlog Management

Jira supports a range of agile methodologies, with two primary project templates available to teams. The Scrum template is designed for teams that deliver work in time-boxed sprints, providing backlog management, sprint planning and capacity tracking in a single view. The Kanban template, by contrast, is built around a continuous flow of work, helping teams visualise tasks as they move through each stage of a process without the constraint of fixed iterations. Both templates support custom configurations for teams whose ways of working do not map neatly to either model.

Reporting and Analytics

Jira's reporting suite provides visibility into project progress through various charts and metrics. The Burndown chart tracks remaining story points against the time left in a sprint, offering an indication of whether the team is on course to complete its committed work. The Burnup chart takes a complementary view, tracking how much work has been completed over time and making it straightforward to compare planned scope against actual delivery. These tools are useful for identifying patterns in team performance, though they are most informative when used consistently over several sprints rather than in isolation.

Custom Workflows

Teams can design workflows that reflect their own processes, defining the states an issue passes through and the transitions between them. Automation rules can be applied to handle repetitive steps without manual intervention, reducing administrative overhead on routine tasks. This flexibility is one of the more frequently cited reasons for adopting Jira, though it does require ongoing governance to prevent workflows from becoming inconsistent or unwieldy as teams and processes evolve.

Jira Query Language

Jira Query Language (JQL) provides a structured way to search and filter issues across projects, enabling teams to construct precise queries based on any combination of fields, statuses, assignees, dates and custom attributes. For organisations that invest time in learning it, JQL is a practical tool for building custom reports and dashboards. It is also the underlying mechanism for many of Jira's more advanced filtering and automation features.

Integration Options

Jira connects with a range of tools both within and outside the Atlassian ecosystem. Confluence handles documentation, Bitbucket manages code repositories and links commits directly to Jira issues, and Loom, acquired by Atlassian in 2023, adds asynchronous video communication. Third-party integrations, including Zoom and a broad catalogue of tools available through the Atlassian Marketplace, extend this further for teams with specific requirements.

Test Management With Xray

Jira does not include dedicated test management functionality by default, and teams that need to manage structured test cases alongside their development work typically turn to the Xray plugin, one of the most widely used additions in the Atlassian Marketplace. Xray operates as a native Jira application, meaning it adds new issue types directly to the Jira instance rather than sitting as a separate external tool. The issue types it introduces include Test, Test Set, Test Plan and Test Execution, all of which behave like standard Jira issues and can be searched, filtered and reported on using JQL.

A key capability is requirements traceability: Xray links test cases directly to the user stories and requirements they cover, and connects those in turn to any defects raised during execution. This gives teams a clear picture of test coverage and release readiness without having to leave Jira or reconcile data from separate systems. Test executions can be manual or automated, and Xray integrates with CI/CD toolchains (including Jenkins and Robot Framework) via a REST API, allowing automated test results to be published back into Jira and associated with the relevant requirements.

Xray also supports Behaviour-Driven Development (BDD), enabling teams to write tests in Gherkin syntax and manage them alongside their other Jira work. For organisations already using Jira as their central project management tool, Xray offers a practical route to bringing QA activities into the same workflow rather than maintaining a separate test management system.

Who is Jira Best Suited For?

Jira is generally considered most suitable for larger teams that require detailed control over workflows, reporting and resource allocation, and that have the capacity to dedicate administrative effort to the platform. Smaller teams or those without a dedicated Jira administrator may find the learning curve significant, particularly when configuring custom workflows or working with more advanced reporting features. Pricing is subscription-based, with tiers determined by user count and deployment model (cloud-hosted or self-managed), which means costs can increase substantially as an organisation grows.

Project Types: Tailoring Access to Needs

Jira divides its project spaces into two categories that serve different audiences. Team-managed projects offer simplified configuration for smaller, autonomous teams that want to get started without involving a Jira administrator. Company-managed projects grant administrators full control over customisation, permissions and settings, making them more appropriate for enterprises with complex requirements and multiple teams operating within the same instance. The two types can coexist within the same deployment, giving organisations the option to apply different governance models to different teams as their needs dictate.

Strengths and Limitations

Jira's scalability is one of its more consistent strengths, in terms of both the size of the user base it can support and the complexity of workflows it can accommodate. Its query functions give teams a precise way to interrogate project data, and its breadth of integrations means it can be connected to most standard development and collaboration toolchains.

A significant consideration for any Jira deployment is the degree of upfront decision-making it requires. Because the platform places few constraints on how it is configured, teams must establish their own conventions around workflow design, issue hierarchy, naming and permissions before adoption begins in earnest. Without that groundwork, it is straightforward for individual teams to configure Jira in incompatible ways, making cross-team reporting difficult and creating inconsistencies that become harder to unpick over time. Organisations that treat Jira as something to be governed, rather than simply installed, tend to get considerably more out of it.

The principal technical limitation is its dependence on the wider Atlassian ecosystem. Advanced portfolio planning, capacity forecasting and cross-programme dependency management typically require either a higher-tier plan or additional tooling. Advanced Roadmaps (now called Plans) are available natively within Jira Premium and Enterprise, providing cross-team timeline planning and scenario modelling. For capacity planning, budget tracking and timesheet management, many organisations turn to third-party Marketplace tools such as Tempo. Teams evaluating Jira should factor in both the cost of the appropriate licence tier and any supplementary tooling they are likely to need.

Where to Go From Here

Jira has grown considerably from the issue tracker it was when first released in 2002, and is now used by over 300,000 organisations worldwide. Its capabilities are broad, and its configurability makes it adaptable to a wide range of team structures and workflows. That same configurability, however, means the platform rewards investment in setup and ongoing administration, and organisations should assess whether they have the resources to realise that potential before committing. For those looking to explore further, Atlassian's official guides, its wider documentation, the support portal, the Atlassian Community and the developer documentation are useful starting points, and there are courses from an independent provider too.

Hardening WordPress on Ubuntu and Apache: A practical layered approach

1^st March 2026

Protecting a WordPress site rarely depends on a single control. Practical hardening layers network filtering, a web application firewall (WAF), careful browser-side restrictions and sensible log-driven tuning. What follows brings together several well-tested techniques and the precise commands needed to get them working, while also calling out caveats and known changes that can catch administrators out. The focus is on Ubuntu and Apache with ModSecurity and the OWASP Core Rule Set for WordPress, but complementary measures round out a cohesive approach. These include a strict Content Security Policy, Cloudflare or Nginx rules for form spam, firewall housekeeping for UFW and Docker, targeted network blocks and automated abuse reporting with Fail2Ban. Where solutions have moved on, that is noted so you do not pursue dead ends.

The Web Application Firewall

ModSecurity and the OWASP Core Rule Set

ModSecurity remains the most widespread open-source web application firewall and has been under the custodianship of the OWASP Foundation since January 2024, having previously been stewarded by Trustwave for over a decade. It integrates closely with the OWASP Core Rule Set (CRS), which aims to shield web applications from a wide range of attacks including the OWASP Top Ten, while keeping false alerts to a minimum. There are two actively maintained engines: 2.9.x is the classic Apache module and 3.x is the newer, cross-platform variant. Whichever engine you pick, the rule set is the essential companion. One important update is worth stating at the outset: CRS 4 replaces exclusion lists with plugins, so older instructions that toggle CRS 3's exclusions no longer apply as written.

Installing ModSecurity on Ubuntu

On Ubuntu 24.04 LTS, installing the Apache module is straightforward. The universe repository ships libapache2-mod-security2 at version 2.9.7, which meets the 2.9.6 minimum required by CRS 4.x, so no third-party repository is needed. You can fetch and enable ModSecurity with the following commands:

sudo apt install libapache2-mod-security2
sudo a2enmod security2
sudo systemctl restart apache2

It is worth confirming the module is loaded before you proceed:

apache2ctl -M | grep security

The default configuration runs in detection-only mode, which does not block anything. Copy the recommended file into place and then edit it so that SecRuleEngine On replaces SecRuleEngine DetectionOnly:

sudo cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Open /etc/modsecurity/modsecurity.conf and make the change, then restart Apache once more to apply it.

Pulling in the Core Rule Set

The next step is to pull in the latest Core Rule Set and wire it up. A typical approach is to clone the upstream repository, move the example setup into place and move the directory named rules into /etc/modsecurity:

cd
git clone https://github.com/coreruleset/coreruleset.git
cd coreruleset
sudo mv crs-setup.conf.example /etc/modsecurity/crs-setup.conf
sudo mv rules/ /etc/modsecurity/

Now adjust the Apache ModSecurity include so that the new crs-setup.conf and all files in /etc/modsecurity/rules are loaded. On Ubuntu, that is governed by /etc/apache2/mods-enabled/security2.conf. Edit this file to reference the new paths, remove any older CRS include lines that might conflict, and then run:

sudo systemctl restart apache2

On Ubuntu 26.04 (due for release in April 2026), the default installation includes a pre-existing CRS configuration at /etc/modsecurity/crs/crs-setup.conf. If this is left in place alongside your own cloned CRS, Apache will fail to start with a Found another rule with the same id error. Remove it before restarting:

sudo rm -f /etc/modsecurity/crs/crs-setup.conf

WordPress-Specific Allowances in CRS 3

WordPress tends to work far better with CRS when its application-specific allowances are enabled. With CRS 3, a variable named tx.crs_exclusions_wordpress can be set in crs-setup.conf to activate those allowances. The commented "exclusions" block in that file includes a template SecAction with ID 900130 that sets application exclusions. Uncomment it and reduce it to the single line that enables the WordPress flag:

SecAction 
 "id:900130,
  phase:1,
  nolog,
  pass,
  t:none,
  setvar:tx.crs_exclusions_wordpress=1"

Reload Apache afterwards with sudo service apache2 reload. If you are on CRS 4, do not use this older mechanism. The project has replaced exclusions with a dedicated WordPress rule exclusions plugin, so follow the CRS 4 plugin documentation instead. The WPSec guide to ModSecurity and CRS covers both the CRS 3 and CRS 4 approaches side by side if you need a reference that bridges the two versions.

Log Retention and WAF Tuning

Once the WAF is enforcing, logs become central to tuning. Retention is important for forensics as well as for understanding false positives over time, so do not settle for the default two weeks. On Ubuntu, you can extend Apache's logrotate configuration at /etc/logrotate.d/apache2 to keep weekly logs for 52 weeks, giving you a year of history to hand.

If you see Execution error – PCRE limits exceeded (-8) in the ModSecurity log, increase the following in /etc/modsecurity/modsecurity.conf to give the regular expression engine more headroom:

SecPcreMatchLimit 1000000
SecPcreMatchLimitRecursion 1000000

File uploads can generate an Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required error. One remedy used in practice is to comment out the offending check around line 86 of modsecurity.conf and then reload. The built-in Theme Editor can trigger Request body no files data length is larger than the configured limit. Bumping SecRequestBodyLimit to 6000000 addresses that, again followed by a reload of Apache.

Whitelisting Rule IDs for Specific Endpoints

There are occasions where whitelisting specific rule IDs for specific WordPress endpoints is the most pragmatic way to remove false positives without weakening protection elsewhere. Creating a per-site or server-wide include works well; on Ubuntu, a common location is /etc/apache2/conf-enabled/whitelist.conf. For the Theme Editor, adding a LocationMatch block for /wp-admin/theme-editor.php that removes a small set of well-known noisy IDs can help:

<LocationMatch "/wp-admin/theme-editor.php">
  SecRuleRemoveById 300015 300016 300017 950907 950005 950006 960008 960011 960904 959006 980130
</LocationMatch>

For AJAX requests handled at /wp-admin/admin-ajax.php, the same set with 981173 added is often used. This style of targeted exclusion mirrors long-standing community advice: find the rule ID in logs, remove it only where it is truly safe to do so, and never disable ModSecurity outright. If you need help finding noisy rules, the following command (also documented by InMotion Hosting) summarises IDs, hostnames and URIs seen in errors:

grep ModSecurity /usr/local/apache/logs/error_log | grep "[id" | 
  sed -E -e 's#^.*[id "([0-9]*).*hostname "([a-z0-9-_.]*)"].*uri "(.*?)".*"#1 2 3#' | 
  cut -d" -f1 | sort -n | uniq -c | sort -n

Add a matching SecRuleRemoveById line in your include and restart Apache.

Browser-Side Controls: Content Security Policy

Beyond the WAF, browser-side controls significantly reduce the harm from injected content and cross-site scripting. A Content Security Policy (CSP) is both simple to begin and very effective when tightened. An easy starting point is a report-only header that blocks nothing but shows you what would have been stopped. Adding the following to your site lets you open the browser's developer console and watch violations scroll by as you navigate:

Content-Security-Policy-Report-Only: default-src 'self'; font-src 'self'; img-src 'self'; script-src 'self'; style-src 'self'

From there, iteratively allowlist the external origins your site legitimately uses and prefer strict matches. If a script is loaded from a CDN such as cdnjs.cloudflare.com, referencing the exact file or at least the specific directory, rather than the whole domain, reduces exposure to unrelated content hosted there. Inline code is best moved to external files. If that is not possible, hashes can allowlist specific inline blocks and nonces can authorise dynamically generated ones, though the latter must be unpredictable and unique per request. The 'unsafe-inline' escape hatch exists but undermines much of CSP's value and is best avoided.

Once the console is clean, you can add real-time reporting to a service such as URIports (their guide to building a solid CSP is also worth reading) by extending the header:

Content-Security-Policy-Report-Only: default-src 'self'; ...; report-uri https://example.uriports.com/reports/report; report-to default

Pair this with a Report-To header so that you can monitor and prioritise violations at scale. When you are satisfied, switch the key from Content-Security-Policy-Report-Only to Content-Security-Policy to enforce the policy, at which point browsers will block non-compliant content.

Server Fingerprints and Security Headers

While working on HTTPS and header hardening, it is useful to trim server fingerprints and raise other browser defences, and this Apache security headers walkthrough covers the rationale behind each directive clearly. Apache's ServerTokens directive can be set in /etc/apache2/apache.conf to mask version details. Options range from Full to Prod, with the latter sending only Server: Apache. Unsetting X-Powered-By in /etc/apache2/httpd.conf removes PHP version leakage. Adding the following headers in the same configuration file keeps responses out of hostile frames, asks browsers to block detected XSS and prevents MIME type sniffing:

X-Frame-Options SAMEORIGIN
X-XSS-Protection 1;mode=block
X-Content-Type-Options nosniff

These are not replacements for fixes in application code, but they do give the browser more to work with. If you are behind antivirus products or corporate HTTPS interception, bear in mind that these can cause certificate errors such as SEC_ERROR_UNKNOWN_ISSUER or MOZILLA_PKIX_ERROR_MITM_DETECTED in Firefox. Disabling encrypted traffic scanning in products like Avast, Bitdefender or Kaspersky, or ensuring enterprise interception certificates are correctly installed in Firefox's trust store, resolves those issues. Some errors cannot be bypassed when HSTS is used or when policies disable bypasses, which is the intended behaviour for high-value sites.

Contact Form Spam

Contact form spam is a different but common headache. Analysing access logs often reveals that many automated submissions arrive over HTTP/1.1 while legitimate traffic uses HTTP/2 with modern browser stacks, and this GridPane analysis of a real spam campaign confirms the pattern in detail. That difference gives you something to work with.

Filtering by Protocol in Cloudflare

You can block or challenge HTTP/1.x access to contact pages at the edge with Cloudflare's WAF by crafting an expression that matches both the old protocol and a target URI, while exempting major crawlers. A representative filter looks like this:

(http.request.version in {"HTTP/1.0" "HTTP/1.1" "HTTP/1.2"}
  and http.request.uri eq "/contact/"
  and not http.user_agent contains "Googlebot"
  and not http.user_agent contains "Bingbot"
  and not http.user_agent contains "DuckDuckBot"
  and not http.user_agent contains "facebot"
  and not http.user_agent contains "Slurp"
  and not http.user_agent contains "Alexa")

Set the action to block or to a managed challenge as appropriate.

Blocking Direct POST Requests Without a Valid Referrer

Another useful approach is to cut off direct POST requests to /wp-admin/admin-ajax.php and /wp-comments-post.php when the Referer does not contain your domain. In Cloudflare, this becomes:

(http.request.uri contains "/wp-admin/admin-ajax.php"
  and http.request.method eq "POST"
  and not http.referer contains "yourwebsitehere.com")
or
(http.request.uri contains "/wp-comments-post.php"
  and http.request.method eq "POST"
  and not http.referer contains "yourwebsitehere.com")

The same logic can be applied in Nginx with small site includes that set variables based on $server_protocol and $http_user_agent, then return 403 if a combination such as HTTP/1.1 on /contact/ by a non-whitelisted bot is met. It is sensible to verify with Google Search Console or similar that legitimate crawlers are not impeded once rules are live.

Complementary Mitigations Inside WordPress

Three complementary tools work well alongside the server-side measures already covered. The first is WP Armour, a free honeypot anti-spam plugin that adds a hidden field to comment forms, contact forms and registration pages using JavaScript. Because spambots cannot execute JavaScript, the field is never present in a genuine submission, and any bot that attempts to fill it is rejected silently. No CAPTCHA, API key or subscription is required, and the plugin is GDPR-compliant with no external server calls.

The second measure is entirely native to WordPress. Navigate to Settings, then Discussion and tick "Automatically close comments on articles older than X days." Spammers disproportionately target older content because it tends to be less actively monitored, so setting this to 180 days significantly reduces spam without affecting newer posts where discussion is still active. The value can be adjusted to suit the publishing cadence of the site.

The third layer is Akismet, developed by Automattic. Akismet passes each comment through its cloud-based filter and marks likely spam before it ever appears in the moderation queue. It is free for personal sites and requires an API key obtained from the Akismet website. Used alongside WP Armour, the two cover different vectors: WP Armour stops most bot submissions before they are processed at all, while Akismet catches those that reach the comment pipeline regardless of origin. Complementing both, reCAPTCHA v3 or hCaptcha (where privacy demands it) and simple "bot test" questions remain useful additions, though any solution that adds heavy database load warrants testing before large-scale deployment.

Host-Level Firewalls: UFW and Docker

Host-level firewalls remain important, particularly when Docker is in the mix. Ubuntu's UFW is convenient, but Docker's default iptables rules can bypass UFW and expose published ports to the public network even when ufw deny appears to be in place. One maintained solution uses the kernel's DOCKER-USER chain, so UFW regains control without disabling Docker's iptables management.

Appending a short block to /etc/ufw/after.rules that defines ufw-user-forward, a ufw-docker-logging-deny target and a DOCKER-USER chain, then jumps from DOCKER-USER into ufw-user-forward, allows UFW to govern forwarded traffic. Returning early for RELATED,ESTABLISHED connections, dropping invalid ones, accepting docker0-to-docker0 traffic and returning for RFC 1918 source ranges preserves internal communications. New connection attempts from public networks destined for private address ranges are logged and dropped, with a final RETURN handing off to Docker's own rules for permitted flows.

Restart UFW to activate the change:

sudo systemctl restart ufw
# or
sudo ufw reload

From that point, you can allow external access to a container's service port:

ufw route allow proto tcp from any to any port 80

Or scope to a specific container IP if needed:

ufw route allow proto tcp from any to 172.17.0.2 port 80

UDP rules follow the same pattern. If you prefer not to edit by hand, the UFW-docker helper script can install, check and manage these rules for you. It supports options to auto-detect Docker subnets, supports IPv6 by enabling ip6tables and a ULA (Unique Local Address) range in /etc/docker/daemon.json and can manage Swarm service exposure from manager nodes.

Should you instead use Firewalld, note that it provides a dynamically managed firewall with zones, a D-Bus API and runtime versus permanent configuration separation. It is the default in distributions such as RHEL, CentOS, Fedora and SUSE, and it also works with Docker's iptables backend, though the interaction model differs from UFW's.

Keeping Firewall Rules Tidy

Keeping firewall rules tidy is a small but useful habit. UFW can show verbose and numbered views of its state, as Linuxize's UFW rules guide explains in full:

sudo ufw status verbose
sudo ufw status numbered

Delete rules safely by number or by specification:

sudo ufw delete 4
sudo ufw delete allow 80/tcp

If you are scripting changes, the --force flag suppresses the interactive prompt. Take care never to remove your SSH allow rule when connected remotely, and remember that rule numbers change after deletions, so it is best to list again before removing the next one.

Logging Abusers with Fail2Ban and AbuseIPDB

Logging abusers and reporting them can reduce repeat visits. Fail2Ban watches logs for repeated failures and bans IPs by updating firewall rules for a set period. It can also report to AbuseIPDB via an action that was introduced in v0.10.0 (January 2017), which many installations have today.

Confirm that /etc/fail2ban/action.d/abuseipdb.conf exists and that your /etc/fail2ban/jail.local defines action_abuseipdb = abuseipdb. Within each jail that you want reported, add the following alongside your normal ban action, using categories that match the jail's purpose, such as SSH brute forcing:

%(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]

Reload with fail2ban-client reload and watch your AbuseIPDB reported IPs page to confirm submissions are flowing. If reports do not arrive, check /var/log/fail2ban.log for cURL errors and ensure your API key is correct, bearing in mind default API limits and throttling. Newer Fail2Ban versions (0.9.0 and above) use a persistent database, so re-reported IPs after restart are less of a concern. If you run older releases, a wrapper script can avoid duplicates by checking ban times before calling the API.

Blocking Provider Ranges

Occasionally, administrators choose to block traffic from entire provider ranges that are persistent sources of scanning or abuse. There are scripts such as the AWS-blocker tool that fetch the official AWS IPv4 and IPv6 ranges and insert iptables rules to block them all, and community posts such as this rundown of poneytelecom.eu ranges that shares specific ranges associated with problematic hosts for people who have seen repeated attacks from those networks. Measures like these are blunt instruments that can have side effects, so they warrant careful consideration and ongoing maintenance if used at all. Where possible, it is preferable to block based on behaviour, authentication failures and reputation rather than on broad ownership alone.

Final ModSecurity Notes: Chasing False Positives

Two final ModSecurity notes help when chasing false positives. First, WordPress comments and posting endpoints can trip generic SQL injection protections such as rule 300016 when text includes patterns that appear dangerous to a naive filter, a well-documented occurrence that catches many administrators out. Watching /etc/httpd/logs/modsec_audit.log or the Apache error log immediately after triggering the offending behaviour, and then scoping SecRuleRemoveById lines to the relevant WordPress locations such as /wp-comments-post.php and /wp-admin/post.php, clears real-world issues without turning off protections globally. Second, when very large responses are legitimately expected in parts of wp-admin, increasing SecResponseBodyLimit in an Apache or Nginx ModSecurity context can be more proportionate than whitelisting many checks at once. Always restart or reload Apache after changes so that your edits take effect.

Defence in Depth

Taken together, these layers complement each other well. ModSecurity with CRS gives you broad, configurable protection at the HTTP layer. CSP and security headers narrow the browser's attack surface and put guardrails in place for any client-side content issues. Targeted edge and server rules dampen automated spam without hindering real users or crawlers. Firewalls remain the bedrock, but modern container tooling means integrating UFW or Firewalld with Docker requires a small amount of extra care. Logs feed both your WAF tuning and your ban lists, and when you report abusers you contribute to a wider pool of threat intelligence. None of this removes the need to keep WordPress core, themes and plugins up to date, but it does mean the same attacks are far less likely to succeed or even to reach your application in the first place.

Catching keyboard interruptions in a Python script for a more orderly exit

17^th April 2024

A while back, I was using a Python script to watch a folder and process photos in there, whenever a new one was added. Eventually, I ended up with a few of these because I was unable to work out a way to get multiple folders watched in the same script.

In each of them, though, I needed a tidy way to exit a running script in place of the stream of consciousness that often emerges when you do such things to it. Because I knew what was happening anyway, I needed a script to terminate quietly and set to uncover a way to achieve this.

What came up was something like the code that you see below. While I naturally did some customisations, I kept the essential construct to capture keyboard interruption shortcuts, like the use of CTRL + C in a Linux command line interface.

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        print('Interrupted')
        try:
            sys.exit(130)
        except SystemExit:
            os._exit(130)

What is happening above is that the interruption operation is captured in a nested TRY/EXCEPT block. The outer block catches the interruption, while the inner one runs through different ways of performing the script termination. For the first termination function call, you need to call the SYS package and the second needs the OS one, so you need to declare these at the start of your script.

Of the two, SYS.EXIT is preferable because it invokes clean-up activities while OS._EXIT does not, which might explain the "_" prefix in the second of these. In fact, calling SYS.EXIT is not that different to issuing RAISE SYSTEMEXIT instead because that lies underneath it. Here OS._EXIT is the fallback for when SYS.EXIT fails, and it is not all that desirable given the abrupt action that it invokes.

The exit code of 130 is fed to both, since that is what is issued when you terminate a running instance of an application on Linux anyway. Using 0 could negate any means of picking up what has happened if you have downstream processing. In my use case, everything was standalone, so that did not matter so much.

Installing Firefox Developer Edition in Linux Mint

22^nd April 2018

Having moved beyond the slow response and larger memory footprint of Firefox ESR, I am using Firefox Developer Edition in its place, even if it means living without a status bar at the bottom of the window. Hopefully, someone will create an equivalent of the old add-on bar extensions that worked before the release of Firefox Quantum.

Firefox Developer Edition may be pre-release software with some extras for web developers like being able to drill into an HTML element and see its properties, but I am finding it stable enough for everyday use. It is speedy too, which helps, and it has its own profile so it can co-exist on the same machine as regular releases of Firefox like its ESR and Quantum variants.

Installation takes a little added effort though and there are various options available. My chosen method involved Ubuntu Make. Installing this involves setting up a new PPA as the first step and the following commands added the software to my system:

sudo add-apt-repository ppa:ubuntu-desktop/ubuntu-make
sudo apt-get update
sudo apt-get install ubuntu-make

With the above completed, it was simple to install Firefox Developer edition using the following command:

umake web firefox-dev

Where things got a bit more complicated was getting entries added to the Cinnamon Menu and Docky. While the former was sorted using the cinnamon-menu-editor command, the latter needed some tinkering with my firefox-developer.desktop file found in .local/share/applications/ within my user area to get the right icon shown. Discovering this took me into .gconf/apps/docky-2/Docky/Interface/DockPreferences/%gconf.xml where I found the location of the firefox-developer.desktop that needed changing. Once this was completed, there was nothing else to do from the operating system side.

Within Firefox itself, I opted to turn off warnings about password logins on non-HTTPS websites by going to about:config using the address bar, then looking for security.insecure_field_warning.contextual.enabled and changing its value from True to False. While some may decry this, there are some local websites on my machine that need attention at times. Otherwise, Firefox is installed with user access so I can update it as if it were a Windows or macOS application, and that is useful given that there are frequent new releases. All is going as I want it so far.

Installing Citrix Receiver 13.0 in Ubuntu GNOME 13.10 64-bit

28^th November 2013

Installing the latest version of Citrix Receiver (13.0 at the time of writing) on 64-bit Ubuntu should be as simple as downloading the required DEB package and double-clicking on the file so that Ubuntu Software Centre can work its magic. Unfortunately, the 64-bit DEB file is faulty, so that means that the Ubuntu community how-to guide for Citrix still is needed. In fact, any user of Linux Mint or another distro that uses Ubuntu as its base would do well to have a look at that Ubuntu link.

For the sake of completeness, I still am going to let you in on the process that worked for me. Once the DEB file has been downloaded, the first task is to create a temporary folder where the DEB file's contents can be extracted:

mkdir ica_temp

With that in place, it then is time to do the extraction, and it needs two commands with the second of these need to extract the control file while the first extracts everything else.

sudo dpkg-deb -x icaclient- ica_temp
sudo dpkg-deb --control icaclient- ica_temp/DEBIAN

It is the control file that has been the cause of all the bother because it refers to unavailable dependencies that it really doesn't need anyway. To open the file for editing, issue the following command:

sudo gedit ica_temp/DEBIAN/control

Then change line 7 (it should begin with Depends:) to: Depends: libc6-i386 (>= 2.7-1), lib32z1, nspluginwrapper. While there are other software packages in there that Ubuntu no longer supports, they are not needed anyway. With the edit made, and the file saved, the next step is to build a new DEB package with the corrected control file:

dpkg -b ica_temp icaclient-modified.deb

Once you have the package, the next step is to install it using the following command:

sudo dpkg -i icaclient-modified.deb

If it fails, then you have missing dependencies and the following command should sort these before a re-run of the above command again:

sudo apt-get install libmotif4:i386 nspluginwrapper lib32z1 libc6-i386

With Citrix Receiver installed, there is one more thing that is needed before you can use it freely. This is to put Thawte security certificate files into /opt/Citrix/ICAClient/keystore/cacerts. What I had not realised until recently was that many of these already are in /usr/share/ca-certificates/mozilla and linking to them with the following command makes them available to Citrix Receiver:

sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts/

Another approach is to download the Thawte certificates and extract the archive to /tmp/. From there they can be copied to /opt/Citrix/ICAClient/keystore/cacerts and I copied the Thawte Personal Premium certificate as follows:

sudo cp /tmp/Thawte Root Certificates/Thawte Personal Premium CA/Thawte Personal Premium CA.cer /opt/Citrix/ICAClient/keystore/cacerts/

Until I found out about what was in the Mozilla folder, I simply picked out the certificate mentioned in the Citrix error message and copied it over like the above. Of course, all of this may seem like a lot of work to those who are non-tinkerers and I have added a repaired 64-bit DEB package that incorporates all of the above and should not need any further intervention aside from installing it using GDebi, Ubuntu's Software Centre, dpkg or anything else that does what's needed.