Technology Tales

What to do an error appears when using pip to install Python packages on Linux Mint 22

After upgrading to Linux Mint 22, the following message appeared when attempting to install Python packages using the pip command:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

This will frustrate anyone following how-tos on the web, so users will need to know about it. On something like Linux Mint, the repositories may not be as up-to-date as PyPI, so picking up the very latest version has its advantages. Thus, I initially used the unrecommended --break-system-packages switch to get things going as before, since doing never broke anything before. While the way of working feels like an overkill in some ways, using pipx probably is the way forward as long as things work as I want them to do.

There is wisdom in using virtual environments too, especially when AI models are involved. For most of what I get to do, that may be getting too elaborate. Then, deleting or renaming the message file in /usr/lib/python3.12/EXTERNALLY-MANAGED is tempting if that gets around things, as retrograde as that probably is. After all, I never broke anything before this message started to appear, possibly since my interests are data related.

Rendering Markdown into HTML using PHP

One of the good things about using virtual private servers for hosting websites instead of shared hosting or using a web application service like WordPress.com or Tumblr is that you get added control and flexibility. There was a time when HTML, CSS and client-side scripting were all that was available from the shared hosting providers that I was using back then. Then, static websites were my lot until it became possible to use Perl server side scripting. PHP predominates now, but Python or Ruby cannot be discounted either.

Being able to install whatever you want is a bonus as well, though it means that you also are responsible for the security of the containers that you use. There will be infrastructure security, but that of your own machine will be your own concern. Added power always means added responsibility, as many might say.

The reason that these thought emerge here is that getting PHP to render Markdown as HTML needs the installation of Composer. Without that, you cannot use the CommonMark package to do the required back-work. All the command that you see here will work on Ubuntu 22.04. First, you need to download Composer and executing the following command will accomplish this:

curl https://getcomposer.org/installer -o /tmp/composer-setup.php

Before the installation, it does no harm to ensure that all is well with the script before proceeding. That means that capturing the signature for the script using the following command is wise:

HASH=`curl https://composer.github.io/installer.sig`

Once you have the script signature, then you can check its integrity using this command:

php -r "if (hash_file('SHA384', '/tmp/composer-setup.php') === '$HASH') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

The result that you want is "Installer verified". If not, you have some investigating to do. Otherwise, just execute the installation command:

sudo php /tmp/composer-setup.php --install-dir=/usr/local/bin --filename=composer

With Composer installed, the next step is to run the following command in the area where your web server expects files to be stored. That is important when calling the package in a PHP script.

composer require league/commonmark

Then, you can use it in a PHP script like so:

define("ROOT_LOC",$_SERVER['DOCUMENT_ROOT']);
include ROOT_LOC . '/vendor/autoload.php';
use League\CommonMark\CommonMarkConverter;
$converter = new CommonMarkConverter();
echo $converter->convertToHtml(file_get_contents(ROOT_LOC . '<location of markdown file>));

The first line finds the absolute location of your web server file directory before using it when defining the locations of the autoload script and the required markdown file. The third line then calls in the CommonMark package, while the fourth sets up a new object for the desired transformation. The last line converts the input to HTML and outputs the result.

If you need to render the output of more than one Markdown file, then repeating the last line from the preceding block with a different file location is all you need to do. The CommonMark object persists and can be used like a variable without needing the reinitialisation to be repeated every time.

The idea of building a website using PHP to render Markdown has come to mind, but I will leave it at custom web pages for now. If an opportunity comes, then I can examine the idea again. Before, I had to edit HTML, but Markdown is friendlier to edit, so that is a small advance for now.

Contents not displaying for Shared Folders on a Fedora 32 guest instance in VirtualBox

While some Linux distros like Fedora install VirtualBox drivers during installation time, I prefer to install the VirtualBox Guest Additions themselves. Before doing this, it is best to remove the virtualbox-guest-additions package from Fedora to avoid conflicts. After that, execute the following command to ensure that all prerequisites for the VirtualBox Guest Additions are in place before mounting the VirtualBox Guest Additions ISO image and installing from there:

sudo dnf -y install gcc automake make kernel-headers dkms bzip2 libxcrypt-compat kernel-devel perl

During the installation, you may encounter a message like the following:

ValueError: File context for /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf already defined

This is generated by SELinux, so the following commands need to be executed before repeating the installation of VirtualBox Guest Additions:

sudo semanage fcontext -d /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf
sudo restorecon /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf

Without doing the above step and fixing the preceding error message, I had an issue with mounting of Shared Folders whereby the mount point was set up, but no folder contents were displayed. This happened even when my user account was added to the vboxsf group, and it proved to be the SELinux context issue that was the cause.

Updating Flatpack applications on Linux Mint 19

Since upgrading to Linux Mint 19, I have installed some software from Flatpak. The cause for my curiosity was that you could have the latest versions of applications like GIMP or LibreOffice without having to depend on a third-party PPA. Installation is straightforward given the support built into Linux Mint. You just need to download the relevant package from the Flatpak website and run the file through the GUI installer. Because the packages come with extras to ensure cross-compatibility, more disk space is used, but there is no added system overhead beyond that, from what I have seen. Updating should be as easy as running the following single command too:

flatpak update

However, I needed to do a little extra work before this was possible. The first step was to update the configuration file at ~/.local/share/flatpak/repo/config to add the following lines:

[remote "flathub"]
gpg-verify=true
gpg-verify-summary=true
url=https://flathub.org/repo/
xa.title=Flathub

Once that was completed, I ran the following commands to import the required GPG key:

wget https://flathub.org/repo/flathub.gpg
flatpak --user remote-modify --gpg-import=flathub.gpg flathub

With this complete, I was able to run the update process and update any applications as necessary. After that first run, it has been integrated in to my normal processes by adding the command to the relevant alias definition.

Trying out a new way to upgrade Linux Mint in situ while going from 17.3 to 18.1

There was a time when the only recommended way to upgrade Linux Mint from one version to another was to do a fresh installation with back-ups of data and a list of the installed applications created from a special tool.

Even so, it never stopped me doing my own style of in situ upgrade, though some might see that as a risky option. More often than not, that actually worked without causing major problems in a time when Linux Mint releases were more tightly tied to Ubuntu's own six-monthly cycle.

Linux Mint releases now align with Ubuntu's Long Term Support (LTS) editions. This means major changes occur only every two years, with minor releases in between. These minor updates are delivered through Linux Mint's Update Manager, making the process simple. Upgrades are not forced, so you can decide when to upgrade, as all main and interim versions receive the same extended support. The recommendation is to avoid upgrading unless something is broken on your installation.

For a number of reasons, I stuck with that advice by sticking on my main machine with Linux Mint 17.3 instead of upgrading to Linux Mint 18. The fact that I broke things on another machine using an older method of upgrading provided even more encouragement.

However, I subsequently discovered another means of upgrading between major versions of Linux Mint that had some endorsement from the project. There still are warnings about testing a live DVD version of Linux Mint on your PC first and backing up your data beforehand. Another task is ensuring that you are upgraded from a fully up-to-date Linux Mint 17.3 installation.

When you are ready, you can install mintupgrade using the following command:

sudo apt-get install mintupgrade

When that is installed, there is a sequence of tasks that you need to do. The first of these is to simulate an upgrade to test for the appearance of untoward messages and resolve them. Repeating any checking, until all is well, gets a recommendation. The command is as follows:

mintupgrade check

Once you are happy that the system is ready, the next step is to download the updated packages so they are on your machine ahead of their installation. Only then should you begin the upgrade process. The two commands that you need to execute are below:

mintupgrade download
mintupgrade upgrade

After these complete, restart your system. In my case, the process worked well, with only my PHP installation requiring attention. I resolved a clash between different versions of the scripting interpreter by removing the older one, as PHP 7 is best kept for testing. Apart from reinstalling VMware Player and upgrading from version 18 to 18.1, I had almost nothing else to do and experienced minimal disruption. This is fortunate as I rely heavily on my main PC. The alternative of a full installation would have left me sorting things out for several days afterwards because I use a customised selection of software.

Updating Piwik using the Linux Command Line

Because updating Piwik using its web interface has proved tempestuous, I have decided to update the self-hosted analytics application on an SSH session. The production web servers that I use are hosted on Linux systems, so that is why any commands apply to the Linux or UNIX command line only. What is needed for Windows servers may differ.

The first step is to down the required ZIP file with this command:

wget https://builds.piwik.org/piwik.zip

Once the download is complete, the contents of the ZIP archive are extracted into a new subfolder. This is a process that I carry out in a separate folder to that where the website files are kept before copying everything from the extraction folder in there. Here is the unzip command, and the -o switch turns on overwriting of any previously existing files:

unzip -o piwik.zip

Without the required folder in the web server area to be updated, the next step is to do the actual system update that includes any updates to the Piwik database that you are using. There are two commands that you can use once you have specified the location of your Piwik installation. The second is needed when the first option cannot find where the PHP executable is stored. My systems had something more specific than these because both PHP 5.6 and PHP 7.0 are installed. Looking in /usr/bin was enough to find what I needed to execute in place of PHP below. Otherwise, the command was the same.

./[path to piwik]/console core:update

php [path to piwik]/console core:update

While the upgrade is ongoing, it prompts you to permit it to continue before it goes and modifies the database. This did not take long on my systems, but that depends on how much data there is. Once, the process has completed, you can delete any extraneous files using the rm command.

Forcing an upgrade to Windows 10 Anniversary Update

There remain people who advise those on Windows 7 or 8.x to hold fire on upgrading to Windows 10. Now that the free upgrade no longer is available, that advice may hold more weight than it did. Even so, there are those among us who jumped ship who are open to having the latest versions of things at no monetary cost to see what is available, and I must admit to being one of those.

After all, I do have a virtual machine with a pre-release version of the next update to Windows 10 installed on there to see what might be coming our way and to get a sense of what changes that may bring so that I am ready for those. Otherwise, I am usually happy to wait, but I noticed that the Windows 10 Anniversary Update only came to my HP Pavilion dm4 laptop and not other machines with Windows 10 installed, so I started to wonder why there was a lag when it came to automatic upgrades.

So that these things do not arrive when it is least convenient, I took advantage of a manual method to choose my timing. This did not involve installation from a disk image, but was in-situ. The first part of the process is standard enough in that the Settings app was started and the Update & security item chosen. That dropped me onto the Windows Update, and I first clicked on the Check for updates button to see what would happen. When nothing came of that, the Learn more link was clicked to bring me onto part of the Microsoft support website where I found that the Windows 10 Anniversary Update installer could be downloaded, so I duly did just that.

Running it produced a screen asking whether I wanted to proceed. Since I wanted to go ahead, the appropriate button was clicked and the machine left alone until the process completed. Because the installer purely is a facilitator, the first stage is to download the rest of the files needed, and that will take a while on any connection. Once downloading was completed, the actual process of installation commenced with several restarts before a log-in screen was again on offer. On logging in to the machine, the last part of the process started.

Though the process took quite a while, it seemingly worked without a hitch. If there was anything that I needed to do, it was the re-installation of VirtualBox Guest Additions to restore access to shared folders, as well as dealing with a self-inflicted irritation. Otherwise, I have found that previously installed software worked as expected and no file has been missed. Waiting a while may have had its advantages too because initial issues with the Anniversary Update will have been addressed, though it is best not to leave it too long, or you could have the feeling of being forgotten. A happy balance needs striking.

Compressing a VirtualBox VDI file for a Windows guest running on a Linux Host

Recently, I had a situation where my the VDI files for my Windows 10 virtual machine expanded in size all of a sudden and I needed to reduce them. My downloading maps for use with RouteBuddy may have been the cause, so I moved the ISO installation files onto the underlying Linux Mint drives. With that space, I then set to uncovering how to compact the virtual disk file, and the Sysinternals sdelete tool was recommended for clearing unused space. After downloading, I set it to work in a PowerShell session running on the guest operating system from its directory using the following command:

.\sdelete -z [drive letter designation; E: is an example]

From the command prompt, the following should do:

sdelete -z [drive letter designation; E: is an example]

Once, that had completed, I shut down the VM and executed a command like the following from a bash terminal session:

VBoxManage modifyhd [file location/file name].vdi --compact

Where there was space to release, VDI files were reduced in size to return more disk space. More could be done, so I will look into the Windows 10 drives to see what else needs to be moved out of them.

Turning off Apport crash reporting on Ubuntu

Last week, I kept getting a multitude of messages from Ubuntu's crash reporting tool, Apport. So many would appear at once on reaching the desktop session during system start-up that I actually downloaded an installation ISO disk image intending to perform a fresh installation to rid myself of the problem. In the end, it never came to that because another remedy produced the result that I needed.

Emptying /etc/crash was a start, but it did not do what I needed, and I disabled Apport altogether. This meant editing its configuration file, which is named apport and is found in /etc/default/. The following command should open it up in Gedit on supplying your password:

gksudo gedit /etc/default/apport

With the file opened, look for the line with enabled=1 and change this to enabled=0. Once that is done, restart Apport as follows:

sudo restart apport

This will need your account password before working, with any messages appearing afterwards. While I would not have done this for a real system problem, my Ubuntu GNOME installation was working smoothly, so it was the remedy I needed. The tool lets Ubuntu developers get information about application crashes, but it sends me to the Ubuntu Launchpad bug reporting website, which requires login details. This is enough to stop me continuing, making me wonder if developers could get what they need without this extra manual step. This would provide them with additional information and give us a more stable operating system in return.

Installing VMware Player 4.04 on Linux Mint 13

Curiosity about the Release Preview of Windows 8 saw me running into bother when trying to see what it's like in a VirtualBox VM. While doing some investigations on the web, I saw VMware Player being suggested as an alternative. Before discovering VirtualBox, I did have a licence for VMware Workstation and was interested in seeing what Player would have to offer. The, it was limited to running virtual machines that were created using Workstation. Now, it can create and manage them itself and without any need to pay for the tool either. Registration on VMware's website is a must for downloading it, though, but that's no monetary cost.

Once I had downloaded Player from the website, I needed to install it on my machine. There are Linux and Windows versions; it was the former that I needed, and there are 32-bit and 64-bit variants, so you need to know what your system is running. With the file downloaded, you need to set it as executable and the following command should do the trick once you are in the right directory:

chmod +x VMware-Player-4.0.4-744019.i386.bundle

Then, it needs execution as a superuser. With sudo access for my user account, it was a matter of issuing the following command and working through the installation screens to instate the Player software on the system:

sudo ./VMware-Player-4.0.4-744019.i386.bundle

Those screens proved easy for me to follow, so life would have been good if that were all that was needed to get Player working on my PC. Having Linux Mint 13 means that the kernel is of the 3,2 stock and that means using a patch to finish off the Player installation because the required VMware kernel modules seem to silently fail to compile during the installation process. This only manifests itself when you attempt to start VMware Player afterwards to find a module installation screen appear. That wouldn't be an issue of itself were it not for the compilation failure of the vmnet module and subsequent inability to start VMware services on the machine. There is a prompt to peer into the log file for the operation, and that is a little uninformative for the non-specialist.

Rummaging around the web brought me to the requisite patch, and it works for Player 4.0.3 and Workstation 8.0.2 by default. Doing some tweaking allowed me to make it work for Player 4.04 too. My first step was to extract the contents of the tarball to /tmp where I could edit patch-modules_3.2.0.sh. Line 8 was changed to the following:

plreqver=4.0.4

With the amendment saved, it was time to execute the shell script as a superuser, having made it executable beforehand. This can be accomplished using the following command:

chmod +x patch-modules_3.2.0.sh && sudo ./patch-modules_3.2.0.sh

With that completed successfully, VMware Player ran as it should. An installation of Windows 8 into a new VM ran very smoothly, and I was impressed with the performance and responsiveness of the operating system within a Player VM. There are a few caveats, though. First, it doesn't run at all well with VMware Tools, so it's best to leave them uninstalled since it doesn't seem to need them either; it was possible to set the resolution to the same as my screen and use the CTRL+ALT+ENTER shortcut to drop in and out of full screen mode anyway. Second, the unattended Windows installation wasn't the way forward for setting up the VM, but it was no big deal to have that experiment thwarted. The feature remains an interesting one, though.

With Windows 8 running so well in Player, I was reminded of the sluggish nature of my Windows 7 VM and an issue with a Fedora 17 one too. The result was that I migrated the Windows 7 VM from VirtualBox to VMware, and all is so much more responsive. Getting it there took not a little tinkering, so that's a story for another entry. Based on my experiences so far, I reckon that VMware Player will remain useful to me for a little while yet. Resolving the installation difficulty was worth that extra effort.