Technology Tales

Dealing with this Python error message on Windows: UnicodeEncodeError: 'charmap' codec can't encode characters in position 56-57: character maps to <undefined>

Recently, I got caught out by the above message when summarising some text using Python and Open AI's API while working within VS Code. There was no problem on Linux or macOS, but it was triggered on the Windows command line from within VS Code. Unlike the Julia or R REPL's, everything in Python gets executed in the console like this:

& "C:/Program Files/Python313/python.exe" script.py

The Windows command line shell operated with cp1252 character encoding, and that was tripping up the code like the following:

with open("out.txt", "w") as file:
    file.write(new_text)

The cure was to specify the encoding of the output text as utf-8:

with open("out.txt", "w", encoding='utf-8') as file:
    file.write(new_text)

After that, all was well and text was written to a file like in the other operating systems. One other thing to note is that the use of backslashes in file paths is another gotcha. Adding an r before the quotes gets around this to escape the contents, like using double backslashes. Using forward slashes is another option.

with open(r"c:\temp\out.txt", "w", encoding='utf-8') as file:
    file.write(new_text)

Finding human balance in an age of AI code generation

Recently, I was asked about how I felt about AI. Given that the other person was not an enthusiast, I picked on something that happened to me, not so long ago. It involved both Perplexity and Google Gemini when I was trying to debug something: both produced too much code. The experience almost inspired a LinkedIn post, only for some of the thinking to go online here for now. A spot of brainstorming using an LLM sounds like a useful exercise.

Going back to the original question, it happened during a meeting about potential freelance work. Thus, I tapped into experiences with code generators over several decades. The first one involved a metadata-driven tool that I developed; users reported that there was too much imperfect code to debug with the added complexity that dealing with clinical study data brings. That challenge resurfaced with another bespoke tool that someone else developed, and I opted to make things simpler: produce some boilerplate code and let users take things from there. Later, someone else again decided to have another go, seemingly with more success.

It is even more challenging when you are insufficiently familiar with the code that is being produced. That happened to me with shell scripting code from Google Gemini that was peppered with some Awk code. There was no alternative but to learn a bit more about the language from Tutorials Point and seek out an online book elsewhere. That did get me up to speed, and I will return to these when I am in need again.

Then, there was the time when I was trying to get a Julia script to deal with Google Drive needing permissions to be set. This started Google Gemini into adding more and more error checking code with try catch blocks. Since I did not have the issue at that point, I opted to halt and wait for its recurrence. When it did, I opted for a simpler approach, especially with the gdrive CLI tool starting up a web server for completing the process of reactivation. While there are times when shell scripting is better than Julia for these things, I added extra robustness and user-friendliness anyway.

During that second task, I was using VS Code with the GitHub Copilot plugin. There is a need to be careful, yet that can save time when it adds suggestions for you to include or reject. The latter may apply when it adds conditional logic that needs more checking, while simple code outputting useful text to the console can be approved. While that certainly is how I approach things for now, it brings up an increasingly relevant question for me.

How do we deal with all this code production? In an environment with myriads of unit tests and a great deal of automation, there may be more capacity for handling the output than mere human inspection and review, which can overwhelm the limitations of a human context window. A quick search revealed that there are automated tools for just this purpose, possibly with their own learning curves; otherwise, manual working could be a better option in some cases.

After all, we need to do our own thinking too. That was brought home to me during the Julia script editing. To come up with a solution, I had to step away from LLM output and think creatively to come up with something simpler. There was a tension between the two needs during the exercise, which highlighted how important it is to learn not to be distracted by all the new technology. Being an introvert in the first place, I need that solo space, only to have to step away from technology to get that when it was a refuge in the first place.

For anyone with a programming hobby, they have to limit all this input to avoid being overwhelmed; learning a programming language could involve stripping out AI extensions from a code editor, for instance, LLM output has its place, yet it has to be at a human scale too. That perhaps is the genius of a chat interface, and we now have Agentic AI too. It is as if the technology curve never slackens, at least not until the current boom ends, possibly when things break because they go too far beyond us. All this acceleration is fine until we need to catch up with what is happening.

Incorporating tmux in a terminal workflow

As part of a recent workstation upgrade and subsequent AI explorations to see what runs on a GPU, I got to use tmux to display two panes within a terminal session on Linux Mint, each with output from a different system monitoring command; one of these was top for monitoring system processes in a more in-depth way. Some of that need has passed, yet I retain tmux and even set to open in a new terminal session by adding the following code to my .bashrc file:

if command -v tmux &> /dev/null && [ -z "$TMUX" ]; then
    tmux new
fi

This tests if tmux is installed and that this is not running in an existing tmux session before opening a new tmux session. You can also attach to an existing session or use a new default session if you like. That changes the second line of the above code to this:

tmux attach -t default || tmux new -s default

Wanting to have everything fresh in a new session, I decided against that. While I have gone away from using tmux panes for the moment, there is a cheat sheet that could have uses if I do, and another post elsewhere describes resizing the panes too, which came in very useful for that early dalliance while system monitoring.

From convex to concave: reflections on decades of computer monitor usage

Within the last week, I changed my monitor and am without an Iiyama in my possession for the first time since 1997. The first one was a 17" CRT screen that accompanied my transition from education into work. Those old screens were not long-lasting, though, especially since it replaced a 15" Dell screen that had started to work less well than I needed; the larger size was an added attraction after I saw someone with a 21" Iiyama at the university where I was pursuing a research degree.

Work saw me using a 21" Philips screen myself for a time before Eizo flat screen displays were given to us as part of a migration to Windows 2000. That inspired me to get a 17" Iiyama counterpart to what I had at work. Collecting that sent me on an errand to a courier's depot on the outskirts of Macclesfield. The same effort may have been accompanied by my dropping my passport, which I was using for identification. That thankfully was handed into the police, so I could get it back from them, even if I was resigned to needing a new one. More care has been taken since then to avoid a repeat.

The screen worked well, though I kept the old one as a backup for perhaps far too long. It took some years to pass before I eventually hauled it to the recycling centre; these days, I might try a nearby charity shop before setting off on such a schlep. In those times, LCD screens lasted so well that they could accumulate if you were not careful. The 17" Iiyama accompanied my migration from Windows to Linux and a period of successful and ill-fated PC upgrades, especially a run of poor luck in 2009.

2010 saw me change my place of work, and a 24" Iiyama was acquired just before then. Again, its predecessor was retained in case anything went awry and eventually went to a charity shop from where I could go into a new life. There was no issue with the new acquisition, and it went on to do nearly twelve years of work for me. A 34" Iiyama replaced it a few years ago, yet I wonder if that decision was the best. Apart from more than a decade of muck on the screen, nothing else was amiss. Even a major workstation upgrade in 2021 did little to challenge it. Even so, it too went to a charity shop searching for a new home.

This year's workstation overhaul did few favours to that 34" successor. While it was always sluggish to wake, it did nothing like going into a cycle of non-responsiveness that it had on numerous occasions in the last few months. Compatibility with a Mac Mini could be better, too. The result is that I am writing these words using a Philips B346C1 instead, and it has few of the issues that beset the Iiyama, save for needing to remove and insert an HDMI cable for a Mac Mini at times.

Screen responsiveness is a big improvement, especially when switching between machines using a KVMP switch. Wake up times are noticeably shorter, and there is much better reliability. However, it did take a deal of time to optimise its settings to my liking. The OSD may be more convenient than the Iiyama, yet having Windows software that did the same thing made configuration at lot easier. While getting acceptable output across Windows, Linux and macOS has been a challenge, there is a feeling that things are nearly there.

Another matter is the fact that this is a curved screen. In some ways, that is akin to the move from a 24" screen to a 34" one when fonts and other items needing enlarging for the bigger screen. After a burst of upheaval, eventually things do settle down and acclimatisation ensues. Even though further tinkering cannot be ruled out, there is a sounder base for computing after the changeover.

A retrograde way to keep an old scanner going?

For making a copy of a document for official purposes, I needed to get my scanner going with the new workstation. The device is an Epson Perfection 4490 Photo that I acquired in 2007 after its Canon predecessor, a CanoScan 5000F, began to malfunction. It has served me well since then, though digital photography has meant that scanning images is not something that I do very often these days, the last time being in early 2022 to get larger images into my online photo gallery.

The age means that software support is an issue, more particularly for Windows 11. However, Linux can leave old devices behind it too. It does not help that there is little incentive for Epson to update its drivers either. Thus, Linux Mint's move from LIBSANE to LIBSANE1 makes things less straightforward when the Epson software needs the former.

While you can take apart a DEB file, re-edit its components before creating a new version, that sounds tricky to me. Nevertheless, it may be the way to go for others. Instead, I downloaded a DEB file for LIBSANE from Ubuntu and installed that instead. With that installed, the Epson software installed fully, allowing VueScan to work as I needed. Thus, the document got copied as I needed, and the rest then could happen as required.

When I went looking up solutions to my conundrum on Perplexity, it kept telling me that it was not the best way to go. However, I still took the chance, knowing that I could roll things back if needed. Computers never know you that well without a multitude of data, so the safety first approach has its merits, even if it can be overly cautious in some cases.

If I ever do need to replace the scanner, I probably would replace the printer with a multi-function device at the same time. The move would save some desk space, and I have had a good experience with such a device elsewhere. For now, though, such a move is on the long finger; securing a new freelance contract is higher up any to-do list.

Upheaval and miniaturisation

The ongoing AI boom got me refreshing my computer assets. One was a hefty upgrade to my main workstation, still powered by Linux. Along the way, I learned a few lessons:

• Processing with LLM's only works on a graphics card when everything can remain within its onboard memory. It is all too easy to revert to system memory and CPU usage, given the amount of memory you get on consumer graphics cards. That applies even with the latest and greatest from Nvidia, when the main use case is for gaming. Things become prohibitively expensive when you go on from there.
• Even with water cooling, keeping a top of the range CPU cool and its fans running quietly remains a challenge, more so than when I last went for a major upgrade. It takes time for things to settle down.
• My Iiyama monitor now feels flaky with input from the latest technology. This is enough to make me look for a replacement, and it is waking up from dormancy that is the real issue. While it was always slow, plugging out from mains electricity and then back in again is a hack that is needed all too often.
• KVM switches may need upgrading to work with the latest graphical input. The monitor may have been a culprit with the problems that I was getting, yet things were smoother once I replaced the unit that I had been using with another that is more modern.
• AMD Ryzen 9 chips now have onboard graphics, a boon when things are not proceeding too well with a dedicated graphics card. Even though this was not the case when the last major upgrade happened, there were no issues like what I faced this time around.
• Having LED's on a motherboard to tell what might be stopping system startup is invaluable. This helped in July 2021 and averted confusion this time around as well. While only four of them were on offer, knowing which of CPU, DRAM, GPU or system boot needs attention is a big help.
• Optical drives are not needed any longer. Booting off a USB drive was enough to get Linux Mint installed, once I got the image loaded on there properly. Rufus got used, and I needed to select the low-level writing option before things proceeded as I had hoped.

Just like 2021, the 2025 upgrade cycle needed a few weeks for everything to settle down. The previous cycle was more challenging, and this was not just because of an accompanying heatwave. The latest one was not so bedevilled.

Given the above, one might be tempted to go for a less arduous path, like my acquisition of an iMac last year for another place that I own. After all, a Mac Mini packs in quite a lot of power, and it is not the only miniature option. Now that I have one, I have moved image processing off the workstation and onto it. The images are stored on the Linux machine and edited on the Mac, which has plenty of memory and storage of its own. There is also an M4 chip, so processing power is not lacking either.

It could have been used for work affairs, yet I acquired a Geekom A8 for just that. Though seeking work as I write this, my being an incorporated freelancer means that having a dedicated machine that uses my main monitor has its advantages. Virtualisation can allow drift from business affairs to business matters, that is not so easy when a separate machine is involved. There is no shortage of power either with an AMD Ryzen 9 8945HS and Radeon 780M Graphics on board. Add in 32 GB of memory and 2 TB of storage and all is commodious. It can be surprising what a small package can do.

The Iiyama's travails also pop up with these smaller machines, less so on the Geekom than with the Mac. The latter needs the HDMI cable to be removed and reinserted after a delay to sort out things. Maybe that new monitor may not be such an off the wall idea after all.

Displaying superscripted text in Hugo website content

In a previous post, there was a discussion about displaying ordinal publishing dates with superscripted suffixes in Hugo and WordPress. Here, I go further with inserting superscripted text into Markdown content. Because of the default set up for the Goldmark Markdown renderer, it is not as simple as adding ^... constructs to your Markdown source file. That will generate a warning like this:

WARN Raw HTML omitted while rendering "[initial location of Markdown file]"; see https://gohugo.io/getting-started/configuration-markup/#rendererunsafe
You can suppress this warning by adding the following to your site configuration:
ignoreLogs = ['warning-goldmark-raw-html']

Because JavaScript can be added using HTML tags, there is an added security hazard that could be overlooked if you switch off the warning as suggested. Also, Goldmark does not interpret Markdown specifications of superscripting without an extension whose incorporation needs some familiarity with Go development.

That leaves using a Shortcode. These go into layouts/shortcodes under your theme area; the file containing mine got called super.html. The content is the following one-liner:

<sup>{{ .Get 0 | markdownify }}⁢/sup>

This then is what is added to the Markdown content:

{{< super "th" >}}

What happens here is that the Shortcode picks up the content within the content within the quotes and encapsulates it with the HTML superscript tags to give the required result. This approach can be extended for subscripts and other similar ways of rendering text, too. All that is required is a use case, and the rest can be put in place.

Adding superscripts to ordinal publishing dates for entries in Hugo and WordPress

These web publishing tools differ and so the solutions, yet the use case is the same: displaying ordinal dates for entries in a blog or website. Also, the wish is to have the ordinal suffix superscripted as in normal English usage. Let us take each platform in turn.

Hugo

Given that is programming in Go, it is little surprise that Hugo uses Go's time formatting syntax. Thus, my starting point was as follows:

{{ .Date.Format "15:04, January 2, 2006" }}

The result from the above looks like this: 20:56, January 2, 2025. Unfortunately, Go does not support ordinal dates in its time formatting, so adding them needs more extensive conditional logic like what you see below. The default suffix is th, while nd is added for the second and twenty-second days of the month and rd is added for the third and twenty-third days. This how things now look:

{{ .Date.Format "15:04, January 2" }}{{ if eq (.Date.Format "2") "2" }}nd{{ else if eq (.Date.Format "2") "22" }}nd{{ else if eq (.Date.Format "2") "1" }}st{{ else if eq (.Date.Format "2") "21" }}st{{ else if eq (.Date.Format "2") "3" }}rd{{ else if eq (.Date.Format "2") "23" }}rd{{ else }}th{{ end }}, {{ .Date.Format "2006" }}

That gives you something like 20:56, January 2nd, 2025. The handy thing about Hugo is that it primarily is an HTML output engine, so adding superscripting tags in the right places, like below, superscripts the ordinal suffixes as needed. Here, the tags are shown in bold for emphasis.

{{ .Date.Format "15:04, January 2" }}<sup>{{ if eq (.Date.Format "2") "2" }}nd{{ else if eq (.Date.Format "2") "22" }}nd{{ else if eq (.Date.Format "2") "1" }}st{{ else if eq (.Date.Format "2") "21" }}st{{ else if eq (.Date.Format "2") "3" }}rd{{ else if eq (.Date.Format "2") "23" }}rd{{ else }}th{{ end }}</sup>, {{ .Date.Format "2006" }}

Once you have that: this is how things appear: 20:56, January 2^nd, 2025, which gets the job done.

WordPress

Hugo produces the website for you to upload it to the web server; it is static after that. In contrast, WordPress is based on PHP and dynamically renders a web page when it is requested. That means that components are generated at that time. Thus, the following code snippet outputs a date when a website post or page has been published:

<?php the_time('jS F Y') ?>

PHP time and date formatting does account for ordinal dates, unlike what you get in Go. Here, the jS F Y format controls how the date gets displayed. The codes do the following: j outputs the day in the month without a leading zero, S adds the ordinal suffix, F adds the full month name and Y adds the four digit year. The result then is something like this: 1st January 2025. To superscript the ordinal suffix, the following change is needed (the addition is emboldened for emphasis):

<?php the_time('j\<\s\u\p\>S\<\/\s\u\p\> F Y') ?>

Here, the HTML superscript tags are inserted into the format with every character escaped by a leading backslash (\). While PHP does generate HTML, it needs the escaping to preserve the intended input here. Security considerations like preventing cross-site scripting also matter, though maybe not so much in this context. Regardless of the technicality, the result becomes 1^st January 2025, which is what was sought.

Changing tab titles in the macOS Terminal app using the command line

One thing that I have noticed with the macOS terminal app that I have not seen with its Linux counterparts is that the tab titles can get stuck after an SSH session to a remote server. Thus, I decided to see if they could be changed or reset. Handily, a single command will do just that:

echo -ne "\033]1;New Tab Title\007"

In a UNIX shell (BASH, ZSH, etc.), the echo command outputs text, and it is the text that changes a tab title. Here, the -ne options both negate the generation of a newline (which would be the function of the -n switch if used on its own) and interprets the escape characters included in the text (which would be the function of the -e switch if used on its own).

Within the text string \033 is the octal representation of the escape character that initiates the control sequence that follows it. This is ]1;, the Operating System Sequence (OSC) for setting the tab title in this case, more generally the icon and window title in other circumstances. The text New Tab Title should be self-explanatory, while \007 is the octal representation of the bell character (BEL) that terminates the OSC.

Because I wanted to have the current working directory path as the title, I made a small modification to do this dynamically:

echo -ne "\033]1;$(pwd)\007"

It is the $(pwd) portion that does just that, taking the output of the pwd command and adding it into the string. Thus, I see what is open in each tab. That stopped me ending up in the wrong one, and I even added an alias into the .zshrc file to make it easier to invoke. The functionality may be a more general UNIX or Linux feature, though I have not had opportunity or reason to try it just yet.

Getting Pylance to recognise locally installed packages in VSCode running on Linux Mint

When using VSCode on Linux Mint, I noticed that it was not finding any Python package installed into my user area, as is my usual way of working. Thus, it was being highlighted as being missing when it was already there.

The solution was to navigate to File > Preferences > Settings and click on the Open Settings (JSON) icon in the top right-hand corner of the app to add something like the following snippet in there.

"python.analysis.extraPaths": [
"/home/[user]/.local/bin"
]

Once you have added your user account to the above, saving the file is the next step. Then, VSCode needs a restart to pick up the new location. After that, the false positives get eliminated.