Expanding the coding toolkit: Adding R and Python in a changing landscape
10th April 2021
Over the years, I have taught myself a number of computing languages, with some coming in useful for professional work while others came in handy for website development and maintenance. The collection has grown to include HTML, CSS, XML, Perl, PHP and UNIX Shell Scripting. The ongoing pandemic allowed to me add two more to the repertoire: R and Python.
My interest in these arose from my work as an information professional concerned with standardisation and automation of statistical results delivery. To date, the main focus has been on clinical study data, but ongoing changes in the life sciences sector could mean that I may need to look further afield, so having extra knowledge never hurts. Though I have been a SAS programmer for more than twenty years, its predominance in the clinical research field is not what it was, which causes me to rethink things.
As it happens, I would like to continue working with SAS since it does so much and thoughts of leaving it after me bring sadness. It also helps to know what the alternatives might be and to reject some management hopes about any newcomers, especially regarding the amount of code being produced and the quality of graphs being created. Use cases need to be assessed dispassionately, even when emotions loom behind the scenes.
Since both R and Python bring large scripting ecosystems with active communities, the attraction of their adoption makes a deal of sense. SAS is comparable in the scale of its own ecosystem, though there are considerable differences and the platform is catching up when it comes to Data Science. While the aforementioned open-source languages may have had a head start, it appears that others are not standing still either. It is a time to have wider awareness, and online conference attendance helps with that.
The breadth of what is available for any programming language more than stymies any attempt to create a truly all encompassing starting point, and I have abandoned thoughts of doing anything like that for R. Similarly, I will not even try such a thing for Python. Consequently, this means that my sharing of anything learned will be in the form of discrete postings from time to time, especially given ho easy it is to collect numerous website links for sharing.
The learning has been facilitated by ongoing pandemic restrictions, though things are opening up a little now. The pandemic also has given us public data that can be used for practice, since much can be gained from having one's own project instead of completing exercises from a book. Having an interesting data set with which to work is a must, and COVID-19 data contain a certain self-interest as well, while one remains mindful of the suffering and loss of life that has been happening since the pandemic first took hold.
Using .htaccess to control hotlinking
10th October 2020
There are times when blogs cease to exist and the only place to find the content is on the Wayback Machine. Even then, it is in danger of being lost completely. One such example is the subject of this post.
Though this website makes use of the facilities of Cloudflare for various functions that include the blocking of image hot linking, the same outcome can be achieved using .htaccess files on Apache web servers. It may work on Nginx to a point too, but there are other configuration files that ought to be updated instead of using .htaccess when some frown upon the approach. In any case, the lines that need adding to .htaccess are listed below, while the web address needs to include your own domain in place of the dummy example provided:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule .*\.(gif|jpe?g|png|bmp)$ [F,NC]
The first line activates the mod_rewrite engine, which you might have already done. For this to work, the module must be enabled in your Apache configuration, and you need permission to make these changes. This requires modifying the Apache configuration files. The next two lines examine the HTTP referrer strings. The third line permits images to be served only from your own web domain, not from others. To include additional domains, copy the third line and change the web address as needed. Any new lines should be placed before the final line that specifies which file extensions are blocked for other web addresses.
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourdomain.com(/)?.*$ [NC]
RewriteRule \.(gif|jpe?g|png|bmp)$ /images/image.gif [L,NC]
Another variant of the previous code involves changing the last line to display a default image showing others what is happening. That may not reduce the bandwidth usage as much as complete blocking, but it may be useful for telling others what is happening.
Getting rid of the Windows Resizing message from a Manjaro VirtualBox guest
27th July 2020
Like Fedora, Manjaro also installs a package for VirtualBox Guest Additions when you install the Linux distro in a VirtualBox virtual machine. However, it does have certain expectations when doing this. On many systems, and my own is one of these, Linux guests are forced to use the VMSVGA virtual graphics controller while Windows guests are allowed to use the VBoxSVGA one. It is the latter that Manjaro expects, so you get a message like the following appearing when the desktop environment has loaded:
Windows Resizing
Set your VirtualBox Graphics Controller to enable windows resizing
After ensuring that gcc, make, perl and kernel headers are installed, I usually install VirtualBox Guest Additions myself from the included ISO image, and so I did the same with Manjaro. Doing that and restarting the virtual machine got me extra functionality like screen resizing and being able to copy and paste between the VM and elsewhere after choosing the Bidirectional setting in the menus under Devices > Shared Clipboard.
That still left an unwanted message popping up on startup. To get rid of that, I just needed to remove /etc/xdg/autostart/mhwd-vmsvga-alert.desktop. While it can be deleted, I just moved it somewhere else and a restart proved that the message was gone as needed. Now everything is working as I wanted.
Resolving VirtualBox Shared Folders problems in Fedora 32 through a correct Guest Additions installation
26th July 2020
While some Linux distros like Fedora install VirtualBox drivers during installation time, I prefer to install the VirtualBox Guest Additions themselves. Before doing this, it is best to remove the virtualbox-guest-additions package from Fedora to avoid conflicts. After that, execute the following command to ensure that all prerequisites for the VirtualBox Guest Additions are in place before mounting the VirtualBox Guest Additions ISO image and installing from there:
sudo dnf -y install gcc automake make kernel-headers dkms bzip2 libxcrypt-compat kernel-devel perl
During the installation, you may encounter a message like the following:
ValueError: File context for /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf already defined
This is generated by SELinux, so the following commands need to be executed before repeating the installation of VirtualBox Guest Additions:
sudo semanage fcontext -d /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf
sudo restorecon /opt/VBoxGuestAdditions-<VERSION>/other/mount.vboxsf
Without doing the above step and fixing the preceding error message, I had an issue with mounting of Shared Folders whereby the mount point was set up, but no folder contents were displayed. This happened even when my user account was added to the vboxsf group, and it proved to be the SELinux context issue that was the cause.
Restoring the menu bar on GNOME Terminal in a GNOME Shell session
25th July 2020
By default, a GNOME Terminal instance does not display a menu bar and that applies not only in GNOME Shell but also on the Cinnamon Desktop environment. In the latter, it is easy enough to display the menu bar using the context menu produced by right-clicking in the window before going to Edit > Preferences and ticking the box for Show menubar by default in new terminals in the General section. After closing the Preferences dialogue, every new GNOME Terminal session will show the menu bar.
Unfortunately, it is not so easy in GNOME Shell, though the context menu route does allow you to unhide the menu bar on a temporary basis. That is because the requisite tick box is missing from the Preferences dialogue box displayed after navigating to Edit > Preferences in the menus. To address, you need to execute the following command in a terminal session:
gsettings set org.gnome.Terminal.Legacy.Settings headerbar false
This change permanently adds the menu bar and includes the previously missing tick box, which is selected when necessary. Although GNOME Shell has a minimalist design in some aspects, making this function difficult to access seems excessive.
Removing obsolete libraries from Flatpak
1st February 2020
Along with various pieces of software, Flatpak also installs KDE and GNOME libraries needed to support them. However, it does not always remove obsolete versions of those libraries whenever software gets updated. One result is that messages regarding obsolete versions of GNOME may be issued and this has been known to cause confusion because there is the GNOME instance that is part of a Linux distro like Ubuntu and using Flatpak adds another one for its software packages to use. My use of Linux Mint may lessen the chances of misunderstanding.
Thankfully, executing a single command will remove any obsolete Flatpak libraries so the messages no longer appear and there then is no need to touch your actual Linux installation. This then is the command that sorted it for me:
flatpak uninstall --unused && sudo flatpak repair
The first part that removes any unused libraries is run as a normal user, so there is no error in the above command. Administrative privileges are needed for the second section that does any repairs that are needed. It might be better if Flatpak did all this for you using the update command, but that is not how the thing works. At least, there is a quick way to address this state of affairs and there might be some good reasons for having things work as they do.
One way to fix slow CyberGhost VPN connections on Windows 10
31st January 2020
Due to a need to access websites with country blocking, I have decided to give CyberGhost a go, and it also will come in handy when connecting devices to other Wi-Fi connections. What I have got is the three-year subscription package and all went well on the first day of use. However, things became unusable on the second and a reboot did not sort it.
Since the problem seemed to affect a phone running Android too, I even got to suspect my router and broadband provider. Even terminating the subscription came to mind, but it did not come to that. Instead, I did a bit more research and tried changing the maximum transition unit (MTU) for the connection to 1300 as suggested in a CyberGhost help article. Because using the Control Panel meant that it was resetting to 1500 on my Windows 10 machine, I then turned to a command line-based solution.
To accomplish that, I started PowerShell in administrator mode from the context menu produced by right-clicking on the Start Menu icon on the taskbar. Then, I entered the following command to see what connections I had and what the MTU settings were:
netsh interface ipv4 show subinterfaces
From looking through the Settings and Control Panel applications, I already had worked out what network interface belonged to the CyberGhost connection. Seeing that the MTU setting was 1500, I then issued a command like the following to change that to 1300.
netsh interface ipv4 set subinterface "<name of ethernet interface>" mtu=1300 store=persistent
Here, <name of ethernet interface> gets replaced by the name of your connection and the string is quoted to avoid spaces in the name causing problems with executing the command. Once that second command had been run, the first one was issued again and the output checked to ensure that the MTU setting was as expected.
While this was done when the VPN connection was inactive, it may work also with an active connection. After making the change, I again reconnected to the VPN and all has been as expected since then, and I found a better connection for my Android phone too.
Using multi-line commenting in Perl to inactivate blocks of code during testing
26th December 2019
Recently, I needed to inactivate blocks of code in a Perl script while doing some testing. Since this is something that I often do in other computing languages, I sought the same in Perl. To accomplish that, I need to use the POD methodology. This meant enclosing the code as follows.
=start
<< Code to be inactivated by inclusion in a comment >>
=cut
While the =start line could use any word after the equality sign, it seems that =cut is required to close the multi-line comment. If this was actual programming documentation, then the comment block should include some meaningful text for use with perldoc. However, that was not a concern here because the commenting statements would be removed afterwards anyway. It also is good practice not to leave commented code in a production script or program to avoid any later confusion.
In my case, this facility allowed me to isolate the code that I had to alter and test before putting everything back as needed. It also saved time since I did not need to individually comment out every executable line because multiple lines could be inactivated at a time.
EVF or OVF?
22nd December 2019
In photography, some developments are temporary fads, while others create enduring changes. Special effects filters and high dynamic range techniques generated excitement before their usage became more restrained. The same applies to most image processing techniques, as good taste eventually prevails. Some developments, however, signal more substantial shifts.
The biggest example of the latter is the move away from film photography to digital image capture. There still are film photographers who largely depend on older cameras, since very few are made any more. While my own transition came later than others, I hardly use film any more, and a lack of replacement parts for cameras that are more than fifteen years old only helps to keep things that way. Another truth is that digital photography makes me look at my images more critically, which helps me improve.
Also, mobile phone cameras have become so capable that the compact camera market has shrunk dramatically. In fact, I gave away my Canon PowerShot G11 earlier this year because there was little justification in hanging onto it. After all, it dated back to 2010 and a phone would do now what it once did, though the G11 did more for me than I might have expected. Until 2017, my only photos of Swedish locations were made with that camera. If I ever was emotional at its departure, and I doubt that I was, that is not felt now.
If you read photography magazines, you get the sense that mirrorless cameras have captured a lot of the limelight, and that especially is the case with the introduction of full frame models. Some writers even are writing off the chances of SLR's remaining in production, though available model ranges remain extensive despite the new interlopers. Whatever about the departure of film, the possible loss of SLR's with their bright optical viewfinders (OVF's) does make me a little emotional since they were the cameras that, so many like me aspired to owning during my younger years and the type has served me well over the decades.
Even so, I too have used mirrorless cameras and an Olympus PEN E-PL5 came into my possession in 2013. However, I found that using the screen on the back of a camera was not to my liking and the quality of mobile phone cameras is such that I no longer need any added portability. However, it needs to be remembered that using a Tamron 14 to 150 mm zoom lens with the body cannot have helped either. Wishing to sample a counterpart with an electronic view finder, I replaced it with an Olympus OM-D E-M10 Mark III earlier this year and have been getting on fine with that.
The camera body is compact, yet handles like an SLR. I turned off the automatic switching between viewfinder and screen because I found it distracting. I prefer to switch between them manually. Getting used to the electronic viewfinder (EVF) took some time, but adding a spirit level overlay proved both useful and educational. The camera produces images with strong greens and blues, which suits me since I like both colours. The 16.1 megapixel sensor creates smaller files that upload quickly to back-up services. My Tamron lens works flawlessly, and keeping this lens was a key reason I stayed with Olympus despite a shutter failure on my previous camera. The repair was done efficiently and at a reasonable price.
Despite the quality of the new Olympus, it has not replaced my Canon EOS 5D Mark II and Pentax K5 II SLRs. The Olympus has a much smaller frame. In January, I bought a new Sigma 24 to 105 mm zoom lens for the Canon after my older lens developed a fault that could not be fixed. The new lens performs well and produces impressively sharp images. However, the full frame Canon setup is heavy, even when used handheld. This is why the Pentax remains my choice for travelling abroad. Both the Canon and Pentax offer brighter viewfinders, which I value. Therefore, the OM-D complements rather than replaces my other cameras.
Although I can work with electronic viewfinders (EVF's) if single-lens reflex cameras (SLRs) become obsolete, I plan to use mainly SLRs for now. Canon recently launched a new enthusiast model, indicating ongoing interest in SLRs. Canon appears to envision a hybrid approach where using the screen on the back of the camera might provide faster autofocus or other functions, while the optical viewfinder (OVF) allows traditional operation. This makes me wonder if future cameras might include viewfinders that switch between EVF and OVF modes. This idea might be both far-fetched and fascinating, but other unforeseen possibilities might exist. One thing is certain: we live in a time of rapid change.
Generating PNG files in SAS using ODS Graphics
21st December 2019
Recently, I had someone ask me how to create PNG files in SAS using ODS Graphics, so I sought out the answer for them. Normally, the suggestion would have been to create RTF or PDF files instead, but there was a specific need that needed a different approach. Adding something like the following lines before an SGPLOT, SGPANEL or SGRENDER procedure should do the needful:
ods listing gpath='E:\';
ods graphics / imagename="test" imagefmt=png;
Here, the ODS LISTING statement declares the destination for the desired graphics file, while the ODS GRAPHICS statement defines the file name and type. In the above example, the file test.png would be created in the root of the E drive of a Windows machine. However, this also works with Linux or UNIX directory paths.