TOPIC: GOOGLE
Limiting Google Drive upload & synchronisation speeds using Trickle
9th October 2021Having had a mishap that lost me some photos in the early days of my dalliance with digital photography, I have been far more careful since then and that now applies to other files as well. Doing regular backups is a must that you find reiterated by many different authors, and the current computing climate makes doing that more vital than it ever was.
So, as well as having various local backups, I also have remote ones in the form of OneDrive, Dropbox and Google Drive. While these more correctly are file synchronisation services, disciplined use can make them useful as additional storage facilities in the interests of maintaining added resilience. There also are dedicated backup services that I have seen reviewed in the likes of PC Pro magazine, but I have to make use of those.
Insync
Part of my process for dealing with new digital photo files is to back them up to Google Drive, and I did that with a Windows client in the early days but then moved to Insync running on Linux Mint. One drawback to the approach is that this hogs the upload bandwidth of an internet connection that has yet to move to fibre from copper cabling. While having fibre connections to a local cabinet helps, a 100 KiB/s upload speed is easily overwhelmed and digital photo file sizes keep increasing. It does not help that I insist on using more flexible raw formats like DNG, CR2 or CR3 either.
While making fewer images could help to cut the load, I still come away from an excursion with many files because I get so besotted with my surroundings. This means that upload sessions take numerous hours and can extend across calendar days. Ultimately, this makes my internet connection far less usable; hence I want to throttle upload speed, much like what is possible in the Transmission BitTorrent client or in the Dropbox client. Since this is not available in Insync, I have tried using the trickle command instead, and an example is below:
trickle -d 2000 -u 50 insync
Here, the upload speed is limited to 50 KiB/s while the download speed is limited to 2000 KiB/s. In my case, the latter of these hardly matters, while the former leaves me with acceptable internet usability. Insync does not work smoothly with this, though, so occasional restarts are needed to keep file uploads progressing and CPU load also is higher. As rough as the user experience feels, uploads can continue in parallel with other work.
gdrive
One other option that I am exploring is the use of the command-line tool gdrive and this appears to work well with trickle. After downloading and installing the tool, getting going is a matter of issuing the following command and following the instructions:
gdrive about
On web servers, I even have the tool backing up things to Google Drive on a scheduled basis. Because of a Google Drive limitation that I have encountered not only with gdrive but also with Insync and Google's own Windows Google Drive client, synchronisation only happens with two new folders, one local and the other remote. Handily, gdrive supports the usual bash style commands for working with remote directories, so something like the following will create a directory on Google Drive:
gdrive mkdir ttdc [ID for parent folder]
Here, the ID for the parent folder may be omitted, though it can be obtained by going to Google Drive online and getting a link location by right-clicking on a folder and choosing the appropriate context menu item. This gets you something like the following and the required identifier is found between the last slash and the first question mark in the address string (so as not to share any real links, I made the address more general below):
https://drive.google.com/drive/folders/[remote folder ID]?usp=sharing
Then, synchronisation uses a command like the following:
gdrive sync upload [local folder or file path] [remote folder ID]
There also is the option to do a one-way upload, and this is the form of the command used:
gdrive upload [local folder or file path] -p [remote folder ID]
Because every file or folder object has its own ID on Google Drive, it is possible to create two objects on there that appear to have the same name, though that is sure to cause confusion even if you know what is happening. It is possible in each of the above to throttle them using trickle as well:
trickle -d 2000 -u 50 gdrive sync upload [local folder or file path] [remote folder ID]
trickle -d 2000 -u 50 gdrive upload [local folder or file path] -p [remote folder ID]
Handily, this works without the added drama seen with Insync and lends itself to scripting as well, so it could be something that I will incorporate into my current workflow. One thing that needs to be watched is file upload failures, but there may be ways to catch those and retry them, which would be another thing that needs doing. This is built into Insync, and it would be a learning opportunity if I were to stick with gdrive instead.
Some books and other forms of documentation on R
11th September 2021The thrust of an exhortation from a computing handbook publisher comes to mind here: don't just look things up on Google, read a book so you really understand what you are doing. A form of words like that was used to sell an eBook on GitHub, but the same sentiment applies to R or any other computing language. While using a search engine will get you going or add to existing knowledge, only a book or a training course will help to embed real competence.
In the case of R, there is a myriad of blogs out there that can be consulted, as well as function and package documentation on RDocumentation or rrdr.io. For the former, R-bloggers or R Weekly can make good places to start, while ones like Stats and R, Statistics Globe, STHDA, PSI's VIS-SIG and anything from Posit (including their main blog as well as their AI one) can be worth consulting. Additionally, there is also RStudio Education and the NHS-R Community, which also have a GitHub repository together with a YouTube channel. Many packages have dedicated websites as well, so there is no lack of documentation with all of these, so here is a selection:
To come to the real subject of this post, R is unusual in that books that you can buy also have companion websites that contain the same content with the same structure. Whatever funds this approach (and some appear to be supported by RStudio itself by the looks of things), there certainly are many books available freely online in HTML as you will see from the list below, while a few do not have a print counterpart as far as I know:
R Programming for Data Science
R Markdown: The Definitive Guide
bookdown: Authoring Books and Technical Documents with R Markdown
blogdown: Creating Websites with R Markdown
pagedown: Create Paged HTML Documents for Printing from R Markdown
Dynamic Documents with R and knitr
Engineering Production-Grade Shiny Apps
Outstanding User Interfaces with Shiny
Happy Git and GitHub for the useR
Outstanding User Interfaces with Shiny
Engineering Production-Grade Shiny Apps
Many of the above have counterparts published by O'Reilly or Chapman & Hall, to name the two publishers that I have found so far. Aside from sharing these with you, there is also the personal motivation of having the collection of links somewhere so I can close tabs in my Firefox session. There are other web articles open in other tabs that I need to retain and share, but these will need to do for now, and I hope that you find them as useful as I do.
Pondering travel device consolidation using an Apple iPad Pro 12.9"
18th September 2016It was a change of job in 2010 that got me interested in using devices with internet connectivity on the go. Until then, the attraction of smartphones had not been strong, but I got myself a Blackberry on a pay as you go contract, but the entry device was painfully slow, and the connectivity was 2G. It was a very sluggish start.
It was supplemented by an Asus Eee PC that I connected to the internet using broadband dongles and a Wi-Fi hub. This cumbersome arrangement did not work well on short journeys, and the variability of mobile network reception even meant that longer journeys were not all that successful either. Usage in hotels and guest houses though went better and that has meant that the miniature laptop came with me on many a journey.
In time, I moved away from broadband dongles to using smartphones as Wi-Fi hubs and that largely is how I work with laptops and tablets away from home unless there is hotel Wi-Fi available. Even trips overseas have seen me operate in much the same manner.
One feature is that we seem to carry quite a number of different gadgets with us at a time, which can cause inconvenience when going through airport security since they want to screen each device separately. When you are carrying a laptop, a tablet, a phone and a camera, it does take time to organise yourself, and you can meet impatient staff, as I found recently when returning from Oslo. Since checking in whatever you can as hold luggage helps to get around at least some of the nuisance, it also might be time for the use of better machinery to cut down on having to screen everything separately.
When you come away after an embarrassing episode, as I once did, the attractions of consolidating devices start to become plain. In fact, most probably could get with having just their phone. It is when you take activities like photography more seriously than the gadget count increases. After all, the main reason a laptop comes on trips beyond Britain and Ireland at all is to back up photos from my camera in case an SD card fails.
Parking that thought for a while, let's go back to March this year, when temptation overcame what should have been a period of personal restraint. The result was that a 32 GB 12.9" Apple iPad Pro came into my possession along with an Apple Pencil and a Logitech CREATE Backlit Keyboard Case. It should have done so, but the size of the screen did not strike me until I got it home from the Apple Store. That was one of the main attractions because maps can be shown with a greater field of view in a variety of apps, a big selling point for a hiker with a liking for maps, who wants more than what is on offer from Apple, Google or even Bing. The precision of the Pencil is another boon that makes surfing the website so much easier, and the solid connection between the case and the iPad means that keyboard usage is less fiddly than it would if it used Bluetooth. Having tried them with the BBC iPlayer app, I can confirm that the sound from the speakers is better than any other mobile device that I have used.
Already, it has come with me on trips around England and Scotland. These weekend trips saw me leave the Asus Eee PC stay at home when it normally might have come with me, and taking just a single device along with a camera or two had its uses too. While the screen is large for reading on a train, I find that it works just as well so long as you have enough space. Otherwise, combining use of a suite of apps with recourse to the web does much of the information seeking needed while on a trip away, which meant that I was not found wanting. Battery life is good too, which helps.
Those trips allowed for a little light hotel room blog post editing too and the iPad Pro did what was needed, though the ergonomics of reaching for the screen with the Pencil meant that my arm was held aloft more than was ideal. Another thing that raised questions in my mind is the appearance of word suggestions at the bottom of the screen as if this were a mobile phone, given that I wondered if these were more of a hindrance than a help given that I just fancied typing and not pointing at the screen to complete words. Though copying and pasting works too, I have found the screen-based version a little clunky. Thus, I must see if the keyboard one works just as well, though the keyboard set up is typical of a Mac and that affects word selection. You need to use the OPTION key in the keyboard shortcut that you use for this and not COMMAND or CONTROL as you might do on a PC.
Even with these eccentricities, I was left wondering if it had any utility when it came to backing up photos from digital cameras, and there is an SD card adapter that makes this possible. A failure of foresight on my part meant that the 32 GB capacity now is an obvious limitation, but I think I might have hit on a possible solution that does not need to upload to an iCloud account. It involves clearing off the photos onto a 128 GB Transcend JetDrive Go 300 so they do not clog up the iPad Pro's storage. That the device has both Lightning and USB connectivity means that you can plug it into a laptop or desktop PC afterwards too. If that were to work as I would hope, then the laptop/tablet combination that I have been using for all overseas trips could be replaced to allow a weight reduction as well as cutting the hassle at airport security.
Trips to Ireland still may see my sticking with a tried and tested combination though because I often have needed to do some printing while over there. While I have been able to print a test document from an iPad Mini on my home network-connected printer, not every model supports this and that for NFC or Air Print is not universal either. If this were not an obstacle, apps like Pages, Numbers and Keynote could have their uses for business-related work and there are web-based offerings from Google, Microsoft and others too.
In conclusion, I have found that my iPad Pro does so much of what I need on a trip away that retiring the laptop/tablet combination for most of these is not as outrageous as it once would have seemed. In some ways, iOS has a way to go yet, before it could take over from macOS, yet it remains in development so it will be interesting to see what happens next. All the while, hybrid devices running Windows 10 are becoming more pervasive, so that might provide Apple with the encouragement that it needs.
Batch conversion of DNG files to other file types with the Linux command line
8th June 2016At the time of writing, Google Drive is unable to accept DNG files, the Adobe file type for RAW images from digital cameras. While the uploads themselves work fine, the additional processing at the end that, I believe, is needed for Google Photos appears to be failing. Because of this, I thought of other possibilities like uploading them to Dropbox or enclosing them in ZIP archives instead; of these, it is the first that I have been doing and with nothing but success so far. Another idea is to convert the files into an image format that Google Drive can handle, and TIFF came to mind because it keeps all the detail from the original image. In contrast, JPEG files lose some information because of the nature of the compression.
Handily, a one line command does the conversion for all files in a directory once you have all the required software installed:
find -type f | grep -i "DNG" | parallel mogrify -format tiff {}
The find and grep commands are standard, with the first getting you a list of all the files in the current directory and sending (piping) these to the grep command, so the list only retains the names of all DNG files. The last part uses two commands for which I found installation was needed on my Linux Mint machine. The parallel package is the first of these and distributes the heavy workload across all the cores in your processor, and this command will add it to your system:
sudo apt-get install parallel
The mogrify command is part of the ImageMagick suite along with others like convert and this is how you add that to your system:
sudo apt-get install imagemagick
In the command at the top, the parallel command works through all the files in the list provided to it and feeds them to mogrify for conversion. Without the use of parallel, the basic command is like this:
mogrify -format tiff *.DNG
In both cases, the -format switch specifies the output file type, with the tiff portion triggering the creation of TIFF files. The *.DNG portion itself captures all DNG files in a directory, but {} does this in the main command at the top of this post. If you wanted JPEG ones, you would replace tiff with jpg. Should you ever need them, a full list of what file types are supported is produced using the identify command (also part of ImageMagick) as follows:
identify -list format
Turning off Advanced Content Filtering in CKEditor
3rd February 2015On one of my websites, I use Textpattern with CKEditor for editing of articles on there. This was working well until I upgraded CKEditor to a version with a number of 4.1 or newer because it started to change the HTML in my articles when I did not want it to do so, especially when it broke the appearance of the things. A search on Google revealed an unhelpful forum exchange that produced no solution to the issue so I decided to share one on here when I found it.
What I needed to do was switch off what is known as Advanced Content Filtering. It can be tuned but I felt that would take too much time so I implemented something like what you see below in the config.js with the ckeditor folder:
CKEDITOR.editorConfig = function( config ) {
config.allowedContent = true;
};
All settings go with the outer function wrapper and setting the config.allowedContent property to true within there sorted my problem as I wanted. Now, any HTML remains untouched and I am happy with the outcome. It might be better for features like Advanced Content Filtering to be switched off by default and turned on by those with the time and need for it, much like the one of the principles adopted by the WordPress project. Still, having any off switch is better than none at all.
Upgrading a 2012 Google Nexus 7 to Android 5.0
19th November 2014Today, I was lured into upgrading my 2012 Google (ASUS) Nexus 7 to the final version of Android 5.0 (also known as Lollipop) by an icon in the device's top panel. Initially, it felt as it was working OK but a certain sluggish could not be overlooked and there have been complaints about this with some questioning the sense of what Google has done. However, there would have been remarks about grandfathering the device if they had not left us to have the latest release of Android, so there was no victory either way. We humans are fickle creatures, and there is an example of exactly that in a well observed double-ended short story by the Irish writer Maura Laverty.
While my impressions of how the upgrade had lumbered the tablet had me wondering about replacing the thing with either an Apple iPad Mini 2 or a Google (HTC) Nexus 9, a much less expensive option came to mind: doing a full factory reset of the device using its recovery mode. Though that may sound drastic, much of what I had on there was in the cloud anyway, so there was nothing to lose. So these are the instructions from Google themselves and I will leave you to use them at your own risk:
- If your tablet is on, turn it off.
- Press and hold the Volume Down button, then press and hold the Power button at the same time until the tablet turns on. You'll see the word "Start" with an arrow around it.
- Press the Volume Down button twice to highlight "Recovery mode".
- Press the Power button to start Recovery mode. You'll see an image of an Android robot with a red exclamation mark and the words "No command."
- While holding down the Power button, press the Volume Up button.
- Use the volume buttons to scroll to "wipe data/factory reset," then press the Power button to select it.
- Scroll down to "Yes - erase all user data," then press the Power button to select it.
Note: If your tablet becomes unresponsive at any point during these steps, you can restart it by holding down the Power button for several seconds.
Once that was completed and the tablet restarted, the set-up routine began and took around an hour to reinstate the various apps that had been lost by the rest. Much of that was down to the time taken for re-installation rather than that taken by the actual downloads themselves over a wired broadband connection. The wait was worth it because the Nexus 7 feels more responsive again. While there are times when little lags are noticeable, they are nothing next to the slowdown that I had witnessed before the rest. It might have been a better option than attempting to return to Android 4.4.4 using a factory image, which was another option that I was considering. So long as there is no deterioration in speed, the effort expended to do a reset will have been worthwhile.
Turning off the full height editor option in WordPress 4.0
10th September 2014Though I casually follow WordPress development, it's not nearly as rigorous as when I submitted a patch that earned me a mention on a main WordPress release's contributor list. This may explain why I barely noticed the full editor setting, which is turned on by default.
WordPress has become so mature now that I almost do not expect major revisions like the overhauls received by the administration back-end in 2008. The second interface was got so right that it still is with us, even if there were concerns in my mind at the time as to how usable it would be. Sometimes, those initial suspicions can come to nothing.
However, WordPress 4.0 introduced a major editor change that I'm not sure is successful. A full-height editor sounds good in principle, but its implementation has rough edges that make me wonder if any UX person reviewed it. Scrolling becomes strange, with the editor's toolbar fixing in place when you scroll down far enough. The sidebar then scrolls out of sync with the editor box, creating an odd sensation. Keyboard shortcuts like CTRL + HOME and CTRL + END don't work properly, which convinced me this new arrangement wasn't for me and I wanted to disable it.
A Google search found nothing useful, so I tried the WordPress.org forum. This revealed I should have looked in the screen options dropdown box for "Expand the editor to match the window height" to deselect it. Because of a Visual Editor control there, I'd checked the user profile screen but found nothing, showing the setup logic is poor. Perhaps the Visual Editor option should be a screen option too. Thankfully, the window height editor setting only needs changing once for both posts and pages, covering all situations.
With a distraction-free editing option available, I'm not sure why someone added the full height editor too. If WordPress keeps this feature, it needs refinement to behave more conventionally. I wouldn't build a website with such ill-synchronised scrolling. This needs work, as does the Visual Editor setting location. Perhaps both settings should be at the user level, rather than having one above that level. Before finding the solution, I considered using distraction-free mode permanently and installed the WP Editor plugin. I kept the plugin for its code highlighting, even though entering code view always creates a new revision. Despite this issue, things are now better.
Turning off seccomp sandbox in vsftpd
21st September 2013Within the last week, I set up a virtual web server using Arch Linux to satisfy my own curiosity, since the DIY nature of Arch means that you can build up exactly what you need without having any real constraints put upon you. Something that didn't surprise me about this was that it took me more work than the virtual server that I created using Ubuntu Server, yet I didn't expect Proftpd to be missing from the main repositories. Though the package can be found in the AUR, I didn't fancy the prospect of dragging more work on myself, so I went with vsftpd (Very Secure FTP Daemon) instead. In contrast to Proftpd, this is available in the standard repositories and there is a guide to its use in the Arch user documentation.
However, while vsftpd worked well just after installation, connections to the virtual FTP soon failed with FileZilla began issuing uninformative messages. In fact, it was the standard command line FTP client on my Ubuntu machine that was more revealing. It issued the following message that let me to the cause after my engaging the services of Google:
500 OOPS: priv_sock_get_cmd
With version 3.0 of vsftpd, a new feature was introduced, and it appears that this has caused problems for a few people. That feature is seccomp_sandbox and it can be turned off by adding the following line in /etc/vsftpd.conf:
seccomp_sandbox=NO
That solved my problem, and version 3.0.2 of vsftpd should address the issue with seccomp sandboxing anyway. In case, this solution isn't as robust as it should be because seccomp is not supported in the Linux kernel that you are using, turning off the new feature still needs to be an option, though.
Piggybacking an Android Wi-Fi device off your Windows PC's internet connection
16th March 2013One of the disadvantages of my Google/Asus Nexus 7 is that it needs a Wi-Fi connection to use. Most of the time, this is not a problem since I also have a Huawei mobile Wi-Fi hub from T-Mobile and this seems to work just about anywhere in the U.K. Away from the U.K. though, it won't work because roaming is not switched on for it and that may be no bad thing with the fees that could introduce. While my HTC Desire S could deputise, I need to watch costs with that too.
There's also the factor of download caps, and those apply both to the Huawei and to the HTC. Recently, I added Anquet's Outdoor Map Navigator (OMN) to my Nexus 7 through the Google Play Store for a fee of £7 and that allows access to any walking maps that I have bought from Anquet. However, those are large downloads, so the caps start to come into play. Frugality would help, but I began to look at other possibilities that make use of a laptop's Wi-Fi functionality.
Looking on the web, I found two options for this that work on Windows 7 (8 should be OK too): Connectify Hotspot and Virtual Router Manager. The first of these is commercial software, yet there is a Lite edition for those wanting to try it out; that it is not a time limited demo is not something that I can confirm though that did not seem to be the case since it looked as if only features were missing from it that you'd get if you paid for the Pro variant. The second option is an open source one that is free of charge apart from an invitation to donate to the project.
Though online tutorials show the usage of either of these to be straightforward, my experiences were not all that positive at the outset. In fact, there was something that I needed to do, which is how this post has come to exist at all. That happened even after the restart that Connectify Hotspot needed as part of its installation; it runs as a system service, which is why the restart was needed. In fact, it was Virtual Router Manager that told me what the issue was, and it needed no reboot. Neither did it cause network disconnection of a laptop like the Connectify offering did on me and that was the cause of its ejection from that system; limitations in favour of its paid addition aside, it may have the snazzier interface, but I'll take effective simplicity any day.
Using Virtual Router Manager turns out to be simple enough. It needs a network name (also known as an SSID), a password to restrict who accesses the network and the internet connection to be shared. In my case, the was Local Area Connection on the dropdown list. With all the required information entered, I was ready to start the router using the Start Network Router button. The text on this button changes to Stop Network Router when the hub is operational, or at least it should have done for me on the first time that I ran it. What I got instead was the following message:
The group or resource is not in the correct state to perform the requested operation.
While the above may not say all that much, it becomes more than ample information if you enter it into the likes of Google. Behind the scenes, Virtual Router Manager uses native Windows functionality to create a Wi-Fi hub from a PC, and it appears to be the Microsoft Virtual Wi-Fi Miniport Adapter from what I have seen. When I tried setting up an ad hoc Wi-Fi network from a laptop to the Nexus 7 using Windows' own network set up capability via its Control Panel, it didn't do what I needed, so there might be something that third party software can do. So, the interesting thing about the solution to my Virtual Router Manager problem was that it needed me to delve into the innards of Windows a little.
Firstly, there's running Command Prompt (All Programs > Accessories) from the Start Menu with Administrator privileges. It helps here if the account with which you log into Windows is in the Administrator group, since all you have to do then is right-click on the Start Menu entry and choose Run as administrator entry in the pop-up context menu. With a command line window now open, you then need to issue the following command:
netsh wlan set hostednetwork mode=allow ssid=[network name] key=[password] keyUsage=persistent
When that had done its thing, Virtual Router Manager worked without a hitch though it did turn itself after a while and that may be no bad thing from the security standpoint. On the Android side, it was a matter of going in Settings > Wi-Fi and choosing the new network that was created on the laptop. This sort of thing may apply to other types of tablet (Dare I mention iPads?) so you could connect anything to the hub without needing to do any more on the Windows side.
For those wanting to know what's going on behind the scenes on Windows, there's a useful tutorial on Instructables that shows what third party software is saving you from having to do. Even if I never go down the more DIY route, I probably have saved myself having to buy a mobile Wi-Fi hub for any trips to Éire. For now, the Irish 3G dongle that I already have should be enough.
Changing to web fonts
12th February 2012While you can add Windows fonts to Linux installations, I have found that their display can be flaky to say the least. Linux Mint and Ubuntu display them as sharp as I'd like, but I have struggled to get the same sort of results from Arch Linux, while I am not so sure about Fedora or openSUSE either.
This led me to explore web fonts for my websites, with Google Web Fonts meeting my needs through options like Open Sans and Arimo. There have been others with which I have dallied, such as Droid Sans, but these are the ones on which I have settled for now. Both are in use on this website now, and I added calls for them to the web page headers using the following code (lines are wrapping due to space constraints):
<link href="http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,600italic,700italic,400,300,600,700" rel="stylesheet" type="text/css">
<link href='http://fonts.googleapis.com/css?family=Arimo:400,400italic,700,700italic' rel='stylesheet' type='text/css'>
With those lines in place, it then is a matter of updating font-family and font declarations in CSS style sheets with "Open Sans" or "Arimo" as needed, while keeping alternatives defined in case the Google font service goes down for whatever reason. A look at a development release of the WordPress Twenty Twelve theme caused me to come across Open Sans and I like it for its clean lines and Arimo, which was found by looking through the growing Google Web Fonts catalogue, is not far behind. Looking through that catalogue now causes for me a round of indecision since there is so much choice. For that reason, I think it's better to be open to the recommendations of others.