TOPIC: TEXT FILE
Dealing with this Python error message on Windows: UnicodeEncodeError: 'charmap' codec can't encode characters in position 56-57: character maps to <undefined>
14th March 2025Recently, I got caught out by the above message when summarising some text using Python and Open AI's API while working within VS Code. There was no problem on Linux or macOS, but it was triggered on the Windows command line from within VS Code. Unlike the Julia or R REPL's, everything in Python gets executed in the console like this:
& "C:/Program Files/Python313/python.exe" script.py
The Windows command line shell operated with cp1252 character encoding, and that was tripping up the code like the following:
with open("out.txt", "w") as file:
file.write(new_text)The cure was to specify the encoding of the output text as utf-8:
with open("out.txt", "w", encoding='utf-8') as file:
file.write(new_text)After that, all was well and text was written to a file like in the other operating systems. One other thing to note is that the use of backslashes in file paths is another gotcha. Adding an r before the quotes gets around this to escape the contents, like using double backslashes. Using forward slashes is another option.
with open(r"c:\temp\out.txt", "w", encoding='utf-8') as file:
file.write(new_text)Windows commands for setting default applications for opening certain file types
18th August 2015On Friday, I was working on a system where a session is instantiated from a stored virtual machine that produces a fresh session every time, meaning that all previous changes get lost. What I have is a batch script that I run to reinstate what I need, and I encountered another task that I wanted it to do.
Part of my work involves the creation of plain text files with the extension lst and this is getting associated with SAS instead of Notepad. While you can reassign such associations using the GUI, it would be a bonus to do it via the command line too, so the assoc and ftype commands caught my interest. The first of these associates a file with a given extension with a desired file type, while the second shows the available file types together with the associated applications that open them. The assoc command also shows all the associations that are in place when it is executed with no parameters, and the ftype command does the same for file types.
Once you have picked out a file type with the ftype command, then the assoc can be used like the following:
assoc .lst=txtfile
The above associates an extension with a file type. In this, .lst files are going to get opened by Notepad because of the txtfile association. Though it did not do what I wanted on Friday due to system lockdown, it is good to know that this is possible and that even the Windows command line supports goodies like these.