Technology Tales

Using cURL in Python with PycURL

PycURL is a Python interface to the libcURL library that enables data transfer to and from servers, supporting multiple protocols including HTTPS, FTPS, SMTP, IMAP and SMB, among others. It is particularly well suited for testing REST APIs, downloading files and handling large numbers of concurrent connections, and is notably faster than the popular Python Requests library.

Installation is straightforward across operating systems, with Mac and Linux requiring no additional dependencies, while Windows users may need to address a few prerequisites beforehand. Once installed, PycURL can be used to perform HTTP GET requests to retrieve data from a given URL, examine response headers, send form data via POST, upload files using either multipart POST or PUT requests, send DELETE requests to remove server-side resources and write responses directly to a local file, all through a relatively consistent coding pattern built around the setopt function and the Curl object.

FOSS4Spectroscopy: R vs Python

Bryan Hanson's FOSS for Spectroscopy project, which catalogues free and open-source software for spectroscopic applications, received a major update in July 2022, at which point Python packages outnumbered R packages by more than two to one. Packages are discovered by searching four repositories, namely CRAN, GitHub, PyPi.org and juliapackages.org, across a range of spectroscopic topics including NMR, Raman, FT-IR, XRF and several others. To support automated searching of GitHub and PyPi.org, Hanson developed a package called webu, which handles API queries and incorporates deliberate delays to avoid overloading servers. Raw search results require considerable manual inspection and cleaning before they are suitable for inclusion in the main database, with additional scripts used to remove duplicate entries and resolve inconsistencies in package naming conventions.

htmlwidgets for R

The htmlwidgets package enables the integration of JavaScript-based data visualisations into R, allowing users to generate interactive plots directly at the R console, embed them within R Markdown documents or Shiny applications and develop custom widgets that bridge R and JavaScript seamlessly. Tools such as flexdashboard facilitate the arrangement of multiple widgets into flexible layouts, while crosstalk supports linking visualisations for coordinated interactions. The framework also provides guidance for creating new widgets, expanding the range of available visualisations and enhancing their use in analytical workflows.

Semantic Versioning 2.0.0

Semantic Versioning offers a structured approach to version numbering, enabling developers to communicate changes clearly and manage dependencies effectively. By adhering to a formal specification, version numbers become meaningful indicators of compatibility and stability.

The system divides updates into major, minor and patch levels, with major versions reserved for backward-incompatible changes, minor for new features and patch for bug fixes. This clarity allows dependent projects to specify ranges that ensure compatibility without requiring constant updates.

The specification emphasises the importance of a well-defined public API, particularly when releasing version 1.0.0, which signals a stable foundation for users. During initial development (0.y.z), rapid iteration is encouraged, but once the API solidifies, careful consideration of backward compatibility becomes essential. Deprecating features requires documentation and a minor version release to allow users time to adapt before removal in a major update. Practical challenges, such as accidental versioning errors or managing dependencies without altering the public API, are addressed through guidelines on corrective releases and evaluating the impact of changes.

While the system may seem rigid, it encourages thoughtful development, ensuring that incompatible changes are introduced only when necessary. By linking to the specification in project documentation, developers invite others to benefit from consistent practices, reducing the friction of dependency management. The approach avoids arbitrary versioning, prioritising transparency and predictability. It acknowledges the complexity of real-world software but provides a framework to navigate it systematically, ensuring that updates are both intentional and manageable.

Python Check If File Exists

In Python, two standard approaches exist for checking whether a file exists before performing operations on it. The first uses the exists() function from the os.path module, which accepts a file path as its argument and returns True if the file is found or False if it is not. The second uses the is_file() method from the Path class within the pathlib module, available since Python 3.4, which follows an object-oriented approach and behaves in the same way. When specifying file paths with either method, forward slashes should be used as separators, as this works consistently across Windows, macOS and Linux.

Python – List Files in a Directory

Python offers several built-in methods for retrieving lists of files and directories stored on a computer. The os module provides three key functions for this purpose: os.listdir(), which returns the contents of a specified directory without going deeper than the first level; os.walk(), which traverses an entire directory tree and is useful for locating specific file types across multiple folders; and os.scandir(), a more efficient alternative to os.listdir() that is available in Python 3.5 and above. For more flexible retrieval using pattern matching with wildcards, the glob module offers two options: glob.glob(), which returns a list of matching file paths, and glob.iglob(), which returns an iterator instead and is better suited to large directories due to its greater efficiency.

Create an empty array in Python

Python offers several approaches to initialising an empty array, with the best choice depending on the specific use case. A standard Python list, created using either square brackets or the list() constructor, is the most flexible and commonly used option, as it can hold any data type and grows dynamically. For memory-efficient storage of uniformly typed data, the array module provides a typed alternative that requires all elements to share the same data type.

In data science and machine learning contexts, NumPy is generally the preferred choice due to its speed and mathematical capabilities, offering multiple initialisation methods including np.array(), np.zeros() and np.empty(), the last of which is the fastest as it allocates memory without setting initial values. A key distinction to bear in mind is that standard Python lists are dynamic whilst NumPy arrays are intended to remain static in size, meaning that frequently resizing a NumPy array is a signal that a list may be the more appropriate starting point, with conversion to NumPy carried out only once the data collection is complete.

Pandas Convert List of Dictionaries to DataFrame

In Python, a dictionary (dict) holds key-value pairs where keys serve as column names and values populate the corresponding column data when converted to a Pandas DataFrame. Several methods can be used to perform this conversion, including pd.DataFrame(), pd.DataFrame.from_records(), pd.DataFrame.from_dict() and json_normalize().

When dictionaries share inconsistent keys, Pandas automatically inserts NaN values for any missing entries. Custom indexing can be applied during conversion using the index parameter, while the columns parameter allows control over column order and selection. For large datasets, from_records() is generally the better-performing option, and for dictionaries containing nested structures, json_normalize() flattens the data into a suitable tabular format.

Linux Touch Command

The touch command on Linux is used to create empty files and update their timestamps, which record access, modification and change times. It allows users to modify specific timestamps individually, set custom dates and times using options like -t or -d, or replicate timestamps from another file with -r.

The command supports creating single or multiple files simultaneously, and can be configured to avoid creating new files if they already exist. By adjusting these metadata attributes, the touch command provides a straightforward method for managing file information without altering the file's content.

Pharmaverse is a collaborative network of pharmaceutical companies and individuals dedicated to the open-source development of curated R packages for clinical reporting. Rather than working in isolation on closed, often duplicative solutions, contributors share tools across a post-competitive space, with the aim of easing regulatory review and ultimately bringing new treatments to patients more quickly.

The network hosts a broad catalogue of packages spanning areas such as data standards, metadata, validation, submission and reporting, and anyone is free to adopt whichever packages suit their needs, with the understanding that inclusion in the catalogue does not constitute an endorsement of any code's reliability. Community involvement is central to the project, with decisions about package inclusion driven by open proposals and discussion, and a governing council stepping in for more contentious matters. The network also engages with industry working groups to explore approaches to R package validation in regulated environments.