Technology Tales

Python – List Files in a Directory

Python offers several built-in methods for retrieving lists of files and directories stored on a computer. The os module provides three key functions for this purpose: os.listdir(), which returns the contents of a specified directory without going deeper than the first level; os.walk(), which traverses an entire directory tree and is useful for locating specific file types across multiple folders; and os.scandir(), a more efficient alternative to os.listdir() that is available in Python 3.5 and above. For more flexible retrieval using pattern matching with wildcards, the glob module offers two options: glob.glob(), which returns a list of matching file paths, and glob.iglob(), which returns an iterator instead and is better suited to large directories due to its greater efficiency.

Create an empty array in Python

Python offers several approaches to initialising an empty array, with the best choice depending on the specific use case. A standard Python list, created using either square brackets or the list() constructor, is the most flexible and commonly used option, as it can hold any data type and grows dynamically. For memory-efficient storage of uniformly typed data, the array module provides a typed alternative that requires all elements to share the same data type.

In data science and machine learning contexts, NumPy is generally the preferred choice due to its speed and mathematical capabilities, offering multiple initialisation methods including np.array(), np.zeros() and np.empty(), the last of which is the fastest as it allocates memory without setting initial values. A key distinction to bear in mind is that standard Python lists are dynamic whilst NumPy arrays are intended to remain static in size, meaning that frequently resizing a NumPy array is a signal that a list may be the more appropriate starting point, with conversion to NumPy carried out only once the data collection is complete.

Pandas Convert List of Dictionaries to DataFrame

In Python, a dictionary (dict) holds key-value pairs where keys serve as column names and values populate the corresponding column data when converted to a Pandas DataFrame. Several methods can be used to perform this conversion, including pd.DataFrame(), pd.DataFrame.from_records(), pd.DataFrame.from_dict() and json_normalize().

When dictionaries share inconsistent keys, Pandas automatically inserts NaN values for any missing entries. Custom indexing can be applied during conversion using the index parameter, while the columns parameter allows control over column order and selection. For large datasets, from_records() is generally the better-performing option, and for dictionaries containing nested structures, json_normalize() flattens the data into a suitable tabular format.

Linux Touch Command

The touch command on Linux is used to create empty files and update their timestamps, which record access, modification and change times. It allows users to modify specific timestamps individually, set custom dates and times using options like -t or -d, or replicate timestamps from another file with -r.

The command supports creating single or multiple files simultaneously, and can be configured to avoid creating new files if they already exist. By adjusting these metadata attributes, the touch command provides a straightforward method for managing file information without altering the file's content.

Pharmaverse is a collaborative network of pharmaceutical companies and individuals dedicated to the open-source development of curated R packages for clinical reporting. Rather than working in isolation on closed, often duplicative solutions, contributors share tools across a post-competitive space, with the aim of easing regulatory review and ultimately bringing new treatments to patients more quickly.

The network hosts a broad catalogue of packages spanning areas such as data standards, metadata, validation, submission and reporting, and anyone is free to adopt whichever packages suit their needs, with the understanding that inclusion in the catalogue does not constitute an endorsement of any code's reliability. Community involvement is central to the project, with decisions about package inclusion driven by open proposals and discussion, and a governing council stepping in for more contentious matters. The network also engages with industry working groups to explore approaches to R package validation in regulated environments.

Arrays in Julia

Working with arrays in Julia has been a journey of discovery, particularly when it comes to understanding the breadth of functions available for manipulation and analysis. Arrays form the backbone of data handling in the language and their versatility is evident through methods like axes, which returns valid indices and cat, which allows concatenation along specified dimensions.

Functions such as broadcast and broadcast! enable efficient operations across multiple arrays, while fill and fill! provide ways to initialise or overwrite arrays with specific values. Manipulation of arrays is straightforward, with push! and pop! adding or removing elements and deleteat! allowing targeted deletions. For multidimensional arrays, methods like hcat and vcat handle horizontal and vertical concatenation, while reshape and permute offer flexible reorganisation of data. The getindex and setindex functions provide precise control over accessing and modifying elements and findall, findfirst and findlast aid in locating specific values within arrays.

Beyond basic operations, Julia’s array methods extend to advanced tasks, such as computing strides with stride, creating views with @view and using similar to generate new arrays with the same structure. These tools collectively make array handling in Julia both powerful and intuitive, catering to everything from simple data storage to complex transformations.

Splitting string into array of substrings in Julia – split() and rsplit() Method

Julia provides two functions, split() and rsplit(), for dividing strings into arrays of substrings based on specified delimiters. The split() function processes the string from the beginning, while rsplit() operates from the end, with both allowing parameters to control the maximum number of resulting elements and whether empty substrings are included. These methods are useful for parsing and manipulating text data, with examples demonstrating their application in handling various string formats and delimiter placements.

For loop in Julia

Julia's for loop follows a for in structure rather than the C-style syntax found in many other programming languages, making it closer in behaviour to a for-each loop. The syntax involves a loop keyword, an iterator, a range and a closing end keyword, allowing sequential traversal across various data structures. This includes lists, tuples and strings, each of which can be iterated over in the same straightforward manner. Julia also supports nested for loops, where one loop is placed inside another, enabling more complex iteration patterns such as printing structured numerical output across multiple rows and columns.

How to Compare Strings in Bash

Bash string comparison relies on a set of operators that evaluate equality, inequality, alphabetical ordering and string length within conditional statements. The equality operators = and == check whether two strings match exactly, while != checks for a mismatch, and =~ tests whether a string matches a regular expression. The -z and -n flags determine whether a string is empty or non-empty respectively.

Since comparisons are case-sensitive by default, case-insensitive checks can be achieved either by converting variables to lowercase using the ,, parameter expansion or by enabling the nocasematch shell option, which should be disabled again afterwards to avoid unintended side effects. The double-bracket [[ construct is generally preferred over the single-bracket [ command in Bash scripts, as it supports pattern matching and regular expressions without requiring variables to be quoted to prevent word splitting.

Glob-style patterns can also be used with == to check for substrings or character classes, and the case statement offers a practical alternative when a string needs to be evaluated against several possible patterns. A useful distinction to keep in mind is that string operators compare characters individually, whereas numeric operators such as -eq and -gt evaluate integer values, meaning that strings like 02 and 2 would not be considered equal under string comparison even though they represent the same number.

How to remove Scientific Notation in R

When working with large numbers in R, scientific notation may be used by default, but this can be adjusted using two approaches. One method involves setting a global preference to suppress scientific notation by modifying the scipen option, which affects all outputs in the session. Alternatively, a specific variable can be displayed without scientific notation by applying the format function with the scientific parameter set to false, allowing for direct control over individual results without altering broader settings. Both techniques provide ways to manage numerical display formats depending on the context of the analysis.