Technology Tales

Calmcode is an online learning platform offering 757 short video tutorials spread across 106 courses, designed to teach modern programming concepts and open-source tools in a calm, accessible manner. The platform is particularly focused on Python developers, covering a broad range of topics including code formatting, testing, data science, machine learning, web development and productivity tooling, with additional content for R users and general software development practices. New courses and lessons are published once or twice a month and are communicated through a newsletter, which the platform positions as the primary way for learners to stay informed. The overall philosophy behind Calmcode is one of reducing skill anxiety by presenting technical knowledge in short, straightforward lessons that begin from first principles, with the aim of making professional life in software development more manageable and enjoyable.

Understanding the Parquet File Format

Apache Parquet is a columnar storage file format designed for efficient data storage and querying, widely used in big data systems such as Hadoop. It organises data by columns rather than rows, enabling faster access to specific fields and reducing storage requirements through techniques like run-length encoding, dictionary encoding and compression.

This approach minimises file size, particularly beneficial for large datasets, and supports cross-platform compatibility. Compared to formats like RDS, which are specific to R and can store complex objects, Parquet prioritises storage efficiency and interoperability. It also contrasts with Feather, which focuses on speed and is part of the Apache Arrow ecosystem. The format's structure allows for metadata storage, facilitating efficient data processing and is implemented in tools such as the R package {arrow}, which enables reading and writing Parquet files with options for compression and encoding optimisation.

Statistics with Julia From the Ground Up

Here is a workshop that introduces the Julia programming language to data scientists and statisticians, focusing on statistical applications rather than general programming. Designed for those with experience in languages like R but no prior Julia knowledge, it covers foundational probability, statistical inference, regression and data manipulation using packages such as StatsBase, Distributions and GLM.

The session emphasises practical, goal-oriented scripting through examples and code snippets from a related book, with an accompanying Jupyter notebook for participants to follow along. The approach prioritises statistical methods and tools, treating Julia as a means to achieve analytical tasks rather than a programming language in isolation.

How to Find Files in Linux Using the Command Line

The Linux command line offers a powerful utility called find for locating files and directories within a file system by recursively filtering objects based on specified conditions. Users can search by file name or extension, modification time, file type and ownership, with results refined further using flags such as -iname for case-insensitive searches, -maxdepth to limit directory depth and -not to exclude certain results. Performance can be tuned through three optimisation levels, -O1, -O2 and -O3, with the default being -O1, which filters by file name first.

For content-based searching, the grep command can be paired with find using the -exec flag, which also allows matched files to be processed immediately, for example by changing permissions with chmod. The -execdir variant runs commands in the directory where the match is found rather than the root directory, which can offer security and performance benefits. A -delete flag can remove matched files, though this should be used with considerable caution, and interactive prompting before any action is taken can be enabled by substituting -exec with -ok or -execdir with -okdir.

Excel TRIM function - quick way to remove extra spaces

Excel's TRIM function offers a straightforward way to remove unwanted spaces from cell data, resolving common formula errors caused by hidden leading, trailing or extra spaces between values. The basic syntax requires only a single cell reference as its argument, instantly stripping all excess spaces whilst preserving single spaces between words. For numeric data, TRIM must be combined with the VALUE function to ensure results behave as numbers rather than as strings, whilst the MID, FIND and LEN functions can be used together to remove only leading spaces whilst keeping multiple spaces between words intact.

Counting excess spaces is achievable by comparing the original string length against the trimmed length using the LEN function, and conditional formatting can highlight affected cells before any changes are made. When the TRIM function fails to remove certain spaces, this is typically due to non-breaking spaces or non-printing characters that require the SUBSTITUTE, CHAR and CLEAN functions in combination to resolve, with the CODE function helping to identify the specific character values causing the problem.

GxP Compliance in Pharma Made Easier: Good Documentation Practices with R Markdown and {officedown}

In regulated pharmaceutical industries, maintaining rigorous documentation standards is essential for ensuring consumer safety and product reliability. GxP, a globally recognised framework covering practices such as Good Clinical Practice and Good Manufacturing Practice, places significant emphasis on Good Documentation Practices, requiring that records be traceable, accountable and data-integrity compliant. Meeting these standards manually is time-consuming and error-prone, which is where programmatic tools offer a practical advantage. R Markdown enables the creation of automated, reproducible and testable regulatory documents in multiple formats, reducing the burden of repetitive manual reporting.

However, its formatting flexibility is limited when precise structural or stylistic requirements must be met. The R package officedown addresses these shortcomings by extending R Markdown with capabilities more suited to generating Microsoft Word and PowerPoint documents, allowing users to control document structure with greater precision, apply custom styling to paragraphs and tables, use Word-based templates for consistent formatting, merge multiple documents while preserving references and numbering, and switch selected pages to landscape orientation. Together, these features make it considerably more straightforward for pharmaceutical teams to produce documentation that satisfies the detailed requirements of regulatory bodies such as the FDA and the European Medicines Agency.

Metaprogramming workshop at JuliaCon 2021

This workshop on metaprogramming in Julia, held during JuliaCon 2021, provided an in-depth exploration of techniques for writing programs that manipulate or generate other programs, highlighting Julia's strengths in this area due to its flexible code structure. Led by David P. Sanders, the session covered foundational concepts such as symbols, expressions and abstract syntax trees, alongside practical applications like macros, code generation using eval and generated functions, which enable the creation of optimised, type-specific code.

Participants worked with Jupyter notebooks offering step-by-step exercises, including examples of substituting values in expressions and building domain-specific language features. The workshop emphasised the balance between leveraging metaprogramming for efficiency and avoiding excessive complexity, underscoring its role in enhancing Julia's performance and expressiveness for scientific and computational tasks.

Jedi SAS Tricks: Explicit SQL Pass-through in DS2

DS2, a SAS programming procedure, offers significant advantages over traditional DATA step processing through its tight integration with SQL and its ability to handle data retrieval and manipulation more efficiently. Unlike a conventional data step, which requires data to be pre-sorted or indexed before using a BY statement, DS2 retrieves data using an implicit SQL query, making pre-sorting unnecessary regardless of whether the data resides in SAS or in a relational database management system such as Oracle.

A particularly powerful feature of DS2 is its support for explicit SQL pass-through queries within the SET statement, allowing programmers to leverage database-specific functions, such as Oracle's DECODE function, while still applying DS2 logic to the results. Ordering of results can be handled either within the database using an ORDER BY clause or within DS2 using a BY statement, though the latter is generally more reliable across different processing environments, particularly in distributed or multithreaded configurations such as SAS Cloud Analytic Services. It is worth noting that row ordering behaviour in DS2 can vary depending on the platform and whether code is executed in-database or through threaded processing, so programmers are advised to consult the relevant SAS documentation to better understand these nuances before relying on any particular ordering approach.

SAS FILENAME Statement: EMAIL (SMTP) Access Method

The FILENAME statement's EMAIL access method in SAS allows users to send electronic mail programmatically via SMTP, and it supports a wide range of options for customising messages, including specifying recipients, carbon copy and blind carbon copy addresses, subject lines, message priority, sensitivity levels, expiration dates and file attachments. When SAS is operating in a locked-down state, the feature is unavailable unless re-enabled by a server administrator.

Users can incorporate conditional logic within a data step to control which recipients receive which messages and can direct procedure output, images and HTML content through email. PUT statement directives provide a way to override or modify message attributes at runtime, such as changing the recipient address, subject or attached files, and they also allow actions like sending a message mid-step or clearing existing message attributes. Secure communication with SMTP servers is supported through Transport Layer Security, though message-level encryption and digital signing are not currently available. Additional system options allow users to adjust the server response wait time and manage UTC offset settings for messages sent across time zones.

How to send email using SAS

Sending emails via SAS presents several challenges, from configuration issues to security considerations. Users often encounter problems such as connection refusals, authentication failures, or emails not appearing in sent folders. These issues typically stem from incorrect SMTP settings, authentication methods, or limitations in how SAS interacts with email servers. For instance, using SMTP does not allow emails to be logged in sent folders, unlike MAPI, which is less reliable for automation.

Security is another concern, as the FROM address can be spoofed, though ISPs and email services may block such attempts. Specific examples include configuring SendGrid with SAS, where proper authentication (like using the correct API key format) is crucial. Solutions often involve verifying SMTP server details, ensuring correct port numbers and using appropriate authentication types. Additionally, restricting the FROM address to a fixed value or the user's account requires careful configuration, as SAS does not enforce this by default.