Technology Tales

Descriptive Statistics in Julia

Performing descriptive statistics in Julia involves leveraging built-in packages such as Distributions.jl, StatsBase.jl, CSV.jl, DataFrames.jl and StatsPlots.jl to analyse data characteristics, generate summaries and visualise results. The process includes installing and importing these packages, generating random data, calculating statistical measures like mean, median, variance and standard deviation, creating data frames for structured manipulation and using functions such as describe and summarystats to derive insights. Categorical variables can be analysed separately using the by function, while visualisations like density plots and box-and-whisker plots help illustrate distributions and relationships within the data.

Choosing how to store your strings in Julia involves evaluating performance, memory usage and data characteristics, with recommendations varying based on whether strings are treated as categorical data, require pooled storage, or need inline allocation for efficiency. When dealing with large datasets, categorical data should be handled using CategoricalArrays.jl, while pooled storage for low-unique-value strings is suitable with PooledArrays.jl. InlineStrings.jl offers memory-efficient types for short, uniform strings and Symbol types may be used for comparison-heavy tasks with immutable labels, though they consume persistent memory. The decision hinges on balancing speed, memory constraints and the specific use case, with immediate conversion to appropriate formats upon data loading being emphasised for optimal outcomes.

Clip your data with ClipData.jl

The ClipData.jl package for Julia provides a straightforward way to transfer tabular data between a Julia session and the system clipboard in both directions. Data scientists who regularly work with tools such as Google Sheets can copy a table to their clipboard and ingest it directly into Julia as a DataFrame, or conversely export a DataFrame from Julia back to the clipboard for pasting elsewhere.

The package handles both tabular data with headers, using the cliptable function, and arrays without headers, using the cliparray function, with column element types detected automatically in the process. Additional features include options for controlling how table cells are parsed, with further details available on the package's homepage.

Animated Unicode Plots with Julia

Creating animated Unicode plots in Julia involves using the UnicodePlots.jl package to generate visualisations directly in the terminal, with a custom function written to sequence and display multiple frames sequentially. The process includes a helper function to reposition the cursor for overlapping output, followed by printing each frame with a specified delay, concluding with the final frame displayed without cursor movement adjustments. Thus, you can generate a series of line plots with varying parameters and animating them through a loop, highlighting the technical approach to achieving dynamic visual output in a terminal environment.

Speeding up Julia precompilation

Creating a custom Julia version with precompiled packages using PackageCompiler.jl reduces startup time for data science projects by generating a sysimage file and configuring a Jupyter kernel, which involves installing the package, creating the sysimage and setting up the kernel to include specific libraries such as Plots.jl and Flux.jl. The process involves running commands to compile selected packages into a shared object file, which is then used to launch Julia with those packages preloaded, minimising repeated compilation delays during notebook sessions or interactive work.

Plotting in Julia with Makie

Makie.jl is a relatively new but rapidly growing data visualisation library for the Julia programming language, gaining a strong reputation within the Julia community for its versatility and active development. It operates through a system of interchangeable backends, namely CairoMakie for static plots, GLMakie for interactive and three-dimensional visualisations using OpenGL and WGLMakie for interactive web-based visualisations.

The library follows a structure built around three core objects, being the Figure, the Axis and the Plot, which can be created and modified individually to allow precise control over the appearance and layout of visualisations, including subplots, legends, axis labels and titles. Rather than relying on a grammar of graphics approach, Makie provides a set of ready-to-use plotting functions such as scatter, lines and density, with a syntax broadly comparable to that of Matplotlib in Python.

Plots can be saved in various formats, including PNG and SVG, the latter being a particular advantage of the CairoMakie backend. Beyond static visualisations, the library also supports animations and a wide range of further customisation options, with comprehensive documentation and dedicated coverage in the Julia Data Science book serving as useful learning references.

Welcome to DataFramesMeta.jl

The DataFramesMeta.jl package simplifies data manipulation in Julia by streamlining operations such as aggregation, column selection and transformation within the DataFrames.jl ecosystem. It introduces macros like @combine, @select and @transform to perform these tasks, with @select retaining only computed columns and @transform preserving original data alongside new ones.

By-row operations, indicated by the r prefix, automate broadcasting for element-wise computations, while the ! suffix enables in-place modifications to avoid duplicating large datasets. Examples demonstrate how to calculate aggregated values, apply mathematical transformations and efficiently manage data modifications, highlighting the package's utility in reducing complexity during data processing workflows.

Getting Started with Python in VS Code

Visual Studio Code's Python extension allows developers to write, run and debug Python code directly within the editor. Getting started requires installing Python 3, VS Code and the Python extension, after which users can set up a project-specific virtual environment to keep package dependencies isolated and avoid version conflicts.

A simple script can then be created, run via the built-in terminal or play button, and stepped through using the integrated debugger, which supports breakpoints, variable inspection and a debug console. Additional functionality can be added by installing third-party packages such as NumPy via pip or the package management interface, and dependencies can be tracked using a requirements.txt` file to ensure consistency across different environments. From this foundation, developers can extend their work into web frameworks such as Django, Flask and FastAPI, or explore further features including linting, formatting, testing and remote development.

Rob J Hyndman is a statistician and researcher whose blog, Hyndsight, covers topics spanning forecasting, time series analysis, anomaly detection, data science and statistical computing. His recent work includes updates to the forecast and fpp3 R packages, the development of Quarto templates for Monash University, and the addition of instructional videos to the third edition of his co-authored textbook, Forecasting: Principles and Practice. He has also explored methodological questions around forecast accuracy metrics, AIC calculations, prediction intervals and the Ljung-Box test, while writing a forthcoming textbook on anomaly detection. Beyond his own research, he has contributed to open data advocacy in Australia, organised forecasting workshops in cities including New York, Chicago and Canberra, and highlighted Python implementations of his statistical models and algorithms for those outside the R ecosystem.

The Invisible JavaScript Backdoor

Researchers have demonstrated how invisible and visually deceptive Unicode characters can be exploited to embed hidden backdoors in JavaScript code that are effectively undetectable during manual code reviews. By using a Unicode character known as HANGUL FILLER (U+3164), which is recognised by JavaScript as a valid identifier yet renders as completely invisible in most editors, an attacker can introduce additional variables and inject arbitrary operating system commands through what appears to be a routine network health check endpoint.

A separate but related technique, known as a homoglyph attack, involves substituting standard code characters with visually similar Unicode equivalents, such as replacing an exclamation mark with an ALVEOLAR CLICK character, which can silently alter the logic of conditional statements. Both approaches pose a particular risk in open-source projects, where contributions may come from anonymous or untrusted developers. Since non-ASCII characters are rarely necessary in most codebases, especially those using English as the primary development language, a practical defensive measure would be to disallow non-ASCII characters entirely, and some development tools have since introduced features to flag such characters when they are detected.