Technology Tales

Create and Preview RMarkdown Documents with QBit Workspace

RMarkdown is a document format rooted in the concept of Literate Programming, a paradigm introduced by Donald Knuth that combines code outputs with written content. While it is widely associated with the R programming community and the RStudio IDE, it supports a broad range of language engines, with version 1.33 of the knitr package offering 44 options including Python, SQL, Julia and many others.

Through the pandoc document converter, RMarkdown can produce a variety of output formats, such as static HTML files, PDF documents generated via LaTeX, Microsoft Word documents, PowerPoint presentations and flexdashboard layouts, all configurable through the YAML header. QBit Workspace allows users to author and preview RMarkdown documents directly in a browser, with an instant preview feature in the Viewer pane designed to accelerate the development process and support the creation of all the aforementioned output formats.

Compare data frames in R

Comparing data frames in R can be achieved using several packages, each offering different levels of detail. The dplyr package provides a straightforward approach through its all_equal function, which returns TRUE when two data frames are identical and describes the specific row differences when they are not. The arsenal package offers a more comprehensive comparison via its comparedf function, producing detailed summaries that highlight differing variables, unequal values and observations that appear in one data frame but not the other. The diffdf package similarly identifies and reports differences between data frames, flagging unequal values by variable and row number.

These tools handle various scenarios, including comparing identical data frames, those with differing values and those with different numbers of rows altogether. Of the three approaches, dplyr is generally considered the most accessible for quick comparisons.

Top 5 tricks to make Matplotlib plots look better

Creating visually appealing data visualisations is an important skill for data scientists, and several straightforward techniques can significantly improve the quality of graphs produced using Python libraries such as Matplotlib and Seaborn. Applying a predefined plot theme, such as ggplot or one of the many Seaborn styles, instantly changes the overall look and feel of a chart with minimal effort.

Adjusting the colours of individual bars or lines, drawing from a library of over 950 named colours, allows specific data points to be highlighted through contrast. Changing the font family ensures that charts blend cohesively with the typography used in a wider presentation, while Seaborn's set context feature scales all visual elements, including fonts and titles, to suit different display settings such as paper, poster or talk formats. Finally, applying a colour palette to a plot unifies its tones and creates a more polished, harmonious appearance compared to using default, uncoordinated colours.

Choosing Colormaps in Matplotlib

Matplotlib offers a wide range of built-in colormaps, organised into several categories to suit different types of data. Selecting the right one depends on factors such as whether the data has a natural ordering, a critical midpoint value or repeating endpoints, as well as any conventions expected by the intended audience.

Perceptually uniform colormaps, in which equal steps in data correspond to equal perceived steps in colour, are generally the best choice, since the human brain responds more reliably to changes in lightness than to changes in hue. Sequential colormaps are suited to ordered data, diverging colormaps work well when the data varies around a meaningful central value, cyclic colormaps are appropriate for values that wrap around at their endpoints and qualitative colormaps are used for unordered categorical data.

A separate miscellaneous category includes colormaps designed for specific purposes, such as topographic or depth visualisation. Lightness values are also important when considering how a plot will appear when printed in greyscale, as colormaps that increase monotonically in lightness tend to reproduce more clearly, whereas those with irregular lightness patterns can result in unreadable output. Awareness of colour vision deficiencies is also advisable, and avoiding colormaps that combine red and green reduces the risk of problems for a significant portion of viewers.

Rotate Tick Labels in Matplotlib

Rotating axis labels in Matplotlib can be achieved through several methods, applicable to both the X and Y axes. At the figure level, plt.xticks() and plt.yticks() allow rotation to be set directly, while at the axes level, options include using ax.set_xticklabels() or ax.set_yticklabels(), iterating over tick labels and applying tick.set_rotation() to each, or using ax.tick_params() with a labelrotation argument. It is important to call plt.draw() before accessing tick labels when working at the axes level, as labels are only populated after the plot is drawn. For plots displaying dates, which often overlap and become unreadable without adjustment, Matplotlib provides the fig.autofmt_xdate() and fig.autofmt_ydate() functions as a convenient alternative to manual rotation.

Create a grouped bar chart with Matplotlib and Pandas

A developer working through freeCodeCamp's Data Analysis with Python certification encountered difficulties creating a grouped bar chart using Matplotlib and pandas, and documented their solution for the Page View Time Series Visualiser project. The dataset used contains daily page view recordings, which are loaded via Pandas, cleaned by removing outliers in the top and bottom 2.5 percentiles, and enriched with year and month columns derived from date/time index attributes. The months are stored as categorical data to preserve chronological order.

The key step in producing the grouped bar chart is reshaping the DataFrame into a pivot table, with years as the index, months as the columns and mean page views as the cell values, after which calling the plot method with the bar type on the reshaped DataFrame is sufficient for Matplotlib to render the grouped visualisation correctly. The author notes that while the final plotting call is straightforward, the real challenge lies in understanding how to manipulate the data into the required shape beforehand, and that comparable results can be achieved more simply using Plotly, which requires only two additional parameters and no pivoting step.

Within() - Base R’s Mutate() function

Base R's built-in within() function serves as a direct alternative to the mutate() function from the popular dplyr package, with both performing the same task of creating new variables within a data frame. Using the classic iris dataset as a demonstration, both functions can be used to calculate a sepal length-to-width ratio, producing identical results regardless of whether tidy or base R syntax is employed.

A benchmark comparison using the rbenchmark package reveals that within() is considerably faster than mutate(), completing 1,000 replications in 0.18 seconds compared to 1.60 seconds for mutate(). Memory usage between the two is negligibly different, with mutate() consuming 1,312 bytes and within() consuming 1,296 bytes. For those seeking an even faster approach, the data.table package offers a further alternative, and the newer base R pipe operator has also been noted as a slightly quicker option than the magrittr pipe commonly associated with tidy workflows.

10 Tips And Tricks For Data Scientists Vol.9

This ninth instalment in a series of practical tips for data scientists covers a range of techniques across R, Python, SQL and Postman. In R, the tips cover writing cross-platform file paths using the file.path() command, repeating vectors using the rep() function and performing circular shifts on vectors with a custom-built function. The Python tips demonstrate how to retrieve the source code of a function using the inspect module, remove elements from a NumPy array based on a specific value, generate filenames that include a creation date or timestamp, identify the most recently modified file in a directory and collect all files of a given type across directories and subdirectories using the os module. The SQL section explains how to extract key-value pairs from JSON objects in PostgreSQL, including nested structures, and the final tip highlights a Postman feature that automatically generates code for API calls in a chosen programming language.

Error Bar Plot in R - Adding Error Bars

Error bars are graphical tools used in data visualisation to represent the variability or uncertainty within a dataset, commonly expressing one standard deviation, one standard error or a 95% confidence interval. A smaller standard deviation bar suggests that data points cluster closely around the mean, indicating greater reliability, whilst a larger one signals wider spread and less reliability. Overlapping standard deviation bars may hint that differences between groups are not statistically significant, whereas non-overlapping bars suggest a potentially significant difference, though a formal statistical test is always required before drawing any firm conclusion.

In R, the ggplot2 package provides a straightforward means of creating error bar plots, with data first summarised to calculate means and standard deviations using functions such as ddply or aggregate. From there, bar charts and line graphs can each be enhanced with error bars using the geom_errorbar function, and these can reflect either full symmetrical ranges or upper bars only, depending on the analytical need.

Problem Note 41684: RTF output appears truncated when a very long text string spans multiple pages

A known issue in SAS Base affects RTF output when the content within a single table cell is long enough to span multiple pages, causing Microsoft Word to display only the first page despite the full content being present in the underlying file. In SAS 9.4M4, the problem was resolved through the introduction of the NOTRKEEP option in the ODS RTF statement. For earlier versions, two workarounds are available: using the MSOFFICE2K tagset to generate the RTF output, or employing a data step to split the long content into multiple observations, as demonstrated in Sample 24672.