Technology Tales

Create a grouped bar chart with Matplotlib and Pandas

A developer working through freeCodeCamp's Data Analysis with Python certification encountered difficulties creating a grouped bar chart using Matplotlib and pandas, and documented their solution for the Page View Time Series Visualiser project. The dataset used contains daily page view recordings, which are loaded via Pandas, cleaned by removing outliers in the top and bottom 2.5 percentiles, and enriched with year and month columns derived from date/time index attributes. The months are stored as categorical data to preserve chronological order.

The key step in producing the grouped bar chart is reshaping the DataFrame into a pivot table, with years as the index, months as the columns and mean page views as the cell values, after which calling the plot method with the bar type on the reshaped DataFrame is sufficient for Matplotlib to render the grouped visualisation correctly. The author notes that while the final plotting call is straightforward, the real challenge lies in understanding how to manipulate the data into the required shape beforehand, and that comparable results can be achieved more simply using Plotly, which requires only two additional parameters and no pivoting step.

Within() - Base R’s Mutate() function

Base R's built-in within() function serves as a direct alternative to the mutate() function from the popular dplyr package, with both performing the same task of creating new variables within a data frame. Using the classic iris dataset as a demonstration, both functions can be used to calculate a sepal length-to-width ratio, producing identical results regardless of whether tidy or base R syntax is employed.

A benchmark comparison using the rbenchmark package reveals that within() is considerably faster than mutate(), completing 1,000 replications in 0.18 seconds compared to 1.60 seconds for mutate(). Memory usage between the two is negligibly different, with mutate() consuming 1,312 bytes and within() consuming 1,296 bytes. For those seeking an even faster approach, the data.table package offers a further alternative, and the newer base R pipe operator has also been noted as a slightly quicker option than the magrittr pipe commonly associated with tidy workflows.

10 Tips And Tricks For Data Scientists Vol.9

This ninth instalment in a series of practical tips for data scientists covers a range of techniques across R, Python, SQL and Postman. In R, the tips cover writing cross-platform file paths using the file.path() command, repeating vectors using the rep() function and performing circular shifts on vectors with a custom-built function. The Python tips demonstrate how to retrieve the source code of a function using the inspect module, remove elements from a NumPy array based on a specific value, generate filenames that include a creation date or timestamp, identify the most recently modified file in a directory and collect all files of a given type across directories and subdirectories using the os module. The SQL section explains how to extract key-value pairs from JSON objects in PostgreSQL, including nested structures, and the final tip highlights a Postman feature that automatically generates code for API calls in a chosen programming language.

Error Bar Plot in R - Adding Error Bars

Error bars are graphical tools used in data visualisation to represent the variability or uncertainty within a dataset, commonly expressing one standard deviation, one standard error or a 95% confidence interval. A smaller standard deviation bar suggests that data points cluster closely around the mean, indicating greater reliability, whilst a larger one signals wider spread and less reliability. Overlapping standard deviation bars may hint that differences between groups are not statistically significant, whereas non-overlapping bars suggest a potentially significant difference, though a formal statistical test is always required before drawing any firm conclusion.

In R, the ggplot2 package provides a straightforward means of creating error bar plots, with data first summarised to calculate means and standard deviations using functions such as ddply or aggregate. From there, bar charts and line graphs can each be enhanced with error bars using the geom_errorbar function, and these can reflect either full symmetrical ranges or upper bars only, depending on the analytical need.

Problem Note 41684: RTF output appears truncated when a very long text string spans multiple pages

A known issue in SAS Base affects RTF output when the content within a single table cell is long enough to span multiple pages, causing Microsoft Word to display only the first page despite the full content being present in the underlying file. In SAS 9.4M4, the problem was resolved through the introduction of the NOTRKEEP option in the ODS RTF statement. For earlier versions, two workarounds are available: using the MSOFFICE2K tagset to generate the RTF output, or employing a data step to split the long content into multiple observations, as demonstrated in Sample 24672.

Adding a Column to a Pandas DataFrame Based on an If-Else Condition

A dataset of over 4,000 tweets was analysed using Python to determine whether posts containing images receive more likes and retweets than those without. Using NumPy's where() function, a new Boolean column was added to a Pandas DataFrame to flag whether each tweet contained an image, and the results indicated that image-based tweets averaged nearly three times as many likes and retweets as those without images.

To explore the data further, NumPy's select() function was applied to categorise tweets into four engagement tiers based on like counts, revealing that while images appeared to improve performance, they were not a guarantee of success, with over 83% of the highest-performing tweets still having no image attached. The broader technical takeaway is that both np.where() and np.select() offer straightforward and practical methods for adding new columns to a Pandas DataFrame based on conditional logic applied to existing data.

5 Tasks To Automate With Python

Python offers a range of practical automation capabilities that can save time and reduce repetitive effort in daily workflows. A Mac-based script using the mac-say library can convert any file into an audiobook, while a simple requests-based script can retrieve weather reports for any city on demand. Currency conversion can be handled just as easily through the currencyconverter library, which allows quick conversions between currencies directly from the command line. For those who struggle with a disorganised folder of downloads, a watchdog-based script can monitor a specified directory and automatically sort incoming files into subfolders based on their type, covering images, PDFs, videos and audio files. Finally, a morning setup script using Python's built-in webbrowser module can open a predefined set of browser tabs automatically, removing the need to do so manually each day. Together, these examples illustrate how Python can be applied to everyday tasks with relatively little code and a handful of third-party libraries.

Usage Note 64615: The SAS log displays the error "Invalid JSON in input near line XXX column XXX: Some code points did not transcode"

When using a LIBNAME statement with the JSON engine in SAS, an error may appear in the log stating that invalid JSON was encountered and that some code points did not transcode. This occurs because UTF-8 characters in the dataset do not map to the default SAS session encoding. The recommended fix is to open SAS in Unicode mode by navigating to the Start menu and selecting the Unicode Support option under SAS 9.4.

Working With JSON Data in Python

Handling data in modern applications often involves working with structured formats like JSON. This format is widely used for transferring information between systems or storing data in document-oriented databases. Python provides robust tools to manage JSON data, enabling seamless conversion between Python objects and JSON strings. Understanding the syntax of JSON is essential, as it relies on key-value pairs and supports nested structures. When converting between Python and JSON, care must be taken due to differences in data types, such as the absence of JSON equivalents for Python-specific types like sets.

Writing and reading JSON files is straightforward with Python’s built-in libraries, allowing for both simple and complex data structures to be persisted or retrieved efficiently. Validating JSON syntax ensures that data remains consistent and error-free, which is crucial when dealing with external sources. Techniques such as pretty-printing JSON in the terminal enhance readability, while minifying JSON reduces file size for storage or transmission.

These practices are particularly useful when working with APIs or managing large datasets. Python’s flexibility in handling JSON data makes it a valuable tool for developers, whether they are building web applications, processing data, or interacting with external services. Mastery of these concepts equips developers to manage data effectively across various domains, from backend systems to data science workflows.

json.dumps() in Python

Python's json.dumps() function serialises Python objects such as dictionaries, lists and strings into JSON-formatted strings, making it useful for tasks like sending data through APIs or storing structured data. The function accepts several optional parameters that control its behaviour, including indent for formatting output with readable spacing, sort_keys for arranging dictionary keys alphabetically, skipkeys for automatically ignoring incompatible key types such as tuples, ensure_ascii for handling non-ASCII characters, allow_nan for permitting special numerical values, and separators for customising how items and key-value pairs are divided. The function always returns a string object, meaning that a Python dictionary passed through it will produce a string representation rather than a dictionary, and the same principle applies to lists, which are converted into JSON arrays.