Technology Tales

Tips for Selecting Columns in a Pandas DataFrame

Working with large datasets in Python often requires efficient column selection techniques, and Pandas offers several useful approaches for this purpose. The iloc function enables integer-location based indexing, allowing users to select individual columns, lists of columns or ranges using slice notation. When combining non-sequential ranges, NumPy's r_ object can translate a mix of slice notation and individual integers into a single array that iloc can process. Boolean arrays offer another powerful method, using string accessor functions on a column index to filter columns based on their names, including support for regular expressions to match multiple patterns simultaneously.

The Pandas filter function provides a more straightforward alternative for selecting columns by name or pattern, accepting either partial string matching or regular expression inputs. Practical utilities such as list and dictionary comprehensions can also help users build reference mappings of column names and their indices, reducing the need to repeatedly consult original data files during analysis. One important consideration when using numerical indexing is ensuring that column positions remain consistent across different data inputs, as changes in column order could cause errors in subsequent processing steps.

What is Spaghetti Code (And Why You Should Avoid It)

Spaghetti code is a derogatory term in IT jargon used to describe poorly structured, convoluted programming that arises from a lack of planning, inexperience, unclear project scopes and the gradual accumulation of changes made by multiple developers over time.

Rather than following sound architectural principles, spaghetti code tends to rely on shortcuts such as GOTO statements, making programmes difficult to maintain and scale. The term has been in use since at least the late 1970s, with historical references linking it to early programming languages such as FORTRAN.

For enterprise businesses, the consequences can be significant, as untangling years of disorganised code wastes time, money and developer effort. To prevent it, developers are encouraged to plan architecture carefully from the outset, conduct regular unit testing, seek peer reviews and use lightweight frameworks with clearly defined layers. Related coding problems include ravioli code, which affects object-oriented projects, lasagna code, which results from overly interdependent layers and pizza code, which describes an architecture that is too flat.

Spyder: The Scientific Python Development Environment

Spyder is an Integrated Development Environment designed for scientific computing in Python, offering an editor, an interactive console, a variable explorer and various other tools to support program development. Users can write and execute code directly within the editor, run individual lines or sections using keyboard shortcuts, and interact with defined objects through the IPython Console.

The environment supports incremental coding and debugging by allowing objects to persist between executions, though this can occasionally cause issues if code inadvertently relies on variables defined in a previous session rather than within the script itself. Spyder includes a step-by-step debugger that lets users move through code line by line, inspect variables at each stage and modify them as needed.

Plotting with Matplotlib is supported either inline within the console or in a separate window, and documentation strings can be formatted using reStructuredText and the Numpydoc standard to produce well-rendered output in the built-in help panel. Configuration options allow users to manage code style compliance, symbolic mathematics via SymPy and a range of run settings, while keyboard shortcuts and customisable preferences help streamline the development workflow.

Python KeyError Exceptions and How to Handle Them

A KeyError is a common Python exception raised when attempting to access a key that does not exist in a dictionary or similar mapping structure. It is a subtype of the LookupError exception and can occasionally appear in standard library modules such as zipfile, where it signals that a requested item cannot be located.

When encountered, the traceback provides useful details, including the missing key and the line of code responsible for the error. There are several practical ways to handle a KeyError, depending on the situation.

The most common approach is to use the .get() method on a dictionary, which returns a default value rather than raising an exception when a key is absent. In cases where it is important to confirm whether a key exists without retrieving its value, the in operator offers a straightforward check. For more general scenarios, particularly when dealing with third-party code or modules that do not support .get(), a try-except block provides reliable control over program flow by catching the exception and executing a fallback action instead.

10 Tips And Tricks For Data Scientists Vol.10

Volume 10 of a series aimed at data scientists covers a range of practical coding techniques in Python and R. In Python, the tips include retrieving the key of the maximum value in a dictionary, sorting dictionaries by value in ascending or descending order, shuffling a Pandas data frame using fractional sampling, repositioning a column to the end of a data frame through re-indexing, performing circular shifts on arrays using NumPy, replacing data frame values by specifying column and index positions with the loc function, automatically generating a requirements file for a Python project using the PIGAR package, producing random names with the names library and reading header-free CSV files by manually defining column names. The single R tip addresses a statistics problem typical of data science interviews, demonstrating how to calculate the standard deviation of a normal distribution from a known probability threshold using the standard normal distribution and confirming the result through simulation.

Passing arguments to an R script from command lines

There are two main approaches to passing external arguments to an R script from the command line. The first uses the built-in R function commandArgs, which scans the arguments supplied when the R session was invoked, storing them in a vector that the script can then reference by position, with the ability to set default values or return errors when required arguments are missing. The second approach uses the optparse package, which works in a Python-like style by allowing named flags and options to be declared with their types, default values and help messages, producing a more structured and user-friendly interface that also generates automatic help documentation when requested.

Think of && as a stricter &

R provides two versions each of the logical AND and OR operators, namely the shorter vectorised forms (& and |) and the longer scalar forms (&& and ||). The shorter forms compare pairs of elements across two vectors and return a result of the same length, making them well-suited for filtering rows in a data frame. The longer forms, by contrast, only examine the first element of each input and return a single scalar value, with R 4.2.0 introducing warnings when vectors longer than one element are passed to them.

The longer forms also support short-circuit evaluation, meaning the second operand is not assessed if the result can be determined from the first alone. Because if() statements require a scalar condition, the longer forms are generally preferred in control-flow programming, though care must be taken to supply genuinely scalar inputs.

The functions all() and any() offer a clean way to reduce a logical vector to a scalar by applying AND or OR across all elements respectively. Finally, NA values complicate logical operations by infecting results, though the isTRUE() and isFALSE() functions provide a reliable means of testing whether a value is strictly and unambiguously TRUE or FALSE.

Speeding Up R Shiny – The Definitive Guide

Optimising the performance of R Shiny applications is achievable through a combination of thoughtful development practices and targeted technical interventions. Proper data handling is a foundational concern, including preprocessing static data in advance and choosing appropriate storage solutions based on data size, with faster reading and writing functions offering measurable gains over base R defaults.

Because R is single-threaded, delegating long-running tasks to separate processes using packages such as promises or shiny.worker can prevent one user's activity from blocking others, resulting in a smoother concurrent experience. Understanding Shiny's scoping rules allows developers to share objects efficiently across sessions where appropriate, while keeping user-specific data private.

Reducing reliance on renderUI in favour of update functions and leveraging JavaScript where possible minimises unnecessary communication between the browser and the server. Caching repeated, resource-heavy operations using either the memoise package or Shiny's built-in bindCache mechanism can significantly reduce rendering times, particularly when outputs depend on a finite set of input combinations.

Serving an application via a platform that supports multiple concurrent R processes improves scalability further. Finally, profiling tools such as profvis help identify which specific functions are consuming the most time, enabling developers to focus optimisation efforts precisely where they are needed rather than relying on guesswork.

Code performance in R: Parallelization

Parallelisation in R allows multiple independent tasks to run simultaneously across processor cores, potentially reducing computation time significantly. By default, R uses only one core, but the parallel package enables multi-core processing. On Linux and Mac systems, this is achieved by replacing the standard lapply function with mclapply, which distributes tasks across available cores.

On Windows, the process is more involved, requiring the creation of a cluster using makeCluster, running computations with parLapply and then closing the cluster with stopCluster. Windows also differs in that each parallel process requires its own copy of data in working memory, rather than sharing a single copy, and each process starts in an empty environment, meaning objects must be manually exported before use.

Parallelisation is not always beneficial, as the initialisation process itself takes time, making it counterproductive for fast tasks but worthwhile for those running for minutes or longer. The optimal number of cores to use depends on available working memory and whether the computer needs to remain usable for other tasks during computation.

Common Format and MIME Type for Comma-Separated Values (CSV) Files

RFC 4180, published in October 2005 by Yakov Shafranovich, formally documents the Comma-Separated Values (CSV) format and registers the associated MIME type text/csv with IANA. Although CSV had long been used for exchanging and converting data between spreadsheet programmes, no formal specification had previously existed, leading to inconsistent implementations across different systems.

The format structures data in records separated by line breaks, with fields within each record divided by commas, and allows for an optional header line identifying field names. Fields may be enclosed in double quotes, which is required when a field contains commas, line breaks or double quotes, with any double quote appearing within a quoted field escaped by a preceding double quote.

The registered MIME type supports optional parameters for character set encoding and header presence, defaults to US-ASCII and uses CRLF for line breaks. Security considerations note that whilst CSV data are generally passive, there is a theoretical risk of malicious binary data being embedded to exploit buffer overruns in processing programmes, and that private data may be inadvertently shared through the format.