Technology Tales

How to Check if a File or Directory Exists in Python

Python offers several methods for checking whether a file or directory exists, each suited to different scenarios. The simplest approach requires no imported modules and works across Python 2 and 3, using a try-except block to attempt opening a file and catching an IOError if it is not found. Using the with keyword alongside this method ensures the file is properly closed after operations are completed.

For situations where a developer needs to verify a file's existence before performing actions such as copying or deleting, the os.path module offers useful functions including os.path.exists, os.path.isfile and os.path.isdir, all of which are compatible with both Python versions. A more modern alternative is the pathlib module, available in Python 3.4 and above, which takes an object-oriented approach and allows developers to work with file paths as Path objects rather than plain strings, though it can be installed for Python 2 via pip. A notable consideration when checking for file existence is the risk of race conditions, which can occur when multiple processes access the same file in the time between a check and a subsequent operation.

Vaex: Pandas but 1000x faster

Vaex is a Python library designed to handle large datasets far more efficiently than Pandas, capable of processing up to one billion rows per second through memory mapping and lazy computations, meaning it avoids copying data unless explicitly instructed to do so. Unlike Pandas, which struggles with memory limitations and slow processing speeds on large datasets, Vaex can work with datasets as large as the available hard drive space.

It can be installed via pip or conda and supports reading from CSV and HDF5 file formats, with benchmarks showing it reads files dramatically faster than Pandas. The library offers a broad range of functionality including statistical operations such as correlation, covariance and groupby aggregations, as well as data cleaning tools for handling missing values and dropping columns. It also includes string operation methods, plotting capabilities for one and two-dimensional visualisations and a virtual columns feature that allows expressions to be stored and computed on the fly without consuming additional memory.

Sample 24820: Creating a Directory Listing Using SAS for Windows

Creating a directory listing using SAS for Windows allows users to document project structures by generating lists of files and folders, which can be annotated for clarity. This is achieved by invoking the DOS DIR command through a FILENAME statement with the pipe device type, enabling the processing of directory information within a data step.

The %DIRLISTWIN macro further enhances this process by filtering files based on size, date, or subdirectory inclusion and producing reports or datasets with details such as file paths, sises, dates and owners. This tool is particularly useful for efficiently locating files within large directories or specific timeframes, though execution time varies depending on the complexity of the selected path.

Python Data Wrangling Solutions — Dynamically Creating Variables When Slicing Data Frames

When working on data science projects, a significant portion of time is spent on data wrangling, and one common challenge is splitting a single dataframe into multiple dataframes based on the categorical values of a variable. While manual splitting is feasible for a few categories, it becomes impractical when dealing with tens or hundreds of distinct values. A practical workaround in Python involves using dictionaries, where each key represents a unique category and its corresponding value holds the relevant dataframe slice. The process involves importing a dataset using Pandas, applying the groupby method to the chosen categorical column, converting the resulting object into a tuple to pair each category with its associated data, and finally converting that tuple into a dictionary. This approach replicates the outcome of creating individual variables for each category, with the dictionary keys serving in place of distinct variable names, allowing the sliced data to be accessed cleanly and efficiently regardless of how many categories exist.

% Macro Core - Production Ready Macros for SAS Application Developers

The Macro Core library is an open-source, MIT-licensed collection of production-quality SAS macros designed to reduce development time and effort for application developers working on the SAS platform. The library is organised into several folders, each targeting a specific platform or environment, including BASE for all platforms, META and METAX for SAS 9 environments, VIYA for SAS Viya, SERVER for the open-source SASjs REST API and XPLATFORM for macros that function across multiple server types.

It also incorporates LUA and FCMP components, allowing developers to embed LUA modules within SAS macros and generate compiled functions, respectively. The entire library can be downloaded and compiled with just two lines of SAS code, and installation involves updating the sasautos path to include the relevant folders.

Strict coding and documentation standards are enforced, including Doxygen-formatted headers, two-space indentation, lowercase filenames, one macro per file and clearly defined naming prefixes for each category. Dependencies must be declared explicitly in macro headers to support the SASjs command-line interface, which can extract and insert them automatically during project compilation.

How to Update All Python Packages

Updating Python packages involves using tools like pip to maintain environment stability and security, with best practices including pinning versions in requirements.txt files to ensure reproducibility. Outdated packages can be identified using pip list --outdated and upgraded through commands tailored to operating systems such as Windows PowerShell or Linux utilities like grep and awk.

Virtual environments require specific scripts or Pipenv commands for updates, while the ActiveState Platform offers an alternative method for managing dependencies and resolving conflicts, though its use is optional. The process highlights the importance of careful upgrades to avoid breaking dependencies, with considerations for both development and production environments.

Python os.system() method

The OS module in Python offers functions for interacting with the operating system, providing a portable way to access system-dependent functionality. The os.system() method executes a command string in a subshell by calling the Standard C system() function, which has inherent limitations.

It sends any generated output to the interpreter's standard output stream and opens the relevant operating system shell to execute the command. The method's syntax involves passing a string parameter representing the command, with return values dependent on the operating system; Unix returns the exit status of the process, while Windows returns the shell's output. Examples include running system-specific commands such as retrieving the current date or launching applications like Notepad on Windows, demonstrating its utility for interacting with the underlying operating system through Python.

Pandas Split strings into two List/Columns using str.split()

The Pandas str.split() method enables splitting string data in a DataFrame column using a specified delimiter, allowing results to be stored as lists within a Series or expanded into separate columns for structured analysis.

By setting the expand parameter to True, strings can be divided into multiple columns, as demonstrated by splitting full names into first and last names, while expand=False retains split values as lists.

Additional flexibility is achieved by combining str.split() with the apply() function for custom splitting logic, such as dynamically separating complex strings into distinct parts. This technique is particularly useful for reorganising textual data in preparation for further processing or analysis.

SettingwithCopyWarning: How to Fix This Warning in Pandas

The SettingWithCopyWarning in Pandas arises from the ambiguity of whether operations on dataframes modify the original data or create copies. This warning was introduced to address silent failures in chained assignments, where changes to a subset of data might not propagate back to the source.

The root of the issue lies in design of Pandas, which balances flexibility with the efficiency of the underlying array structures of NumPy. When slices of a dataframe contain a single data type, they can be returned as views, which are memory-efficient but may lead to unintended side effects if modified.

Multi-type slices, however, require copies, which are safer but less efficient. Developers are advised to avoid chained indexing, which can obscure whether changes affect the original data or a copy. Instead, using .copy() explicitly ensures modifications are applied to a separate instance, or working directly on the original dataframe with loc or iloc maintains clarity.

Understanding this warning is crucial for reliable data manipulation, as it highlights the need for intentional coding practices. The evolution of Pandas, from its early reliance on the ix indexer to the preference for loc and iloc, reflects a broader effort to make indexing more predictable. While the warning may seem cumbersome, it serves as a safeguard against subtle bugs, reinforcing the importance of deliberate data handling in analysis workflows.

Create a nice looking table using R

Make Beautiful Tables with the Formattable Package

Create stylish tables in R using formattable