Benford's Law Application and Interpretation

Sat, 10 Feb 2024 00:00:00 +0000

Benford's Law, also known as the Newcomb-Benford law or the first-digit law, is a surprising observation about the leading digits of numbers in real-world datasets. In many naturally occurring collections of data, smaller leading digits (like 1 and 2) are significantly more common than larger ones (like 8 and 9).

Financial records
Scientific measurements
Astronomical distances
Street addresses

Why does this happen?¶

Real-world data often involves growth, multiplication, and comparisons across different scales. This "scaling invariance" creates a natural bias towards smaller leading digits.

An In-depth Comparison of Pandas String dtypes

Sat, 27 May 2023 00:00:00 +0000

In [1]:

import sys
import pandas as pd
import numpy as np

print(f"python version {sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}")
print(f"pandas version {pd.__version__}")
print(f"numpy version {np.__version__}")

python version 3.10.11
pandas version 1.5.2
numpy version 1.23.5

The default data type for strings in Pandas DataFrames is the object type. However, pandas' documentation recommendeds explicitly using the StringDtype for storing strings as it's more efficient and allows for more specific string operations.

pd.StringDtype() is a dedicated data type for storing strings. It is an It allows for more specific string operations. StringDtype is still considered experimental as of pandas 2.0.1.

Pandas on Ye Joo Park's Blog

Benford's Law Application and Interpretation

Why does this happen?¶

An In-depth Comparison of Pandas String dtypes