To reduce pandas memory usage, measure with df.memory_usage(deep=True), then fix dtypes column by column: low-cardinality text to category, small integers to int8/int16, scores to float32, date strings to datetime64, Y/N flags to bool, and pass those dtypes to read_csv. On a simulated 2,000,000-row stock mentions table this took memory from 682.6 MB to 76.1 MB (−88.9%) on pandas 3.0.6. It will not help much with columns that are almost all unique, such as URLs or headline IDs; leave those as the pyarrow-backed str dtype.
Facts
- What: an 8-item checklist for shrinking a pandas DataFrame, with each item measured on the same table.
- Who it is for: anyone holding a long news-mentions or event log in pandas that is close to filling RAM.
- Coverage: 2,000,000 rows, 8 columns (date, ticker, source, board, url, mentions, sentiment, is_dup), 300 placeholder tickers, 215 trading days.
- Window & units: memory in MB (1 MB = 1,048,576 bytes) from
memory_usage(deep=True), index excluded; timings are single runs. - Source & as-of: simulated with NumPy
default_rng(20261123), run on 2026-11-23 (Asia/Shanghai) with Python 3.13.5, pandas 3.0.6, NumPy 2.5.5, pyarrow 26.0.1.
All data here is synthetic. Tickers are random six-digit codes, mention counts are Poisson noise, and nothing in this post says anything about a real stock.
Why memory matters before the table is "too big"
The pandas guide on scaling to large datasets puts it plainly: "Even datasets that are a sizable fraction of memory become unwieldy, as some pandas operations need to make intermediate copies." A 700 MB frame on a 4 GB laptop is already in trouble once a merge or a groupby makes a copy. The same guide recommends the two moves this checklist builds on: categorical columns and numeric downcasting.
The test table
import numpy as np import pandas as pd rng = np.random.default_rng(20261123) N = 2_000_000 tickers = np.array([f"{i:06d}" for i in rng.choice(np.arange(1, 690000), 300, replace=False)]) sources = np.array([f"source_{i:02d}" for i in range(20)]) boards = np.array([f"board_{i:02d}" for i in range(40)]) days = pd.bdate_range("2026-01-05", "2026-10-30").strftime("%Y-%m-%d").to_numpy().astype("U10") raw = { "date": rng.choice(days, N), "ticker": rng.choice(tickers, N), "source": rng.choice(sources, N), "board": rng.choice(boards, N), "url": np.char.add("news/", rng.integers(0, 10**9, N).astype(str)), "mentions": rng.poisson(3, N).astype("int64"), "sentiment": rng.uniform(-1, 1, N), "is_dup": rng.choice(np.array(["Y", "N"]), N, p=[0.08, 0.92]), } df_obj = pd.DataFrame({k: (pd.Series(v, dtype=object) if v.dtype.kind == "U" else v) for k, v in raw.items()}) # pandas 2.x style: text as object df = pd.DataFrame(raw) # pandas 3.0 default: text columns become str (pyarrow-backed) To reduce pandas memory usage step by step, the checklist starts from df_obj, which is what most existing code on pandas 2.x holds. The pandas 3.0 default frame (df) is shown as a second baseline because it changes the starting point a lot.
The checklist at a glance: memory usage before and after each item
| # | Item | Column(s) | Before (MB) | After (MB) |
|---|---|---|---|---|
| 1 | Measure with deep=True | whole frame | 122.1 (shallow) | 682.6 (deep) |
| 2 | Low-cardinality text → category | ticker, source, board | 324.2 | 7.6 |
| 3 | Unique text stays str | url | 120.0 | 41.8 |
| 4 | Downcast integers | mentions | 15.3 | 1.9 |
| 5 | float64 → float32 | sentiment | 15.3 | 7.6 |
| 6 | Parse dates and flags once | date, is_dup | 207.9 | 17.2 |
| 7 | Put dtypes in read_csv | 6 columns from CSV | 202.0 | 32.4 |
| 8 | Align categories before concat | ticker | 26.7 | 3.8 |
1. Why does df.info() under-report pandas memory usage?
df.info() and memory_usage() default to a shallow count, which for object columns adds up only the 8-byte pointers, not the Python strings they point to.
print(df_obj.memory_usage(deep=False).sum() / 2**20) # 122.1 print(df_obj.memory_usage(deep=True).sum() / 2**20) # 682.6 df_obj.info(memory_usage="deep") # memory usage: 682.6 MB The DataFrame.memory_usage reference says deep=True will "introspect the data deeply by interrogating object dtypes for system-level memory consumption". Here the shallow figure was 5.6 times too low, and info() marks it with a + (122.1+ MB) that is easy to miss. Every number below uses deep=True.
2. Reduce memory usage of low-cardinality text with category
The biggest memory cut came from the three text columns with few distinct values. A category column stores each distinct value once plus a small integer code per row.
for c in ["ticker", "source", "board"]: df_obj[c] = df_obj[c].astype("category") | Column | Distinct values | object (MB) | str, pyarrow (MB) | category (MB) | Code dtype |
|---|---|---|---|---|---|
| ticker | 300 | 104.9 | 26.7 | 3.8 | int16 |
| source | 20 | 110.6 | 32.4 | 1.9 | int8 |
| board | 40 | 108.7 | 30.5 | 1.9 | int8 |
Even against the newer pyarrow str dtype, category was 7 to 17 times smaller. The board column in this table is random, but in real work the ticker-to-board mapping usually comes from a stock-to-concept-board lookup, and joining it in as a category keeps a 40-value column at 1.9 MB instead of 100+ MB.
3. When does category dtype use more memory than str?
Category loses when almost every value is different, because it stores all the values plus a code array on top.
df["url"].nunique() # 1,997,990 of 2,000,000 (99.9%) df["url"].astype("category").memory_usage(deep=True, index=False) / 2**20 # 49.6 The url column took 120.0 MB as object, 41.8 MB as pyarrow str and 49.6 MB as category. A simple rule from this run: only convert when distinct values are a small share of rows (here 300 out of 2,000,000 for ticker). For unique text on pandas 3.0, str is already the better choice; on pandas 2.x, astype("string[pyarrow]") gives the same storage.
4. Downcast integers to cut memory, then watch arithmetic
To reduce memory usage of count columns, pd.to_numeric(..., downcast="integer") picks the smallest integer type that fits, and the mentions column (max 15) went from int64 to int8, 15.3 MB to 1.9 MB.
m8 = pd.to_numeric(df_obj["mentions"], downcast="integer") # int8 m8.sum() # 6001870, returned as int64: reductions upcast m8.cumsum().dtype # int64 (m8 * 10).max() # 120 -> wrong, the true max is 150 Reductions and cumsum returned int64, so totals were correct. Element-wise arithmetic did not: m8 * 10 stayed int8 and wrapped past 127. In this run 30 of 2,000,000 rows came out wrong, with a minimum of −126. Thirty rows is easy to miss in a spot check, so cast up (m8.astype("int32") * 10) before scaling a downcast column.
5. Is float32 precise enough to halve the memory of sentiment scores?
For scores bounded in [−1, 1], float32 was enough here: the largest absolute error after the round trip was 2.98e-08 and the column mean matched to 10 decimal places.
s32 = df_obj["sentiment"].astype("float32") # 15.3 MB -> 7.6 MB (s32.astype("float64") - df_obj["sentiment"]).abs().max() # 2.98e-08 Keep float64 for values with more than about seven significant digits, such as cumulative amounts or IDs stored as floats. float32 halves memory; it does not shrink it further.
6. Parse dates and flags once to reduce memory
Date strings and Y/N flags look harmless, yet together they held 207.9 MB of the 682.6 MB object frame.
df_obj["date"] = pd.to_datetime(df_obj["date"], format="%Y-%m-%d") # 112.5 MB -> 15.3 MB df_obj["is_dup"] = df_obj["is_dup"].eq("Y") # 95.4 MB -> 1.9 MB Passing an explicit format avoids per-row guessing. After this step the frame was 76.1 MB: 88.9% below the object frame and 64.3% below the pandas 3.0 default frame (213.4 MB).
7. Reduce memory at load time: put dtypes into read_csv
The cheapest way to reduce pandas memory usage is to never load the wide dtypes at all. Reading the same table from a 144.3 MB CSV with defaults gave a 202.0 MB frame in 1.74 s and silently turned ticker '021674' into the integer 21674.
dt = {"ticker": "category", "source": "category", "board": "category", "mentions": "int8", "sentiment": "float32"} df2 = pd.read_csv("mentions.csv", usecols=["date", "ticker", "source", "board", "mentions", "sentiment"], dtype=dt, parse_dates=["date"]) With dtype=, usecols= and parse_dates= the result was 32.4 MB in 1.17 s and the leading zero survived. Saving the slim frame with to_parquet produced a 34.2 MB file, and read_parquet brought back every dtype (category, int8, float32, bool) without a second pass.
8. Keep categories aligned before concat so memory usage stays low
Two category columns with different category lists do not stay categorical when concatenated; on pandas 3.0.6 the ticker column fell back to str and grew from 3.8 MB to 26.7 MB.
from pandas.api.types import union_categoricals cats = union_categoricals([a["ticker"], b["ticker"]]).categories a["ticker"] = a["ticker"].cat.set_categories(cats) b["ticker"] = b["ticker"].cat.set_categories(cats) pd.concat([a, b], ignore_index=True)["ticker"].dtype # category, 3.8 MB The trigger in this test was a single extra ticker code in one half. Daily batch files hit it all the time, since a new listing or a newly tagged stock adds a category.
What the lower memory usage buys you
A frame at 76.1 MB leaves room for the groupby and merge copies the pandas guide warns about. The usual next step on a mentions log is to group by ticker and date and plot something like a daily mention heat series for one stock; with categorical keys, pass observed=True (the pandas 3.0 default) so only combinations that exist are returned. In this run groupby("board", observed=True) returned all 40 boards.
FAQ
Does pandas 3.0 make this checklist unnecessary?
No. The pyarrow-backed str default cut this table from 682.6 MB to 213.4 MB with no code changes, but the checklist took it to 76.1 MB.
Should I convert every text column to category?
No. Check nunique() first; the 99.9%-unique url column was larger as category (49.6 MB) than as str (41.8 MB).
Is downcasting safe for sums?
sum() and cumsum() upcast to int64 in this test. Element-wise multiplication did not, so cast up before scaling.
Do I need pyarrow?
For the str numbers above and for to_parquet, yes. Without it, pandas 3.0 falls back to Python-backed str storage; that setup was not measured here, so check it with memory_usage(deep=True) on your own machine.
snippets.py and snippets_output.txt reproduce every number in this post; make_figures.py draws both charts from the same run.
Lava (熔岩) — A-share stock heat & sentiment monitoring by AI-Dolphin. All data in this post is simulated, and nothing here is investment advice.
中文简介: 本文是一份 pandas 降内存清单,在一张模拟的 200 万行个股新闻提及表上逐项实测(pandas 3.0.6 + pyarrow):先用 memory_usage(deep=True) 看真实占用(浅统计低估 5.6 倍),再把低基数文本列转 category、整数降到 int8、情绪分转 float32、日期和 Y/N 标记各解析一次,并在 read_csv 里直接指定 dtype。全表从 682.6 MB 降到 76.1 MB;同时提醒几处坑:几乎全唯一的 URL 列转 category 反而更大,int8 乘法会悄悄溢出,类别不一致的列拼接后会退回 str。


