To reduce pandas memory usage, measure with df.memory_usage(deep=True), then fix dtypes column by column: low-cardinality text to category, small integers to int8/int16, scores to float32, date strings to datetime64, Y/N flags to bool, and pass those dtypes to read_csv. On a simulated 2,000,000-row stock mentions table this took memory from 682.6 MB to 76.1 MB (−88.9%) on pandas 3.0.6. It will not help much with columns that are almost all unique, such as URLs or headline IDs; leave those as the pyarrow-backed str dtype.

Facts

  • What: an 8-item checklist for shrinking a pandas DataFrame, with each item measured on the same table.
  • Who it is for: anyone holding a long news-mentions or event log in pandas that is close to filling RAM.
  • Coverage: 2,000,000 rows, 8 columns (date, ticker, source, board, url, mentions, sentiment, is_dup), 300 placeholder tickers, 215 trading days.
  • Window & units: memory in MB (1 MB = 1,048,576 bytes) from memory_usage(deep=True), index excluded; timings are single runs.
  • Source & as-of: simulated with NumPy default_rng(20261123), run on 2026-11-23 (Asia/Shanghai) with Python 3.13.5, pandas 3.0.6, NumPy 2.5.5, pyarrow 26.0.1.

All data here is synthetic. Tickers are random six-digit codes, mention counts are Poisson noise, and nothing in this post says anything about a real stock.

Why memory matters before the table is "too big"

The pandas guide on scaling to large datasets puts it plainly: "Even datasets that are a sizable fraction of memory become unwieldy, as some pandas operations need to make intermediate copies." A 700 MB frame on a 4 GB laptop is already in trouble once a merge or a groupby makes a copy. The same guide recommends the two moves this checklist builds on: categorical columns and numeric downcasting.

The test table

import numpy as np import pandas as pd rng = np.random.default_rng(20261123) N = 2_000_000 tickers = np.array([f"{i:06d}" for i in rng.choice(np.arange(1, 690000), 300, replace=False)]) sources = np.array([f"source_{i:02d}" for i in range(20)]) boards = np.array([f"board_{i:02d}" for i in range(40)]) days = pd.bdate_range("2026-01-05", "2026-10-30").strftime("%Y-%m-%d").to_numpy().astype("U10") raw = { "date": rng.choice(days, N), "ticker": rng.choice(tickers, N), "source": rng.choice(sources, N), "board": rng.choice(boards, N), "url": np.char.add("news/", rng.integers(0, 10**9, N).astype(str)), "mentions": rng.poisson(3, N).astype("int64"), "sentiment": rng.uniform(-1, 1, N), "is_dup": rng.choice(np.array(["Y", "N"]), N, p=[0.08, 0.92]), } df_obj = pd.DataFrame({k: (pd.Series(v, dtype=object) if v.dtype.kind == "U" else v) for k, v in raw.items()}) # pandas 2.x style: text as object df = pd.DataFrame(raw) # pandas 3.0 default: text columns become str (pyarrow-backed) 

To reduce pandas memory usage step by step, the checklist starts from df_obj, which is what most existing code on pandas 2.x holds. The pandas 3.0 default frame (df) is shown as a second baseline because it changes the starting point a lot.

The checklist at a glance: memory usage before and after each item

# Item Column(s) Before (MB) After (MB)
1 Measure with deep=True whole frame 122.1 (shallow) 682.6 (deep)
2 Low-cardinality text → category ticker, source, board 324.2 7.6
3 Unique text stays str url 120.0 41.8
4 Downcast integers mentions 15.3 1.9
5 float64 → float32 sentiment 15.3 7.6
6 Parse dates and flags once date, is_dup 207.9 17.2
7 Put dtypes in read_csv 6 columns from CSV 202.0 32.4
8 Align categories before concat ticker 26.7 3.8

Grouped horizontal bar chart of per-column memory in MB for a simulated 2,000,000-row stock mentions table, comparing object dtype (grey), the pandas 3.0 pyarrow str default (blue) and the frame after the checklist (orange). ticker, source and board drop from about 105-111 MB as object and 27-32 MB as str to 3.8, 1.9 and 1.9 MB as category. url stays at 41.8 MB as str versus 120.0 MB as object. date falls from 112.5 MB to 15.3 MB as datetime64, is_dup from 95.4 MB to 1.9 MB as bool, mentions from 15.3 MB to 1.9 MB as int8 and sentiment from 15.3 MB to 7.6 MB as float32.

1. Why does df.info() under-report pandas memory usage?

df.info() and memory_usage() default to a shallow count, which for object columns adds up only the 8-byte pointers, not the Python strings they point to.

print(df_obj.memory_usage(deep=False).sum() / 2**20) # 122.1 print(df_obj.memory_usage(deep=True).sum() / 2**20) # 682.6 df_obj.info(memory_usage="deep") # memory usage: 682.6 MB 

The DataFrame.memory_usage reference says deep=True will "introspect the data deeply by interrogating object dtypes for system-level memory consumption". Here the shallow figure was 5.6 times too low, and info() marks it with a + (122.1+ MB) that is easy to miss. Every number below uses deep=True.

2. Reduce memory usage of low-cardinality text with category

The biggest memory cut came from the three text columns with few distinct values. A category column stores each distinct value once plus a small integer code per row.

for c in ["ticker", "source", "board"]: df_obj[c] = df_obj[c].astype("category") 
Column Distinct values object (MB) str, pyarrow (MB) category (MB) Code dtype
ticker 300 104.9 26.7 3.8 int16
source 20 110.6 32.4 1.9 int8
board 40 108.7 30.5 1.9 int8

Even against the newer pyarrow str dtype, category was 7 to 17 times smaller. The board column in this table is random, but in real work the ticker-to-board mapping usually comes from a stock-to-concept-board lookup, and joining it in as a category keeps a 40-value column at 1.9 MB instead of 100+ MB.

3. When does category dtype use more memory than str?

Category loses when almost every value is different, because it stores all the values plus a code array on top.

df["url"].nunique() # 1,997,990 of 2,000,000 (99.9%) df["url"].astype("category").memory_usage(deep=True, index=False) / 2**20 # 49.6 

The url column took 120.0 MB as object, 41.8 MB as pyarrow str and 49.6 MB as category. A simple rule from this run: only convert when distinct values are a small share of rows (here 300 out of 2,000,000 for ticker). For unique text on pandas 3.0, str is already the better choice; on pandas 2.x, astype("string[pyarrow]") gives the same storage.

4. Downcast integers to cut memory, then watch arithmetic

To reduce memory usage of count columns, pd.to_numeric(..., downcast="integer") picks the smallest integer type that fits, and the mentions column (max 15) went from int64 to int8, 15.3 MB to 1.9 MB.

m8 = pd.to_numeric(df_obj["mentions"], downcast="integer") # int8 m8.sum() # 6001870, returned as int64: reductions upcast m8.cumsum().dtype # int64 (m8 * 10).max() # 120 -> wrong, the true max is 150 

Reductions and cumsum returned int64, so totals were correct. Element-wise arithmetic did not: m8 * 10 stayed int8 and wrapped past 127. In this run 30 of 2,000,000 rows came out wrong, with a minimum of −126. Thirty rows is easy to miss in a spot check, so cast up (m8.astype("int32") * 10) before scaling a downcast column.

5. Is float32 precise enough to halve the memory of sentiment scores?

For scores bounded in [−1, 1], float32 was enough here: the largest absolute error after the round trip was 2.98e-08 and the column mean matched to 10 decimal places.

s32 = df_obj["sentiment"].astype("float32") # 15.3 MB -> 7.6 MB (s32.astype("float64") - df_obj["sentiment"]).abs().max() # 2.98e-08 

Keep float64 for values with more than about seven significant digits, such as cumulative amounts or IDs stored as floats. float32 halves memory; it does not shrink it further.

6. Parse dates and flags once to reduce memory

Date strings and Y/N flags look harmless, yet together they held 207.9 MB of the 682.6 MB object frame.

df_obj["date"] = pd.to_datetime(df_obj["date"], format="%Y-%m-%d") # 112.5 MB -> 15.3 MB df_obj["is_dup"] = df_obj["is_dup"].eq("Y") # 95.4 MB -> 1.9 MB 

Passing an explicit format avoids per-row guessing. After this step the frame was 76.1 MB: 88.9% below the object frame and 64.3% below the pandas 3.0 default frame (213.4 MB).

Bar chart of total DataFrame memory after each checklist step on simulated data: 682.6 MB with object dtypes, 366.0 MB after converting ticker, source and board to category, 287.8 MB after url to pyarrow str, 274.5 MB after mentions to int8, 266.8 MB after sentiment to float32 and 76.1 MB after parsing date to datetime64 and is_dup to bool. A dashed blue line marks the pandas 3.0 default str frame at 213.4 MB.

7. Reduce memory at load time: put dtypes into read_csv

The cheapest way to reduce pandas memory usage is to never load the wide dtypes at all. Reading the same table from a 144.3 MB CSV with defaults gave a 202.0 MB frame in 1.74 s and silently turned ticker '021674' into the integer 21674.

dt = {"ticker": "category", "source": "category", "board": "category", "mentions": "int8", "sentiment": "float32"} df2 = pd.read_csv("mentions.csv", usecols=["date", "ticker", "source", "board", "mentions", "sentiment"], dtype=dt, parse_dates=["date"]) 

With dtype=, usecols= and parse_dates= the result was 32.4 MB in 1.17 s and the leading zero survived. Saving the slim frame with to_parquet produced a 34.2 MB file, and read_parquet brought back every dtype (category, int8, float32, bool) without a second pass.

8. Keep categories aligned before concat so memory usage stays low

Two category columns with different category lists do not stay categorical when concatenated; on pandas 3.0.6 the ticker column fell back to str and grew from 3.8 MB to 26.7 MB.

from pandas.api.types import union_categoricals cats = union_categoricals([a["ticker"], b["ticker"]]).categories a["ticker"] = a["ticker"].cat.set_categories(cats) b["ticker"] = b["ticker"].cat.set_categories(cats) pd.concat([a, b], ignore_index=True)["ticker"].dtype # category, 3.8 MB 

The trigger in this test was a single extra ticker code in one half. Daily batch files hit it all the time, since a new listing or a newly tagged stock adds a category.

What the lower memory usage buys you

A frame at 76.1 MB leaves room for the groupby and merge copies the pandas guide warns about. The usual next step on a mentions log is to group by ticker and date and plot something like a daily mention heat series for one stock; with categorical keys, pass observed=True (the pandas 3.0 default) so only combinations that exist are returned. In this run groupby("board", observed=True) returned all 40 boards.

FAQ

Does pandas 3.0 make this checklist unnecessary?
No. The pyarrow-backed str default cut this table from 682.6 MB to 213.4 MB with no code changes, but the checklist took it to 76.1 MB.

Should I convert every text column to category?
No. Check nunique() first; the 99.9%-unique url column was larger as category (49.6 MB) than as str (41.8 MB).

Is downcasting safe for sums?
sum() and cumsum() upcast to int64 in this test. Element-wise multiplication did not, so cast up before scaling.

Do I need pyarrow?
For the str numbers above and for to_parquet, yes. Without it, pandas 3.0 falls back to Python-backed str storage; that setup was not measured here, so check it with memory_usage(deep=True) on your own machine.

snippets.py and snippets_output.txt reproduce every number in this post; make_figures.py draws both charts from the same run.

Lava (熔岩) — A-share stock heat & sentiment monitoring by AI-Dolphin. All data in this post is simulated, and nothing here is investment advice.

中文简介: 本文是一份 pandas 降内存清单,在一张模拟的 200 万行个股新闻提及表上逐项实测(pandas 3.0.6 + pyarrow):先用 memory_usage(deep=True) 看真实占用(浅统计低估 5.6 倍),再把低基数文本列转 category、整数降到 int8、情绪分转 float32、日期和 Y/N 标记各解析一次,并在 read_csv 里直接指定 dtype。全表从 682.6 MB 降到 76.1 MB;同时提醒几处坑:几乎全唯一的 URL 列转 category 反而更大,int8 乘法会悄悄溢出,类别不一致的列拼接后会退回 str。