Jaccard similarity in pandas takes three lines once your data is in the right shape: turn a long table of (theme, ticker) pairs into a 0/1 incidence matrix with pd.crosstab, multiply it by its own transpose to get every pairwise intersection, and divide by the union |A| + |B| − |A∩B|. The result is a theme-by-theme matrix where 1 means "identical member list" and 0 means "nothing in common".
That matters whenever you work with concept or theme classifications. In A-share data, the same company can belong to a dozen concept boards, and two boards with different names ("computing power", "AI servers", "data center") often share most of their constituents. If you aggregate news attention by theme, highly overlapping themes will look like they move together simply because they contain the same stocks. Measuring the overlap first tells you which themes are genuinely distinct and which are near-duplicates.
This walkthrough uses synthetic data: 300 tickers and 20 themes, built so that some themes share a common core. It covers the vectorised Jaccard matrix, why you also want the overlap coefficient, how to tell real overlap from chance with a hypergeometric test, clustering themes into families, and tracking membership drift over time.
The data: a long membership table
Most sources deliver constituents as a long table, one row per membership. Here four "families" of three themes each draw from a shared core of 40 stocks, and eight unrelated themes are drawn at random.
import numpy as np import pandas as pd rng = np.random.default_rng(11) universe = [f"T{i:03d}" for i in range(300)] cores = {fam: rng.choice(universe, 40, replace=False) for fam in ["compute", "robotics", "storage", "power"]} themes = {} for fam, core in cores.items(): for v in range(3): keep = rng.choice(core, int(rng.integers(22, 36)), replace=False) extra = rng.choice(universe, int(rng.integers(5, 20)), replace=False) themes[f"{fam}_{v}"] = sorted(set(keep) | set(extra)) for j in range(8): themes[f"misc_{j}"] = sorted(rng.choice(universe, int(rng.integers(15, 60)), replace=False)) membership = pd.DataFrame([(t, s) for t, ss in themes.items() for s in ss], columns=["theme", "ticker"]) print(membership.shape, membership["theme"].nunique(), membership["ticker"].nunique()) # (777, 2) 20 276 Step 1: the incidence matrix and a vectorised Jaccard
Avoid the double loop over theme pairs with Python sets. It is fine for 20 themes but slow for several hundred. A matrix product computes every intersection at once.
X = pd.crosstab(membership["theme"], membership["ticker"]).clip(upper=1) A = X.to_numpy(dtype=np.int32) # shape (20, 276) inter = A @ A.T # |A ∩ B| for every pair sizes = A.sum(axis=1) # |A| union = sizes[:, None] + sizes[None, :] - inter J = pd.DataFrame(inter / union, index=X.index, columns=X.index) cols = ["compute_0", "compute_1", "robotics_0", "misc_0"] print(J.loc[cols, cols].round(3)) theme compute_0 compute_1 robotics_0 misc_0 compute_0 1.000 0.288 0.087 0.083 compute_1 0.288 1.000 0.056 0.066 robotics_0 0.087 0.056 1.000 0.032 misc_0 0.083 0.066 0.032 1.000 .clip(upper=1) protects against duplicate rows in the source, which would otherwise count a stock twice. To check the result against a reference implementation, SciPy's pdist with metric="jaccard" returns the Jaccard distance, so 1 - distance should match:
from scipy.spatial.distance import pdist, squareform D = squareform(pdist(A.astype(bool), metric="jaccard")) print(np.allclose(1 - D, J.to_numpy())) # True For a few thousand themes and tens of thousands of tickers, swap the dense array for scipy.sparse.csr_matrix(A). The product S @ S.T works the same way and keeps memory use proportional to the number of memberships.
Step 2: a pairs table you can sort and filter
A square matrix is hard to scan. Flatten its upper triangle into one row per pair and keep the raw counts next to the ratio:
iu = np.triu_indices_from(J, k=1) pairs = pd.DataFrame({ "a": J.index[iu[0]], "b": J.columns[iu[1]], "jaccard": J.to_numpy()[iu], "overlap": inter[iu], "size_a": sizes[iu[0]], "size_b": sizes[iu[1]], }) pairs["containment"] = pairs["overlap"] / np.minimum(pairs["size_a"], pairs["size_b"]) print(pairs.sort_values("jaccard", ascending=False).head(5).round(3).to_string(index=False)) a b jaccard overlap size_a size_b containment robotics_0 robotics_2 0.382 21 37 39 0.568 robotics_1 robotics_2 0.379 22 41 39 0.564 compute_1 compute_2 0.365 19 38 33 0.576 power_1 power_2 0.346 18 40 30 0.600 storage_1 storage_2 0.339 20 45 34 0.588 Step 3: Jaccard vs overlap coefficient
Jaccard penalises size differences. If a small theme of 30 stocks sits entirely inside a broad theme of 150, Jaccard is only 0.2, even though every member of the small theme is in the big one. The overlap coefficient (also called containment or the Szymkiewicz–Simpson coefficient), |A∩B| / min(|A|, |B|), is 1.0 in that case.
Use both, for different questions:
- Jaccard answers "are these two lists essentially the same?" Use it for de-duplicating themes.
- Overlap coefficient answers "is one list mostly contained in the other?" Use it for finding sub-themes and parent themes.
Sorted by containment, power_0 / power_2 comes first at 0.667: two thirds of the smaller theme sits inside the larger one, although their Jaccard is only 0.333.
Step 4: is the overlap more than chance?
Two large themes drawn from a small universe will share members by chance. A raw Jaccard of 0.18 might be meaningful for two 20-stock themes and meaningless for two 150-stock themes. Two quick corrections:
Expected Jaccard under random membership. With universe size N, the expected intersection is |A|·|B|/N, which gives a baseline and a lift ratio:
N = len(universe) e_int = pairs["size_a"] * pairs["size_b"] / N pairs["j_exp"] = e_int / (pairs["size_a"] + pairs["size_b"] - e_int) pairs["lift"] = pairs["jaccard"] / pairs["j_exp"] The 12 within-family pairs come out at a lift of 4.0–5.8, so they overlap four to six times more than random lists of the same size would.
A hypergeometric p-value. The probability of seeing at least this much overlap if B were a random draw from the universe:
from scipy.stats import hypergeom pairs["p_hyper"] = hypergeom.sf(pairs["overlap"] - 1, N, pairs["size_a"], pairs["size_b"]) print((pairs["p_hyper"] < 0.001).sum(), "of", len(pairs)) # 12 of 190 Exactly the 12 within-family pairs pass p < 0.001. The strongest chance-only pair, misc_3 / storage_1, has a Jaccard of 0.18 (the highest of any cross-family pair) and p = 0.004. That would look meaningful if you ranked by Jaccard alone, but it falls well short once you account for misc_3 holding 58 stocks. With 190 pairs, use a strict threshold or a Benjamini–Hochberg correction (scipy.stats.false_discovery_control) rather than 0.05.
Step 5: cluster themes into families
Jaccard distance (1 − J) is a proper metric, so it plugs straight into hierarchical clustering:
from scipy.cluster.hierarchy import linkage, fcluster Z = linkage(pdist(A.astype(bool), metric="jaccard"), method="average") families = pd.Series(fcluster(Z, t=0.75, criterion="distance"), index=X.index) print(families.groupby(families).apply(lambda s: list(s.index)).to_string()) With a cut at distance 0.75 (Jaccard ≥ 0.25 on average within a cluster), the four designed families come back as four clusters of three, and each misc_* theme stays on its own. On real data, scipy.cluster.hierarchy.dendrogram(Z, labels=X.index) shows where a natural cut lies. Pick it from the plot rather than fixing it in advance.
Step 6: track membership drift between snapshots
The same function measures how much one theme's constituent list changes over time. Store dated snapshots and compare each to the previous one:
def jaccard(a, b): a, b = set(a), set(b) return len(a & b) / len(a | b) if a | b else np.nan old = set(themes["compute_0"]) r = np.random.default_rng(5) drop = set(r.choice(sorted(old), 6, replace=False)) add = set(r.choice([u for u in universe if u not in old], 9, replace=False)) new = (old - drop) | add print(len(old), len(new), round(jaccard(old, new), 3)) # 38 41 0.681 Removing 6 and adding 9 members to a 38-stock theme drops snapshot-to-snapshot Jaccard to 0.68. A rule such as "flag any theme whose Jaccard to last week's list falls below 0.8" catches reclassifications that would otherwise silently change a theme-level time series. It also means historical theme aggregates should use the membership that was valid on each date, not today's list.
Practical notes
- Normalise tickers first. Mixed formats (
600000,600000.SH,SH600000) split one stock into several columns and depress every similarity. - Empty or tiny themes. A theme with 2 members produces extreme ratios. Set a minimum size before ranking pairs.
- Weighted overlap. If memberships carry weights (relevance scores, index weights), the weighted Jaccard
Σ min(w_a, w_b) / Σ max(w_a, w_b)is the natural extension. It works on the same matrix withnp.minimumandnp.maximumbroadcasts. - Stock-level view.
A.Tgives the ticker-by-theme matrix, and the same code measures which stocks share the most themes, which is useful when picking a diverse comparison basket.
FAQ
How do I calculate Jaccard similarity between two columns in pandas?
For two sets of values, use len(set(a) & set(b)) / len(set(a) | set(b)). For many groups at once, build a 0/1 matrix with pd.crosstab and use the matrix-product approach above.
Does scikit-learn have Jaccard similarity?
sklearn.metrics.jaccard_score compares label vectors in classification. For pairwise set similarity, scipy.spatial.distance.pdist(..., metric="jaccard") or sklearn.metrics.pairwise_distances(..., metric="jaccard") on a boolean matrix return distances; subtract them from 1.
What is a "high" Jaccard similarity for concept lists?
It depends on list sizes and universe size, which is why the expected value and hypergeometric test help. As a rough guide, above 0.5 usually means near-duplicate themes, and a lift below 2 means the overlap is close to what chance would produce.
Should overlapping themes be merged?
Not necessarily. They can carry different narratives even with shared members. Keep them separate, but account for the overlap when comparing their attention series, for example by also reporting a version that excludes the shared constituents.
Conclusion
Jaccard similarity in pandas is a matrix product and a division, and it scales to thousands of themes with a sparse matrix. The useful analysis comes from what you put around it: the overlap coefficient for containment, an expected-value baseline and hypergeometric test to separate real overlap from chance, hierarchical clustering to group near-duplicate themes, and snapshot-to-snapshot Jaccard to catch membership changes. Run these checks before comparing theme-level attention, so that two series do not look related just because the themes share stocks. To browse concept memberships for A-share stocks, try the concept stocks lookup in Lava.
中文简介:本文讲如何用 pandas 和 SciPy 批量计算概念/题材成分股列表之间的 Jaccard 相似度:用 crosstab 构建 0/1 关联矩阵,用矩阵乘法一次算出全部两两交集;再介绍重叠系数(包含度)、随机基准与超几何检验、基于 Jaccard 距离的层次聚类,以及用快照间 Jaccard 追踪成分股变动。数据为合成样本,仅作方法演示。