A dataset with 109 rows and 16 fields should not need a user manual. And yet five of those fields will give you quietly wrong answers if you treat them like ordinary columns.
This is a guide to the five traps in the Chinese Surnames Open Dataset, written because I have now watched each of them happen — once to a reader, twice to myself. None of them throw an exception. That's what makes them interesting.
1. slug is a join key, not a pronunciation index
The dataset has both pinyin (lǐ, lǚ) and slug (li3, lv3). The slug is tone-numbered ASCII and doubles as the detail-page URL. It is great for exact joins and terrible for sound-based matching.
A reader asked me why /^li/ only returned five surnames when there are clearly more li-sounding ones. Because the filter is literal:
li3 李 liang2 梁 liao4 廖 lin2 林 liu2 刘 lv3 吕 lu2 卢 lu4 陆 luo2 罗 /^li/ catches the first five. 吕 is lv3 (ü written as v), and 卢/陆/罗 are lu*/luo*. There is no single prefix for a sound.
If you genuinely want sound-based grouping, normalize the tone marks rather than guessing prefixes:
import unicodedata def sound_key(p): # ü and its toned forms become v, then strip combining tone marks p = p.translate(str.maketrans({ "ü": "v", "ǖ": "v", "ǘ": "v", "ǚ": "v", "ǜ": "v", })) return "".join( c for c in unicodedata.normalize("NFKD", p) if not unicodedata.combining(c) ) sound_key("lǚ") # 'lv' sound_key("lú") # 'lu' Then group on sound_key(pinyin). But for joins, just use slug exactly. It's the only field guaranteed to be unique and stable across versions.
2. A compound surname is one record, not two characters
30 of the 109 records have a two-character surname: 欧阳, 司马, 诸葛, 尉迟, 万俟. surname is a string, and len() on it is a perfectly reasonable way to find them.
compounds = df[df["surname"].str.len() == 2] # 30 rows What breaks is the instinct to take the first character as a key. There is no standalone 欧 in this dataset — only 欧阳 — so a join on 欧 silently drops the row. And 万俟 is read mò qí, not wàn sì, so generating its slug from the characters gives you a key that matches nothing.
The warning extends beyond compounds: several surnames have two readings depending on context. Never build your own key from the display string. slug already exists for that.
3. Null means unknown. Zero means zero people.
This is the one that corrupts aggregates without throwing anything.
| Category | Records | What the population fields hold |
|---|---|---|
mps2020 | 12 | Official figure from the 2020 report's compound-surname section |
estimate | 30 | Other statistics / estimates of varying vintage |
qualitative | 18 | "人数稀少" — rare, no number |
| empty source | 49 | Nothing at all |
population_est is non-null for only 42 records. The mean of those 42 is about 20.4 million. Fill the other 67 with zero and the mean drops to 7.8 million — a 61% understatement, produced by a single innocent-looking fillna(0).
The right default is to leave the nulls alone and state the denominator:
df["population_est"].mean() # 20,365,190 over 42 rows df["population_est"].fillna(0).mean() # 7,847,138 over 109 rows — wrong df["population_est"].count() # 42 — say this out loud The same rule applies to rank_num. Zero is not rank zero; it's no rank.
4. The two population columns are not one column
The newest trap, and the one I found the hard way. rank_num and population_est look like a pair — rank and figure, figure and rank — but they come from different sources. The 2020 MPS report ranks the top 100 surnames and prints no headcounts for them; the headcounts in this file come from other estimates.
Sort by population_est and the order contradicts the official ranking five times:
林 #18: 14,200,000 > 何 #17: 13,000,000 高 #20: 14,500,000 > 罗 #19: 14,000,000 宋 #24: 11,200,000 > 谢 #23: 10,500,000 韩 #27: 9,800,000 > 许 #26: 9,000,000 冯 #29: 9,500,000 > 邓 #28: 9,200,000 So: sort on rank_num. Use population_est for order of magnitude only. And if you need to say "according to the 2020 report", filter first:
official = df[df["population_source"] == "mps2020"] # 12 rows Those 12 rows are the only figures in the file that are printed in that report verbatim. (The full story of how that got mislabeled is here.)
5. ancestor and origin_note are multi-valued in disguise
Both look like prose, which means it is tempting to treat them as a single fact. They are not.
Six records carry two documented founding ancestors in one string — 白 is "白公胜、公子白", 吴 is "泰伯、仲雍" — because the sources genuinely disagree. 45 origin_note fields contain 另有 ("another account holds…"), laying out competing branches in parallel. This is deliberate: forcing one root per surname would be a claim the literature doesn't support.
multi_ancestor = df[df["ancestor"].str.contains("、", regex=False)] # 6 multi_origin = df[df["origin_note"].str.contains("另有", regex=False)] # 45 If your pipeline expects one ancestor per surname, split on 、 and keep both values — or keep the string and mark it multi-valued. Do not pick the first one and call it resolved.
Three questions, ten lines
Putting it together:
import pandas as pd df = pd.read_csv("https://naming-cn.pages.dev/data/surnames.csv") top10 = (df.dropna(subset=["rank_num"]) .nsmallest(10, "rank_num")[["surname", "rank_num", "population_source"]]) compound_with_figures = (df[df["surname"].str.len() == 2] .dropna(subset=["population_est"]) [["surname", "population"]]) # 12 rows, all official disputed = df[df["origin_note"].str.contains("另有", regex=False)] # 45 rows No regex on Chinese numerals, no guessing at compound boundaries, no invented zeros.
The general version
Every field that can be "unknown" needs a null, not a sentinel. Every number needs its provenance in the same row as the value. Every key needs to be generated, not re-derived from a display string. And every dataset needs one check that can fail — in this case the sync script sorts by rank and refuses to let the five inversions become six.
The machine-readable schema, the validator and a local mirror are in the GitHub repo. Data is CC BY 4.0: use it, modify it, redistribute it, with attribution and a licence link.
Corrections are welcome. If you find a sixth trap, that's a new version number, not a comment.