We run a chat application used in nine languages, and a surprising amount of its behaviour depends on regular expressions: recognising that a user said "yes", detecting that a message asks for a photo, catching a stated age in the safety layer. In one month we found the same defect three times, in three places, each time without an error, a log line or a failing test. A rule simply never fired.
The cause: in JavaScript, \b does not know about Unicode.
The behaviour
A word boundary \b is a position between a character in [A-Za-z0-9_] and one that is not. That definition is fixed: the u and v flags do not change it. Every accented Latin letter, and every letter of every other script, counts as a non-word character.
/\b(si|sì)\b/i.test('si') // true /\b(si|sì)\b/i.test('sì') // false: nothing after ì is a boundary /\b(yes|はい)\b/.test('はい') // false /\bâge\b/.test('âge') // false: nothing before â is a boundary either /\b(ok|好的)\b/.test('好的!') // false An alternative that ends with a non-ASCII character can never be followed by \b at the end of a string or before a space. The pattern reads correctly, compiles, and matches the ASCII alternatives next to it, which is exactly why nobody notices.
Three incidents in one codebase
- "Yes" in Arabic, Chinese, Japanese and Russian. The pattern that recognises a short "yes" had been extended with نعم, 好的, はい and да. None of them had ever matched.
- Accented Latin. sì (Italian) and sí (Spanish), the normal way to say yes in two of our languages, had never been recognised. The unaccented si worked, so every quick test looked fine.
- A safety rule. A deterministic rule that catches stated ages under 18 ended with
\bafter the Chinese, Japanese and Arabic words for "years old", so in those scripts it could not fire. (Our other moderation layers, including an independent classifier that does not use these patterns, were not affected.)
Six of our nine languages were affected somewhere.
The fix
Replace \b with lookarounds over Unicode property classes, and add the u flag:
// before const YES = /\b(yes|si|sì|sí|oui|ja)\b/i; // after: a "word character" is any letter or digit in any script const B_START = '(?<![\\p{L}\\p{N}_])'; const B_END = '(?![\\p{L}\\p{N}_])'; const YES = new RegExp(`${B_START}(yes|si|sì|sí|oui|ja)${B_END}`, 'iu'); YES.test('sì'); // true YES.test('sì!'); // true YES.test('sìx'); // false Chinese and Japanese need no trailing boundary at all. They do not put spaces between words, so a correct match is usually followed directly by another letter: 15歳です ("I am 15") has no break after 歳, and even the Unicode lookahead would reject it. Tokens in these scripts are specific enough to match on their own.
Finding it automatically
Reading the patterns does not work: they look right. A mechanical check does. This script scans a source tree for the two shapes of the bug:
// find-ascii-boundaries.mjs — usage: node find-ascii-boundaries.mjs src import fs from 'node:fs'; import path from 'node:path'; const nonAscii = (ch) => ch && ch.codePointAt(0) > 127; function* files(dir) { for (const e of fs.readdirSync(dir, { withFileTypes: true })) { const p = path.join(dir, e.name); if (e.isDirectory() && e.name !== 'node_modules') yield* files(p); else if (/\.(m?[jt]sx?)$/.test(e.name)) yield p; } } for (const f of files(process.argv[2] || '.')) { fs.readFileSync(f, 'utf8').split('\n').forEach((line, i) => { // 1) an alternative ending in a non-ASCII char right before ")\b" for (const m of line.matchAll(/\(([^()]*)\)\\b/g)) for (const alt of m[1].split('|')) if (nonAscii([...alt].pop())) console.log(`${f}:${i + 1} "${alt}" before )\\b`); // 2) \b written directly next to a non-ASCII character if (/[^\x00-\x7F]\\b|\\b[^\x00-\x7F]/.test(line)) console.log(`${f}:${i + 1} \\b touches non-ASCII`); }); } In our codebase it found one more case, in an unrelated rule, after all the manual fixes were considered complete.
What we changed in how we work
Every multilingual pattern is now tested with one real sentence per language before it counts as working, including the accented and non-Latin forms. If your app speaks anything other than English, run the script above on it: it takes a second, and the bugs it finds are the silent kind.
Originally published as a research note at lovora.ai/research/unicode-word-boundaries (CC BY 4.0).
