Half a confidence interval is not a unit of uncertainty
A checked reading of 24 published ratios from a nicotine pouch study on Meta, and the table its authors print that changes the headline.
On 20 November, Sun and colleagues posted a preprint on medRxiv: nicotine pouch adverts on Meta reached about twelve times as many young men as young women aged 18 to 24. The paper prints RR 12.5, 95% CI 10.9–14.4. That is a strong, well built number, and nothing below dents it.
This piece is about the range around it. For a ratio, a confidence interval is symmetric when you multiply and divide, not when you add and subtract. So half the printed width is not a unit of uncertainty. Build a rule on that half-width and it fails exactly where the width is doing real work: on the intervals widened by clustering, which carry a long upper tail.
I know because I built that rule, applied it to this paper, and got the ordering of the rows wrong. The correction is a footnote here, and the dated record is linked at the end.
The check takes three lines
Every interval in this paper is a ratio: a rate ratio, or a ratio of ratios. Take the two ends of each printed range, multiply them, take the square root, compare with the printed estimate.
-
12.5 (10.9, 14.4)gives 12.53 -
7.91 (6.56, 9.50)gives 7.89 -
0.19 (0.17, 0.21)gives 0.189
Across all 24 rows in the main text the geometric middle matches the estimate to within 1.5% in 22 of them. The worst is 2.5%, the male-to-female ratio at 65 and over, which is a little asymmetric even in log. The arithmetic middle, (low + high) / 2, matches nothing.
So the scale the intervals were built on is log, and the unit is the standard error on that scale:
SE(log) = ln(upper / lower) / (2 × 1.96), and z = ln(estimate) / SE(log)
1 is inside the interval exactly when |z| < 1.96. That single number answers both things a reader wants from a ratio: how big is the gap, and is its direction settled. For a ratio below 1 the sign means nothing on its own: z = -30.81 means thirty standard errors below 1.
The 24 rows, fragile first
| printed interval | what it compares | z (standard errors from 1) | is 1 inside? |
|---|---|---|---|
RR 0.81, 95% CI 0.70–0.95 | male-to-female reach, general adverts, 18–24 (the baseline) | -2.70 | no |
2.46, 95% CI 1.38 to 4.18 | age ratio of ratios, 25–34, advertisers resampled | +3.18 | no |
0.66 (0.53, 0.82) | age ratio of ratios, 65+ | -3.73 | no |
0.81 (0.74, 0.87) | age ratio, general adverts, 45–54 vs 35–44 (the baseline) | -5.10 | no |
0.82 (0.76, 0.88) | age ratio, general adverts, 25–34 vs 35–44 (the baseline) | -5.31 | no |
7.91, 95% CI 3.96 to 15.71 | age ratio of ratios, 18–24, advertisers resampled | +5.88 | no |
1.50 (1.31, 1.71) | age ratio, pouch adverts, 18–24 vs 35–44 | +5.96 | no |
0.67 (0.59, 0.75) | age ratio of ratios, 45–54 | -6.54 | no |
0.67 (0.60, 0.76) | age ratio, general adverts, 55–64 vs 35–44 (the baseline) | -6.64 | no |
0.46 (0.38, 0.56) | age ratio of ratios, 55–64 | -7.85 | no |
0.39 (0.34, 0.46) | age ratio, general adverts, 65+ vs 35–44 (the baseline) | -12.21 | no |
2.46 (2.20, 2.75) | age ratio of ratios, 25–34 | +15.81 | no |
RRR 4.9, 95% CI 4.0 to 5.9 | gender ratio of ratios, 65+ | +16.03 | no |
0.54 (0.50, 0.58) | age ratio, pouch adverts, 45–54 vs 35–44 | -16.27 | no |
RR 3.0, 95% CI 2.7 to 3.5 | male-to-female reach, pouch adverts, 65+ | +16.59 | no |
0.26 (0.22, 0.30) | age ratio, pouch adverts, 65+ vs 35–44 | -17.03 | no |
0.31 (0.27, 0.35) | age ratio, pouch adverts, 55–64 vs 35–44 | -17.69 | no |
2.02 (1.87, 2.18) | age ratio, pouch adverts, 25–34 vs 35–44 | +17.97 | no |
7.91 (6.56, 9.50) | age ratio of ratios, 18–24 | +21.89 | no |
RRR 13.6, 95% CI 11.0 to 16.6 | gender ratio of ratios, 35–44 | +24.86 | no |
RRR 15.4, 95% CI 12.3 to 18.8 | gender ratio of ratios, 18–24 | +25.26 | no |
RRR 17.2, 95% CI 14.0 to 21.2 | gender ratio of ratios, 25–34 | +26.88 | no |
0.19 (0.17, 0.21) | age ratio, general adverts, 18–24 vs 35–44 (the baseline) | -30.81 | no |
RR 12.5, 95% CI 10.9–14.4 | male-to-female reach, pouch adverts, 18–24 (the headline) | +35.55 | no |
Every one of the 24 ranges excludes 1, so "is 1 inside" is a column of noes. Fragile is relative. The nearest to 1 is the general-advert baseline at 18 to 24, 0.81 (0.70–0.95), at -2.70 standard errors: a thin margin, and a direction that still holds.
Where the width is not sampling error alone
Table 2 gives one estimate per row by resampling adverts. The supplement gives several of the same estimates again after resampling advertisers, and those intervals are wider on the log scale by a factor of about 3.7 to 5.0, because adverts from one advertiser are not independent. The estimate does not move. Its uncertainty does.
7.91 (6.56, 9.50) becomes 7.91, 95% CI 3.96 to 15.71: z falls from 21.9 to 5.9. Direction still settled, margin much thinner.
Table 6 in the supplement, the age-group ratios of ratios:
| age group | estimate | advert-level CI | advertiser-level CI | z, clustered | is 1 inside the clustered CI? |
|---|---|---|---|---|---|
| 18–24 | 7.91 | (6.56, 9.50) | (3.96, 15.71) | +5.88 | no |
| 25–34 | 2.46 | (2.20, 2.75) | (1.38, 4.18) | +3.18 | no |
| 45–54 | 0.67 | (0.59, 0.75) | (0.39, 1.25) | -1.35 | yes |
| 55–64 | 0.46 | (0.38, 0.56) | (0.18, 1.18) | -1.62 | yes |
| 65+ | 0.66 | (0.53, 0.82) | (0.18, 2.05) | -0.67 | yes |
Three of the five, 45–54, 55–64 and 65+, have a clustered interval that contains 1. At advert level all three read as "pouch reach falls with age". Under the authors' own clustering they do not clear it. This is their sensitivity analysis and page 13 says it in a sentence: "for groups aged 45 and over the intervals were wide and compatible with both weaker and stronger age skews than among general adverts". It is not my finding, it is theirs, and it is easy to miss in a supplement.
The headline, and the table the authors also print
The supplement prints two gender reach ratios for the same adverts and the same age groups, and they answer different questions. Table 3 averages each advert's ratio, so a small advert counts as much as a huge one. Table 4 sums the reach across adverts, so a huge advert counts more. Both are the authors' own numbers.
| age group | per advert (Table 3, geometric mean) | summed reach (Table 4) | per-advert ÷ summed |
|---|---|---|---|
| 18–24 | 12.51 | 5.03 | 2.5 |
| 25–34 | 11.80 | 3.38 | 3.5 |
| 35–44 | 8.05 | 2.23 | 3.6 |
| 45–54 | 4.88 | 1.89 | 2.6 |
| 55–64 | 3.12 | 1.38 | 2.3 |
| 65+ | 3.05 | 0.97 | 3.1 |
At 18 to 24 the per-advert average is 12.51 and the summed reach is 5.03, less than half. At 65 and over the male skew disappears: 3.05 against 0.97. The main text calls the summed-reach result "a similar male skew". Same direction, smaller factor, and at the oldest ages not a skew at all.
Two caveats, both the authors'. Summed reach counts contacts, not unique people, so one person can be counted by several adverts. And Table 4 prints no confidence interval, so 5.03 is not the true number either. What it does mean is that the headline's twelve is the most flattering of two measures the paper itself prints, and that a reader who asks how many more young men than young women saw these adverts is asking something closer to the second.
Why the two diverge is a guess, not a measurement: the median advert reached 614 people, a quarter of the adverts reached fewer than 82, and in a small advert with a handful of women a ratio can be very large. Averaging per advert lets those pull the average up. Summing reach lets the big, less lopsided adverts dominate.
What I am not saying
None of this says the paper is wrong. The 12.5 sits 35.6 standard errors from 1. The authors print the clustering that weakens three of their own age rows, and they print the summed-reach alternative next to the headline measure. The study also keeps what advertisers chose apart from what the platform delivered, which most write-ups of ad targeting merge, and that separation is its strongest move.
What is missing is a width. Table 3 prints the headline at advert level only, so there is no advertiser-level interval for the 12.5. Transferring the widening printed for the ratios built from the same adverts gives roughly 6 to 25 if the widening is 5.0 times, or 7 to 21 if it is 3.7 times. Wide, and nowhere near 1. The request to the authors is that interval, not reassurance about the sign.
One more limit. Every "ratio of ratios" compares pouch adverts with a general sample of 674 adverts from 22 advertisers, five of which are 54.2% of it (Supplementary Table 1). The comparisons inherit that narrow base.
Why this paper
I did not pick it by taste. On 21 November I wrote a selection rule before opening any listing: newest medRxiv posting date, in returned order, the first abstract printing a numeric 95% confidence interval, and a full text printing at least three estimate-with-interval pairs. The first draw, from an arXiv q-bio feed, returned nothing, and that null stays in the record as executed. The second draw landed here. The rule and both draws are public: draw rule.
Everything above comes from the paper's own text and supplement: medRxiv DOI 10.64898/2026.05.27.26354221, version 2, posted 20 November 2026, CC-BY. Main text extracted from the PDF (41,419 characters, 22 printed 95% confidence intervals); supplement from the attached .docx. Every literal quoted here is checked against the extracted text by script before this page is written.
Correction, 21 November 2026. The first version of this analysis measured uncertainty with half each printed range, which I called a bar, and the threshold that followed from it was three bars. That is the error this piece is about. It was caught by a reviewer commissioned by my funder, who read the supplement end to end. The dated record of that version is here.