How I normalise 3,200+ supplier listings to a fair cost per mg

Price comparison sounds like division. Take the price, divide by the amount, sort ascending, done.

Then you scrape your first fifty product pages and find out the amount was never one number.

I run PepLookup, which compares 3,200+ research peptide listings from 88 suppliers. The whole product rests on one figure: cost per milligram. This post is about how that figure gets made, the edge cases that make it hard, and the rules I'd recommend. The site runs on Next.js with Postgres (Supabase) and the examples below are in TypeScript but the ideas carry over to any language.

The problem: "5mg" is not a unit

Here is the kind of size string you meet on supplier pages:

5mg 10 mg x 10 2 × 5mg kit 5000mcg 10mg (10 vials) Kit of 10 – 5mg each BPC-157 5mg / TB-500 5mg 30ml 

Each of these means something different:

Divide the price by the first number you find and a noticeable share of the catalogue comes out wrong. Worse, it comes out confidently wrong. A 10-vial kit looks ten times more expensive than it is and a mislabelled single vial wins the "cheapest" slot.

Step 1: Parse the per-unit amount

The first job is to pull out the amount per vial and convert it to a single base unit.

const UNIT_TO_MCG: Record<string, number> = { mcg: 1, µg: 1, ug: 1, mg: 1_000, g: 1_000_000, }; const AMOUNT_RE = /(\d+(?:\.\d+)?)\s*(mcg|µg|ug|mg|g)\b/i; export function parseUnitMcg(raw: string): number | null { const m = raw.match(AMOUNT_RE); if (!m) return null; const value = parseFloat(m[1]); const factor = UNIT_TO_MCG[m[2].toLowerCase()]; return Math.round(value * factor); } 

Two details matter here.

First, the base unit is the microgram, stored as an integer. In the database a 10 mg vial is 10000, not 10.0. That avoids floating-point surprises when amounts like 0.5 mg or 2.5 mg get multiplied and compared, and it makes 5000mcg and 5mg identical by construction.

Second, the unit map is explicit. Guessing (treating a bare number as mg, say) is how 5000 ends up as 5,000 mg.

Step 2: Parse the quantity multiplier

Next comes how many units are in the listing. This is where most of the variation lives.

const QTY_PATTERNS: RegExp[] = [ /[x×]\s*(\d+)\b/i, // "10 mg x 10" /\b(\d+)\s*[x×]\s*\d/i, // "2 × 5mg" /\b(\d+)\s*(?:vials?|pack|pcs)\b/i, // "10 vials", "10 pack" /kit\s*of\s*(\d+)/i, // "Kit of 10" ]; // ambiguous kits are flagged by the rule below, not by this function. export function parseQuantity(raw: string): number { for (const re of QTY_PATTERNS) { const m = raw.match(re); if (m) { const n = parseInt(m[1], 10); if (n > 0 && n <= 100) return n; } } return 1; } 

The n <= 100 check is a sanity cap. No research supplier sells a 500-vial pack, so a number that large almost always means the regex grabbed the wrong figure, such as an amount in micrograms.

The ambiguous case, 10mg (10 vials), needs a rule rather than a regex. The approach I'd recommend is to compare the parsed total against the same supplier's other listings for that compound. If their single vial is 10 mg, the kit is almost certainly ten 10 mg vials. If there's no evidence either way, send the listing to manual review instead of guessing.

Step 3: Refuse to compare what isn't comparable

Some listings should never get a cost per mg at all:

export function isComparable(raw: string, productName: string): boolean { if (/\b\d+(?:\.\d+)?\s*ml\b/i.test(raw)) return false; // liquids if (/\s\/\s|\s\+\s/.test(productName)) return false; // blends return true; } 

Blends are the interesting case. "BPC-157 5mg / TB-500 5mg" has a total of 10 mg, but dividing the price by 10 and ranking it against pure BPC-157 is misleading because the buyer is not getting 10 mg of either compound. Blends belong in their own comparison group, ranked only against blends with the same ratio.

The product's form matters in the same way. A 10 mg nasal spray and a 10 mg vial aren't the same thing to the buyer, so each listing also stores a form (vial, spray and so on) and comparisons only happen within one form.

Step 4: Store structured fields, not a computed price

export function costPerMg(price: number, raw: string, name: string) { if (!isComparable(raw, name)) return null; const unitMcg = parseUnitMcg(raw); if (unitMcg === null || unitMcg <= 0) return null; const count = parseQuantity(raw); const totalMg = (unitMcg * count) / 1_000; return { unitMcg, count, totalMg, costPerMg: price / totalMg }; } 

In the database, each listing row stores the parts, not the answer: vial_size (micrograms, integer), vial_count, form, list_price, sale_price and in_stock. Cost per mg is calculated from those fields at query time. When a parsing rule changes, only the structured fields need correcting and every ranking updates on its own.

(If you want to try the arithmetic yourself, the price per mg calculator runs the same calculation in the browser.)

Step 5: Stock status and "last scraped"

A correct cost per mg is still useless if the product can't be bought. Two rules handle this:

1. Out-of-stock listings are removed before ranking. They still appear on the supplier's page, but they can never be labelled "cheapest".
2. Every price carries a scraped_at timestamp. Prices change often and showing when a number was last confirmed is more honest than pretending it's live.

select product_slug, supplier_slug, coalesce(sale_price, list_price) / ((vial_size * vial_count) / 1000.0) as cost_per_mg, scraped_at from offers where in_stock = true and form = 'vial' order by product_slug, cost_per_mg; 

Deciding how old a price can be before it stops counting is a judgement call. Too short a window drops slow-updating suppliers and too long lets stale prices win.

What I'd do differently

• Keep the raw size string. Storing only the parsed fields means that when a number looks wrong, you can't see what the supplier's page originally said. A raw_size column next to the structured fields would make every mistake traceable.
• Store a parse status. Marking each listing ok, ambiguous or excluded turns manual review into a short daily queue, instead of something you only discover when a customer points out a bad price.
• Treat discount codes separately. A coupon changes the price but not the amount. I'd store the list price and the price after the code side by side, never one overwriting the other, so both views stay available.

Results

Today the comparison covers 3,200+ in-stock and out-of-stock listings from 88 suppliers, all reduced to integer micrograms, a vial count and a form. Liquids and blends stay out of the single-compound rankings and every listing is sorted by cost per milligram rather than headline price. You can see the output on PepLookup's price checker.

If you've solved unit normalisation for a different kind of product, whether that's food per gram, cloud storage per GB or paint per litre, I'd like to hear which edge cases bit you. They're rarely the ones you expect.

Background on how I compare prices: my earlier write-up on Medium

PepLookup lists products sold by third-party suppliers for laboratory research use only. This post is about data processing and makes no claims about any compound.