I have been scraping catalogs, prices, product cards and listings for years. All that time a small homegrown HTML parser did the work: millions of pages, no complaints. It never went public because its tests covered only my own sites and there were no docs.
Last week I cleaned it up and released it as z3l.FastHtmlParser. The first version answered "how do I find a tag fast". Since then it has gone through five releases in one week, driven by feedback: XPath 1.0, CSS selectors, a review against hostile markup, and parsing straight from a char[] buffer. This post covers the idea, the numbers and the trade-offs.
dotnet add package z3l.FastHtmlParser A DOM is more than a scraper needs
In a high-volume scraping service the parser is the hottest loop, and your server bill grows with it.
HtmlAgilityPack and AngleSharp are excellent libraries, but they are built to be browsers. They create an object for every node, synthesize the implicit <html>, <head> and <tbody>, and keep a mutable tree so you can edit it and walk it with selectors.
A scraper does far less: find an element, read its text or an attribute, move on. So the idea is simple: don't build what you won't use. The document is the source string plus a flat list of tags, each one a set of integer offsets into that string.
The numbers
BenchmarkDotNet 0.15.8, .NET 10, Intel Xeon E-2236, Windows Server 2025. Pages are real saved pages of books.toscrape.com, quotes.toscrape.com and webscraper.io test sites. AngleSharp 1.10.0, HtmlAgilityPack 1.13.1.
Parse a page, time (lower is better):
| Page | FastHtmlParser | HtmlAgilityPack | AngleSharp |
|---|---|---|---|
| quotes, 11 KB | 17 μs | 155 μs | 229 μs |
| laptops, 38 KB | 53 μs | 544 μs | 959 μs |
| books, 51 KB | 59 μs | 610 μs | 958 μs |
| books, no end tags* | 59 μs | 671 μs | 949 μs |
| books ×10, 510 KB | 0.61 ms | 19.6 ms | 12.2 ms |
Parse a page, memory allocated:
| Page | FastHtmlParser | HtmlAgilityPack | AngleSharp |
|---|---|---|---|
| quotes, 11 KB | 11 KB | 243 KB | 189 KB |
| laptops, 38 KB | 31 KB | 713 KB | 608 KB |
| books, 51 KB | 39 KB | 812 KB | 663 KB |
| books, no end tags* | 39 KB | 756 KB | 644 KB |
| books ×10, 510 KB | 391 KB | 8.1 MB | 6.3 MB |
* The books page with all 75 </li> and 60 </p> removed, as on sloppy sites.
Parse and extract 20 books (title, price, rating). All libraries return exactly the same records; the setup checks it.
| Library | Time | Allocated |
|---|---|---|
| z3l.FastHtmlParser | 69 μs | 45 KB |
| … with CSS selectors | 75 μs | 45 KB |
| … with XPath | 109 μs | 173 KB |
| HtmlAgilityPack (XPath) | 760 μs | 1,038 KB |
| AngleSharp (CSS selectors) | 1,043 μs | 707 KB |
The interesting row is the CSS one: the same selectors AngleSharp runs are about 14x faster here, with 16x less memory. You don't have to learn a new API to get most of the speed.
On a real 1 MB Amazon search results page (100+ inline scripts, ~4,000 tags) parsing takes 0.65 ms against 11.3 ms (HtmlAgilityPack) and 15.1 ms (AngleSharp), and extracting 27 products takes 0.87 ms against 16.3 ms and 18.1 ms, with 23–29x less memory. That page isn't in the repo for copyright reasons; save your own as benchmarks/local/amazon*.html and run LocalScrapeBenchmarks.
What the API looks like
Three ways to query, pick whichever you like. All of them work on the document or inside a tag.
using z3l.FastHtmlParser; var doc = new HtmlParser(html, "https://books.toscrape.com/"); // 1. Find / FindAll: enums for tag and attribute names, no allocations while matching foreach (var book in doc.FindAll(new TagQuery(HtmlTag.Article) { Class = "product_pod" })) { var title = book.Find(HtmlTag.H3).Find(HtmlTag.A).AttributeByName(HtmlAttribute.Title); var price = book.Find(HtmlTag.P, cls: "price_color").Text(); var link = book.Find(HtmlTag.A).AttributeHref(); // absolute URL, resolved against the page } // 2. CSS selectors, as querySelectorAll / querySelector in a browser doc.Css("article.product_pod h3 > a"); doc.Css("div.quote:has(a.tag[href*=love]) span.text"); doc.CssFirst("ul#menu > li:last-child a")?.AttributeHref(); // 3. XPath 1.0 doc.XPathValues("//a[contains(@href, '/product/')]/@href"); doc.XPathValue("//dt[.='Price']/following-sibling::dd[1]"); // compile once, reuse on any number of pages and threads static readonly CssQuery Prices = CssQuery.Compile("div.card span.price"); page.Css(Prices); Chains are null-safe: every method accepts null and returns null, so a missing element never throws in the middle of a chain.
var phone = doc.Find(HtmlTag.Div, id: "contacts") .Find(HtmlTag.A, cls: "phone") .AttributeHrefTelephone(); // null if any step is missing How it works
1. A tag list instead of a tree
Each tag stores only integers: where it starts and ends, where its inner content starts and ends, its depth and its parent. Text, inner HTML and attributes are computed on demand as slices of the source. Descendants of a tag are simply the tags that follow it in the list until one starts past its end, so "search inside a tag" is a plain indexed for loop over a range.
2. Memory allocated once, at the exact size
'<' can't appear fewer times than there are tags, so a vectorized Count('<') gives an upper bound. Working arrays are rented from ArrayPool<T>.Shared, the final list is created once with CollectionsMarshal.SetCount, and the rented arrays are cleared before they go back to the pool, so they don't keep the page alive. No List<T> doubling, no copies.
3. Vectorized scanning
Outside a tag only '<' matters, found with vectorized IndexOf. Inside a tag only '<', '=' and '>' matter, found with IndexOfAny. Quoted attribute values are skipped as a block, so data-x="<p>...</p>" doesn't break anything. script, style, textarea, title and the other raw-text elements are scanned straight to their end tag, as HTML5 says.
4. Tag names without strings
Names are ReadOnlySpan<char> looked up through Dictionary<string, HtmlTag>.AlternateLookup<ReadOnlySpan<char>> (new in .NET 9) and turned into an enum. I tried FrozenDictionary and dropped it: about 16 ms of JIT warm-up on first use against ~2 ms for a plain Dictionary, and slower lookups on this workload. A custom comparer hashes the length plus three case-folded characters. The same trick drives the table of all 2,125 HTML5 named entities.
5. Queries without IEnumerable
Find, Parent, Next and NextSibling return a single tag through an indexed loop; TagQuery is a struct passed by in. A chain like doc.Find(...).Find(...).NextSibling(...) allocates nothing on the heap; the only allocation is the final Text() string.
Broken HTML
Clean test pages are easy. Real pages leave out </li>, </p>, </td> and </option>, and a naive parser then nests every <li> inside the previous one.
I tried three fixes:
- Full HTML5 implicit-close rules in the main loop. Correct, but 4–9% slower.
- Fast pass, then a local re-parse on trouble. Fell apart on the books page without end tags: all 20 books sit in one
<ol>, so most of the page was rescanned. 61 μs became 119 μs. - Check the top of the open-tag stack when a tag opens. If a new
<li>opens while an unclosed<li>is on top, they're siblings: close the previous one. One integer comparison. This is the one that shipped.
The result is in the table above: the books page with all 75 </li> and 60 </p> removed parses in the same 59 μs as the original.
Then came a review against hostile and random markup (version 1.3.0), and it found real bugs:
-
<a title="see id=5" id="real">reported the id as5: attribute lookup now never looks inside another attribute's quoted value. -
<br/>without a space was namedbr/. - Thousands of unclosed tags followed by stray end tags took seconds. Parsing is now linear on any markup.
- In 1.4.1, XPath like
//div//span//aon deeply nested or unclosed markup was quadratic: 20,000 nested tags took 16 s and gigabytes of memory. Now 3 ms.
The test suite runs every saved page and a set of broken-markup snippets through AngleSharp too, and checks that both find the same elements, texts, attribute values and parents, CSS selector results included. The only differences are the trees a browser builds implicitly, like an inserted <tbody>.
Parsing from a buffer
If your HTTP client already reads into a pooled char[], you don't have to make a string out of it:
var owner = MemoryPool<char>.Shared.Rent(maxChars); int length = reader.Read(owner.Memory.Span); using var page = HtmlParser.Parse(owner, length, url); // Dispose returns the buffer to the pool When to use something else
- You need to modify the document. It's read-only (which is also why it's safe to query from many threads).
- You need the browser's implicit tree. No synthesized
<html>,<head>or<tbody>;table > trmatches as written in the source. - You're on .NET 8 or older. The design depends on
AlternateLookup, so it targets .NET 9 and .NET 10 only. - You need JavaScript-rendered content. It's a parser, not a browser.
A week ago CSS selectors and XPath were on this list. They aren't anymore.
License
Free for individuals and for companies with under $1M annual revenue and fewer than 100 employees, commercial use included. Larger companies pay once: $1,000 per organization, perpetual, all versions, and it covers every z3l library. The source is on GitHub.
Links
- GitHub: https://github.com/romanti3172/z3l.FastHtmlParser
- NuGet: https://www.nuget.org/packages/z3l.FastHtmlParser
- Docs: https://romanti3172.github.io/z3l.FastHtmlParser/
- Benchmarks and how to run them: benchmarks/README.md
I'd like to hear where it breaks: send a page it parses differently from your browser, and I'll add it to the tests.