Download every image a web page shows (and keep a CSV manifest)

"Grab the images from this page" sounds like a five-line script until you try it on a real article. The markup mixes img[src], srcset, lazy-load attributes, root-relative paths, and the occasional PDF. Half the "images" are tracking pixels. One hero URL returns a login wall with Content-Type: text/html. Another logo appears three times under three CDNs. If you only keep the files that landed on disk, you cannot tell what was skipped or why.

I maintain a small Python tool aimed at that job: page-image-downloader (MIT, standard library only, Python 3.9+). This post is about the problem shape and a pattern you can run locally. A hosted runner is optional later — start with the free repo.

Page images are not the same problem as a CSV

If someone already handed you a spreadsheet of direct image URLs, use a list downloader. We publish one of those too: bulk-image-downloader. A page is different: you must discover candidates from HTML first, resolve them against the page URL, then download.

What page-image-downloader collects from HTML (at commit 263a6f6):

• img via lazy-load attrs (data-src, data-lazy-src, data-original) before src
• largest srcset / data-srcset / data-lazy-srcset candidate (highest w, else highest x)
• each <picture><source srcset="..."> largest candidate
• og:image / twitter:image (and a few related meta keys)
• optional linked files with --links pdf (or other extensions)

Relative URLs are resolved against <base href> and the page URL. The same absolute URL is fetched once per page. Whatever you collect, write a manifest with one row per candidate. Status beats vibes.

Minimal local run

Verified CLI at commit 263a6f6 (python3 page_image_downloader.py --help): URLs are positional (there is no --url). Output directory is -o / --out (default images).

git clone https://github.com/fecklesspleb/page-image-downloader cd page-image-downloader python3 page_image_downloader.py \ "https://example.com/" \ -o page-images 

Or pass a text file of URLs with -i / --input (one URL per line, # comments allowed).

What that run writes:

• Folder page-images/<page-host>/ with files named <stem>-<first 10 hex of sha256>.<ext> (extension from sniffed bytes, not the URL)
• page-images/manifest.csv (always)
• A JSON summary printed to stdout (not a separate .json file). Progress lines go to stderr (-q hides them).

manifest.csv columns (exact header):

page_url, image_url, final_url, found_in, alt, status, file, content_type, bytes, width, height, sha256, error

status values the tool writes: saved, duplicate, too_small, too_narrow, too_large, not_an_image, html_instead_of_file, robots_disallowed, error.

Exit code is 1 only when nothing was saved and at least one request failed.

Pattern: resolve, fetch, sniff, record

Even if you write your own script, keep this shape. The snippets below are illustrative and use only the standard library (Python 3.9+; Optional keeps the annotation 3.9-safe).

from urllib.parse import urljoin, urlparse from urllib.request import Request, urlopen import hashlib from pathlib import Path from typing import Optional UA = "page-image-example/0.1 (+https://github.com/fecklesspleb/page-image-downloader)" def fetch(url: str, timeout: float = 30) -> bytes: req = Request(url, headers={"User-Agent": UA}) with urlopen(req, timeout=timeout) as resp: return resp.read() def looks_like_image(data: bytes) -> Optional[str]: if data.startswith(b"\x89PNG\r\n\x1a\n"): return "png" if data[:3] == b"\xff\xd8\xff": return "jpg" if data[:4] == b"GIF8": return "gif" if data[:4] == b"RIFF" and data[8:12] == b"WEBP": return "webp" head = data[:200].lstrip().lower() if head.startswith(b"<svg") or b"<svg" in head: return "svg" return None def save_unique(out_dir: Path, url: str, data: bytes) -> Path: kind = looks_like_image(data) if not kind: raise ValueError("not_an_image") digest = hashlib.sha256(data).hexdigest()[:10] host = urlparse(url).netloc or "unknown" path = out_dir / host path.mkdir(parents=True, exist_ok=True) dest = path / f"asset-{digest}.{kind}" if not dest.exists(): dest.write_bytes(data) return dest page = "https://example.com/articles/one" candidate = "/static/hero.jpg" absolute = urljoin(page, candidate) print(absolute) 

Running only the resolve lines prints:

https://example.com/static/hero.jpg 

Sniffing the first bytes matters more than trusting Content-Type or a .jpg suffix. Soft 404 pages and interstitial HTML with Content-Type: text/html fail the sniff and should be recorded as not_an_image, not written as a corrupt JPEG that breaks the next pipeline step. (page-image-downloader does that when the bytes do not sniff as an image and the MIME type is not an image type either.)

Manifest columns worth keeping

At minimum — and these are the columns the tool actually writes:

With that table you can answer "how many assets did we actually get?" without opening the folder.

Politeness and rights

• Send a User-Agent that names the project and points at the repo.
• Cap concurrency; the tool defaults to a 0.5 s per-host delay (--delay).
• It checks robots.txt unless you pass --ignore-robots (only for sites you run).
• Only download pages and assets you have a right to use. "It was on a public page" is not automatically a license to redistribute.

Free repo first

The free GitHub project is the default. Use it in CI, on a laptop, or inside a container with no API keys. If you later want the same workflow on a hosted crawler platform, treat that as an optional convenience for people who already live there — not as the primary link in a how-to.

Try it: fecklesspleb/page-image-downloader. For CSV/text URL lists instead of HTML discovery, use bulk-image-downloader.

Disclosure

We built and maintain page-image-downloader (and the sibling bulk-image-downloader for CSV lists). This article and the tool/README were written with AI assistance and reviewed before publishing.