DevOps
LinkDigest: turn Xiaohongshu, Douyin, TikTok, YouTube and X links into text your agent can read
Programming with Jack Chew Dev.to (EN Zone)
9 views
I build LinkDigest. Every number below is from a measured run recorded in the project log; nothing is projected.
The problem
Agents are good at reading and bad at opening. Give Claude Code, Cursor, or a Dify workflow a link from Xiaohongshu, Douyin, TikTok, YouTube or X and the usual tool — fetch the URL, parse the HTML — returns one of three things: an app-download shell, a login wall, or a page whose body is 264 characters of navigation because the post itself lives in a JSON blob or in a video. The agent reports "nothing useful here" and it is telling the truth about the HTML.
The substance of these posts is not text. It is speech in a video, words printed on images, screenshots of code, a recipe laid out in a photo. A plain fetch throws all of that away, and a share link from the app carries a token that expires in a few weeks, so even the HTML you do get is different depending on which link and which user-agent you used.
The solution
LinkDigest does the reading on its own servers and hands back text. One call:
curl -X POST https://linkdigest.dev/api/v1/digest \
-H "Authorization: Bearer $LINKDIGEST_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.youtube.com/watch?v=CEvIs9y1uog", "format": "json"}'
returns the post's transcript with timecodes, every fragment of on-screen text, a description and OCR of each image, the caption and metadata, and a list of key points — as JSON, or as Markdown if you ask for it. Long media answers in two steps: a 202 with a job id, then the digest when it is done.
Measured, not estimated:
post
what came back
time
a 17-image Xiaohongshu note
17 images described and read, 381 on-screen text fragments, 13 key points
119 s
a Xiaohongshu video note
ASR transcript, 56 on-screen text fragments, 12 key points
~4 min
a YouTube talk (native captions)
426 transcript segments, 190 on-screen text fragments
~1 s cached
a two-photo X post
both photos described, 3 text fragments
~20 s
For agents: it is an MCP server
claude mcp add --transport http linkdigest https://linkdigest.dev/mcp \
--header "Authorization: Bearer $LINKDIGEST_KEY"
One tool, digest_url(url, format). The tool description tells the model exactly when to reach for it and which platforms are out, so an agent does not confidently hand you a failure. It is in the official MCP registry as dev.linkdigest/linkdigest, on Apify as three Actors, and submitted to the Dify marketplace.
What it does not do
This is the part I would want to know before pasting a key:
Bilibili: not supported. Bilibili returns HTTP 412 to our server's address.
Instagram: wired, not verified end to end. Facebook: out of scope.
X long-form articles expose only their lead image; regular photo posts come back in full.
A YouTube link is read by Gemini watching the video, because YouTube blocks datacenter IPs; if Google refuses a video, you get an honest stub, and it costs nothing.
Anything that could not be read is named in a degraded field. A digest that read nothing is free.
Pricing
Three digests free, no card. Then $9 a month for 500 credits — a typical post is 1 credit, video 2 per started minute, and links anyone has already digested are free forever. https://linkdigest.dev/pricing
Why I built it
I kept pasting Xiaohongshu and Douyin links into agents and watching them fail, and the failures were silent — a confident summary of a login page. Two of the bugs I hit while building this are written up separately: share links have two domains and one of them fails silently, and what it actually takes to read a Xiaohongshu post from a server.
If you try it and something reads thin, reply here — I fix these the same day.
Read original: https://dev.to/programming_withjackche/linkdigest-turn-xiaohongshu-douyin-tiktok-youtube-and-x-links-into-text-your-agent-can-read-ip
← Previous
I Built One Project in a Year and Another in Two Months: What Changed With AI-Assisted Development
Next →
Claude Code vs Cursor vs Copilot: An Honest Review After 40 Production Automations
Related
What WhatsApp, Instagram and Telegram actually do to your photos
DevOps
0
DEV Community
The 2-Hour Bash Bug That Taught Me How Quoting Actually Works
DevOps
2
Dev.to (EN Zone)
Choosing free on-prem git server - Gitea is the winner!
DevOps
5
DEV Community
Daily Dose of DevOps — Terraform remote state explained
DevOps
3
DEV Community
Comments0
No comments yet — be the first