PDF to Markdown for RAG: did the exceptions survive?

A support assistant says an order can be returned within 30 days. The PDF says the same thing—and adds that personalized items cannot be returned unless they arrived damaged.

The answer contains a real number from the document. It can still be wrong for the customer asking the question.

When evaluating a parser such as DocSlight, test the conditions that change a decision, not just whether the exported Markdown looks readable. This guide uses a small fictional returns policy to build that check. It is an evaluation procedure, not a parser ranking or a measured improvement in RAG accuracy.

Start with a policy whose answers you already know

Create a one-page document in Word or another editor and export it to PDF. Put the general rule near the top, the exclusions below it, and the shipping note at the bottom. Use this fictional text:

Example returns policy — version R1 Standard items can be returned within 30 calendar days of delivery. Personalized items cannot be returned unless they arrived damaged. Opened hygiene items are not eligible for a change-of-mind return. Return shipping: the customer pays unless the item arrived damaged or the seller sent the wrong item. 

Keep the input PDF. These sentences are the ground truth for this exercise; they are not advice about any retailer's actual policy.

Write the questions before processing the file:

That last question matters. Preserving an exception is different from filling in details the source never supplied.

Choose the processing route deliberately

DocSlight's repository provides local and cloud modes, with Markdown and JSON output options. They are different processing routes: a cloud result does not establish local-mode quality or deployment compatibility. Check the current release requirements and the mode you actually intend to use.

If you want to inspect cloud parsing before setting up a local environment, use the ComPDF document-parsing demo with the fictional PDF. This is a hosted ComPDF parsing path, not a test of the local open-source DocSlight engine. Check the available output and processing conditions before uploading business documents.

For either route, keep the source PDF, exported result, processing mode, and version or run identifier together. Do not compare a local run on one file with a cloud run on a revised file and attribute the difference entirely to the parser.

Inspect the exceptions before building the index

Open the result beside the PDF. Check four relationships:

1. Rule and subject: does “30 calendar days” still apply to standard items, rather than every item?
2. Negation and exception: are “cannot,” “not eligible,” and “unless” intact and attached to the right item?
3. Scope: is the shipping note still recognizable as a shipping condition, rather than another eligibility rule?
4. Missing information: does the result leave unspecified details unspecified?

Readable text is useful, but structure also matters. Where the selected output provides headings, blocks or page information, inspect how you will carry them into your application. Do not assume that every output format or edition exposes the same provenance fields.

Illustration: fictional policy text and an intentionally incomplete snippet. This is not a DocSlight output screenshot or a retrieval benchmark.

A small text check can flag omissions

Save the Markdown export as policy.md. The following Python script uses only the standard library. It flags missing phrases; it does not verify their meaning or prove that retrieval works.

from pathlib import Path import re text = Path("policy.md").read_text(encoding="utf-8") text = re.sub(r"[*_`#]", "", text) text = " ".join(text.casefold().split()) clauses = { "general_rule": "within 30 calendar days of delivery", "personalized_exception": "personalized items cannot be returned unless they arrived damaged", "hygiene_exclusion": "opened hygiene items are not eligible for a change-of-mind return", "shipping_exception": "unless the item arrived damaged or the seller sent the wrong item", } missing = [name for name, phrase in clauses.items() if phrase not in text] if missing: print("Inspect these clauses against the PDF:", ", ".join(missing)) else: print("All test phrases found; inspect their context before indexing.") 

Run python3 check_policy.py after saving the code under that name. A line-end hyphen, punctuation change or OCR substitution can trigger a flag even when a human can recover the meaning. Conversely, all phrases can be present in the wrong order. Review the surrounding text rather than treating either message as a quality score.

Test the chunk and the answer separately

Now apply your own chunking method. For the undamaged personalized-item question, inspect the actual retrieved context. Does it contain both the general return window and the personalized-item restriction? If only the 30-day rule is retrieved, the answer model has not received the condition it needs.

Possible adjustments include keeping the short policy section together, attaching the relevant exclusion to an item-specific chunk, or returning a wider parent section. Choose based on the document structure; a fixed overlap size does not guarantee that an exception stays with its rule.

For each test question, record:

source version | parser mode/version | question supporting source text | retrieved text | answer | review outcome 

Then separate the failure:

Use the same questions and downstream settings when comparing parser routes. One small policy can expose a failure, but it cannot establish performance across contracts, scanned forms or a production collection.

Decide what to evaluate next

DocSlight is relevant when your application needs to turn source documents into content it can inspect and use. Its local and cloud options provide different starting points; application-side retrieval, authorization and answer evaluation still need their own checks.

If the requirement extends to managed review, workflow integration or deployment controls, compare the DocSlight editions and workflow scope against that requirement. Do not assume the open-source Lite package includes the Enterprise workflow features.

For launch background, DocSlight also has a Product Hunt listing. The useful evaluation remains the same: try a document containing a rule, an exception and an unanswered detail, then check what your own application can actually support.