3. We Converted 698 Pages of IRS Guidance and MarkItDown 0.1.8 Scrambled Every Publication

This is the job MarkItDown is built for: take a folder of real documents and turn it into Markdown an LLM can search and quote. Our folder was the IRS's own guidance for small businesses and employees, the kind of thing you would feed a model so it can answer "can I deduct this" with a page reference. We downloaded twelve publications, 698 pages in total, and converted them in one batch.

The batch ran without a single error. Then a quality check we wrote found that on all eleven real PDFs, MarkItDown 0.1.8 had scrambled the text, keeping between 1 and 9 of every 50 sampled sentences intact. The same files through MarkItDown 0.1.5, from February 2026, kept between 39 and 48 of 50. This part shows how we found that, why it happens, and the pipeline we would actually use.

Run over the IRS folder	Wall time	Sentences intact	Peak memory, 142-page Pub 17
MarkItDown 0.1.8, 1 worker	55.32 s	1 to 9 of 50	210 MB
MarkItDown 0.1.8, 8 workers	17.31 s	1 to 9 of 50	Not measured
MarkItDown 0.1.5, one CLI call per file	100.35 s	39 to 48 of 50	2,065 MB

The corpus, and the PDF that was not a PDF

Same environment as parts 1 and 2: MarkItDown 0.1.8, Python 3.12.14, an Apple M4 Pro with 12 cores and 48 GB, macOS 26.5.1. You will also need Poppler for pdftotext and pdfinfo:

brew install poppler
cd ~/markitdown-test && source .venv/bin/activate
mkdir -p irs
for p in p334 p583 p463 p535 p946 p587 p15 p505 p501 p17 p970 p596; do
  curl -sSL -o irs/$p.pdf https://www.irs.gov/pub/irs-pdf/$p.pdf
done
file irs/*.pdf

That last line caught something we would otherwise have missed. Eleven files said PDF document. One, p535.pdf, said HTML document text. Publication 535, Business Expenses, has been discontinued, and the IRS now redirects its PDF address to a web page called "Guide to business expense resources". Curl followed the redirect and saved the page under a .pdf name.

We kept it in the folder on purpose, because real archives are full of files like this. MarkItDown handed it to its HTML converter and produced 24,716 characters of IRS navigation menus with exit code 0. So a "tax guide" in your knowledge base can quietly be a cookie banner.

We also dropped in the scanned W-9 from part 2 as zz-scanned-w9.pdf, a real PDF with no text in it, to see if the batch would notice.

The batch script

Calling the command once per file wastes 0.42 seconds per file on Python start-up, as we measured in part 1. The Python API avoids that, and a process pool lets you use more than one core. Save this as batch.py:

import re
import sys
import time
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path

from markitdown import MarkItDown

SRC = Path(sys.argv[1])
OUT = Path(sys.argv[2])
WORKERS = int(sys.argv[3]) if len(sys.argv) > 3 else 1

MAGIC = {".pdf": b"%PDF", ".docx": b"PK", ".pptx": b"PK", ".xlsx": b"PK"}


def check(path, text):
    problems = []
    magic = MAGIC.get(path.suffix.lower())
    if magic and not path.read_bytes()[:4].startswith(magic):
        problems.append("extension lies about the contents")
    words = re.findall(r"\S+", text)
    if len(text.strip()) < 200:
        problems.append(f"almost empty ({len(text.strip())} chars), scanned?")
    elif words:
        glued = sum(1 for w in words if len(w) >= 25 and w.isascii() and "/" not in w)
        if glued / len(words) > 0.02:
            problems.append(f"{glued} glued words out of {len(words)}")
    if text.count(" NaN ") > 20:
        problems.append(f"{text.count(' NaN ')} NaN cells")
    return problems


def convert(path):
    md = MarkItDown()
    start = time.perf_counter()
    try:
        text = md.convert(str(path)).markdown
    except Exception as exc:
        return path.name, 0.0, 0, [f"failed: {type(exc).__name__}"]
    elapsed = time.perf_counter() - start
    (OUT / f"{path.stem}.md").write_text(text)
    return path.name, elapsed, len(text), check(path, text)


if __name__ == "__main__":
    OUT.mkdir(exist_ok=True)
    files = sorted(p for p in SRC.iterdir() if p.is_file())
    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=WORKERS) as pool:
        results = list(pool.map(convert, files))
    total = time.perf_counter() - start
    flagged = 0
    for name, secs, size, problems in results:
        flagged += bool(problems)
        status = "; ".join(problems) or "ok"
        print(f"{name:14} {secs:6.2f}s {size:>9,} chars  {status}")
    print(f"{len(files)} files, {flagged} flagged, {total:.2f}s wall with {WORKERS} worker(s)")

cp docs/w9-scanned.pdf irs/zz-scanned-w9.pdf
python batch.py irs irs-md 1
python batch.py irs irs-md 8

The checks are the three silent failures from part 2 turned into code: output that is nearly empty, words glued together, and a file whose first bytes do not match its extension. One worker took 55.32 seconds. Eight workers took 17.31 seconds, and could not go lower, because the 142-page Publication 17 alone takes about 16 seconds and one file cannot be split across workers. Both runs flagged exactly the two files we planted:

p535.pdf         0.07s    24,716 chars  extension lies about the contents
zz-scanned-w9.pdf   0.00s         0 chars  almost empty (0 chars), scanned?
13 files, 2 flagged, 17.31s wall with 8 worker(s)

Everything else said ok. We nearly stopped there.

What the checker missed

We opened irs-md/p334.md, the Tax Guide for Small Business, to spot-check a paragraph. Here is a line exactly as MarkItDown wrote it:

Options to pay electronically include using your bank ac- (Form 1040). Use Schedule C (Form 1040) to figure your
count with Direct Pay, your debit or credit card, your digital net profit or loss from your business.

Two different paragraphs from two different columns, woven together line by line. IRS publications are set in two columns, and MarkItDown was reading straight across the page. None of our checks could see it: there are spaces between the words, there is plenty of text, and the file really is a PDF.

An LLM given that text will still produce a fluent answer. It just might attach the Schedule C rules to the Direct Pay sentence. For a tax knowledge base, that is worse than an error.

A canary that catches scrambled text

The fix for detection is to compare against an extractor that reads columns properly. pdftotext does, and it takes a fraction of a second. The canary samples 50 sentences from pdftotext's output and checks how many appear, word for word, in MarkItDown's. Save it as canary.py:

import random
import re
import subprocess
import sys
from pathlib import Path

PDF_DIR = Path(sys.argv[1])
MD_DIR = Path(sys.argv[2])
SAMPLE = 50


def norm(text):
    text = re.sub(r"(\w)-\s+(\w)", r"\1\2", text)
    return re.sub(r"\s+", " ", text)


for pdf in sorted(PDF_DIR.glob("*.pdf")):
    md_file = MD_DIR / f"{pdf.stem}.md"
    ref = subprocess.run(["pdftotext", str(pdf), "-"], capture_output=True, text=True).stdout
    sentences = [norm(s).strip() for s in re.split(r"(?<=[.!?])\s+", ref)]
    sentences = [s for s in sentences if 80 <= len(s) <= 200]
    if not sentences or not md_file.exists():
        print(f"{pdf.name:18} skipped (no text layer or no output)")
        continue
    random.seed(pdf.name)
    sample = random.sample(sentences, min(SAMPLE, len(sentences)))
    md = norm(md_file.read_text())
    intact = sum(s in md for s in sample)
    verdict = "ok" if intact / len(sample) >= 0.7 else "SCRAMBLED"
    print(f"{pdf.name:18} {intact:>2}/{len(sample)} sentences intact  {verdict}")

python canary.py irs irs-md

The norm function rejoins words hyphenated across line breaks, so "ac- count" does not count as a miss. Every one of the eleven real publications failed:

Publication	Pages	0.1.8 intact	0.1.5 intact
15, Employer's Tax Guide	59	5/50	48/50
17, Your Federal Income Tax	142	4/50	42/50
334, Tax Guide for Small Business	57	3/50	46/50
463, Travel, Gift, and Car Expenses	62	3/50	45/50
587, Business Use of Your Home	35	1/50	43/50
946, How To Depreciate Property	113	9/50	39/50

The other five (501, 505, 583, 596, 970) scored between 5 and 8 on 0.1.8 and between 42 and 46 on 0.1.5. We checked the method against pdfminer, the library MarkItDown itself uses, run directly with its defaults: on 200-sentence samples it kept 87% to 94% intact, in line with 0.1.5. So the extractor underneath is fine. The problem is in how MarkItDown decides to use it.

Why it happens: one changed condition

We installed 0.1.1, 0.1.3, 0.1.5, 0.1.6 and 0.1.7 side by side and ran the canary on Publications 334 and 15 with each. The first three were fine. From 0.1.6 onward, every version scrambled. So we read the diff between the 0.1.5 and 0.1.6 tags. Only 38 lines of the PDF converter changed, in a pull request (#1612) that fixed memory growth on long PDFs. Inside it, this:

# 0.1.5
if plain_pages > form_pages and plain_pages > 0:
    markdown = pdfminer.high_level.extract_text(pdf_bytes)

# 0.1.6 and later
if form_page_count == 0:
    markdown = pdfminer.high_level.extract_text(pdf_bytes)

MarkItDown looks at every page and asks whether it looks like a form or table. In 0.1.5, if most pages were plain prose, the whole document went through pdfminer, which handles columns. In 0.1.6 and later, pdfminer is only used if no page looks like a form. Otherwise every page, including the prose ones, goes through pdfplumber's simpler extraction, which reads straight across. An IRS publication has worksheets and tables on a handful of pages, so one table page flips the entire document. We did not find an issue on the tracker that names this change. Open pull request #1847, about prose being mistaken for tables, is the closest.

Pinning 0.1.5 is not free

The obvious workaround is uv pip install 'markitdown[all]==0.1.5'. It works for text, and we measured what it costs. On Publication 17, from the command line, it took 27.38 seconds and peaked at 2,065 MB of memory, against 16.00 seconds and 210 MB on 0.1.8. That memory is exactly what #1612 was fixing. Eight workers on 0.1.5 could need 16 GB. It also loses the table extraction from part 2: on the Attention paper, 0.1.5 kept the prose (5,868 words, 7 glued tokens) but produced zero table rows, where 0.1.8 produced 72.

The pipeline we would actually run

What we settled on: send real PDFs to pdftotext, send everything else to MarkItDown, and flag anything that comes back empty. Save as route.py:

import subprocess
import sys
from pathlib import Path

from markitdown import MarkItDown

SRC, OUT = Path(sys.argv[1]), Path(sys.argv[2])
OUT.mkdir(exist_ok=True)
md = MarkItDown()

for path in sorted(p for p in SRC.iterdir() if p.is_file()):
    if path.read_bytes()[:4] == b"%PDF":
        text = subprocess.run(["pdftotext", str(path), "-"], capture_output=True, text=True).stdout
        engine = "pdftotext"
    else:
        try:
            text = md.convert(str(path)).markdown
        except Exception as exc:
            print(f"{path.name:22} FAILED {type(exc).__name__}")
            continue
        engine = "markitdown"
    (OUT / f"{path.stem}.md").write_text(text)
    warning = "  EMPTY, probably a scan" if len(text.strip()) < 200 else ""
    print(f"{path.name:22} {engine:10} {len(text):>9,} chars{warning}")

On a mixed folder of seven files (Publication 334, the fake 535, the scan, and the Word, PowerPoint, Excel and EPUB files from part 1) it finished in 1.29 seconds. All eleven real IRS PDFs through pdftotext alone took 1.53 seconds, against 55.32 for MarkItDown. Be clear about what you give up: pdftotext produces plain text, with no Markdown tables. Also note that the fake p535.pdf still goes to MarkItDown as HTML and comes out as a web page. Checking the magic bytes routes it correctly, but it cannot tell you the IRS swapped the document for a menu. A human still has to look.

We are not reporting a canary score for pdftotext, because the canary uses it as the reference. It would score itself 50 out of 50, which proves nothing.

Bonus: the MCP server

MarkItDown also ships an MCP server so AI assistants can call it as a tool. It installed in 0.61 seconds as version 0.0.1a7, an alpha:

uv pip install markitdown-mcp
markitdown-mcp --help

We did not register it in any assistant. Instead we started it and spoke MCP's JSON-RPC protocol to it directly over standard input, which shows what any client would get. It answered the handshake in 0.63 seconds and offers one tool, convert_to_markdown, taking a single uri. Three findings:

• A plain path like /Users/you/doc.docx is rejected with Unsupported URI scheme. It needs file:///Users/you/doc.docx.
• Wikipedia still returns 403, and the server has no way to set the User-Agent fix from part 2.
• It read file:///etc/hosts without complaint. It will read any file your user account can, so an assistant that is tricked by a malicious document could ask it for your SSH config. The README says as much under Security Considerations. Believe it.

Also, by default the HTTP mode listens on port 3001, which a lot of local dev servers also use.

What broke, and common mistakes

• Every real IRS publication was scrambled by 0.1.8, with exit 0 and no warning.
• A discontinued publication's URL served HTML, which was converted as if it were the document.
• Our own first checker passed scrambled output as ok. Checking for spaces and size is not enough.
• Expecting more workers to help past your largest file. The longest single document sets the floor.
• Pinning 0.1.5 and running eight workers on a machine with less than 16 GB free.
• Assuming Markdown saves tokens. Across the eleven publications, 0.1.8 used 1,145,162 tokens (OpenAI's o200k_base tokenizer) and pdftotext used 928,017, about 23% fewer.

What we did not test

We tested one family of PDFs: US government publications typeset in two columns. Single-column reports, slides saved as PDF, and non-English or right-to-left documents may behave differently, and there are open issues about Arabic. We did not bisect below the 0.1.5 to 0.1.6 boundary to a single commit, only read that diff. We ran the batch on one 12-core machine. We did not wire the MCP server into Claude, Cursor or any other client, and we did not test its HTTP mode.

Next

Part 4 puts MarkItDown against Pandoc, Docling and plain pdftotext on the same files, including the scan it could not read, and ends with who should adopt it and who should not.