digital-convert reads a born-digital document, one made on a computer, such as a PDF or an office file. It writes the text as an ATRIUM document record: pages, lines in reading order, headings, tables and, for a PDF, exact positions. Later ATRIUM stages add their results to the same record. Use it when the text is already in the file. A scan belongs to the OCR route. It needs no OCR, no model and no network.
Releases · MIT licence · ATRIUM documentation
| Input | Output |
|---|---|
| PDF, DOCX, ODT, ODS, XLSX, RTF, recognised by content; DOC and XLS through LibreOffice | the record: source, pages, lines, content, tables, provenance, validated against the frozen schema |
optionally, AMČR's seed record (doc_id, source.sha512), checked against the file |
per page, text_layer and, where the text cannot be used as it is, needs_ocr with its reason |
on request, the record as annotated Markdown; with /describe, a per-page assessment and where each page goes next |
The record follows the hub's document contract. Each format, how it is read and what reaches the record: docs/reference/formats.md.
The service and the command line take the same inputs and give the same results. They run the same code for the record, the Markdown and the per-page assessment:
| Result | Service | Command line (api_util/digital_to_json.py) |
|---|---|---|
| the record | POST /reformat → document_json |
<doc_id>.document.json (or --document-json-out PATH) |
| the record as annotated Markdown | markdown=true → markdown |
--markdown → <doc_id>.md |
| the per-page assessment, the routes | POST /describe → pages, summary, stages |
--describe → <doc_id>.describe.json (--stages, --classify-pages, --no-text) |
| a seed record to add to | the document_json part |
--document-json |
| the run's paradata | paradata, a CreateAction in the answer |
a file, with --paradata-dir |
Only the command line has the opt-in Docling engine (--engine docling, a local build); the service image runs
the light engine. Every difference, and why: docs/usage.md.
Two offline research tools, outside the images and the service, ask a decision model the AMČR document type of
PDF pages and documents, from the rendered pages alone
(usage).
What you need: Docker for the service; for the sample files and the command line, a checkout of this
repository and Python 3.11 (the version of the image and of CI). No network is needed to convert a file, and no
GPU. The /describe endpoint calls page-classification and ocr-postprocess only when their URLs are set. DOC
and XLS files need LibreOffice, which the images carry.
The service, from the published image, on small documents generated from this repository's test fixtures:
python3 tests/fixtures/digital/make_fixtures.py --outdir fixtures # 17 small PDFs and DOCX files
docker run --rm -p 8000:8000 ghcr.io/ufal/atrium-digital-convert-api:<version> # the release without its v
curl --fail-with-body -sS -X POST localhost:8000/reformat -F "file=@fixtures/minimal.pdf" -o out.jsonExpected: HTTP 200 and out.json with document_json, whose source.origin is digital-born-pdf, whose two
pages are digital, and whose first lines read "Excavation report, block one, line one." POST /describe
with the same file adds pages[] with a route for each page; it calls page-classification and ocr-postprocess
only when their URLs are set, and says so in stages[].
The same conversion on the command line, from a checkout with its setup done:
python3 api_util/digital_to_json.py fixtures/minimal.pdf --out-dir out --markdown --describeExpected: out/minimal.document.json (the record /reformat returns), out/minimal.md (its markdown) and
out/minimal.describe.json (the pages, summary and stages of /describe; without the stage URLs both stages
read not_configured). With the stage URLs set, --describe keeps what the stages write into the record, and the
record and the Markdown then show it, as /describe's document_json does.
Each page's text layer is checked before anything is trusted:
digital: the layer decodes;garbled: it does not (broken font encodings, glyphs without a Unicode mapping, words losing letters);ocr: it is an earlier OCR run;none: the page has no text, or its only text is a page number over a scan.
Every page that is not digital (or blank) is flagged needs_ocr with a reason. Those pages go to the OCR
route, and the record is still written. /describe then decides, page by page, where a page goes next: to NLP,
to OCR, to handwriting recognition or nowhere. It uses page-classification's page types and ocr-postprocess's
line quality when those services are deployed. The rules, with an example:
docs/how-it-works.md.
For an AMČR upload that is born-digital, digital-convert is the first stage and the record's originator: it alone
writes the positional part of the record. Its record goes on to nlp-enrich and keyword-extract; the pages it flags
go to the OCR/HTR route, and ocr-postprocess merges their new text back into the same record, page by page. In
the AMČR deployment Temporal chains the stages, and /reformat calls no other service. The whole chain: the
hub's workflow page. Its SSH Open Marketplace
record: Born-digital document converter: PDF and office files into structured text without OCR -
ATRIUM.
- Born-digital only. A PDF whose pages are mostly an earlier OCR run (
OCR_LAYER_DOCUMENT_SHARE, 0.5) is refused asocr_text_layer, so send it to the OCR route. Such pages inside a born-digital PDF are flagged for OCR (errors). - Uploads up to 250 MB by default. AMČR's originals reach about 1.5 GB, so a deployment must raise
MAX_UPLOAD_MBfor them. Above 250 MB, memory, concurrency and timeouts are not yet tested, and the upload is held in memory (Limits). - Only PDF has geometry. The other formats give pages and lines without positions (formats).
- Tried on 15 AMČR excavation reports, with no accuracy figure. The defects they showed were fixed or are listed below (what they showed).
- Known, not handled. Unruled tables are read by column, and letter-spaced captions and map labels split into pieces (the AMČR samples).
- DOC and XLS need LibreOffice. The images carry it. Without it these files fail with
dependency_missing, so use the image or convert them to DOCX or XLSX first (errors). - The Docling engine is opt-in. It reads PDF only and needs a GPU and its models (
--engine docling); the default light engine needs neither (usage).
| For | To | Read |
|---|---|---|
| using it | set it up, run the command line, the service, Docker | docs/usage.md |
| read the record as annotated Markdown | docs/reference/markdown.md | |
| look up the formats and the record's fields | docs/reference/formats.md | |
| see the paradata, licences and the record schema | docs/reference/paradata.md | |
| running the service | deploy and operate it, read its errors and limits | service/README.md |
| see the API contract | service/openapi.json | |
| set every option | docs/configuration.md | |
| judging the results | see what the AMČR samples showed | docs/evaluation/amcr-samples.md |
| run the document-understanding benchmark | docs/evaluation/benchmark.md | |
| changing the code | understand the text layers, needs_ocr and the routes |
docs/how-it-works.md |
| set up, test, release; find the history | docs/development.md | |
| follow the conventions | CONTRIBUTING.md |
Terms used across the tools are explained in the ATRIUM glossary. The documentation follows the ATRIUM documentation style.
The code is MIT (LICENSE). It reads PDFs with pdfplumber/pdfminer.six and pypdfium2, office files with python-docx and lxml (and the reader shared with atrium-ocr-postprocess), the legacy formats through LibreOffice (MPL-2.0), and, opt-in, Docling; the service runs on FastAPI and uvicorn, and flexiconv is an optional TEITOK adapter (MIT). The inventory of components and their licences is para_config.txt; what a run records about them is in docs/reference/paradata.md. Cite with CITATION.cff.
Developed by ÚFAL, Charles University, with AMČR (the Archaeological Map of the Czech Republic) as the pipeline partner. Funded by ATRIUM (GA 101132163), with the support of the LINDAT/CLARIAH-CZ research infrastructure and the AIS CR infrastructure (LM2023031). Contact: the issue tracker.
Formerly atrium-llm-enrich; its history is in docs/development.md.