Skip to content

About

Digital-born documenta transformation into JSONs/MDs and enrichments of documents with page content information

Topics

Resources

Contributing

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

359 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ATRIUM digital-convert: born-digital documents into ATRIUM document records

digital-convert reads a born-digital document, one made on a computer, such as a PDF or an office file. It writes the text as an ATRIUM document record: pages, lines in reading order, headings, tables and, for a PDF, exact positions. Later ATRIUM stages add their results to the same record. Use it when the text is already in the file. A scan belongs to the OCR route. It needs no OCR, no model and no network.

Releases · MIT licence · ATRIUM documentation

Inputs and outputs

Input Output
PDF, DOCX, ODT, ODS, XLSX, RTF, recognised by content; DOC and XLS through LibreOffice the record: source, pages, lines, content, tables, provenance, validated against the frozen schema
optionally, AMČR's seed record (doc_id, source.sha512), checked against the file per page, text_layer and, where the text cannot be used as it is, needs_ocr with its reason
on request, the record as annotated Markdown; with /describe, a per-page assessment and where each page goes next

The record follows the hub's document contract. Each format, how it is read and what reaches the record: docs/reference/formats.md.

The service and the command line take the same inputs and give the same results. They run the same code for the record, the Markdown and the per-page assessment:

Result Service Command line (api_util/digital_to_json.py)
the record POST /reformat → document_json <doc_id>.document.json (or --document-json-out PATH)
the record as annotated Markdown markdown=true → markdown --markdown → <doc_id>.md
the per-page assessment, the routes POST /describe → pages, summary, stages --describe → <doc_id>.describe.json (--stages, --classify-pages, --no-text)
a seed record to add to the document_json part --document-json
the run's paradata paradata, a CreateAction in the answer a file, with --paradata-dir

Only the command line has the opt-in Docling engine (--engine docling, a local build); the service image runs the light engine. Every difference, and why: docs/usage.md. Two offline research tools, outside the images and the service, ask a decision model the AMČR document type of PDF pages and documents, from the rendered pages alone (usage).

Quick start

What you need: Docker for the service; for the sample files and the command line, a checkout of this repository and Python 3.11 (the version of the image and of CI). No network is needed to convert a file, and no GPU. The /describe endpoint calls page-classification and ocr-postprocess only when their URLs are set. DOC and XLS files need LibreOffice, which the images carry.

The service, from the published image, on small documents generated from this repository's test fixtures:

python3 tests/fixtures/digital/make_fixtures.py --outdir fixtures     # 17 small PDFs and DOCX files
docker run --rm -p 8000:8000 ghcr.io/ufal/atrium-digital-convert-api:<version>   # the release without its v
curl --fail-with-body -sS -X POST localhost:8000/reformat -F "file=@fixtures/minimal.pdf" -o out.json

Expected: HTTP 200 and out.json with document_json, whose source.origin is digital-born-pdf, whose two pages are digital, and whose first lines read "Excavation report, block one, line one." POST /describe with the same file adds pages[] with a route for each page; it calls page-classification and ocr-postprocess only when their URLs are set, and says so in stages[].

The same conversion on the command line, from a checkout with its setup done:

python3 api_util/digital_to_json.py fixtures/minimal.pdf --out-dir out --markdown --describe

Expected: out/minimal.document.json (the record /reformat returns), out/minimal.md (its markdown) and out/minimal.describe.json (the pages, summary and stages of /describe; without the stage URLs both stages read not_configured). With the stage URLs set, --describe keeps what the stages write into the record, and the record and the Markdown then show it, as /describe's document_json does.

How it works

Each page's text layer is checked before anything is trusted:

  • digital: the layer decodes;
  • garbled: it does not (broken font encodings, glyphs without a Unicode mapping, words losing letters);
  • ocr: it is an earlier OCR run;
  • none: the page has no text, or its only text is a page number over a scan.

Every page that is not digital (or blank) is flagged needs_ocr with a reason. Those pages go to the OCR route, and the record is still written. /describe then decides, page by page, where a page goes next: to NLP, to OCR, to handwriting recognition or nowhere. It uses page-classification's page types and ocr-postprocess's line quality when those services are deployed. The rules, with an example: docs/how-it-works.md.

Where it fits in ATRIUM

For an AMČR upload that is born-digital, digital-convert is the first stage and the record's originator: it alone writes the positional part of the record. Its record goes on to nlp-enrich and keyword-extract; the pages it flags go to the OCR/HTR route, and ocr-postprocess merges their new text back into the same record, page by page. In the AMČR deployment Temporal chains the stages, and /reformat calls no other service. The whole chain: the hub's workflow page. Its SSH Open Marketplace record: Born-digital document converter: PDF and office files into structured text without OCR - ATRIUM.

Limits and maturity

  • Born-digital only. A PDF whose pages are mostly an earlier OCR run (OCR_LAYER_DOCUMENT_SHARE, 0.5) is refused as ocr_text_layer, so send it to the OCR route. Such pages inside a born-digital PDF are flagged for OCR (errors).
  • Uploads up to 250 MB by default. AMČR's originals reach about 1.5 GB, so a deployment must raise MAX_UPLOAD_MB for them. Above 250 MB, memory, concurrency and timeouts are not yet tested, and the upload is held in memory (Limits).
  • Only PDF has geometry. The other formats give pages and lines without positions (formats).
  • Tried on 15 AMČR excavation reports, with no accuracy figure. The defects they showed were fixed or are listed below (what they showed).
  • Known, not handled. Unruled tables are read by column, and letter-spaced captions and map labels split into pieces (the AMČR samples).
  • DOC and XLS need LibreOffice. The images carry it. Without it these files fail with dependency_missing, so use the image or convert them to DOCX or XLSX first (errors).
  • The Docling engine is opt-in. It reads PDF only and needs a GPU and its models (--engine docling); the default light engine needs neither (usage).

Documentation

For To Read
using it set it up, run the command line, the service, Docker docs/usage.md
read the record as annotated Markdown docs/reference/markdown.md
look up the formats and the record's fields docs/reference/formats.md
see the paradata, licences and the record schema docs/reference/paradata.md
running the service deploy and operate it, read its errors and limits service/README.md
see the API contract service/openapi.json
set every option docs/configuration.md
judging the results see what the AMČR samples showed docs/evaluation/amcr-samples.md
run the document-understanding benchmark docs/evaluation/benchmark.md
changing the code understand the text layers, needs_ocr and the routes docs/how-it-works.md
set up, test, release; find the history docs/development.md
follow the conventions CONTRIBUTING.md

Terms used across the tools are explained in the ATRIUM glossary. The documentation follows the ATRIUM documentation style.

Licence, citation and acknowledgements

The code is MIT (LICENSE). It reads PDFs with pdfplumber/pdfminer.six and pypdfium2, office files with python-docx and lxml (and the reader shared with atrium-ocr-postprocess), the legacy formats through LibreOffice (MPL-2.0), and, opt-in, Docling; the service runs on FastAPI and uvicorn, and flexiconv is an optional TEITOK adapter (MIT). The inventory of components and their licences is para_config.txt; what a run records about them is in docs/reference/paradata.md. Cite with CITATION.cff.

Developed by ÚFAL, Charles University, with AMČR (the Archaeological Map of the Czech Republic) as the pipeline partner. Funded by ATRIUM (GA 101132163), with the support of the LINDAT/CLARIAH-CZ research infrastructure and the AIS CR infrastructure (LM2023031). Contact: the issue tracker.

Formerly atrium-llm-enrich; its history is in docs/development.md.

About

Digital-born documenta transformation into JSONs/MDs and enrichments of documents with page content information

Topics

Resources

Contributing

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages