| Format | Parser | External Deps | Notes |
|---|---|---|---|
.txt | Stdlib | None | Split by blank lines |
.md | Stdlib regex | None | Headings, lists, code blocks, blockquotes |
.html | Stdlib HTMLParser | None | Extracts body content with structure |
.pdf | pdfplumber | pip install pdfplumber | Text + table extraction |
.docx | python-docx | pip install python-docx | Paragraphs, headings, tables |
| Images | Tesseract OCR | pip install pytesseract | Via OCR interface |
Choose the right chunking strategy based on your documents:
Is the document well-structured (headings, sections)?
├── Yes → Use SEMANTIC chunking
│ (each section becomes one or more chunks with heading context)
└── No
├── Is it prose (articles, reports)?
│ └── Use SENTENCE chunking (preserves sentence boundaries)
├── Is it technical docs with clear paragraphs?
│ └── Use PARAGRAPH chunking (atomic paragraph units)
└── Is it raw/unstructured text?
└── Use FIXED or RECURSIVE chunking| Embedding Model | Recommended Chunk Size | Max Tokens |
|---|---|---|
| OpenAI text-embedding-3-small | 500-800 chars | 8191 |
| Cohere embed-v3 | 500-1000 chars | 512 |
| sentence-transformers | 300-500 chars | 512 |
| E5-large | 300-500 chars | 512 |
Overlap: 10-20% of chunk size prevents information loss at boundaries.
Define a schema for structured data extraction:
from src.schema_extractor import ExtractionSchema, SchemaExtractor
schema = ExtractionSchema("contract_data")
schema.add_field("parties", "list", "Names of contracting parties", required=True)
schema.add_field("effective_date", "date", "Contract start date", required=True)
schema.add_field("termination_date", "date", "Contract end date")
schema.add_field("total_value", "currency", "Total contract value")
schema.add_field("governing_law", "string", "Jurisdiction / governing law")
extractor = SchemaExtractor(schema)
result = extractor.extract(document_text)The regex fallback handles common types (dates, emails, currency, phones) without an LLM.
For complex extraction, provide an llm_fn callback.
| Strategy | Best For | How It Works |
|---|---|---|
| Extractive | Any length, no LLM needed | Picks top sentences by word frequency + position |
| Direct | Short docs (< 3000 chars) | Sends full text to LLM |
| Map-Reduce | Medium docs (3k-30k chars) | Summarize chunks, then combine |
| Refine | Long docs (30k+ chars) | Iteratively refine summary with each chunk |
Get the full Document AI Toolkit and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.