Series: Building Quorentra CRM from Zero — A Modular, ChatGPT-Native AI CRM

Building asynchronous document processing, PDF, DOCX, PPTX, TXT and CSV extraction, normalized text, page and slide provenance, retries, idempotency, failure handling, and AI-ready document content.

Quorentra Building the Document Text Extraction Pipeline: Building from Zero — Part 27

1. Introduction

In Part 26, Quorentra gained a first-class Document domain.

The platform can now securely manage:

			
Document Upload
Document Metadata
Binary Storage
Document Links
CRM Relationships
Permissions
Tenant Isolation
Authorized Downloads
Soft Deletion
Audit Events

		

A proposal can now exist as an authoritative CRM entity:

			
Adventure Works
      ↓
Azure Migration
      ↓
Proposal-v3.pdf

		

Quorentra knows:

			
Document ID
Filename
Display Name
MIME Type
File Size
Checksum
Storage Location
Uploader
Tenant
CRM Relationships

		

But there is still an important limitation.

Quorentra knows that the document exists.

It does not yet know what the document says.

That changes in Part 27.

We will build the:

Document Text Extraction Pipeline

Its responsibility is deliberately narrow:

			
Document Binary
      ↓
Format-Specific Extraction
      ↓
Structured Extracted Content
      ↓
Normalization
      ↓
Provenance
      ↓
Persistent Extracted Text

		

We are still not building:

			
Semantic Chunking
Embeddings
Vector Search
RAG
Document Question Answering

		

Those layers come later.

First we need trustworthy text.

2. Our Target Interaction

Suppose Quorentra contains:

Proposal-v3.pdf

linked to:

			
Adventure Works
      ↓
Azure Migration

The processing pipeline receives:

			
Document ID:
doc_123
MIME Type:
application/pdf
Storage Key:
organizations/.../documents/.../original

		

Quorentra creates:

DocumentProcessingJob

The worker loads the binary and selects:

PDFExtractor

The extractor produces:

			
Page 1
Title
Executive Summary
Page 2
Customer Requirements
Page 3
Proposed Architecture
...
Page 14
Commercial Proposal
Page 15
Terms and Conditions

		

The system stores this as structured extracted content.

Later, other modules will be able to transform it into searchable knowledge.

3. Extraction Is Infrastructure, Not AI Reasoning

The extractor should answer:

What textual content can be recovered from this document?

It should not answer:

What does this document mean?

Those are different responsibilities.

Extraction:

			
PDF
      ↓
"The proposed migration..."

Reasoning:

			
"The proposed migration..."
      ↓
Customer intends to migrate 120 workloads.

Part 27 builds the first operation.

Not the second.

4. Why This Separation Matters

If extraction and AI reasoning are combined:

			
Document
      ↓
LLM
      ↓
Summary

		

we immediately lose several important properties:

			
Determinism
Traceability
Reusability
Page provenance
Extraction debugging
Independent testing

		

Instead:

			
Document
      ↓
Deterministic Extraction
      ↓
Authoritative Extracted Text
      ↓
Future AI Processing

		

This gives us a much stronger foundation.

5. The Document Knowledge Pipeline

We are gradually constructing:

			
Document
      ↓
Text Extraction          ← Part 27
      ↓
Normalization
      ↓
Chunking                 ← Later
      ↓
Embeddings               ← Later
      ↓
Vector Storage           ← Later
      ↓
Retrieval                ← Later
      ↓
CRM Context
      ↓
ChatGPT

		

Each layer has one responsibility.

6. Starting Checkpoint

Before implementing Part 27, verify Part 26.

Run:

			
cd backend
python -m pytest

Then:

			
cd ..\chatgpt-ui
npm run build

Verify a Document can be:

			
Uploaded
      ↓
Validated
      ↓
Stored
      ↓
Linked to CRM Entity
      ↓
Retrieved
      ↓
Downloaded

		

Also test:

Show me the documents for the Azure Migration opportunity.

The result should show document metadata without pretending to understand document contents.

7. New Permission

Introduce:

documents.process

This is distinct from:

			
documents.read
documents.create
documents.delete

Processing may consume substantial system resources.

It should therefore have an explicit authorization boundary.

8. Who Needs `documents.process`?

Human users may not normally call processing directly.

Instead, processing may be initiated automatically after upload.

However, the permission remains useful for:

			
Manual reprocessing
Administrative operations
ChatGPT-triggered reprocessing
Future batch processing

For normal uploads, the application can initiate processing through an internal trusted service.

9. Processing Must Be Asynchronous

Do not build:

			
POST /documents
      ↓
Upload
      ↓
Extract PDF
      ↓
Normalize
      ↓
Store Extraction
      ↓
Return Response

		

A 100-page PDF could make the upload request slow and fragile.

Instead:

			
Upload
      ↓
Document AVAILABLE
      ↓
Create Processing Job
      ↓
Return Upload Success

		

Then:

			
Worker
      ↓
Process Document

The user does not need to wait for extraction.

10. Processing Architecture

Conceptually:

                 Document Upload
                       │
                       ▼
                    Document
                       │
                       ▼
            DocumentProcessingJob
                       │
                       ▼
                 Worker Queue
                       │
                       ▼
              Processing Worker
                       │
                       ▼
               Extractor Router
                       │
        ┌──────────────┼──────────────┐
        │              │              │
        ▼              ▼              ▼
   PDFExtractor   DOCXExtractor   PPTXExtractor
        │              │              │
        └──────────────┼──────────────┘
                       ▼
                 ExtractedDocument
                       │
                       ▼
                   Normalize
                       │
                       ▼
                Persist Content

11. Keep the Queue Abstract

For local development, we do not necessarily need:

			
Kafka
RabbitMQ
Celery
Redis

immediately.

A simple processing abstraction is sufficient.

For example:

DocumentProcessingQueue

with a development implementation.

Later it can be backed by:

			
Celery + Redis
RQ
Dramatiq
Cloud Tasks
Service Bus
SQS

		

without changing the Document Processing domain.

12. Define the Processing Job

Introduce:

DocumentProcessingJob

Conceptually:

			
DocumentProcessingJob
├── id
├── organization_id
├── document_id
├── status
├── extractor_name
├── extractor_version
├── attempt_count
├── started_at
├── completed_at
├── failed_at
├── error_code
├── error_message
├── created_at
└── updated_at

		

This gives processing its own lifecycle.

13. Processing Status

Define:

			
class DocumentProcessingStatus(str, Enum):
    PENDING = "pending"
    PROCESSING = "processing"
    COMPLETED = "completed"
    FAILED = "failed"

		

Later we may introduce:

			
CANCELLED
QUARANTINED
RETRYING

But the MVP does not need them yet.

14. Processing Lifecycle

Normal flow:

			
PENDING
   ↓
PROCESSING
   ↓
COMPLETED

		

Failure:

			
PENDING
   ↓
PROCESSING
   ↓
FAILED

		

Retry:

			
FAILED
   ↓
PENDING
   ↓
PROCESSING

		

if retry policy permits.

15. Processing Job Is Not the Document

Do not overload:

Document.status

with extraction workflow state.

The Document remains:

AVAILABLE

while processing may be:

			
PENDING
PROCESSING
FAILED
COMPLETED

This distinction matters.

A failed extraction does not necessarily mean the original Document is unavailable.

16. Example

			
Document:
Proposal-v3.pdf
Status:
AVAILABLE
Processing:
FAILED
Reason:
Unsupported PDF encoding

		

The user can still download the original PDF.

Only AI knowledge processing has failed.

17. Processing Attempts

Track:

attempt_count

Example:

			
Attempt 1
→ temporary storage failure
Attempt 2
→ successful

This becomes useful for diagnostics and retry control.

18. Avoid Infinite Retries

Do not create:

			
FAILED
      ↓
Retry
      ↓
FAILED
      ↓
Retry
      ↓
...

		

forever.

Define:

MAX_PROCESSING_ATTEMPTS = 3

or another configurable limit.

19. Retryable Versus Permanent Errors

Not every failure should be retried.

Retryable:

			
Temporary storage unavailable
Database timeout
Worker interruption
Transient infrastructure error

Permanent:

			
Unsupported format
Corrupt document
Password-protected document
Invalid encoding
No compatible extractor

		

The processing service should distinguish them.

20. Structured Error Codes

Introduce errors such as:

			
DOCUMENT_PROCESSING_FAILED
DOCUMENT_FORMAT_UNSUPPORTED
DOCUMENT_CORRUPT
DOCUMENT_PASSWORD_PROTECTED
DOCUMENT_EXTRACTION_EMPTY
DOCUMENT_ENCODING_INVALID
DOCUMENT_STORAGE_UNAVAILABLE
DOCUMENT_EXTRACTOR_FAILED

		

Do not rely only on arbitrary exception strings.

21. Extractor Abstraction

Introduce:

DocumentExtractor

Conceptually:

			
class DocumentExtractor(Protocol):
    supported_mime_types: set[str]
    async def extract(
        self,
        *,
        document: Document,
        content: BinaryIO,
    ) -> "ExtractedDocument":
        ...

		

Each format gets its own implementation.

22. Why an Extractor Interface?

Because:

			
PDF
DOCX
PPTX
TXT
CSV

		

are fundamentally different formats.

Trying to handle everything in one function produces brittle code.

Instead:

			
DocumentExtractor
├── PDFExtractor
├── DOCXExtractor
├── PPTXExtractor
├── TXTExtractor
└── CSVExtractor

		

23. Extraction Router

Introduce:

DocumentExtractionRouter

It receives:

			
MIME Type
File Extension

and selects the appropriate extractor.

Example:

			
application/pdf
      ↓
PDFExtractor

24. Router Must Not Trust Extension Alone

A file called:

proposal.pdf

may not actually be a PDF.

The upload layer already performs validation.

The extraction layer should still use authoritative metadata and fail safely if the binary is incompatible.

25. ExtractedDocument

Define a format-neutral intermediate model.

Conceptually:

			
class ExtractedDocument(BaseModel):
    document_id: UUID
    extractor_name: str
    extractor_version: str
    units: list["ExtractedUnit"]
    raw_text: str
    normalized_text: str
    character_count: int
    word_count: int

		

This is the common output from all extractors.

26. Extracted Units

Different formats have different natural units.

PDF:

Page

PPTX:

Slide

DOCX:

Section / Paragraph

TXT:

Text Block

CSV:

Row / Logical Block

We need a common representation without erasing format-specific provenance.

27. ExtractedUnit

Conceptually:

			
class ExtractedUnit(BaseModel):
    unit_type: str
    unit_index: int
    text: str
    page_number: int | None = None
    slide_number: int | None = None
    section_name: str | None = None
    metadata: dict[str, Any] = {}

		

This becomes an important provenance boundary.

28. Why Preserve Units?

Suppose later ChatGPT answers:

The proposed implementation period is six months.

We want to trace that to:

			
Proposal-v3.pdf
Page 17

If extraction immediately flattens everything into one giant string, that provenance becomes much harder to recover.

29. Persistent Extraction Model

Introduce:

DocumentExtraction

Conceptually:

			
DocumentExtraction
├── id
├── organization_id
├── document_id
├── processing_job_id
├── extractor_name
├── extractor_version
├── raw_text
├── normalized_text
├── character_count
├── word_count
├── created_at
└── version

		

This represents the extraction result.

30. Extraction Units Table

Introduce:

DocumentExtractionUnit

Conceptually:

			
DocumentExtractionUnit
├── id
├── organization_id
├── extraction_id
├── document_id
├── unit_type
├── unit_index
├── page_number
├── slide_number
├── section_name
├── raw_text
├── normalized_text
├── character_count
└── created_at

		

This preserves granular provenance.

31. Why Store Both Document and Extraction IDs?

document_id provides direct provenance.

extraction_id identifies the specific extraction run.

This matters if we later reprocess a Document with a better extractor.

32. Extractor Version

Always record:

			
extractor_name
extractor_version

Example:

			
pdf_text_extractor
1.0.0

Later:

			
pdf_text_extractor
1.1.0

may produce better results.

Without version metadata, we cannot explain why extraction output changed.

33. Extraction Is Reproducible Processing

Think of:

			
Document Binary
+
Extractor Version
=
Extraction Result

		

This gives us a clear lineage.

Later:

			
Extraction
+
Chunker Version
=
Chunks

		

and:

			
Chunk
+
Embedding Model
=
Vector

		

This provenance chain becomes extremely important.

34. PDF Extraction

PDF is likely to be the most important format.

But PDF is also one of the most difficult.

A PDF may contain:

			
Native text
Scanned images
Tables
Columns
Headers
Footers
Embedded fonts
Rotated text
Forms

		

Part 27 should start with digitally readable PDFs.

OCR can be added separately.

35. PDFExtractor

Conceptually:

			
class PDFExtractor(DocumentExtractor):
    async def extract(
        self,
        *,
        document: Document,
        content: BinaryIO,
    ) -> ExtractedDocument:
        ...

		

It should process each page independently.

36. PDF Page Units

For a 20-page PDF:

			
DocumentExtraction
      ↓
20 DocumentExtractionUnits

Each unit:

			
unit_type = "page"
page_number = 1
...
page_number = 20

37. Preserve Page Numbers

Do not use only zero-based internal indexes.

Store user-facing:

page_number = 1

while:

unit_index = 0

may remain useful internally.

This makes later citations intuitive.

38. Scanned PDFs

Suppose the PDF contains only scanned images.

Native extraction returns almost no text.

Do not silently treat that as successful high-quality extraction.

Detect:

			
Very low extracted character count
relative to page count

and flag:

OCR_REQUIRED

or an equivalent processing outcome.

39. OCR Is Not Part 27

We deliberately defer:

			
Image
      ↓
OCR
      ↓
Text

		

to a later extension.

For Part 27:

			
Native text PDF
→ supported
Scanned PDF
→ detected and reported

This keeps the module focused.

40. Password-Protected PDFs

If extraction requires a password:

DOCUMENT_PASSWORD_PROTECTED

Do not attempt to bypass protection.

The Document remains stored but extraction fails.

41. DOCX Extraction

DOCX is structurally different from PDF.

It may contain:

			
Headings
Paragraphs
Tables
Lists
Headers
Footers
Text boxes

		

For the MVP, prioritize:

			
Paragraph text
Heading text
Table text

in document order where practical.

42. DOCX Natural Units

Possible unit strategy:

			
Heading / Section
      ↓
Paragraphs

For the first implementation, we can store logical blocks.

Example:

			
Unit 1
Heading:
Executive Summary
Unit 2
Paragraph:
Adventure Works requires...
Unit 3
Table:
Service | Price | Duration

		

43. Preserve Headings

Headings are valuable later for semantic chunking.

Therefore if the parser exposes:

			
Heading 1
Heading 2
Heading 3

preserve that information in metadata.

Example:

			
{
  "unit_type": "paragraph",
  "section_name": "Commercial Proposal",
  "metadata": {
    "style": "Heading 2"
  }
}

		

44. Tables in DOCX

Do not discard tables.

A pricing table may contain some of the most important customer information.

For Part 27, convert tables into a stable textual representation.

For example:

			
Service | Quantity | Price
Migration Assessment | 1 | €10,000
Migration Delivery | 1 | €80,000

Preserve row ordering.

45. PPTX Extraction

Presentations are common CRM attachments.

A PPTX contains:

			
Slides
Titles
Text boxes
Tables
Speaker notes

		

For the MVP, extract:

			
Slide title
Visible text
Tables

Speaker notes can be added if the chosen library supports them reliably.

46. PPTX Units

Each slide becomes:

DocumentExtractionUnit

with:

			
unit_type = "slide"
slide_number = 1

This gives us later provenance such as:

Azure-Architecture.pptx, slide 7.

47. Preserve Slide Order

Extract text in a deterministic order.

Perfect visual reading order can be difficult.

The MVP does not need full layout reconstruction.

It does need:

			
Stable extraction
Slide provenance
Repeatable results

48. TXT Extraction

TXT looks simple but encoding matters.

Possible encodings include:

			
UTF-8
UTF-8 BOM
Windows-1252
Latin-1

Prefer UTF-8.

If decoding fails, use a controlled detection/fallback strategy.

Do not silently replace large amounts of invalid text.

49. TXT Units

For a plain text document, units can be:

paragraphs

or bounded text blocks.

Avoid creating one enormous unit for a multi-megabyte TXT file.

50. CSV Extraction

CSV is technically structured data rather than ordinary prose.

But CRM Documents may contain:

			
Pricing
Asset inventories
Customer lists
Project schedules
Requirements matrices

		

For Part 27, extract a textual representation while preserving row structure.

51. CSV Example

Input:

			
Service,Quantity,Price
Assessment,1,10000
Migration,1,80000

Extracted representation:

			
Service | Quantity | Price
Assessment | 1 | 10000
Migration | 1 | 80000

Later we may build a dedicated structured spreadsheet ingestion pipeline.

Not yet.

52. XLSX Is Different

Part 26 allowed XLSX uploads.

But robust spreadsheet extraction requires handling:

			
Workbooks
Worksheets
Cells
Formulas
Merged cells
Tables
Hidden sheets
Large datasets

		

That deserves more deliberate treatment.

Therefore Part 27 can initially mark:

			
XLSX
→ upload supported
→ extraction not yet supported

rather than pretending CSV extraction is equivalent to spreadsheet extraction.

53. Images

Likewise:

			
PNG
JPEG

can remain valid CRM Documents.

But text extraction requires OCR or multimodal processing.

Therefore:

			
Image upload
✓
Native text extraction
—

until OCR is implemented.

54. Extraction Support Matrix

At the end of Part 27:

Format	Upload	Native Extraction
PDF with text	✓	✓
Scanned PDF	✓	OCR required
DOCX	✓	✓
PPTX	✓	✓
TXT	✓	✓
CSV	✓	✓
XLSX	✓	Later
PNG	✓	OCR required
JPEG	✓	OCR required

This makes system capabilities explicit.

55. Raw Text Versus Normalized Text

Store both:

raw_text

and:

normalized_text

Why?

Because normalization rules may improve later.

The raw extraction allows us to re-normalize without re-reading the original binary.

56. Raw Text

Raw text should stay as close as practical to extractor output.

For example:

			
Adventure   Works
Azure Migration
Commercial Proposal

This is useful for debugging.

57. Normalized Text

Normalization may transform it into:

			
Adventure Works
Azure Migration
Commercial Proposal

The goal is not semantic rewriting.

The goal is consistent text hygiene.

58. Normalization Pipeline

Conceptually:

			
Raw Extracted Text
      ↓
Unicode Normalization
      ↓
Line Ending Normalization
      ↓
Whitespace Cleanup
      ↓
Control Character Removal
      ↓
Repeated Blank Line Reduction
      ↓
Normalized Text

		

59. Unicode Normalization

Use a consistent Unicode normalization form.

For example:

NFC

This prevents visually identical text from having unnecessarily different byte representations.

60. Line Endings

Normalize:

			
\r\n
\r
\n

to one internal representation:

\n

This makes downstream processing more predictable.

61. Whitespace

Avoid aggressive normalization.

Do not transform:

€ 90,000

into something semantically different.

Only normalize obvious formatting noise.

62. Preserve Paragraph Boundaries

Do not collapse the entire document into one line.

Paragraph boundaries contain structural information useful for later chunking.

Preserve:

			
Paragraph A
Paragraph B
Paragraph C

63. Preserve Table Boundaries

Likewise:

			
Service | Price
Migration | €80,000

should remain distinguishable from ordinary prose.

Later chunking may treat tables differently.

64. Header and Footer Noise

PDFs often repeat:

			
Adventure Works Confidential
Page 1 of 20

on every page.

Part 27 does not need sophisticated header/footer removal.

We may preserve the text initially.

Later normalization can detect repeated boilerplate.

Avoid premature heuristics that accidentally remove meaningful content.

65. Character Count

Store:

character_count

for:

			
Extraction
Extraction Unit

This helps:

			
Quality checks
OCR detection
Chunk planning
Metrics

66. Word Count

Store:

word_count

at least at the full extraction level.

This gives another simple quality signal.

67. Empty Extraction Detection

A technically successful parser may return:

""

That should not automatically become:

COMPLETED

Define a minimum useful extraction threshold.

For example:

character_count < threshold

may produce:

DOCUMENT_EXTRACTION_EMPTY

or:

OCR_REQUIRED

depending on file type.

68. Extraction Quality Metadata

We can introduce simple metadata such as:

			
page_count
unit_count
character_count
word_count
empty_unit_count

		

Do not invent an elaborate AI quality score yet.

Deterministic measurements are sufficient.

69. Extraction Provenance

Every extraction must trace to:

			
Tenant
Document
Processing Job
Extractor
Extractor Version
Extraction Unit
Page / Slide / Section

		

This gives us a provenance chain.

70. Provenance Chain

Conceptually:

			
Adventure Works
      ↓
Azure Migration
      ↓
Proposal-v3.pdf
      ↓
Document ID
      ↓
Extraction ID
      ↓
Page 14
      ↓
Extracted Text

		

Later:

			
Extracted Text
      ↓
Chunk
      ↓
Embedding
      ↓
Search Result
      ↓
ChatGPT Answer

		

Nothing should lose its origin.

71. Idempotent Processing

Suppose the same processing job is accidentally delivered twice.

We must avoid:

			
Extraction A
Extraction B

being created unintentionally for the same job.

Processing must be idempotent.

72. Processing Idempotency Key

A useful identity may be:

			
document_id
+
document checksum
+
extractor name
+
extractor version

		

This tells us whether equivalent extraction already exists.

73. Example

Document:

			
checksum =
abc123

Extractor:

			
pdf_text_extractor
1.0.0

If a completed extraction already exists for:

			
abc123
+
pdf_text_extractor
+
1.0.0

		

the system can reuse it or avoid duplicate work according to policy.

74. Reprocessing

Reprocessing is different.

User:

Reprocess this document.

Maybe a new extractor version exists:

1.1.0

Then:

			
same Document
+
new Extractor Version
=
new Extraction

		

This should be allowed.

75. Current Extraction

If multiple extractions exist, the Document needs a way to identify which one is current.

Possible approach:

Document.current_extraction_id

But this introduces a cross-domain state relationship.

Another approach:

DocumentExtraction.is_current

For the MVP, a service can define:

latest successfully completed extraction

as current.

Keep it simple.

76. Do Not Delete Old Extractions Immediately

Old extraction results may be useful for:

			
Debugging
Regression comparison
Auditability
Reprocessing analysis

They can be retained initially.

Later retention policies may remove them.

77. Worker Processing Flow

The worker performs:

			
Claim PENDING Job
      ↓
Set PROCESSING
      ↓
Load Document
      ↓
Verify Tenant
      ↓
Verify AVAILABLE
      ↓
Load Binary
      ↓
Resolve Extractor
      ↓
Extract
      ↓
Normalize
      ↓
Validate Result
      ↓
Persist Extraction
      ↓
Persist Units
      ↓
Set COMPLETED
      ↓
Commit

		

78. Claiming Jobs Safely

Multiple workers may see the same pending job.

We need to avoid:

			
Worker A
→ processes job
Worker B
→ processes same job simultaneously

Use database locking or an atomic claim operation.

79. Example Claim Strategy

Conceptually:

			
SELECT pending job
FOR UPDATE SKIP LOCKED

or the equivalent queue-level guarantee.

The exact implementation depends on the processing backend.

80. Worker Crash

Suppose:

			
Job =
PROCESSING

and the worker crashes.

The job must not remain stuck forever.

Later recovery can detect:

			
PROCESSING
+
started_at older than threshold

and mark it retryable.

81. Processing Timeout

Introduce:

DOCUMENT_PROCESSING_TIMEOUT_SECONDS

or an equivalent worker timeout.

One malformed file should not occupy a worker indefinitely.

82. Resource Limits

Extraction can consume:

			
CPU
Memory
Disk
Time

Therefore future production workers should have resource limits.

For the local MVP, at minimum enforce:

			
File size limits
Processing timeout
Attempt limits

83. Processing API

Introduce:

POST /api/v1/documents/{document_id}/processing

for authorized manual processing.

The normal upload path may enqueue automatically.

84. Processing Status API

Introduce:

GET /api/v1/documents/{document_id}/processing

Return:

			
{
  "document_id": "...",
  "status": "completed",
  "extractor": "pdf_text_extractor",
  "extractor_version": "1.0.0",
  "started_at": "...",
  "completed_at": "...",
  "character_count": 28492,
  "word_count": 4381
}

		

85. Extracted Text API

Introduce:

GET /api/v1/documents/{document_id}/extraction

This requires:

documents.read

and tenant authorization.

86. Should ChatGPT Receive the Entire Extracted Text?

Usually:

No.

A 200-page proposal may contain enormous amounts of text.

The direct extraction endpoint is useful for application and administrative purposes.

ChatGPT should eventually receive relevant chunks through retrieval.

That comes later.

87. ChatGPT Processing Tool

Expose:

get_document_processing_status

This allows questions such as:

Has the Adventure Works proposal finished processing?

ChatGPT can answer using authoritative job state.

88. Manual Reprocessing Tool

Optionally expose:

reprocess_document

behind:

documents.process

Because this triggers resource-consuming work, it should be governed carefully.

For the MVP, this may remain an admin/API operation.

89. Processing Widget

Create:

			
chatgpt-ui/src/documents/
└── DocumentProcessingStatus.tsx

Example:

			
┌────────────────────────────────────────────┐
│ Azure Migration Proposal v3                │
│                                            │
│ PDF · 2.4 MB                               │
│                                            │
│ Text extraction                            │
│ ✓ Completed                                │
│                                            │
│ 20 pages                                   │
│ 4,381 words                                │
│ 28,492 characters                          │
│                                            │
│ Extractor                                  │
│ PDF Text Extractor 1.0.0                   │
└────────────────────────────────────────────┘

		

90. Processing State Widget

While processing:

			
┌────────────────────────────────────────────┐
│ Azure Migration Proposal v3                │
│                                            │
│ Text extraction                            │
│ Processing...                              │
│                                            │
│ The original document remains available.   │
└────────────────────────────────────────────┘

		

This reinforces the distinction between Document availability and processing state.

91. Failed Processing Widget

Example:

			
┌────────────────────────────────────────────┐
│ Scanned Requirements.pdf                   │
│                                            │
│ Text extraction                            │
│ Could not extract readable text            │
│                                            │
│ Reason                                     │
│ OCR is required for this document.         │
│                                            │
│ The original file is still available.      │
└────────────────────────────────────────────┘

		

92. CRM Context Integration

Part 26 added Document metadata to CRM Context.

Part 27 can enrich that metadata with processing status.

Example:

			
Documents
│
├── Proposal-v3.pdf
│   └── text_status: completed
│
├── Requirements-Scan.pdf
│   └── text_status: ocr_required
│
└── Pricing.xlsx
    └── text_status: unsupported

		

Do not include the entire extracted text.

93. Why Processing Status Helps ChatGPT

User:

Can you analyze the Adventure Works proposal?

If processing is:

PENDING

ChatGPT should not pretend the content is available.

It can say:

The proposal is still being processed.

If:

COMPLETED

future retrieval layers can use it.

94. We Still Do Not Have Document Q&A

Even after Part 27, the user should not automatically receive an answer to:

What does the proposal say about pricing?

Why?

Because although extracted text now exists, we have not yet built a bounded retrieval strategy.

Sending an entire 100-page document to ChatGPT is not our architecture.

95. The Next Boundary

We now have:

			
Document
      ↓
Extraction
      ↓
Normalized Text

		

The next problem is:

How do we divide that text into useful, provenance-preserving retrieval units?

That is the chunking problem.

96. Do Not Use Fixed Characters Blindly

A naive implementation might do:

			
text[0:2000]
text[2000:4000]
text[4000:6000]

This can split:

			
sentences
paragraphs
tables
sections

in poor locations.

We will later build structure-aware chunking.

Part 27 only preserves enough structure to make that possible.

97. Extraction Data Minimization

Extracted content may contain sensitive customer information.

Therefore:

DocumentExtraction

must inherit the same tenant boundary as its Document.

Never create globally searchable extracted text.

98. Tenant Isolation

Every extraction query must include:

			
organization_id =
TenantContext.organization_id

The same applies to:

			
DocumentProcessingJob
DocumentExtraction
DocumentExtractionUnit

99. Permission Inheritance

If the user cannot read:

Document X

they cannot read:

Extraction of Document X

Extraction never creates a new permission path around the original Document.

100. Deleted Documents

If:

			
Document.status =
DELETED

normal extraction retrieval must not expose its content.

Existing extraction rows may remain for audit/retention purposes.

But they are no longer part of normal CRM retrieval.

101. Deletion and Future Vector Data

This rule will become important later.

Deleting a Document must eventually ensure that:

			
Chunks
Embeddings
Vector Index Entries

stop appearing in retrieval.

Part 27 establishes the ownership chain needed to enforce that.

102. Processing a Deleted Document

A worker may receive a queued job after the Document was deleted.

The worker must reload the Document.

If:

status != AVAILABLE

processing stops.

Never trust queued state blindly.

103. Processing a Changed Document

Part 26 treats each uploaded binary as an independent Document.

Therefore the binary should be immutable.

If the file changes, create a new Document.

This simplifies extraction provenance enormously.

104. Binary Immutability

Once stored:

Document checksum

must represent the binary permanently.

Do not replace the storage object behind an existing Document ID with new content.

Otherwise:

Extraction

could refer to a different binary than its recorded checksum.

105. Verify Checksum Before Processing

Optionally, the processing worker can recalculate or verify the stored binary checksum.

If it does not match:

Document.checksum_sha256

fail with an integrity error.

This detects storage corruption or unexpected replacement.

106. Extraction Audit

Do we need a full business AuditEvent every time automatic extraction occurs?

Possibly not.

Processing is primarily an operational workflow.

Use structured processing records and logs.

Manual reprocessing may create an audit event if desired.

107. Operational Logging

Useful fields:

			
processing_job_id
document_id
organization_id
mime_type
file_size_bytes
extractor_name
extractor_version
attempt_count
status
unit_count
character_count
word_count
duration_ms
error_code

		

Do not log the entire extracted text.

108. Metrics

Useful metrics include:

			
document_processing_jobs_total
document_processing_completed_total
document_processing_failed_total
document_processing_retries_total
document_processing_duration_ms
document_extracted_characters_total
document_extracted_words_total

		

Per extractor:

			
pdf_extractions_total
docx_extractions_total
pptx_extractions_total
txt_extractions_total
csv_extractions_total

		

109. Failure Metrics

Track:

			
ocr_required_total
password_protected_documents_total
corrupt_documents_total
unsupported_extraction_formats_total
empty_extractions_total

		

This will tell us where future development effort is needed.

110. Extraction Performance

Measure:

			
Document Size
Page Count
Processing Time
Character Count

This will later help with worker sizing and cost estimation.

111. Database Indexes

Useful indexes include:

			
document_processing_jobs
(
    organization_id,
    status,
    created_at
)
document_processing_jobs
(
    document_id,
    created_at
)
document_extractions
(
    organization_id,
    document_id,
    created_at
)
document_extraction_units
(
    extraction_id,
    unit_index
)

		

Tenant ID remains central.

112. Database Migration

Part 27 introduces:

			
document_processing_jobs
document_extractions
document_extraction_units

Run:

			
cd backend
alembic revision --autogenerate -m "add document extraction pipeline"

Review the generated migration carefully.

Then:

alembic upgrade head

Verify:

alembic current

113. Suggested Backend Structure

Extend the Document module:

			
backend/app/documents/
├── models.py
├── schemas.py
├── repository.py
├── service.py
├── routes.py
├── storage/
│   ├── base.py
│   └── local.py
│
└── processing/
    ├── models.py
    ├── schemas.py
    ├── service.py
    ├── queue.py
    ├── worker.py
    ├── router.py
    ├── normalization.py
    └── extractors/
        ├── base.py
        ├── pdf.py
        ├── docx.py
        ├── pptx.py
        ├── txt.py
        └── csv.py

		

This keeps document processing modular.

114. Dependencies

The exact Python libraries should remain implementation choices behind the extractor interfaces.

Conceptually, we need libraries capable of reading:

			
PDF
DOCX
PPTX
CSV
TXT

		

Do not allow third-party parser APIs to leak into application-layer contracts.

115. Library Isolation

For example:

			
PDFExtractor
      ↓
PDF Library

Only:

PDFExtractor

knows which library is used.

The rest of Quorentra receives:

ExtractedDocument

This makes future replacement easier.

116. Why Library Replacement Matters

PDF extraction quality varies significantly between libraries.

We may later replace:

Extractor 1

with:

Extractor 2

without changing:

			
Processing Service
Chunking
Embedding
RAG
ChatGPT

		

That is the value of modular boundaries.

117. PDF Extraction Tests

Test:

			
single-page PDF
multi-page PDF
empty PDF
PDF with headings
PDF with tables
password-protected PDF
corrupt PDF
scanned PDF
large PDF

		

Verify page provenance.

118. DOCX Extraction Tests

Test:

			
paragraphs
headings
lists
tables
empty document
large document
special characters
Unicode text

		

Verify document order remains deterministic.

119. PPTX Extraction Tests

Test:

			
single slide
multiple slides
titles
text boxes
tables
empty slides
Unicode

		

Verify slide numbers.

120. TXT Extraction Tests

Test:

			
UTF-8
UTF-8 BOM
Windows line endings
Unix line endings
Unicode
empty text
large text
invalid encoding

		

121. CSV Extraction Tests

Test:

			
comma delimiter
quoted values
embedded commas
Unicode
empty cells
large rows
multiline values

		

Preserve row structure.

122. Normalization Tests

Input:

Adventure    Works\r\n\r\n\r\nAzure Migration

Expected:

			
Adventure Works
Azure Migration

according to our normalization policy.

123. Preserve Meaning Test

Input:

			
Price: €90,000
Probability: 60%

Normalization must not alter:

			
€90,000
60%

Text cleanup is not semantic rewriting.

124. Unit Provenance Test

For PDF page 14:

			
unit_type =
page
unit_index =
13
page_number =
14

		

must remain correct after persistence.

125. Idempotency Test

Deliver the same job twice.

Expected:

one logical completed extraction

according to the processing identity policy.

126. Reprocessing Test

Process with:

Extractor 1.0.0

Then:

Extractor 1.1.0

Expected:

two traceable extraction versions

without losing the original.

127. Worker Concurrency Test

Two workers attempt to claim the same pending job.

Expected:

one worker processes it

The other moves to another job.

128. Worker Crash Test

Simulate crash after:

PROCESSING

Verify stale processing jobs can later be recovered.

129. Retry Test

Temporary storage failure.

Expected:

			
attempt 1
FAILED / retryable
attempt 2
COMPLETED

according to retry policy.

130. Permanent Failure Test

Password-protected PDF.

Expected:

			
FAILED
error_code =
DOCUMENT_PASSWORD_PROTECTED
retry =
false

		

131. OCR Detection Test

Scanned 20-page PDF with almost no native text.

Expected:

OCR_REQUIRED

or equivalent structured failure.

Do not mark it as a high-quality completed extraction.

132. Deleted Document Test

Queue processing.

Delete Document before worker executes.

Expected:

processing cancelled / rejected

No normal extraction becomes available.

133. Tenant Isolation Processing Test

Tenant B attempts to request processing for Tenant A Document ID.

Expected:

not found / forbidden

No job created.

134. Extraction Permission Test

User cannot read Document.

User requests extraction text.

Expected:

denied

Extraction does not bypass Document permissions.

135. Storage Integrity Test

Stored binary checksum differs from Document checksum.

Expected:

processing fails

No completed extraction is persisted.

136. Transaction Test

Suppose:

			
Extraction row created
Unit 1 created
Unit 2 created
Unit 3 fails

Expected:

no partial completed extraction

Use a transaction for extraction persistence.

137. Processing Job Completion Test

Only after:

			
Extraction persisted
+
Units persisted

should:

			
status =
COMPLETED

be committed.

138. Failed Job Data

A failed job should store enough diagnostic information to understand the failure.

But avoid storing:

			
entire document contents
large stack traces
secrets
filesystem credentials

in ordinary user-facing error fields.

139. User-Facing Errors

User-facing:

Text could not be extracted because this PDF appears to contain scanned pages and requires OCR.

Internal logs can contain more technical details.

Separate:

User Error

from:

Operational Diagnostic

140. Context Tests

Opportunity context contains:

			
Proposal-v3.pdf
text_status: completed

but does not include:

28,492 characters of extracted text

This keeps context bounded.

141. ChatGPT Grounding Test

User:

Can you read Proposal-v3.pdf?

If processing is:

FAILED

ChatGPT should not imply it has document content.

If:

COMPLETED

ChatGPT may say the text has been extracted, but full semantic Q&A still belongs to the retrieval layer.

142. No Premature RAG Test

Verify Part 27 introduces no requirement for:

			
pgvector
embedding model
vector index
semantic similarity
RAG prompt

		

The extraction module must function independently.

143. No LLM Dependency

The core extraction pipeline should not require an LLM.

That is a major architectural property.

If ChatGPT or an AI provider is unavailable:

Document Extraction

should still work.

144. Why This Matters

Quorentra now has a deterministic knowledge ingestion foundation.

That means the AI layer does not need to own basic document parsing.

The platform owns it.

145. Version Update

Part 27 introduces:

			
Document Processing Jobs
Asynchronous Processing
Extractor Architecture
PDF Extraction
DOCX Extraction
PPTX Extraction
TXT Extraction
CSV Extraction
Normalization
Extraction Persistence
Extraction Units
Page Provenance
Slide Provenance
Extractor Versioning
Retry Handling
Idempotency
Processing Metrics

		

Update:

app/core/constants.py

from:

APP_VERSION = "0.13.0"

to:

APP_VERSION = "0.14.0"

146. Quorentra 0.14.0

Our modular MVP now looks like:

			
Platform
├── FastAPI                           ✓
├── PostgreSQL                        ✓
├── SQLAlchemy                        ✓
├── Alembic                           ✓
├── Document Storage                  ✓
└── Processing Worker Boundary        ✓
Identity
├── Organizations                     ✓
├── Users                             ✓
├── Memberships                       ✓
├── Authentication                    ✓
└── JWT                               ✓
Security
├── TenantContext                     ✓
├── Tenant Isolation                  ✓
├── RBAC                              ✓
├── Domain Permissions                ✓
├── Document Authorization            ✓
└── Processing Authorization          ✓
CRM
├── Companies                         ✓
├── Contacts                          ✓
├── Opportunities                     ✓
├── Activities                        ✓
├── Tasks                             ✓
├── Meetings                          ✓
└── Documents                         ✓
Documents
├── Secure Upload                     ✓
├── Metadata                          ✓
├── Storage                           ✓
├── CRM Links                         ✓
├── Search                            ✓
├── Download                          ✓
├── Soft Deletion                     ✓
└── Auditability                      ✓
Document Processing
├── Processing Jobs                   ✓
├── Asynchronous Boundary             ✓
├── Retry Policy                      ✓
├── Attempt Limits                    ✓
├── Failure Classification            ✓
├── Idempotency                       ✓
├── Worker Claiming                   ✓
└── Recovery Strategy                 ✓
Document Extraction
├── Extractor Interface               ✓
├── Extraction Router                ✓
├── PDF                               ✓
├── DOCX                              ✓
├── PPTX                              ✓
├── TXT                               ✓
├── CSV                               ✓
├── XLSX                              -
└── OCR                               -
Extracted Knowledge
├── Raw Text                          ✓
├── Normalized Text                   ✓
├── Extraction Units                 ✓
├── Page Provenance                   ✓
├── Slide Provenance                  ✓
├── Section Metadata                  ✓
├── Extractor Version                 ✓
├── Character Counts                  ✓
└── Word Counts                       ✓
CRM Context
├── Structured CRM Data               ✓
├── Document Metadata                 ✓
├── Processing Status                 ✓
├── Context Budgets                   ✓
└── Provenance                        ✓
AI Knowledge Pipeline
├── Extraction                        ✓
├── Structural Chunking               -
├── Semantic Chunking                 -
├── Embeddings                        -
├── Vector Storage                    -
├── Hybrid Retrieval                  -
└── RAG                               -
ChatGPT
├── CRM Search                        ✓
├── CRM Context                       ✓
├── CRM Mutations                     ✓
├── Document Metadata                 ✓
├── Document Attachment               ✓
├── Processing Status                 ✓
└── Document Q&A                      -

		

147. Acceptance Criteria

Part 27 is complete when:

			
✓ Part 26 regression suite remains green
✓ documents.process exists
✓ processing authorization is server-enforced
✓ DocumentProcessingJob exists
✓ DocumentExtraction exists
✓ DocumentExtractionUnit exists
✓ migrations apply successfully
✓ PENDING exists
✓ PROCESSING exists
✓ COMPLETED exists
✓ FAILED exists
✓ Document availability is separate from processing status
✓ extraction failure does not delete the original Document
✓ original Document remains downloadable after extraction failure
✓ processing is asynchronous
✓ upload does not synchronously perform full extraction
✓ successful upload can enqueue processing
✓ manual processing endpoint exists where appropriate
✓ DocumentExtractor abstraction exists
✓ extraction router exists
✓ extractor selection is deterministic
✓ third-party parsing libraries do not leak into application contracts
✓ PDFExtractor exists
✓ native PDF text extraction works
✓ PDF pages remain separate provenance units
✓ page numbers are preserved
✓ password-protected PDFs fail safely
✓ corrupt PDFs fail safely
✓ scanned PDFs can be detected as requiring OCR
✓ DOCXExtractor exists
✓ paragraphs are extracted
✓ headings are preserved where practical
✓ tables are preserved as textual structure
✓ DOCX extraction order is deterministic
✓ PPTXExtractor exists
✓ slide text is extracted
✓ slide titles are extracted
✓ tables are preserved where practical
✓ slide numbers are preserved
✓ TXTExtractor exists
✓ UTF-8 works
✓ line endings are normalized
✓ encoding failures are handled explicitly
✓ CSVExtractor exists
✓ rows are preserved
✓ quoted values are handled
✓ row ordering is deterministic
✓ XLSX remains explicitly unsupported for extraction if not implemented
✓ image OCR remains explicitly unsupported if not implemented
✓ unsupported formats do not pretend to be processed
✓ ExtractedDocument contract exists
✓ ExtractedUnit contract exists
✓ raw_text exists
✓ normalized_text exists
✓ unit-level raw text exists
✓ unit-level normalized text exists
✓ Unicode normalization exists
✓ line ending normalization exists
✓ whitespace normalization exists
✓ control-character cleanup exists
✓ paragraph boundaries are preserved
✓ table boundaries are preserved where practical
✓ normalization does not semantically rewrite content
✓ character_count exists
✓ word_count exists
✓ unit_count exists
✓ empty extraction detection exists
✓ extractor_name is stored
✓ extractor_version is stored
✓ processing_job_id is stored
✓ document_id is stored
✓ tenant ID is stored
✓ provenance is preserved
✓ page provenance is preserved for PDF
✓ slide provenance is preserved for PPTX
✓ section metadata is preserved where available
✓ processing is idempotent
✓ duplicate job delivery does not create uncontrolled duplicate extraction
✓ processing identity includes authoritative Document content identity
✓ reprocessing with a new extractor version is supported
✓ attempt_count exists
✓ maximum retry attempts exist
✓ retryable failures are distinguishable
✓ permanent failures are distinguishable
✓ infinite retries are impossible
✓ worker claims jobs safely
✓ two workers cannot normally process one job simultaneously
✓ stale PROCESSING jobs can be recovered
✓ processing timeout exists
✓ worker reloads Document before processing
✓ deleted Documents are not processed normally
✓ unavailable Documents are not processed
✓ tenant is revalidated
✓ binary integrity can be verified using checksum
✓ extraction persistence is transactional
✓ partial extraction units do not produce COMPLETED status
✓ COMPLETED is set only after successful extraction persistence
✓ extraction text is tenant-isolated
✓ extraction inherits Document authorization
✓ users cannot read extraction for inaccessible Documents
✓ deleted Documents do not expose extraction through normal APIs
✓ GET processing status exists
✓ GET extraction exists for authorized use
✓ entire extracted text is not automatically injected into ChatGPT context
✓ processing status can appear in CRM Context
✓ processing status is bounded metadata
✓ CRM Context does not contain full document extraction
✓ ChatGPT can report processing status
✓ ChatGPT does not pretend failed Documents are readable
✓ ChatGPT does not pretend OCR has occurred when it has not
✓ ChatGPT does not perform unrestricted whole-document Q&A yet
✓ no embeddings are required
✓ no pgvector is required
✓ no semantic search is required
✓ no RAG pipeline is required
✓ no LLM is required for deterministic extraction
✓ processing logs exist
✓ processing metrics exist
✓ extractor metrics exist
✓ failure metrics exist
✓ logs do not contain full extracted content
✓ Quorentra reports version 0.14.0

		

Most importantly:

Quorentra can now transform an authoritative CRM Document into deterministic, normalized, provenance-rich text without coupling document ingestion to ChatGPT, embeddings, vector databases, or RAG.

148. What We Have Achieved

Our document lifecycle has evolved from:

			
File
      ↓
Upload
      ↓
Document

		

to:

			
File
      ↓
Secure Upload
      ↓
Document
      ↓
CRM Relationship
      ↓
Processing Job
      ↓
Format Detection
      ↓
Extractor
      ↓
Raw Text
      ↓
Normalization
      ↓
Extraction Units
      ↓
Persistent Extracted Content

		

That is a major architectural step.

149. The Knowledge Layer Is Beginning to Form

The Azure Migration opportunity can now conceptually contain:

			
Azure Migration
│
├── CRM Context
│   ├── Company
│   ├── Contacts
│   ├── Activities
│   ├── Tasks
│   └── Meetings
│
└── Documents
    │
    ├── Requirements.pdf
    │   └── Extracted Text ✓
    │
    ├── Architecture.pptx
    │   └── Extracted Text ✓
    │
    ├── Proposal-v3.pdf
    │   └── Extracted Text ✓
    │
    └── Pricing.xlsx
        └── Extraction -

		

The system is starting to possess both:

Structured CRM Knowledge

and:

Unstructured Customer Knowledge

But the latter is not yet efficiently retrievable.

150. Why We Still Cannot Send Everything to ChatGPT

Suppose:

			
Requirements.pdf
= 18,000 words
Architecture.pptx
= 6,000 words
Proposal-v3.pdf
= 24,000 words

		

The Opportunity now has:

48,000 words

of extracted content.

Sending all of that for every question would be:

			
Expensive
Slow
Noisy
Difficult to ground
Poorly scalable

		

We need smaller retrieval units.

151. The Chunking Problem

Consider this section:

			
Commercial Proposal
The total project price is €90,000.
The implementation consists of three phases:
assessment, migration, and optimization.
Payment terms are 30% on project start,
40% after migration,
and 30% following acceptance.

		

We want this information to remain together.

A naive fixed-character split might produce:

			
Chunk 1:
The total project price is €90,000.
The implementation consists of three phases:
assessment, migra
Chunk 2:
tion, and optimization.
Payment terms are 30%...

		

That is poor retrieval structure.

152. Structural Boundaries Already Exist

Fortunately, Part 27 preserved:

			
Pages
Slides
Paragraphs
Sections
Tables

		

Those can guide chunk creation.

Instead of arbitrary character splitting:

			
Extracted Structure
      ↓
Logical Blocks
      ↓
Bounded Chunks

		

This is the next step.

153. Chunks Must Preserve Provenance

A future chunk should know:

			
Document:
Proposal-v3.pdf
Pages:
14–15
Section:
Commercial Proposal
Chunk:
3
Text:
...

		

Then a future ChatGPT answer can cite:

			
Proposal-v3.pdf
Pages 14–15

That is the architecture we want.

154. Chunking Is Not Embedding

These remain separate stages:

			
Extracted Text
      ↓
Chunking
      ↓
Chunks

		

then:

			
Chunks
      ↓
Embeddings
      ↓
Vectors

		

We should be able to inspect and test chunks before any embedding model is involved.

155. Why This Is Important

If retrieval quality is poor later, we need to determine whether the problem is:

			
Bad extraction?
Bad normalization?
Bad chunking?
Bad embeddings?
Bad retrieval?
Bad ranking?
Bad prompting?

		

Modular boundaries let us answer that.

Without them, everything becomes:

"The AI isn't working."

That is not an acceptable engineering diagnosis.

156. Next Article

In Part 28, we will build:

Building the Document Chunking Pipeline — Structure-Aware Segmentation, Token Budgets, Overlap, Provenance, Tables, Sections, and Retrieval-Ready Knowledge Units

We will introduce:

			
DocumentChunk
DocumentChunkingJob
Chunking Status
DocumentChunker
Chunking Policy
Chunk Size
Token Budget
Minimum Chunk Size
Maximum Chunk Size
Chunk Overlap
Paragraph-Aware Splitting
Section-Aware Splitting
Page-Aware Splitting
Slide-Aware Splitting
Table Preservation
Heading Context
Chunk Ordering
Chunk Sequence Numbers
Source Unit References
Page Ranges
Slide Ranges
Section Names
Chunk Metadata
Chunk Character Count
Chunk Token Count
Chunker Name
Chunker Version
Idempotent Chunking
Rechunking
Chunk Provenance
Chunk Quality Validation
Retrieval-Ready Text
Chunking Metrics
Chunking Tests

		

The pipeline will become:

			
Proposal-v3.pdf
      ↓
Extraction
      ↓
Normalized Text
      ↓
Structure-Aware Chunker
      ↓
Chunk 1
Executive Summary
Chunk 2
Customer Requirements
Chunk 3
Solution Architecture
Chunk 4
Commercial Proposal
Chunk 5
Payment Terms

		

Each chunk will retain:

			
Document ID
Extraction ID
Page Range
Section
Sequence
Chunker Version

		

Then Quorentra will finally possess bounded, traceable knowledge units ready for the next major capability:

			
Chunk
      ↓
Embedding
      ↓
Vector
      ↓
Semantic Retrieval

		

That will move Quorentra one major step closer to grounded document intelligence while preserving the modular architecture of A Modular, ChatGPT-Native AI CRM.

Quorentra CRM Building the Document Text Extraction Pipeline: Building from Zero — Part 27

1. Introduction

2. Our Target Interaction

3. Extraction Is Infrastructure, Not AI Reasoning

4. Why This Separation Matters

5. The Document Knowledge Pipeline

6. Starting Checkpoint

7. New Permission

8. Who Needs documents.process?

9. Processing Must Be Asynchronous

10. Processing Architecture

11. Keep the Queue Abstract

12. Define the Processing Job

13. Processing Status

14. Processing Lifecycle

15. Processing Job Is Not the Document

16. Example

17. Processing Attempts

18. Avoid Infinite Retries

19. Retryable Versus Permanent Errors

20. Structured Error Codes

21. Extractor Abstraction

22. Why an Extractor Interface?

23. Extraction Router

24. Router Must Not Trust Extension Alone

25. ExtractedDocument

26. Extracted Units

27. ExtractedUnit

28. Why Preserve Units?

29. Persistent Extraction Model

30. Extraction Units Table

31. Why Store Both Document and Extraction IDs?

32. Extractor Version

33. Extraction Is Reproducible Processing

34. PDF Extraction

35. PDFExtractor

36. PDF Page Units

37. Preserve Page Numbers

38. Scanned PDFs

39. OCR Is Not Part 27

40. Password-Protected PDFs

41. DOCX Extraction

42. DOCX Natural Units

43. Preserve Headings

44. Tables in DOCX

45. PPTX Extraction

46. PPTX Units

47. Preserve Slide Order

48. TXT Extraction

49. TXT Units

50. CSV Extraction

51. CSV Example

52. XLSX Is Different

53. Images

54. Extraction Support Matrix

55. Raw Text Versus Normalized Text

56. Raw Text

57. Normalized Text

58. Normalization Pipeline

59. Unicode Normalization

60. Line Endings

61. Whitespace

62. Preserve Paragraph Boundaries

63. Preserve Table Boundaries

64. Header and Footer Noise

65. Character Count

66. Word Count

67. Empty Extraction Detection

68. Extraction Quality Metadata

69. Extraction Provenance

70. Provenance Chain

71. Idempotent Processing

72. Processing Idempotency Key

73. Example

74. Reprocessing

75. Current Extraction

76. Do Not Delete Old Extractions Immediately

77. Worker Processing Flow

78. Claiming Jobs Safely

79. Example Claim Strategy

8. Who Needs `documents.process`?