Series: Building Quorentra CRM from Zero — A Modular, ChatGPT-Native AI CRM

Building structure-aware document segmentation, token budgets, overlap, heading context, table preservation, provenance, versioned chunking, rechunking, and retrieval-ready knowledge units.

Quorentra Building the Document Chunking Pipeline: Building from Zero — Part 28

1. Introduction

In Part 27, Quorentra learned how to transform uploaded CRM Documents into deterministic, normalized text.

Our document pipeline now looks like:

			
Customer File
      ↓
Secure Upload
      ↓
Document
      ↓
DocumentProcessingJob
      ↓
Format-Specific Extractor
      ↓
Raw Text
      ↓
Normalization
      ↓
DocumentExtraction
      ↓
DocumentExtractionUnit

		

For example:

			
Proposal-v3.pdf
      ↓
PDFExtractor
      ↓
Page 1
Page 2
Page 3
...
Page 20

		

Each extracted unit preserves important provenance such as:

			
Document ID
Extraction ID
Page Number
Slide Number
Section
Extractor
Extractor Version

		

This is an excellent foundation.

But it is not yet suitable for efficient AI retrieval.

A 20-page proposal may contain:

25,000 words

A requirements document may contain:

80,000 words

A large customer workspace may contain millions of characters across hundreds of documents.

We cannot send all of that to ChatGPT every time the user asks a question.

We need smaller knowledge units.

That is the responsibility of Part 28.

We will build the:

Document Chunking Pipeline

Its job is to transform normalized extracted content into bounded, ordered, provenance-preserving units suitable for later embedding and retrieval.

The architecture becomes:

			
Document
      ↓
Extraction
      ↓
Normalized Structure
      ↓
Chunking
      ↓
DocumentChunk
      ↓
Future Embedding
      ↓
Future Retrieval

		

Importantly, Part 28 still does not require:

			
Embedding Models
pgvector
Vector Search
RAG
LLM Summarization

		

Chunking remains an independent deterministic processing stage.

2. Why Chunking Exists

Suppose Proposal-v3.pdf contains:

			
Page 1
Executive Summary
Page 2
Customer Objectives
Pages 3–5
Current Environment
Pages 6–10
Proposed Architecture
Pages 11–13
Migration Approach
Pages 14–15
Commercial Proposal
Pages 16–17
Payment Terms
Pages 18–20
Terms and Conditions

		

A future user asks:

What are the payment terms in the Adventure Works proposal?

We do not want to send all 20 pages to ChatGPT.

Instead, we want retrieval to identify something like:

			
DocumentChunk
Document:
Proposal-v3.pdf
Section:
Payment Terms
Pages:
16–17
Text:
Payment will be made in three stages...

		

That is the fundamental purpose of chunking.

3. The Retrieval Unit

The Document is the authoritative file.

The Extraction is the authoritative recovered text.

The Chunk becomes the:

retrieval unit

Future semantic search will operate primarily over chunks.

Conceptually:

			
Document
      ↓
Extraction
      ↓
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Chunk 5

		

Later:

			
Chunk
      ↓
Embedding
      ↓
Vector

		

And eventually:

			
Question
      ↓
Semantic Search
      ↓
Relevant Chunks
      ↓
ChatGPT

		

4. Chunking Is Not Summarization

This distinction is critical.

Chunking should not transform:

The project will cost €90,000.

into:

Project cost: €90k.

That is summarization.

Instead, chunking should preserve source text as faithfully as practical.

Its responsibility is:

			
Segment
Group
Bound
Order
Annotate
Preserve Provenance

		

not:

			
Interpret
Rewrite
Summarize
Infer

5. Chunking Should Not Require an LLM

The core chunking pipeline should be deterministic.

If ChatGPT or another AI provider is unavailable:

Document Chunking

must still work.

This gives us:

			
Repeatability
Testability
Predictability
Lower Cost
Lower Latency
Provider Independence

		

Later we may experiment with AI-assisted semantic segmentation.

But that should be an optional enhancement, not the foundation.

6. Starting Checkpoint

Before implementing Part 28, verify Part 27.

Run:

			
cd backend
python -m pytest

Then:

			
cd ..\chatgpt-ui
npm run build

Upload:

Proposal-v3.pdf

Verify:

			
Document
      ↓
Processing Job
      ↓
PDFExtractor
      ↓
DocumentExtraction
      ↓
DocumentExtractionUnits

		

Inspect several units.

Verify:

			
page_number
raw_text
normalized_text
character_count

are correct.

Part 28 depends on trustworthy extraction.

7. The Naive Chunking Approach

The simplest possible implementation might be:

			
chunks = [
    text[i:i + 2000]
    for i in range(0, len(text), 2000)
]

This is easy.

It is also poor document architecture.

Consider:

			
The total implementation cost is €90,000.
Payment will be made according to the following
milestones:
30% on project commencement
40% following migration completion
30% after final acceptance

		

A blind character boundary might produce:

			
Chunk 1
The total implementation cost is €90,000.
Payment will be made according to the following
milest

and:

			
Chunk 2
ones:
30% on project commencement
40% following migration completion
30% after final acceptance

		

The semantic unit has been damaged.

8. Structure-Aware Chunking

Instead, Quorentra should use the structure preserved during extraction.

For example:

			
Document
      ↓
Section
      ↓
Paragraph
      ↓
Sentence

		

or:

			
PDF
      ↓
Page
      ↓
Paragraph Blocks

		

or:

			
PPTX
      ↓
Slide
      ↓
Title
      ↓
Text Blocks

		

The chunker should prefer natural boundaries before falling back to smaller boundaries.

9. Chunking Hierarchy

A useful priority order is:

			
Section Boundary
      ↓
Page / Slide Boundary
      ↓
Paragraph Boundary
      ↓
Sentence Boundary
      ↓
Token Boundary

		

The chunker should split at the highest meaningful level that satisfies the configured size budget.

10. Chunking Is a Processing Stage

Part 27 introduced:

DocumentProcessingJob

for extraction.

Chunking deserves its own lifecycle.

Introduce:

DocumentChunkingJob

This gives us:

			
Extraction
      ↓
Chunking Job
      ↓
Chunks

		

rather than burying chunk creation inside the extraction worker.

11. Why Separate Extraction and Chunking Jobs?

Because we may want to:

			
Re-extract without rechunking immediately
Rechunk without re-extracting
Compare chunking strategies
Upgrade chunker versions
Change token budgets
Test retrieval quality

		

If extraction and chunking are one inseparable process, these operations become difficult.

12. DocumentChunkingJob

Conceptually:

			
DocumentChunkingJob
├── id
├── organization_id
├── document_id
├── extraction_id
├── status
├── chunker_name
├── chunker_version
├── policy_name
├── attempt_count
├── started_at
├── completed_at
├── failed_at
├── error_code
├── error_message
├── created_at
└── updated_at

		

13. Chunking Status

Define:

			
class DocumentChunkingStatus(str, Enum):
    PENDING = "pending"
    PROCESSING = "processing"
    COMPLETED = "completed"
    FAILED = "failed"

		

This mirrors our extraction processing model.

14. Chunking Lifecycle

Normal:

			
PENDING
   ↓
PROCESSING
   ↓
COMPLETED

		

Failure:

			
PENDING
   ↓
PROCESSING
   ↓
FAILED

		

Retry:

			
FAILED
   ↓
PENDING
   ↓
PROCESSING

		

according to retry policy.

15. Chunking Depends on Extraction

A Chunking Job requires a successfully completed:

DocumentExtraction

Therefore:

			
Extraction FAILED
      ↓
Chunking impossible

The chunking service must validate the extraction state before processing.

16. Chunking Permission

Introduce:

documents.chunk

if we want explicit manual control.

For automated internal pipelines, the trusted processing service may initiate chunking directly.

Human users generally do not need to manage chunking manually.

However, administrators may eventually need:

Rechunk Document

for troubleshooting or pipeline upgrades.

17. DocumentChunk

Now introduce the central entity:

DocumentChunk

Conceptually:

			
DocumentChunk
├── id
├── organization_id
├── document_id
├── extraction_id
├── chunking_job_id
│
├── sequence_number
├── text
│
├── section_name
├── heading_context
│
├── start_unit_index
├── end_unit_index
│
├── start_page
├── end_page
│
├── start_slide
├── end_slide
│
├── character_count
├── token_count
│
├── chunker_name
├── chunker_version
├── policy_name
│
├── created_at
└── version

		

This becomes our future semantic retrieval object.

18. Sequence Number

Each Chunk should have:

sequence_number

Example:

This preserves document order.

Future retrieval may return Chunk 17.

The system can then easily locate:

			
Chunk 16
Chunk 17
Chunk 18

if neighboring context is useful.

19. Chunk Identity

Use a generated UUID for:

DocumentChunk.id

Do not use:

document_id + sequence_number

as the primary identifier.

Sequence numbers can change during rechunking.

Chunk identity belongs to a specific chunking run.

20. Chunk Text

The most important field is:

text

This should contain retrieval-ready source content.

For example:

			
Commercial Proposal
The total implementation price is €90,000.
Payment will be made in three stages:
30% at project commencement,
40% following migration completion,
and 30% following final acceptance.

		

This is what will later be embedded.

21. Retrieval-Ready Does Not Mean Rewritten

We may add structural context such as:

Commercial Proposal

above the source paragraph.

But we should not rewrite the underlying content.

The Chunk must remain traceable to the extracted source.

22. Heading Context

Suppose the document structure is:

			
5. Commercial Proposal
5.1 Pricing
The total implementation price is €90,000.

The paragraph itself may not contain enough context.

Therefore the chunk can preserve:

			
heading_context:
Commercial Proposal > Pricing

This is valuable for both embeddings and later presentation.

23. Heading Context Should Be Explicit

Prefer storing:

heading_context

as metadata.

Then decide whether to prepend it to:

text

according to chunking policy.

This keeps the original content and retrieval augmentation distinguishable.

24. Page Ranges

For PDF chunks, store:

			
start_page
end_page

Example:

			
start_page = 14
end_page = 15

Later Quorentra can cite:

Proposal-v3.pdf, pages 14–15.

25. Slide Ranges

For PPTX:

			
start_slide
end_slide

Example:

			
start_slide = 6
end_slide = 7

Later:

Azure-Architecture.pptx, slides 6–7.

26. Source Unit Range

Also store:

			
start_unit_index
end_unit_index

This connects the Chunk back to:

DocumentExtractionUnit

even when page or slide numbering does not apply.

27. More Precise Provenance

A Chunk may contain only part of one extraction unit.

Therefore we may later need:

			
start_character_offset
end_character_offset

or a dedicated mapping table.

For the MVP, source-unit ranges plus page/slide metadata may be sufficient.

But the architecture should not prevent more precise provenance later.

28. DocumentChunkSource

A stronger model is to introduce:

DocumentChunkSource

Conceptually:

			
DocumentChunkSource
├── id
├── organization_id
├── chunk_id
├── extraction_unit_id
├── sequence_number
├── start_offset
├── end_offset
└── created_at

		

This lets one Chunk reference multiple extraction units precisely.

29. Why a Source Mapping Table Helps

Suppose Chunk 8 contains:

			
End of Page 14
+
Beginning of Page 15

We can represent:

			
Chunk 8
├── Source A → Page 14
└── Source B → Page 15

rather than only storing:

pages 14–15

This gives us stronger provenance.

30. MVP Decision

For Quorentra, I recommend introducing:

DocumentChunkSource

now.

Provenance is difficult to retrofit after embeddings and retrieval are already built.

A little extra structure now saves substantial complexity later.

31. Chunker Abstraction

Introduce:

DocumentChunker

Conceptually:

			
class DocumentChunker(Protocol):
    async def chunk(
        self,
        *,
        extraction: DocumentExtraction,
        units: list[DocumentExtractionUnit],
        policy: "ChunkingPolicy",
    ) -> list["GeneratedChunk"]:
        ...

		

The chunker should know nothing about:

			
pgvector
embedding models
ChatGPT

It simply creates chunks.

32. Chunking Policy

Do not hard-code:

1000 tokens

throughout the application.

Introduce:

ChunkingPolicy

Conceptually:

			
class ChunkingPolicy(BaseModel):
    name: str
    target_tokens: int
    max_tokens: int
    min_tokens: int
    overlap_tokens: int
    preserve_sections: bool
    preserve_tables: bool
    preserve_pages: bool
    preserve_slides: bool
    include_heading_context: bool

		

This makes chunking behavior explicit.

33. Default Policy

For example:

			
name:
default_v1
target_tokens:
500
max_tokens:
700
min_tokens:
100
overlap_tokens:
75

		

These numbers are starting points, not universal truths.

We will tune them later based on retrieval quality.

34. Target Versus Maximum

Distinguish:

target_tokens

from:

max_tokens

The chunker tries to produce chunks around:

500 tokens

but may allow:

650 tokens

to preserve a coherent paragraph or small table.

It must not normally exceed:

700 tokens

under this example policy.

35. Why Token Budgets?

Future embedding and language models operate on tokens rather than characters.

Therefore token count is a more meaningful boundary than:

2,000 characters

Two 2,000-character passages may have very different token counts.

36. Tokenizer Abstraction

Introduce:

TokenCounter

Conceptually:

			
class TokenCounter(Protocol):
    def count(self, text: str) -> int:
        ...

Do not couple the chunker directly to one AI provider’s SDK.

37. Why Abstract Token Counting?

Today we may use one embedding model.

Later another.

Different tokenizers may behave differently.

The Chunking module should depend on:

TokenCounter

not:

SpecificVendorTokenizer

38. Token Count Is Metadata

Every Chunk stores:

token_count

based on the tokenizer used by the chunking policy.

Also store tokenizer identity if needed:

			
tokenizer_name
tokenizer_version

This improves reproducibility.

39. Character Count Still Matters

Also store:

character_count

because it is:

			
cheap
deterministic
model-independent

Both counts are useful.

40. Minimum Chunk Size

Very small chunks can be poor retrieval units.

Example:

Thank you.

Embedding that independently may add noise.

Therefore define:

min_tokens

and try to merge small adjacent blocks where semantically appropriate.

41. Do Not Merge Across Strong Boundaries Blindly

Suppose:

			
Section:
Pricing
€90,000.

followed by:

			
Section:
Legal Terms

Do not merge them simply because the Pricing chunk is small.

Strong section boundaries can outweigh minimum size targets.

Chunking is a constrained optimization problem.

42. Chunking Priorities

A useful priority order is:

			
Preserve meaning
Preserve provenance
Respect hard token limit
Prefer structural boundaries
Reach useful target size
Apply overlap carefully

		

Not:

Every chunk must be exactly 500 tokens.

43. Paragraph-Aware Chunking

Paragraphs are natural semantic units.

Example:

			
Paragraph 1
Customer objective
Paragraph 2
Current challenge
Paragraph 3
Proposed solution

		

The chunker can accumulate paragraphs until the target budget is reached.

44. Paragraph Accumulation

Conceptually:

			
Chunk
      ↓
Add Paragraph 1
      ↓
150 tokens
Add Paragraph 2
      ↓
330 tokens
Add Paragraph 3
      ↓
520 tokens
Close Chunk

		

This produces coherent boundaries.

45. Oversized Paragraphs

What if one paragraph is:

1,400 tokens

and our hard maximum is:

Then the chunker needs a fallback.

Use:

Sentence-Aware Splitting

before resorting to token-level splitting.

46. Sentence-Aware Splitting

Conceptually:

			
Oversized Paragraph
      ↓
Sentence 1
Sentence 2
Sentence 3
...
      ↓
Accumulate Sentences
      ↓
Bounded Chunks

		

This preserves meaning better than arbitrary token boundaries.

47. Final Token-Level Fallback

A pathological sentence may itself exceed the hard maximum.

Then:

Token-Level Split

may be unavoidable.

But this should be the last fallback.

The hierarchy becomes:

			
Section
↓
Block
↓
Paragraph
↓
Sentence
↓
Token

		

48. Section-Aware Chunking

Suppose:

			
Executive Summary
[400 tokens]
Customer Requirements
[600 tokens]
Commercial Proposal
[450 tokens]

		

These naturally become separate chunks.

Do not combine:

			
Executive Summary
+
Customer Requirements

merely to reach a target size.

Sections often carry strong semantic boundaries.

49. Section Names

Store:

section_name

where available.

Example:

Commercial Proposal

This helps:

			
Retrieval
Ranking
Display
Citations
Debugging

		

50. Nested Headings

Documents may contain:

			
Commercial Proposal
1 Pricing
1.1 Migration Services

Preserve a hierarchy such as:

Commercial Proposal > Pricing > Migration Services

in:

heading_context

51. PDF Page Boundaries

PDF pages are important for provenance but are not always semantic boundaries.

A paragraph may continue:

			
Page 14
...
The migration project will be delivered
Page 15
over a period of six months...

		

Do not automatically split at every page boundary.

Allow a Chunk to span pages when necessary.

Preserve:

			
start_page
end_page

52. Why Not One Chunk Per Page?

Because page sizes vary.

One page may contain:

100 tokens

another:

1,200 tokens

And semantic sections frequently cross page boundaries.

Page provenance matters.

Page-sized chunking is not always good retrieval chunking.

53. PPTX Is Different

Slides are stronger semantic boundaries than PDF pages.

A presentation often has:

			
Slide 1
Executive Summary
Slide 2
Customer Challenges
Slide 3
Solution Architecture

		

Therefore the PPTX policy may prefer:

one slide per chunk

when slides are reasonably sized.

54. Small Slides

If several slides contain very little text:

			
Slide 1
Title
Slide 2
Agenda
Slide 3
Background

		

the chunker may combine adjacent slides if the policy allows.

But provenance must still record all source slides.

55. Oversized Slides

If one slide contains:

1,500 tokens

split its text internally.

Store:

			
start_slide = 7
end_slide = 7

for both resulting chunks.

56. Table Preservation

Tables require special handling.

Consider:

			
Service | Quantity | Price
Assessment | 1 | €10,000
Migration | 1 | €80,000

Do not split randomly between:

Service

and:

Price

or separate rows from their headers when avoidable.

57. Small Tables

If a table fits within:

max_tokens

keep it intact.

This is the preferred behavior.

58. Large Tables

A large table may exceed the maximum.

Then split by:

row groups

while repeating the header.

Example:

			
Chunk 1
Service | Quantity | Price
Assessment | 1 | €10,000
Migration | 1 | €80,000

			
Chunk 2
Service | Quantity | Price
Optimization | 1 | €15,000
Support | 12 months | €24,000

This preserves column meaning.

59. Table Metadata

Chunks containing tables may include:

			
content_type =
table

or:

			
contains_table =
true

This may later help retrieval and presentation.

60. CSV Chunking

CSV content is naturally row-oriented.

Chunk:

			
Header
+
bounded group of rows

Repeat the header in each Chunk.

This is better than treating the entire CSV as ordinary prose.

61. Chunk Overlap

Overlap helps preserve context across boundaries.

Suppose:

			
Chunk 1
Paragraphs A B C
Chunk 2
Paragraphs C D E

Paragraph C appears in both.

This can help retrieval when relevant information lies near a boundary.

62. But Overlap Has a Cost

Overlap increases:

			
Number of Chunks
Embedding Cost
Vector Storage
Retrieval Duplication
Context Duplication

		

Therefore overlap should be deliberate.

63. Default Overlap

A starting point might be:

			
overlap_tokens =
50–100

for approximately:

500-token chunks

But overlap should preferably follow structural boundaries.

64. Structural Overlap

Instead of copying the last exactly:

75 tokens

prefer:

last paragraph

if it fits within the overlap budget.

This preserves coherent context.

65. Do Not Overlap Tables Blindly

Repeating part of a table can create confusing duplicate data.

For tables, use a table-specific policy.

Often:

repeat header

is more useful than arbitrary overlap.

66. Do Not Overlap Section Headers as Fake Content

A heading may be repeated as contextual metadata.

That is different from overlap.

Keep these concepts separate:

Heading Context

versus:

Source Text Overlap

67. GeneratedChunk

Before persistence, the chunker can return:

			
class GeneratedChunk(BaseModel):
    sequence_number: int
    text: str
    section_name: str | None
    heading_context: str | None
    source_units: list["GeneratedChunkSource"]
    character_count: int
    token_count: int
    metadata: dict[str, Any]

		

This keeps chunk generation independent from ORM persistence.

68. Chunk Source Mapping

Each generated source:

			
class GeneratedChunkSource(BaseModel):
    extraction_unit_id: UUID
    sequence_number: int
    start_offset: int | None = None
    end_offset: int | None = None

		

Later we can improve offset precision without redesigning Chunk itself.

69. Chunk Provenance

A future Chunk should be able to answer:

Where did you come from?

For example:

			
Chunk ID:
chunk_123
Document:
Proposal-v3.pdf
Extraction:
ext_456
Pages:
14–15
Section:
Commercial Proposal
Source Units:
unit_14
unit_15
Chunker:
structure_aware
Version:
1.0.0
Policy:
default_v1

		

That is strong provenance.

70. Chunker Version

Store:

			
chunker_name
chunker_version

Example:

			
structure_aware
1.0.0

Later:

			
structure_aware
1.1.0

may improve table handling.

71. Policy Versioning

The Chunker code can stay the same while policy changes.

Example:

			
default_v1
target:
500
max:
700

		

versus:

			
default_v2
target:
700
max:
900

		

Therefore store:

policy_name

and ideally a policy version or immutable policy configuration hash.

72. Configuration Hash

For stronger reproducibility, calculate:

policy_hash

from the normalized ChunkingPolicy configuration.

Then:

			
Extraction
+
Chunker Version
+
Policy Hash
=
Chunking Identity

		

73. Idempotent Chunking

Suppose a job is delivered twice.

Do not create duplicate active chunks.

Use an identity based on:

			
extraction_id
+
chunker_name
+
chunker_version
+
policy_hash

		

Equivalent work should be reusable or safely ignored.

74. Rechunking

Rechunking is intentional creation of a new chunk set.

For example:

			
Old:
500-token chunks
New:
750-token chunks

or:

			
Old:
structure_aware 1.0
New:
structure_aware 1.1

Both chunk sets may temporarily coexist.

75. Chunk Set

It is useful to formalize a group of chunks.

Introduce:

DocumentChunkSet

Conceptually:

			
DocumentChunkSet
├── id
├── organization_id
├── document_id
├── extraction_id
├── chunking_job_id
├── chunker_name
├── chunker_version
├── policy_name
├── policy_hash
├── chunk_count
├── total_tokens
├── created_at
└── is_current

		

76. Why Introduce ChunkSet?

Without it, identifying:

the current 84 chunks for this document

becomes awkward.

With:

DocumentChunkSet

we get:

			
Document
      ↓
Extraction
      ↓
ChunkSet
      ↓
Chunks

		

This also prepares us for embedding versioning.

77. Future Embedding Relationship

Later:

			
ChunkSet
      ↓
EmbeddingSet

may represent:

all chunks embedded using model X

This gives us clean pipeline lineage.

78. Current Chunk Set

Only one ChunkSet should normally be considered:

current

for retrieval.

When rechunking completes successfully:

			
Old ChunkSet
is_current = false
New ChunkSet
is_current = true

Do this transactionally.

79. Never Switch to Partial Chunking

Suppose new chunking produces:

			
Chunk 1
Chunk 2
Chunk 3

then fails.

Do not mark that partial ChunkSet current.

Only after the complete set is persisted successfully should it become current.

80. Chunking Transaction

Conceptually:

			
Create ChunkSet
      ↓
Persist Chunk 1
      ↓
Persist Sources
      ↓
Persist Chunk 2
      ↓
Persist Sources
      ↓
...
      ↓
Validate Set
      ↓
Set ChunkSet Current
      ↓
Mark Job COMPLETED
      ↓
Commit

		

Failure:

Rollback

or retain an explicitly failed non-current set according to implementation policy.

81. Chunk Validation

Before completing a ChunkSet, verify:

			
At least one Chunk
Every Chunk has text
Every Chunk has source provenance
Sequence numbers are continuous
Token counts are valid
No Chunk exceeds hard maximum without explicit exception
Document IDs match
Extraction IDs match
Tenant IDs match

		

82. Empty Chunks

Never persist chunks containing only:

			
Whitespace
Empty headings
Formatting noise

They add no retrieval value.

83. Very Small Chunks

Small chunks may be acceptable when they represent strong standalone semantic units.

Example:

			
Termination
Either party may terminate this agreement with 30 days' written notice.

Do not merge it with unrelated content simply to satisfy a numeric target.

84. Oversized Chunks

A Chunk exceeding:

max_tokens

should normally fail validation unless a specific policy exception exists.

For example, preserving a table may allow a small controlled overage.

Such exceptions should be explicit and observable.

85. Chunk Metadata

Useful metadata may include:

			
content_type
contains_table
contains_list
heading_level
language
source_format

		

Do not turn metadata into an uncontrolled JSON dumping ground.

Only store fields that have a clear downstream use.

86. Language

Language detection can eventually help:

			
Embedding model selection
Search behavior
Display

But automatic language detection is not required for Part 28.

If known from upstream metadata, preserve it.

Otherwise leave it unset.

87. Chunk Text Construction

A retrieval-ready Chunk may be constructed as:

			
[Heading Context]
[Source Text]

Example:

			
Commercial Proposal > Payment Terms
Payment will be made in three stages:
30% at project commencement,
40% following migration completion,
and 30% following final acceptance.

		

This can improve semantic retrieval.

88. But Preserve Original Source Separately

If heading context is prepended, distinguish:

source_text

from:

retrieval_text

This is even stronger.

Conceptually:

			
DocumentChunk
├── source_text
└── retrieval_text

89. Why Two Text Fields?

source_text:

Payment will be made in three stages...

retrieval_text:

			
Commercial Proposal > Payment Terms
Payment will be made in three stages...

The second may produce a better embedding.

The first is closer to the source.

90. Recommended Model

For Quorentra, use:

			
source_text
retrieval_text

rather than a single ambiguous text field.

This makes future RAG citations cleaner.

91. Future ChatGPT Answer

Later ChatGPT may answer:

Payment is structured as 30% at project commencement, 40% after migration completion, and 30% after final acceptance.

The evidence should point to:

source_text

not an augmented heading string pretending to be original document content.

92. Chunk Hash

Calculate:

content_hash

for each Chunk.

For example:

SHA-256(source_text)

This helps:

			
Change detection
Embedding reuse
Debugging
Integrity checks

93. Why Chunk Hash Matters Later

Suppose rechunking produces some identical chunks.

If:

content_hash

matches an existing embedded chunk, we may eventually reuse embeddings depending on model and retrieval metadata requirements.

That can reduce cost.

94. Do Not Optimize Embedding Reuse Yet

Store the hash now.

Use it later.

Part 28 should not become an embedding cache implementation.

95. Chunk Ordering

Chunks must follow source order.

Example:

			
Chunk 1
Executive Summary
Chunk 2
Objectives
Chunk 3
Architecture
Chunk 4
Migration Plan
Chunk 5
Commercial Proposal

		

This allows neighboring chunk expansion later.

96. Neighbor Relationships

We do not need explicit:

			
previous_chunk_id
next_chunk_id

because:

			
chunk_set_id
+
sequence_number

already provides ordering.

Avoid redundant state unless needed.

97. Context Expansion Later

Future retrieval may find:

Chunk 12

and decide to include:

			
Chunk 11
Chunk 12
Chunk 13

because the relevant answer spans boundaries.

Sequence numbers make this easy.

98. Chunking PDF Example

Suppose pages 14–17 contain:

			
Page 14
Commercial Proposal
The total project price is €90,000.
Page 15
The price includes assessment, migration,
testing and production transition.
Page 16
Payment Terms
30% at project commencement.
Page 17
40% following migration completion.
30% after final acceptance.

		

The chunker might produce:

			
Chunk 21
Commercial Proposal
The total project price is €90,000.
The price includes assessment, migration,
testing and production transition.
Pages 14–15

		

and:

			
Chunk 22
Payment Terms
30% at project commencement.
40% following migration completion.
30% after final acceptance.
Pages 16–17

		

That is far more useful than page-sized or fixed-character chunks.

99. Chunking PPTX Example

Slides:

			
Slide 5
Current Architecture
Slide 6
Target Architecture
Slide 7
Migration Phases

		

Possible chunks:

			
Chunk 5
Current Architecture
Slide 5

			
Chunk 6
Target Architecture
Slide 6

			
Chunk 7
Migration Phases
Slide 7

Slides already provide strong semantic units.

100. Chunking DOCX Example

DOCX:

			
Heading 1:
Solution
Heading 2:
Migration Approach
Paragraphs...
Heading 2:
Testing Strategy
Paragraphs...

		

The chunker can naturally preserve:

Solution > Migration Approach

and:

Solution > Testing Strategy

as separate contextual units.

101. Chunking CSV Example

CSV:

			
Service | Quantity | Price
Assessment | 1 | €10,000
Migration | 1 | €80,000
Optimization | 1 | €15,000
Support | 12 | €24,000

		

If all rows fit:

one Chunk

If not:

			
Chunk 1
Header + rows 1–50
Chunk 2
Header + rows 51–100

102. No Cross-Document Chunks

Never create a Chunk containing content from:

			
Document A
+
Document B

A Chunk belongs to exactly one authoritative Document.

Cross-document context assembly happens during retrieval, not ingestion.

103. Tenant Isolation

Every:

			
DocumentChunkingJob
DocumentChunkSet
DocumentChunk
DocumentChunkSource

contains:

organization_id

All queries remain tenant-scoped.

104. Authorization Inheritance

Chunks inherit access from their Document.

If a user cannot read:

Proposal-v3.pdf

they cannot retrieve:

Chunks from Proposal-v3.pdf

Future vector search must enforce the same rule.

105. This Is a Critical Future Security Requirement

Vector databases often tempt developers to perform:

Global Similarity Search

and filter afterward.

That can be dangerous.

Our architecture is establishing:

			
Tenant
Document
Chunk

ownership now so semantic retrieval can remain tenant-aware from the beginning.

106. Deleted Documents

If:

			
Document.status =
DELETED

its Chunks must not appear in normal retrieval.

We may retain Chunk records temporarily for audit or retention purposes.

But retrieval must follow the authoritative Document state.

107. Reprocessed Documents

If a new Extraction becomes current, the old ChunkSet should eventually stop being current.

Pipeline lineage:

			
Document
├── Extraction 1
│   └── ChunkSet 1
│
└── Extraction 2
    └── ChunkSet 2 ← current

		

Future search uses only the current approved pipeline state.

108. Processing Orchestration

After successful extraction:

			
DocumentExtraction
      ↓
Create Chunking Job

This may happen automatically.

But keep the stages logically separate.

109. Pipeline State

A Document may now have:

			
Document:
AVAILABLE
Extraction:
COMPLETED
Chunking:
PENDING

		

Then:

			
Document:
AVAILABLE
Extraction:
COMPLETED
Chunking:
COMPLETED

		

Later:

			
Embedding:
PENDING

This gives us a transparent processing pipeline.

110. Knowledge Status

We can expose a derived status:

Document Knowledge Status

such as:

			
uploaded
extracting
extracted
chunking
chunked
embedding
ready
failed

		

But this should be derived from authoritative stage states rather than becoming a second competing state machine.

111. ChatGPT Processing Status

User:

Is the Adventure Works proposal ready for AI search?

At Part 28, Quorentra can inspect:

			
Extraction:
COMPLETED
Chunking:
COMPLETED
Embedding:
NOT BUILT

		

and answer:

The proposal has been extracted and chunked, but semantic indexing has not been completed yet.

That is precise and grounded.

112. Chunking API

Introduce:

POST /api/v1/documents/{document_id}/chunking

for authorized manual rechunking.

Input may optionally select:

policy_name

from approved server-side policies.

113. Do Not Accept Arbitrary Policy Objects from Normal Users

Bad:

			
{
  "target_tokens": 500000,
  "max_tokens": 1000000
}

Instead:

			
{
  "policy_name": "default_v1"
}

The server loads an approved policy.

Administrative tooling can provide advanced controls later.

114. Chunking Status API

Introduce:

GET /api/v1/documents/{document_id}/chunking

Example:

			
{
  "status": "completed",
  "chunk_set_id": "...",
  "chunk_count": 42,
  "total_tokens": 18324,
  "chunker": "structure_aware",
  "chunker_version": "1.0.0",
  "policy": "default_v1"
}

		

115. Chunk Inspection API

For development and administration:

GET /api/v1/documents/{document_id}/chunks

Return bounded results.

Example:

			
{
  "items": [
    {
      "sequence_number": 1,
      "section_name": "Executive Summary",
      "start_page": 1,
      "end_page": 2,
      "token_count": 482
    }
  ]
}

		

116. Full Chunk Text

An authorized detail endpoint can return:

			
source_text
retrieval_text
provenance

This is useful for debugging retrieval quality later.

117. Do Not Put All Chunks in CRM Context

Part 25 introduced bounded CRM Context.

Part 26 added Document metadata.

Part 27 added processing status.

Part 28 should not suddenly inject:

all Document Chunks

into ordinary context.

That would defeat the entire purpose of retrieval.

118. CRM Context Metadata

CRM Context may include:

			
Proposal-v3.pdf
Extraction:
completed
Chunking:
completed
Chunks:
42

		

That is enough.

Relevant Chunk content will be retrieved later.

119. Chunking Widget

Create:

			
chatgpt-ui/src/documents/
└── DocumentKnowledgeStatus.tsx

Example:

			
┌────────────────────────────────────────────┐
│ Azure Migration Proposal v3                │
│                                            │
│ Document processing                        │
│                                            │
│ ✓ Text extracted                           │
│ ✓ Content chunked                          │
│                                            │
│ 42 knowledge chunks                        │
│ 18,324 tokens                              │
│                                            │
│ Semantic indexing                         │
│ Not started                                │
└────────────────────────────────────────────┘

		

120. Chunk Quality Metrics

Useful metrics include:

			
document_chunking_jobs_total
document_chunking_completed_total
document_chunking_failed_total
document_chunks_created_total
document_chunk_tokens_total
document_chunking_duration_ms

		

121. Distribution Metrics

Track:

			
average_chunk_tokens
minimum_chunk_tokens
maximum_chunk_tokens
average_chunks_per_document

These are valuable for tuning.

122. Boundary Metrics

Later we may track:

			
oversized_chunks_total
undersized_chunks_total
table_chunks_total
multi_page_chunks_total
multi_slide_chunks_total

		

This helps diagnose policy behavior.

123. Operational Logging

Useful structured fields:

			
chunking_job_id
chunk_set_id
document_id
extraction_id
organization_id
chunker_name
chunker_version
policy_name
policy_hash
chunk_count
total_tokens
duration_ms
status
error_code

		

Do not log full Chunk contents routinely.

124. Database Indexes

Useful indexes include:

			
document_chunking_jobs
(
    organization_id,
    status,
    created_at
)
document_chunk_sets
(
    organization_id,
    document_id,
    is_current
)
document_chunks
(
    chunk_set_id,
    sequence_number
)
document_chunks
(
    organization_id,
    document_id
)
document_chunk_sources
(
    chunk_id,
    sequence_number
)

		

125. Uniqueness

Within one ChunkSet:

sequence_number

should be unique.

Constraint:

			
UNIQUE
(
    chunk_set_id,
    sequence_number
)

		

126. One Current ChunkSet

Enforce as strongly as practical:

			
one current ChunkSet
per current Extraction / Document

The exact database constraint depends on our model.

The service must enforce it transactionally.

127. Database Migration

Part 28 introduces:

			
document_chunking_jobs
document_chunk_sets
document_chunks
document_chunk_sources

Run:

			
cd backend
alembic revision --autogenerate -m "add document chunking pipeline"

Review the migration.

Then:

alembic upgrade head

Verify:

alembic current

128. Suggested Backend Structure

Extend:

			
backend/app/documents/
├── processing/
│   ├── extraction/
│   │   ├── service.py
│   │   ├── router.py
│   │   └── extractors/
│   │
│   └── chunking/
│       ├── models.py
│       ├── schemas.py
│       ├── repository.py
│       ├── service.py
│       ├── worker.py
│       ├── queue.py
│       ├── chunker.py
│       ├── policies.py
│       ├── token_counter.py
│       ├── segmentation.py
│       └── validation.py

		

This maintains a clean processing hierarchy.

129. Chunker Components

The structure-aware chunker can internally use:

			
StructureAwareChunker
│
├── SectionSegmenter
├── ParagraphSegmenter
├── SentenceSegmenter
├── TableSegmenter
├── SlideSegmenter
├── TokenCounter
├── OverlapBuilder
└── ChunkValidator

		

Each component has a narrow responsibility.

130. Why This Modularity Matters

Later we may replace:

SentenceSegmenter

without changing:

TableSegmenter

or introduce:

SemanticSegmenter

without rewriting persistence.

Again, modularity lets the system evolve.

131. Unit Tests: Basic Chunking

Test normalized text containing:

10 paragraphs

Verify:

			
chunks are created
sequence is correct
text is preserved
provenance exists

132. Token Budget Test

Policy:

			
target = 500
max = 700

Verify normal Chunks remain:

<= 700 tokens

except explicitly allowed structural exceptions.

133. Minimum Size Test

Create several tiny adjacent paragraphs.

Verify the chunker combines them when doing so does not violate strong semantic boundaries.

134. Oversized Paragraph Test

Create:

			
1 paragraph
1,500 tokens

Verify:

sentence-aware splitting

occurs before token fallback.

135. Oversized Sentence Test

Create one pathological sentence over the hard maximum.

Verify final fallback creates bounded chunks without crashing.

136. Section Boundary Test

Input:

			
Pricing
...
Legal Terms
...

Verify Chunks do not combine unrelated sections merely to hit target size.

137. Heading Context Test

Input:

			
Solution
  Migration
    Testing

Verify:

			
heading_context =
Solution > Migration > Testing

where appropriate.

138. Page Provenance Test

Chunk spans:

			
Page 14
Page 15

Verify:

			
start_page = 14
end_page = 15

and source mappings reference the appropriate extraction units.

139. Slide Provenance Test

Chunk spans:

Slides 6–7

Verify:

			
start_slide = 6
end_slide = 7

140. Table Preservation Test

Small table under maximum.

Expected:

one intact Chunk

not arbitrary row splitting.

141. Large Table Test

Table exceeds maximum.

Expected:

multiple row-group Chunks

with the header repeated.

142. Overlap Test

Verify:

configured overlap

appears where appropriate.

Also verify overlap does not cause uncontrolled Chunk growth.

143. No Duplicate Content Explosion

For a 10,000-token document with modest overlap, we should not accidentally generate:

100,000 tokens

of Chunk content.

Track expansion ratio:

			
total chunk tokens
/
source tokens

144. Expansion Ratio

A useful diagnostic metric:

chunk_expansion_ratio

Example:

			
source:
10,000 tokens
chunks:
11,200 tokens
ratio:
1.12

		

Large ratios may indicate excessive overlap or heading duplication.

145. Idempotency Test

Run identical:

			
Extraction
Chunker Version
Policy

twice.

Expected:

no uncontrolled duplicate current ChunkSets

146. Rechunking Test

Change:

default_v1

to:

default_v2

Expected:

new ChunkSet

while old ChunkSet remains traceable.

147. Atomic Current Switch Test

Simulate failure during new ChunkSet creation.

Expected:

old ChunkSet remains current

Never switch to an incomplete set.

148. Tenant Isolation Test

Tenant B requests Chunks for Tenant A Document.

Expected:

not found / forbidden

No Chunk metadata leakage.

149. Permission Test

User cannot read the Document.

User requests its Chunks.

Expected:

denied

Chunking never creates an alternate permission path.

150. Deleted Document Test

Document is deleted after Chunking.

Normal Chunk retrieval:

returns nothing

or appropriate not-available response.

151. Failed Extraction Test

Attempt Chunking on:

			
Extraction.status =
FAILED

Expected:

CHUNKING_SOURCE_NOT_READY

No ChunkSet created.

152. Empty Extraction Test

Extraction contains no useful text.

Expected:

no empty ChunkSet marked current

Fail with a structured error.

153. Processing Concurrency Test

Two workers claim the same Chunking Job.

Expected:

one processor

not duplicate ChunkSets.

154. Worker Recovery Test

Worker crashes while Chunking Job is:

PROCESSING

Verify stale job recovery works similarly to extraction jobs.

155. Transaction Test

Failure while persisting Chunk 25 of 40.

Expected:

no partial current ChunkSet

The previous valid ChunkSet remains authoritative.

156. Provenance Completeness Test

Every persisted Chunk must have:

			
document_id
extraction_id
chunk_set_id
source mapping
sequence_number
chunker version
policy identity

		

No orphan knowledge units.

157. Source Fidelity Test

Concatenate non-overlapping source portions represented by Chunks and compare them with the extraction.

We should be able to demonstrate that meaningful source content was not silently lost.

158. No Semantic Rewriting Test

Input:

The project value is €90,000.

Chunk source text must remain:

The project value is €90,000.

not:

Project cost: €90k.

Chunking is not summarization.

159. No Embedding Dependency Test

Disable all AI provider credentials.

Run Chunking.

Expected:

works

Part 28 must remain independent of AI APIs.

160. No Vector Database Dependency Test

Disable pgvector.

Run Chunking.

Expected:

works

Vector storage comes later.

161. No ChatGPT Dependency Test

ChatGPT integration unavailable.

Run:

			
Extraction
      ↓
Chunking

Expected:

works

The knowledge ingestion pipeline belongs to Quorentra.

162. Version Update

Part 28 introduces:

			
Document Chunking Jobs
Document Chunk Sets
Document Chunks
Chunk Source Mapping
Structure-Aware Segmentation
Token Budgets
Paragraph-Aware Splitting
Sentence-Aware Splitting
Page Provenance
Slide Provenance
Section Context
Heading Context
Table Preservation
Overlap
Chunk Hashing
Chunker Versioning
Policy Versioning
Idempotent Chunking
Rechunking
Chunk Metrics

		

Update:

app/core/constants.py

from:

APP_VERSION = "0.14.0"

to:

APP_VERSION = "0.15.0"

163. Quorentra 0.15.0

Our modular MVP now looks like:

			
Platform
├── FastAPI                           ✓
├── PostgreSQL                        ✓
├── SQLAlchemy                        ✓
├── Alembic                           ✓
├── Document Storage                  ✓
└── Processing Workers                ✓
Identity
├── Organizations                     ✓
├── Users                             ✓
├── Memberships                       ✓
├── Authentication                    ✓
└── JWT                               ✓
Security
├── TenantContext                     ✓
├── Tenant Isolation                  ✓
├── RBAC                              ✓
├── Document Authorization            ✓
├── Processing Authorization          ✓
└── Chunk Authorization Inheritance   ✓
CRM
├── Companies                         ✓
├── Contacts                          ✓
├── Opportunities                     ✓
├── Activities                        ✓
├── Tasks                             ✓
├── Meetings                          ✓
└── Documents                         ✓
Document Storage
├── Upload                            ✓
├── Validation                        ✓
├── Binary Storage                    ✓
├── Checksums                         ✓
├── Retrieval                         ✓
└── Soft Deletion                     ✓
Document Extraction
├── Processing Jobs                   ✓
├── PDF                               ✓
├── DOCX                              ✓
├── PPTX                              ✓
├── TXT                               ✓
├── CSV                               ✓
├── Normalization                     ✓
├── Extraction Units                 ✓
├── Page Provenance                   ✓
├── Slide Provenance                  ✓
└── Extractor Versioning              ✓
Document Chunking
├── Chunking Jobs                     ✓
├── Chunk Sets                        ✓
├── Document Chunks                   ✓
├── Source Mapping                    ✓
├── Structure-Aware Segmentation      ✓
├── Section-Aware Splitting           ✓
├── Paragraph-Aware Splitting         ✓
├── Sentence-Aware Splitting          ✓
├── Token Fallback                    ✓
├── Token Budgets                     ✓
├── Minimum Size                      ✓
├── Maximum Size                      ✓
├── Overlap                           ✓
├── Heading Context                   ✓
├── Page Ranges                       ✓
├── Slide Ranges                      ✓
├── Table Preservation                ✓
├── Chunk Hashes                      ✓
├── Chunker Versioning                ✓
├── Policy Versioning                 ✓
├── Idempotency                       ✓
└── Rechunking                        ✓
Knowledge Pipeline
├── Document                          ✓
├── Extraction                        ✓
├── Normalization                     ✓
├── Chunking                          ✓
├── Embeddings                        -
├── Vector Storage                    -
├── Semantic Search                   -
├── Hybrid Retrieval                  -
├── Reranking                         -
└── RAG                               -
CRM Context
├── Structured CRM Evidence           ✓
├── Document Metadata                 ✓
├── Extraction Status                 ✓
├── Chunking Status                   ✓
├── Provenance                        ✓
└── Bounded Context                   ✓
ChatGPT
├── CRM Search                        ✓
├── CRM Context                       ✓
├── CRM Mutations                     ✓
├── Document Management               ✓
├── Knowledge Processing Status       ✓
├── Semantic Document Search          -
└── Grounded Document Q&A             -

		

164. Acceptance Criteria

Part 28 is complete when:

			
✓ Part 27 regression suite remains green
✓ DocumentChunkingJob exists
✓ DocumentChunkSet exists
✓ DocumentChunk exists
✓ DocumentChunkSource exists
✓ migrations apply successfully
✓ PENDING exists
✓ PROCESSING exists
✓ COMPLETED exists
✓ FAILED exists
✓ chunking is a separate stage from extraction
✓ chunking requires a completed extraction
✓ chunking does not require re-extraction
✓ DocumentChunker abstraction exists
✓ ChunkingPolicy exists
✓ TokenCounter abstraction exists
✓ structure-aware chunker exists
✓ target_tokens exists
✓ max_tokens exists
✓ min_tokens exists
✓ overlap_tokens exists
✓ approved policies are server-controlled
✓ normal users cannot submit arbitrary unsafe chunking parameters
✓ sections are preferred as boundaries
✓ paragraphs are preferred before sentences
✓ sentences are preferred before token-level splitting
✓ token splitting is the final fallback
✓ PDF page provenance is preserved
✓ Chunks may span PDF pages
✓ page boundaries are not treated as mandatory semantic boundaries
✓ PPTX slide provenance is preserved
✓ slides are preferred as semantic boundaries where practical
✓ oversized slides can be split safely
✓ DOCX heading hierarchy is preserved
✓ section names are preserved
✓ heading context is preserved
✓ small tables remain intact where practical
✓ oversized tables split by logical row groups
✓ table headers can be repeated
✓ tables are not arbitrarily token-split when avoidable
✓ CSV rows remain structured
✓ CSV headers can be repeated across row-group Chunks
✓ overlap exists
✓ overlap is bounded
✓ structural overlap is preferred
✓ table overlap is handled separately
✓ overlap does not cause uncontrolled content expansion
✓ source_text exists
✓ retrieval_text exists
✓ source text remains faithful to extraction
✓ retrieval text may include explicit heading context
✓ augmented retrieval context is distinguishable from source content
✓ sequence_number exists
✓ sequence numbers are deterministic within a ChunkSet
✓ sequence numbers are unique within a ChunkSet
✓ start_page exists where applicable
✓ end_page exists where applicable
✓ start_slide exists where applicable
✓ end_slide exists where applicable
✓ DocumentChunkSource maps Chunks to extraction units
✓ source mappings preserve ordering
✓ architecture supports source offsets
✓ character_count exists
✓ token_count exists
✓ content_hash exists
✓ chunker_name exists
✓ chunker_version exists
✓ policy_name exists
✓ policy_hash exists
✓ Chunking identity includes extraction
✓ Chunking identity includes chunker version
✓ Chunking identity includes policy configuration
✓ duplicate job delivery is idempotent
✓ rechunking creates a new ChunkSet
✓ previous ChunkSets remain traceable
✓ only a completed ChunkSet becomes current
✓ partial ChunkSets never replace a valid current set
✓ every Chunk belongs to one Document
✓ cross-document Chunks are impossible
✓ every Chunk belongs to one Extraction
✓ every Chunk belongs to one ChunkSet
✓ every Chunk is tenant-scoped
✓ Chunk retrieval is tenant-scoped
✓ Chunk access inherits Document authorization
✓ Chunk access cannot bypass Document permissions
✓ deleted Documents do not expose Chunks through normal retrieval
✓ old Chunks remain traceable according to retention policy
✓ empty Chunks are rejected
✓ meaningless whitespace Chunks are rejected
✓ hard token limits are enforced
✓ structural exceptions are explicit
✓ source content is not silently lost
✓ chunk persistence is transactional
✓ ChunkSet current-state switch is atomic
✓ failed rechunking leaves previous ChunkSet current
✓ worker claiming is safe
✓ duplicate worker processing is prevented
✓ stale processing jobs can be recovered
✓ retry policy exists
✓ attempt limits exist
✓ Chunking status API exists
✓ authorized Chunk inspection API exists
✓ Chunk results are bounded
✓ CRM Context can expose chunking status
✓ CRM Context can expose Chunk count
✓ CRM Context does not inject all Chunk contents
✓ ChatGPT can report whether Chunking is complete
✓ ChatGPT can distinguish extracted from chunked
✓ ChatGPT does not claim semantic indexing exists yet
✓ ChatGPT does not perform unrestricted document Q&A yet
✓ chunking metrics exist
✓ Chunk size distribution is observable
✓ overlap expansion is observable
✓ Chunking failures are observable
✓ logs do not routinely contain Chunk contents
✓ no embedding model is required
✓ no vector database is required
✓ no LLM is required
✓ no ChatGPT connection is required
✓ Quorentra reports version 0.15.0

		

Most importantly:

Quorentra can now transform extracted customer documents into bounded, ordered, versioned, provenance-preserving knowledge units without depending on an embedding model, vector database, or LLM.

165. What We Have Achieved

Our document pipeline has evolved considerably.

Part 26 gave us:

			
File
      ↓
Document

Part 27 gave us:

			
Document
      ↓
Extraction
      ↓
Normalized Text

		

Part 28 now gives us:

			
Document
      ↓
Extraction
      ↓
Normalized Structure
      ↓
ChunkSet
      ↓
Chunk 1
Chunk 2
Chunk 3
...
Chunk N

		

Each Chunk knows:

			
Which tenant it belongs to
Which Document it came from
Which Extraction produced it
Which ChunkSet owns it
Where in the source it originated
Which pages or slides it covers
Which section it belongs to
Which chunker created it
Which policy was used
How large it is
What its source content is
What its retrieval content is

		

That is a strong knowledge architecture.

166. The Opportunity Workspace Is Becoming an AI Knowledge Space

Our example Opportunity now looks conceptually like:

			
Azure Migration
Adventure Works
€90,000
Proposal
60%
│
├── Contacts
│
├── Activities
│
├── Tasks
│
├── Meetings
│
└── Documents
    │
    ├── Requirements.pdf
    │   ├── Extracted ✓
    │   └── 38 Chunks ✓
    │
    ├── Azure-Architecture.pptx
    │   ├── Extracted ✓
    │   └── 17 Chunks ✓
    │
    ├── Proposal-v3.pdf
    │   ├── Extracted ✓
    │   └── 42 Chunks ✓
    │
    └── Pricing.xlsx
        └── Extraction not yet supported

		

The documents are no longer opaque binary files.

They are becoming structured knowledge sources.

167. But the Chunks Are Still Not Semantically Searchable

Suppose we now have:

			
Proposal-v3.pdf
42 Chunks

The user asks:

What are the payment terms?

We could search with ordinary keywords:

			
payment
terms

But what if the document says:

Commercial settlement will occur in three milestone-based instalments.

Keyword search may miss it.

We need semantic similarity.

168. From Text to Meaning

The next transformation is:

			
Chunk Text
      ↓
Embedding Model
      ↓
Vector

		

A vector represents the semantic meaning of the Chunk numerically.

Then:

			
"What are the payment terms?"
      ↓
Question Embedding
      ↓
Vector Similarity
      ↓
Relevant Chunks

		

This is where Quorentra’s document knowledge begins becoming semantically searchable.

169. But Embeddings Must Remain Modular

We should not write:

Google embedding API

directly throughout the application.

Just as we created:

			
DocumentStorage
DocumentExtractor
DocumentChunker
TokenCounter

we should introduce:

EmbeddingProvider

The rest of Quorentra should not care which provider creates the vectors.

170. ChatGPT-Native Does Not Mean Provider-Locked

Quorentra is designed around ChatGPT as the primary conversational CRM interface.

But its knowledge infrastructure should still have clean provider boundaries.

We want:

			
Quorentra
      ↓
EmbeddingProvider
      ↓
Selected Embedding Model

		

not:

			
Quorentra domain logic
      ↓
hard-coded vendor calls everywhere

This keeps the platform maintainable.

171. The Next Storage Problem

Embeddings are vectors such as:

			
[
  0.0142,
 -0.0371,
  0.0924,
 ...
]

		

We need to store and search them efficiently.

Quorentra already uses PostgreSQL.

The natural next step is:

			
PostgreSQL
+
pgvector

This keeps:

			
CRM Data
Document Metadata
Chunk Metadata
Vector Metadata

within one strongly governed data platform.

172. Tenant Isolation Becomes Even More Important

When semantic search arrives, we must ensure:

Tenant A Question

can never retrieve:

Tenant B Chunk

even if it is semantically similar.

Our existing architecture already gives every Chunk:

organization_id

That was deliberate.

Future vector queries must filter by tenant as part of retrieval.

173. Provenance Will Survive Embedding

The vector will not replace the Chunk.

Instead:

			
Vector
      ↓
references
      ↓
DocumentChunk
      ↓
DocumentChunkSource
      ↓
DocumentExtractionUnit
      ↓
Document

		

Therefore a semantic search result can always be traced back to the original customer file.

174. This Is the Foundation of Grounded AI

Eventually:

			
User Question
      ↓
Question Embedding
      ↓
Vector Search
      ↓
Chunk 22
      ↓
Proposal-v3.pdf
Pages 16–17
      ↓
ChatGPT
      ↓
Grounded Answer

		

The answer will not come from generic model memory.

It will come from Quorentra’s authoritative CRM knowledge.

175. Next Article

In Part 29, we will build:

Building the Embedding and Vector Storage Layer — Embedding Providers, Batch Generation, PostgreSQL, pgvector, Vector Versioning, Tenant Isolation, and Semantic Indexing

We will introduce:

			
EmbeddingProvider
EmbeddingModel
EmbeddingJob
EmbeddingSet
ChunkEmbedding
Embedding Dimensions
Embedding Model Name
Embedding Model Version
Embedding Provider
Batch Embedding
Embedding Requests
Embedding Responses
Embedding Retry Policy
Embedding Rate Limits
Embedding Token Accounting
Embedding Cost Accounting
Content Hash Reuse
Embedding Idempotency
Re-embedding
Vector Versioning
PostgreSQL pgvector
vector Extension
Vector Columns
Vector Dimensions
Vector Indexes
HNSW
IVFFlat Considerations
Cosine Similarity
Inner Product
Euclidean Distance
Tenant-Scoped Vector Storage
Document-Scoped Filtering
Chunk Metadata Filtering
Current Embedding Set
Embedding Status
Semantic Index Status
Embedding Metrics
Embedding Tests

		

Our knowledge pipeline will become:

			
Document
      ↓
Extraction
      ↓
Normalized Text
      ↓
Chunking
      ↓
DocumentChunk
      ↓
EmbeddingProvider
      ↓
Embedding Vector
      ↓
PostgreSQL + pgvector

		

And Quorentra will move from:

We have structured knowledge chunks.

to:

We can mathematically search those chunks by meaning.

That will establish the semantic foundation needed for the next major stage:

			
Question
      ↓
Semantic Search
      ↓
Relevant CRM Knowledge
      ↓
Grounded ChatGPT Response

		

Part 29 therefore marks the beginning of Quorentra’s semantic knowledge layer—while preserving the same modular, tenant-aware, provider-abstracted architecture we have used throughout Building Quorentra CRM from Zero.

Quorentra CRM Building the Document Chunking Pipeline: Building from Zero — Part 28

1. Introduction

2. Why Chunking Exists

3. The Retrieval Unit

4. Chunking Is Not Summarization

5. Chunking Should Not Require an LLM

6. Starting Checkpoint

7. The Naive Chunking Approach

8. Structure-Aware Chunking

9. Chunking Hierarchy

10. Chunking Is a Processing Stage

11. Why Separate Extraction and Chunking Jobs?

12. DocumentChunkingJob

13. Chunking Status

14. Chunking Lifecycle

15. Chunking Depends on Extraction

16. Chunking Permission

17. DocumentChunk

18. Sequence Number

19. Chunk Identity

20. Chunk Text

21. Retrieval-Ready Does Not Mean Rewritten

22. Heading Context

23. Heading Context Should Be Explicit

24. Page Ranges

25. Slide Ranges

26. Source Unit Range

27. More Precise Provenance

28. DocumentChunkSource

29. Why a Source Mapping Table Helps

30. MVP Decision

31. Chunker Abstraction

32. Chunking Policy

33. Default Policy

34. Target Versus Maximum

35. Why Token Budgets?

36. Tokenizer Abstraction

37. Why Abstract Token Counting?

38. Token Count Is Metadata

39. Character Count Still Matters

40. Minimum Chunk Size

41. Do Not Merge Across Strong Boundaries Blindly

42. Chunking Priorities

43. Paragraph-Aware Chunking

44. Paragraph Accumulation

45. Oversized Paragraphs

46. Sentence-Aware Splitting

47. Final Token-Level Fallback

48. Section-Aware Chunking

49. Section Names

50. Nested Headings

51. PDF Page Boundaries

52. Why Not One Chunk Per Page?

53. PPTX Is Different

54. Small Slides

55. Oversized Slides

56. Table Preservation

57. Small Tables

58. Large Tables

59. Table Metadata

60. CSV Chunking

61. Chunk Overlap

62. But Overlap Has a Cost

63. Default Overlap

64. Structural Overlap

65. Do Not Overlap Tables Blindly

66. Do Not Overlap Section Headers as Fake Content

67. GeneratedChunk

68. Chunk Source Mapping

69. Chunk Provenance

70. Chunker Version

71. Policy Versioning

72. Configuration Hash

73. Idempotent Chunking

74. Rechunking

75. Chunk Set

76. Why Introduce ChunkSet?

77. Future Embedding Relationship

78. Current Chunk Set

79. Never Switch to Partial Chunking