Series: Building Quorentra CRM from Zero — A Modular, ChatGPT-Native AI CRM

Building provider-abstracted embeddings, batch generation, PostgreSQL pgvector storage, vector versioning, tenant isolation, semantic indexing, retries, usage accounting, and re-embedding.

Quorentra Building the Embedding and Vector Storage Layer: Building from Zero — Part 29

1. Introduction

In Part 28, Quorentra gained a structure-aware Document Chunking Pipeline.

Our knowledge ingestion architecture now looks like:

			
Document
      ↓
Extraction
      ↓
Normalized Content
      ↓
ChunkSet
      ↓
DocumentChunk

		

Instead of treating an entire customer document as one enormous block of text, Quorentra can now produce bounded knowledge units such as:

			
Chunk 21
Commercial Proposal
Pages 14–15
487 tokens

and:

			
Chunk 22
Payment Terms
Pages 16–17
312 tokens

Each Chunk preserves:

			
Tenant
Document
Extraction
ChunkSet
Sequence
Section
Heading Context
Page / Slide Provenance
Source Text
Retrieval Text
Token Count
Content Hash
Chunker Version
Policy Version

		

This gives us high-quality retrieval units.

But Quorentra still cannot search those units by meaning.

If the user asks:

How does Adventure Works pay for the migration?

the relevant document may not contain those exact words.

It may say:

			
Commercial settlement will occur in three
milestone-based instalments.

Traditional keyword matching may not connect:

How does the customer pay?

with:

milestone-based instalments

Semantic search can.

To build semantic search, we first need embeddings.

In Part 29, we will build:

The Embedding and Vector Storage Layer

Our pipeline becomes:

			
Document
      ↓
Extraction
      ↓
Chunking
      ↓
DocumentChunk
      ↓
EmbeddingProvider
      ↓
Embedding Vector
      ↓
PostgreSQL + pgvector

		

This is the beginning of Quorentra’s semantic knowledge layer.

2. What Is an Embedding?

An embedding transforms text into a numerical vector.

For example:

"The project will cost €90,000."

may conceptually become:

			
[
   0.0182,
  -0.0421,
   0.0917,
   ...
]

		

The actual vector may contain hundreds or thousands of dimensions.

The important property is not the individual numbers.

It is their spatial relationship.

Texts with similar meaning tend to produce vectors that are closer together.

3. Semantic Similarity

Consider:

			
A:
What are the payment terms?
B:
The customer will pay in three milestone-based instalments.
C:
The solution uses Kubernetes for container orchestration.

		

Semantically:

A ↔ B

are closely related.

But:

A ↔ C

are not.

An embedding model attempts to represent this relationship mathematically.

4. Why Embeddings Matter for CRM

CRM knowledge is rarely phrased exactly like user questions.

A user might ask:

Why is this opportunity at risk?

while CRM documents say:

			
Executive sponsor has not attended
the last three steering meetings.

Or:

What does the customer care about most?

while the proposal says:

			
Business continuity and minimizing production
downtime are the primary migration priorities.

Semantic retrieval helps bridge these vocabulary differences.

5. Embeddings Are an Index

This architectural principle is critical:

An embedding is not authoritative CRM knowledge.

The authoritative chain remains:

			
Document
      ↓
Extraction
      ↓
Chunk

		

The vector is merely an index that helps us find the Chunk.

Conceptually:

			
Embedding
      ↓
points to
      ↓
DocumentChunk

		

Never reverse this relationship.

6. Why This Matters

Embedding models can change.

Vectors can be regenerated.

Indexes can be rebuilt.

But the authoritative source should remain:

Customer Document

and its traceable extracted content.

Therefore:

			
Document = source
Extraction = recovered source content
Chunk = retrieval unit
Embedding = semantic index

This separation gives us a maintainable architecture.

7. Starting Checkpoint

Before Part 29, verify Part 28.

Run:

			
cd backend
python -m pytest

Then:

			
cd ..\chatgpt-ui
npm run build

Upload and process a test document.

Verify:

			
Document
      ↓
Extraction COMPLETED
      ↓
Chunking COMPLETED

		

Then inspect:

GET /api/v1/documents/{document_id}/chunks

Confirm that Chunks contain:

			
sequence_number
source_text
retrieval_text
token_count
content_hash
section_name
page provenance

		

Only then should we generate embeddings.

8. Enable pgvector

Quorentra already uses PostgreSQL 17.

For vector storage, we add:

pgvector

The PostgreSQL extension is enabled with:

CREATE EXTENSION IF NOT EXISTS vector;

In pgAdmin:

			
Database
      ↓
Query Tool
      ↓
CREATE EXTENSION IF NOT EXISTS vector;

		

Then verify:

			
SELECT extname
FROM pg_extension
WHERE extname = 'vector';

Expected:

vector

9. Why PostgreSQL + pgvector?

We could introduce a dedicated vector database.

But for the modular MVP, PostgreSQL + pgvector has significant advantages.

We already store:

			
Organizations
Users
Companies
Contacts
Opportunities
Activities
Tasks
Meetings
Documents
Extractions
Chunks

		

in PostgreSQL.

Adding vectors gives us:

			
PostgreSQL
│
├── CRM Data
├── Document Metadata
├── Knowledge Metadata
└── Vector Index

		

This simplifies:

			
Transactions
Backups
Tenant Filtering
Authorization Joins
Operations
Local Development
Deployment

		

For Quorentra’s MVP, this is an excellent fit.

10. Avoid Premature Vector Infrastructure

Do not immediately introduce:

			
Pinecone
Weaviate
Qdrant
Milvus
Elasticsearch Vector Search
Dedicated Retrieval Cluster

		

unless the system actually requires it.

The modular architecture should allow a different vector backend later.

But the MVP should remain operationally simple.

11. Embedding Provider Abstraction

Do not call a specific embedding API directly from the Chunk service.

Introduce:

EmbeddingProvider

Conceptually:

			
class EmbeddingProvider(Protocol):
    async def embed_texts(
        self,
        texts: list[str],
    ) -> "EmbeddingBatchResult":
        ...

		

This becomes the provider boundary.

12. Why Provider Abstraction?

Today we may use:

Provider A

Tomorrow:

Provider B

or:

Local Embedding Model

The rest of Quorentra should not care.

It should request:

Generate embeddings for these texts.

and receive:

EmbeddingBatchResult

13. ChatGPT-Native Does Not Mean Embedding-Locked

Quorentra is designed to lean heavily on ChatGPT and the Apps SDK for the conversational experience.

That does not mean every infrastructure component must be tightly coupled to one model or provider.

A strong architecture is:

			
ChatGPT
      ↓
Quorentra Tools
      ↓
Quorentra Backend
      ↓
Knowledge Retrieval
      ↓
EmbeddingProvider

		

ChatGPT consumes the retrieval capability.

It does not own the indexing architecture.

14. Embedding Model Configuration

Introduce explicit configuration:

			
EMBEDDING_PROVIDER
EMBEDDING_MODEL
EMBEDDING_DIMENSIONS
EMBEDDING_BATCH_SIZE
EMBEDDING_MAX_RETRIES
EMBEDDING_TIMEOUT_SECONDS

		

Do not scatter model names throughout the codebase.

15. Example Configuration

Conceptually:

			
EMBEDDING_PROVIDER=provider_name
EMBEDDING_MODEL=embedding_model_name
EMBEDDING_DIMENSIONS=1536
EMBEDDING_BATCH_SIZE=64
EMBEDDING_MAX_RETRIES=3

		

The exact model and dimensions depend on the selected provider.

The architecture must not assume one fixed dimension globally forever.

16. Embedding Model Identity

Store more than:

model_name

Prefer:

			
provider_name
model_name
model_version
dimensions

where the provider exposes version information.

This becomes part of semantic-index provenance.

17. Why Dimensions Matter

A pgvector column has a vector dimensionality.

For example:

vector(1536)

A vector with:

768 dimensions

cannot simply be inserted into that column.

Therefore dimensionality is an architectural concern.

18. MVP Dimensionality Strategy

For the MVP, choose one active embedding configuration.

For example:

			
Active Embedding Profile
├── Provider
├── Model
└── Dimensions

All current semantic retrieval uses that profile.

Later we can support multiple profiles more flexibly.

19. Embedding Profile

Introduce:

EmbeddingProfile

Conceptually:

			
EmbeddingProfile
├── name
├── provider
├── model
├── model_version
├── dimensions
├── distance_metric
└── active

		

For the MVP, this may be application configuration rather than a database table.

The concept is still important.

20. Distance Metric

Semantic similarity requires a distance or similarity function.

Common options include:

			
Cosine Distance
Euclidean Distance
Inner Product

For text embeddings, cosine similarity is a common default.

Conceptually:

			
Question Vector
      ↓
Cosine Similarity
      ↓
Nearest Chunk Vectors

		

21. Consistency Is More Important Than Fashion

Do not switch distance metrics casually.

The combination:

			
Embedding Model
+
Vector Normalization
+
Distance Metric

		

should be treated as one retrieval configuration.

Changing it can alter ranking behavior.

22. Embedding Processing Stage

Just as we separated:

Extraction

and:

Chunking

we now separate:

Embedding

Introduce:

DocumentEmbeddingJob

or more generally:

EmbeddingJob

23. EmbeddingJob

Conceptually:

			
EmbeddingJob
├── id
├── organization_id
├── document_id
├── chunk_set_id
├── status
├── provider_name
├── model_name
├── model_version
├── dimensions
├── attempt_count
├── started_at
├── completed_at
├── failed_at
├── error_code
├── error_message
├── created_at
└── updated_at

		

24. Embedding Status

Define:

			
class EmbeddingJobStatus(str, Enum):
    PENDING = "pending"
    PROCESSING = "processing"
    COMPLETED = "completed"
    FAILED = "failed"

		

Again, we reuse a familiar processing lifecycle.

25. Pipeline State

Our Document may now be:

			
Document:
AVAILABLE
Extraction:
COMPLETED
Chunking:
COMPLETED
Embedding:
PENDING

		

Then:

			
Embedding:
PROCESSING

and finally:

			
Embedding:
COMPLETED

Only then is semantic indexing ready.

26. EmbeddingSet

As Part 28 introduced:

DocumentChunkSet

Part 29 introduces:

EmbeddingSet

Conceptually:

			
EmbeddingSet
├── id
├── organization_id
├── document_id
├── chunk_set_id
├── embedding_job_id
├── provider_name
├── model_name
├── model_version
├── dimensions
├── distance_metric
├── embedding_count
├── total_input_tokens
├── created_at
└── is_current

		

27. Why an EmbeddingSet?

Suppose:

ChunkSet A

contains:

42 chunks

and we embed them using:

Model X

That collection of vectors is one coherent semantic index version.

Later we may use:

Model Y

and generate a new EmbeddingSet.

28. Pipeline Lineage

We now have:

			
Document
      ↓
Extraction
      ↓
ChunkSet
      ↓
EmbeddingSet

		

Each stage has explicit provenance.

That is exactly what we want.

29. ChunkEmbedding

Introduce:

ChunkEmbedding

Conceptually:

			
ChunkEmbedding
├── id
├── organization_id
├── document_id
├── chunk_set_id
├── chunk_id
├── embedding_set_id
├── vector
├── dimensions
├── provider_name
├── model_name
├── model_version
├── input_token_count
├── content_hash
├── created_at
└── version

		

The vector belongs to one Chunk.

30. Vector Column

With pgvector:

embedding vector(1536)

for an example 1536-dimensional profile.

The actual dimension must match the selected model.

Do not blindly copy this number if your model uses another dimensionality.

31. SQLAlchemy Model

Conceptually:

			
class ChunkEmbedding(Base):
    __tablename__ = "chunk_embeddings"
    id = mapped_column(UUID, primary_key=True)
    organization_id = mapped_column(
        UUID,
        nullable=False,
        index=True,
    )
    chunk_id = mapped_column(
        UUID,
        nullable=False,
        index=True,
    )
    embedding_set_id = mapped_column(
        UUID,
        nullable=False,
        index=True,
    )
    embedding = mapped_column(
        Vector(settings.embedding_dimensions),
        nullable=False,
    )

		

The exact pgvector SQLAlchemy integration depends on the library version used.

32. Do Not Store Vectors as JSON

Avoid:

"[0.12, -0.44, ...]"

inside a JSON or text column.

That prevents efficient vector indexing and similarity operators.

Use the native:

vector

type.

33. What Text Should Be Embedded?

Part 28 deliberately introduced:

source_text

and:

retrieval_text

For semantic indexing, use:

retrieval_text

because it may contain useful structural context.

Example:

			
Commercial Proposal > Payment Terms
Payment will be made in three stages...

This can produce a more useful semantic representation.

34. But Evidence Still Uses Source Text

The vector finds the Chunk using:

retrieval_text

But future citations should display:

source_text

and provenance.

This prevents augmented retrieval metadata from being presented as if it were literal source content.

35. Embedding Batch

Calling an embedding API once per Chunk is inefficient.

Bad:

			
Chunk 1 → API
Chunk 2 → API
Chunk 3 → API
...
Chunk 100 → API

		

Prefer:

			
Chunks 1–64
      ↓
Batch Request

then:

			
Chunks 65–100
      ↓
Batch Request

according to provider limits.

36. EmbeddingBatchResult

Conceptually:

			
class EmbeddingBatchResult(BaseModel):
    embeddings: list[list[float]]
    provider_name: str
    model_name: str
    model_version: str | None
    input_tokens: int | None

		

The result ordering must correspond exactly to input ordering.

37. Validate Batch Cardinality

If we submit:

64 texts

we must receive:

64 vectors

Otherwise:

EMBEDDING_RESPONSE_INVALID

Do not persist partially mismatched results.

38. Validate Dimensions

Every returned vector must contain:

expected_dimensions

values.

If configured:

and the provider returns:

fail.

Do not silently truncate or pad vectors.

39. Batch Size

Configure:

EMBEDDING_BATCH_SIZE

rather than hard-coding it.

The correct value depends on:

			
Provider Limits
Input Token Limits
Latency
Memory
Rate Limits
Cost

		

40. Batch by Token Budget Too

A fixed number of Chunks may still exceed provider limits.

For example:

64 × 700 tokens

may be too large for a particular API request.

Therefore the batching algorithm should consider:

			
maximum items
+
maximum estimated tokens

where supported.

41. Embedding Workflow

Conceptually:

			
Claim EmbeddingJob
      ↓
Load ChunkSet
      ↓
Verify Current / Valid
      ↓
Load Chunks
      ↓
Build Batches
      ↓
Call EmbeddingProvider
      ↓
Validate Vectors
      ↓
Persist ChunkEmbeddings
      ↓
Create / Complete EmbeddingSet
      ↓
Set Current
      ↓
Mark Job COMPLETED

		

42. Do Not Hold One Huge Database Transaction Across API Calls

This is important.

Bad:

			
BEGIN TRANSACTION
      ↓
Call external API
      ↓
wait
      ↓
call API again
      ↓
wait
      ↓
persist 500 vectors
      ↓
COMMIT

		

External calls may take time.

Avoid holding database locks unnecessarily.

43. Staged Persistence

A better approach:

			
Create EmbeddingSet
      ↓
Process Batch
      ↓
Persist Batch Results
      ↓
Process Next Batch
      ↓
Persist
      ↓
Validate Complete Set
      ↓
Atomically Mark Current

		

An incomplete set remains:

non-current

until all Chunks are embedded.

44. Partial Progress

For a ChunkSet containing:

1,000 chunks

we should not discard 950 successful embeddings because the final batch encounters a temporary rate limit.

Persist completed batches safely.

Then retry missing work.

45. Embedding Item State

For larger-scale processing, we may eventually track item-level status.

For the MVP, the presence of:

ChunkEmbedding

for a Chunk can indicate successful completion.

The worker can calculate:

missing chunks

on retry.

46. Idempotency

Embedding generation must be idempotent.

A useful identity is:

			
chunk_id
+
chunk content_hash
+
provider
+
model
+
model version

		

If an identical embedding already exists, avoid generating it unnecessarily.

47. Embedding Identity

More formally:

			
Embedding Identity
=
Content Hash
+
Embedding Profile

		

This is stronger than using Chunk ID alone.

Why?

Because a different Chunk may contain identical retrieval text.

48. Embedding Reuse

Suppose rechunking creates:

Chunk A

with the same:

content_hash

as a previously embedded Chunk.

If the embedding profile is identical, we may be able to reuse the vector.

This can reduce:

			
API Calls
Cost
Latency

49. Be Careful with Retrieval Text

If the hash is based only on:

source_text

but embeddings use:

retrieval_text

reuse may be incorrect.

Therefore the embedding cache identity should hash the actual embedded input:

			
embedding_input_hash
=
SHA-256(retrieval_text)

50. Store Both Hashes

Useful fields:

			
source_content_hash
embedding_input_hash

The first identifies source content.

The second identifies what the embedding model actually received.

51. Re-Embedding

We need deliberate support for:

Re-embedding

Reasons include:

			
New embedding model
New model version
Changed dimensions
Changed retrieval text
Changed heading augmentation
Retrieval quality improvements
Provider migration

		

52. Re-Embedding Does Not Require Re-Extraction

Our modular architecture pays off here.

We can perform:

			
Existing ChunkSet
      ↓
New Embedding Model
      ↓
New EmbeddingSet

		

without:

			
Re-upload
Re-extract
Rechunk

That is a major operational advantage.

53. Rechunking Versus Re-Embedding

Keep them distinct.

Rechunking:

			
Extraction
      ↓
New ChunkSet

Re-embedding:

			
Existing ChunkSet
      ↓
New EmbeddingSet

This distinction helps diagnose retrieval changes.

54. Current EmbeddingSet

A Document should normally have one current semantic index for the active profile.

Example:

			
EmbeddingSet 1
Model A
is_current = false
EmbeddingSet 2
Model B
is_current = true

		

Future retrieval uses:

EmbeddingSet 2

55. Atomic Current Switch

Do not switch:

is_current = true

until every required Chunk has a valid embedding.

The sequence should be:

			
Build New EmbeddingSet
      ↓
Validate Completeness
      ↓
Begin Transaction
      ↓
Old Current = false
New Current = true
      ↓
Commit

		

56. Failed Re-Embedding

Suppose the new model fails halfway.

Expected:

			
Old EmbeddingSet
remains current

The user should not lose semantic search because an upgrade failed.

57. Retry Policy

Embedding APIs introduce transient failures such as:

			
Timeout
Rate Limit
Temporary Provider Error
Network Failure
Service Unavailable

		

These should be retryable.

58. Permanent Errors

Examples:

			
Invalid API Credentials
Unsupported Model
Invalid Dimensions
Input Too Large
Malformed Provider Response

		

These should not retry indefinitely.

59. Structured Errors

Introduce codes such as:

			
EMBEDDING_PROVIDER_UNAVAILABLE
EMBEDDING_RATE_LIMITED
EMBEDDING_TIMEOUT
EMBEDDING_MODEL_INVALID
EMBEDDING_INPUT_TOO_LARGE
EMBEDDING_DIMENSION_MISMATCH
EMBEDDING_RESPONSE_INVALID
EMBEDDING_AUTHENTICATION_FAILED
EMBEDDING_JOB_FAILED

		

60. Exponential Backoff

For transient failures:

			
Attempt 1
      ↓
Wait
Attempt 2
      ↓
Wait Longer
Attempt 3

		

Use exponential backoff with jitter.

Avoid retry storms.

61. Rate Limits

The provider may enforce:

			
Requests per minute
Tokens per minute
Concurrent requests

The embedding worker should respect these constraints.

Do not let every worker independently overwhelm the provider.

62. Concurrency Control

Configure:

EMBEDDING_MAX_CONCURRENCY

or enforce queue-level concurrency.

This becomes especially important when multiple tenants upload large document sets simultaneously.

63. Token Accounting

Store embedding input usage.

At minimum:

input_token_count

per embedding or batch where available.

Then aggregate into:

EmbeddingSet.total_input_tokens

64. Why Token Accounting Matters

Embedding cost may be small per document.

But across:

			
Thousands of Documents
Millions of Chunks
Multiple Re-Embedding Runs

it becomes meaningful.

Quorentra should understand its AI consumption from the beginning.

65. Cost Accounting

If provider pricing is known, calculate:

estimated_cost

at processing time or through a usage service.

Do not hard-code prices permanently into historical records.

Store:

			
Usage
Model
Timestamp

and calculate cost using versioned pricing data where practical.

66. Future AI Usage Ledger

Part 29 can prepare for a broader:

AIUsageEvent

model.

Conceptually:

			
AIUsageEvent
├── organization_id
├── operation
├── provider
├── model
├── input_tokens
├── output_tokens
├── units
├── estimated_cost
├── related_entity_type
├── related_entity_id
└── created_at

		

For embeddings:

			
operation =
embedding

67. Why Usage Must Be Tenant-Aware

Later Quorentra may support:

			
Usage Limits
Plan Quotas
Cost Dashboards
Tenant Billing
Fair Use

		

Therefore embedding usage should already include:

organization_id

68. Secrets

Embedding API keys must never be stored in:

			
Document
Chunk
EmbeddingJob
EmbeddingSet

Use:

			
Environment Variables
Secret Manager
Deployment Secrets

depending on environment.

69. Never Log API Keys

Structured logs should contain:

			
provider
model
job_id
status
duration

		

not:

			
Authorization Header
API Key
Secret

70. What About the Text Sent to the Provider?

This is a significant privacy boundary.

Embedding generation may send:

Customer Document Content

to an external provider.

That means provider selection has:

			
Privacy
Security
Compliance
Data Residency
Contractual
Retention

		

implications.

71. Provider Policy Must Be Explicit

Quorentra should know:

			
Which provider processes customer data?
Where?
Under what retention terms?
Under what tenant configuration?

For the MVP, use one approved provider.

Do not allow arbitrary user-supplied providers without governance.

72. Future Local Embeddings

Because we use:

EmbeddingProvider

we can later support:

			
CloudEmbeddingProvider
LocalEmbeddingProvider
EnterpriseEmbeddingProvider

without changing the Chunk domain.

That is precisely why the abstraction matters.

73. Tenant Isolation in Vector Storage

Every ChunkEmbedding must contain:

organization_id

This may seem redundant because Chunk already has it.

Keep it anyway.

Why?

Because future vector queries should filter directly at the vector table.

74. Never Search Globally Then Filter in Python

Bad:

			
SELECT nearest vectors globally
      ↓
Return 100
      ↓
Python removes wrong tenants

		

This can:

			
Leak Ranking Information
Miss Correct Tenant Results
Create Security Risk

Tenant filtering belongs in the database query.

75. Correct Conceptual Query

Future retrieval should resemble:

			
SELECT ...
FROM chunk_embeddings
WHERE organization_id = :organization_id
ORDER BY embedding <=> :query_vector
LIMIT :limit;

		

The exact operator depends on the chosen distance metric.

The important rule is:

			
Tenant filter
+
Vector similarity

occur together.

76. Document Filtering

Future semantic search may also restrict by:

document_id

Example:

Search only Proposal-v3.pdf.

Then:

			
organization_id = tenant
AND
document_id = proposal

must be part of retrieval.

77. CRM Entity Filtering

Because Documents are linked to CRM entities, future search may ask:

Search documents attached to the Azure Migration opportunity.

The retrieval layer will eventually need to join:

			
Chunk
      ↓
Document
      ↓
DocumentLink
      ↓
Opportunity

		

or maintain safe derived filter metadata.

Part 29 does not implement the full retrieval service yet.

But the data model must support it.

78. Vector Indexes

Without an approximate nearest-neighbor index, vector search may require scanning many vectors.

For small MVP datasets, exact search may be acceptable.

As data grows, pgvector supports indexes such as:

			
HNSW
IVFFlat

79. HNSW

HNSW generally provides strong query performance and does not require the same training step as IVFFlat.

It is often an attractive default for modern pgvector workloads.

Conceptually:

			
CREATE INDEX ...
USING hnsw (embedding vector_cosine_ops);

The exact index should match:

			
Distance Metric
Vector Type
pgvector Version

80. IVFFlat

IVFFlat is another option.

It may be useful for particular workloads but requires careful configuration such as:

			
lists
probes

For the MVP, we do not need to optimize prematurely.

81. Start Simple

A sensible development sequence is:

			
Correct vector storage
Correct tenant filtering
Exact similarity queries
Measure
Add ANN index
Measure again

		

Do not introduce indexing complexity before correctness.

82. HNSW for the MVP?

If the expected dataset is modest during development, we can implement the schema first and add HNSW either immediately or once semantic search is introduced.

The critical architecture in Part 29 is:

			
correct vectors
correct lineage
correct tenancy
correct versioning

not micro-optimizing query latency.

83. Index Creation Migration

If we choose cosine distance and HNSW:

			
chunk_embeddings
      ↓
HNSW Index
      ↓
vector_cosine_ops

		

should be created through Alembic.

Do not rely on manual production database steps.

84. pgvector Migration

The migration may need to enable:

CREATE EXTENSION IF NOT EXISTS vector;

depending on deployment privileges and database provisioning strategy.

In some production environments, extension installation may be infrastructure-managed.

Keep that distinction explicit.

85. Database Tables

Part 29 introduces:

			
embedding_jobs
embedding_sets
chunk_embeddings

Optionally:

ai_usage_events

if we decide to start usage accounting now.

86. Recommended Model

For the modular MVP, I recommend adding:

ai_usage_events

now.

AI usage will expand rapidly once we introduce:

			
Embeddings
Retrieval
ChatGPT Reasoning
Summarization
Automation

		

A generic usage ledger is easier to establish early.

87. Database Migration

Generate:

			
cd backend
alembic revision --autogenerate -m "add embedding and vector storage"

Review carefully.

Then:

alembic upgrade head

Verify:

alembic current

Then confirm the vector extension:

			
SELECT extversion
FROM pg_extension
WHERE extname = 'vector';

88. Suggested Backend Structure

Extend the knowledge-processing architecture:

			
backend/app/documents/
└── processing/
    ├── extraction/
    ├── chunking/
    │
    └── embedding/
        ├── models.py
        ├── schemas.py
        ├── repository.py
        ├── service.py
        ├── worker.py
        ├── queue.py
        ├── batching.py
        ├── validation.py
        └── providers/
            ├── base.py
            └── configured_provider.py

		

And optionally:

			
backend/app/ai/
└── usage/
    ├── models.py
    ├── repository.py
    └── service.py

		

89. Provider Interface

Conceptually:

			
class EmbeddingProvider(Protocol):
    @property
    def provider_name(self) -> str:
        ...
    @property
    def model_name(self) -> str:
        ...
    @property
    def dimensions(self) -> int:
        ...
    async def embed_texts(
        self,
        texts: list[str],
    ) -> EmbeddingBatchResult:
        ...

		

The provider implementation handles external API details.

90. Provider Responsibilities

The provider adapter may handle:

			
Authentication
Request Serialization
Provider SDK
Response Parsing
Provider-Specific Errors
Model Selection

		

It should not handle:

			
Tenant Authorization
Chunk Selection
Database Persistence
Current EmbeddingSet
CRM Context

		

Those belong elsewhere.

91. Embedding Service Responsibilities

The application service handles:

			
Load ChunkSet
Validate Tenant
Validate State
Select Embedding Profile
Create Job
Build Batches
Invoke Provider
Validate Results
Persist Embeddings
Record Usage
Complete EmbeddingSet

		

This is the orchestration layer.

92. Repository Responsibilities

The repository handles:

			
EmbeddingJob persistence
EmbeddingSet persistence
ChunkEmbedding persistence
Current-set queries
Missing-embedding queries

		

It should not call external APIs.

93. Worker Responsibilities

The worker handles:

			
Claim Job
Invoke Embedding Service
Handle Retryable Failure
Update Job State
Record Operational Metrics

		

Again, narrow responsibilities.

94. Embedding Job Creation

After Chunking completes:

			
ChunkSet
      ↓
Create EmbeddingJob

This can happen automatically.

But the stage remains logically independent.

95. Manual Re-Embedding API

Introduce:

POST /api/v1/documents/{document_id}/embeddings

for administrative or authorized re-indexing.

Input:

			
{
  "profile": "default"
}

The server resolves the actual provider/model configuration.

96. Do Not Let Normal Users Supply Arbitrary Model Names

Bad:

			
{
  "provider": "whatever",
  "model": "whatever"
}

This creates:

			
Security Risk
Cost Risk
Compatibility Risk
Data Governance Risk

Use approved server-side profiles.

97. Embedding Status API

Introduce:

GET /api/v1/documents/{document_id}/embeddings

Example:

			
{
  "status": "completed",
  "embedding_set_id": "...",
  "embedding_count": 42,
  "provider": "configured_provider",
  "model": "configured_model",
  "dimensions": 1536,
  "total_input_tokens": 18324
}

		

98. Semantic Index Status

CRM Context may now include:

			
Proposal-v3.pdf
Extraction:
completed
Chunking:
completed
Embedding:
completed
Semantic Index:
ready

		

This is useful to ChatGPT.

99. Knowledge Status Widget

Extend:

DocumentKnowledgeStatus.tsx

to show:

			
┌────────────────────────────────────────────┐
│ Azure Migration Proposal v3                │
│                                            │
│ Knowledge processing                       │
│                                            │
│ ✓ Text extracted                           │
│ ✓ Content chunked                          │
│ ✓ Semantic index created                   │
│                                            │
│ 42 chunks                                  │
│ 42 embeddings                              │
│                                            │
│ Semantic search                            │
│ Ready                                      │
└────────────────────────────────────────────┘

		

100. Failed Embedding State

Example:

			
┌────────────────────────────────────────────┐
│ Azure Migration Proposal v3                │
│                                            │
│ ✓ Text extracted                           │
│ ✓ Content chunked                          │
│ ! Semantic indexing failed                 │
│                                            │
│ The document remains available.            │
│ Extracted text remains available.          │
└────────────────────────────────────────────┘

		

Again, failure in one processing layer must not invalidate earlier layers.

101. Query Embeddings

We are primarily building stored Chunk embeddings in Part 29.

But semantic search will also require:

			
Question
      ↓
EmbeddingProvider
      ↓
Query Vector

		

Do not persist every query vector permanently by default.

Query embeddings are usually ephemeral.

102. Same Embedding Space

This is critical.

Chunk embeddings and query embeddings must use compatible models.

Bad:

			
Chunks:
Model A
Query:
Model B

unless the models explicitly share a compatible embedding space.

Normally:

			
Chunk Model
=
Query Model

103. Embedding Profile Controls Both

Therefore the future retrieval service should use the active:

EmbeddingProfile

for:

			
Stored Chunk Vectors
+
Query Vector

This avoids accidental model mismatch.

104. Similarity Smoke Test

Although full semantic retrieval belongs to Part 30, Part 29 should include a low-level smoke test.

Store embeddings for:

			
Chunk A:
Payment will be made in three milestone-based instalments.
Chunk B:
The platform runs on Kubernetes.
Chunk C:
The customer requires disaster recovery.

		

Embed:

What are the payment terms?

Run a direct pgvector similarity query.

Expected:

Chunk A

ranks above unrelated Chunks.

This validates the vector infrastructure.

105. This Is Not Yet the Retrieval API

The smoke test proves:

			
Embedding
+
Storage
+
Similarity

		

works.

But we are not yet exposing a production:

semantic_search

tool to ChatGPT.

Part 30 will build the retrieval service properly.

106. Why Wait?

A production semantic retrieval service needs more than:

ORDER BY vector distance

It needs:

			
Tenant Filters
Document Filters
CRM Entity Filters
Similarity Thresholds
Top-K Limits
Current EmbeddingSet Rules
Deleted Document Rules
Deduplication
Provenance Assembly
Authorization
Result Contracts

		

That deserves its own module.

107. Vector Search Must Never Bypass Current State

Suppose old embeddings exist for a deleted or superseded Document.

A raw vector query might still find them.

Therefore future retrieval must filter by:

			
Current EmbeddingSet
Current ChunkSet
Readable Document
Tenant
Authorization

		

Vector similarity alone is never enough.

108. Current-State Invariant

A retrievable Chunk should conceptually satisfy:

			
Document AVAILABLE
AND
ChunkSet CURRENT
AND
EmbeddingSet CURRENT
AND
Tenant MATCHES
AND
Authorization PASSES

		

Part 30 will formalize this.

109. Soft-Deleted Documents

When a Document is deleted:

			
Document.status =
DELETED

we do not necessarily need to synchronously delete every vector.

But semantic retrieval must immediately stop returning them.

Later background cleanup can remove obsolete vectors.

110. Vector Retention

Old EmbeddingSets may be retained temporarily for:

			
Rollback
Comparison
Evaluation
Audit

But they must not participate in normal retrieval once superseded.

111. Storage Growth

Vectors can consume significant storage.

Approximate storage depends on:

			
Dimensions
Vector Representation
Number of Chunks
Indexes
Metadata

		

Therefore monitor:

			
embeddings per tenant
vectors per document
database vector storage
index size

112. Avoid Embedding Everything Forever

Future retention policies may remove:

obsolete EmbeddingSets

after a safe retention period.

The authoritative Chunk and Document can remain.

Vectors can always be regenerated.

113. Why Vectors Are Disposable

This is another important architectural property:

			
Document
= authoritative
Extraction
= reproducible derivative
ChunkSet
= reproducible derivative
EmbeddingSet
= reproducible index

		

The further down the pipeline we go, the more disposable the data becomes.

That helps disaster recovery planning.

114. Backup Strategy

Critical backups prioritize:

			
CRM Data
Document Metadata
Original Documents

Extracted text and Chunks are valuable.

Embeddings are useful but regenerable.

This can influence backup and restore policies later.

115. Observability

Useful metrics:

			
embedding_jobs_total
embedding_jobs_completed_total
embedding_jobs_failed_total
embedding_batches_total
embeddings_generated_total
embedding_input_tokens_total
embedding_duration_ms
embedding_provider_errors_total
embedding_rate_limit_events_total

		

116. Cost Metrics

Also track:

			
embedding_estimated_cost_total
embedding_cost_per_document
embedding_cost_per_tenant

where pricing data is available.

This will become useful for SaaS economics.

117. Vector Metrics

Useful database metrics include:

			
chunk_embeddings_total
embedding_sets_total
current_embedding_sets_total
vector_index_size
vector_query_duration_ms

		

The last becomes more important in Part 30.

118. Logging

Structured logs should include:

			
embedding_job_id
embedding_set_id
organization_id
document_id
chunk_set_id
provider
model
dimensions
batch_size
batch_number
input_tokens
duration_ms
status
error_code

		

Do not routinely log:

			
full customer Chunk text
full vectors
API credentials

119. Why Not Log Vectors?

Vectors are:

			
Large
Operationally Noisy
Potentially Sensitive
Rarely Useful in Logs

Log vector metadata instead.

120. Database Indexes

Traditional indexes:

			
embedding_jobs
(
    organization_id,
    status,
    created_at
)
embedding_sets
(
    organization_id,
    document_id,
    is_current
)
chunk_embeddings
(
    organization_id,
    embedding_set_id
)
chunk_embeddings
(
    chunk_id,
    embedding_set_id
)

		

Plus the vector index when introduced.

121. Uniqueness

Within one EmbeddingSet:

chunk_id

should normally be unique.

Constraint:

			
UNIQUE
(
    embedding_set_id,
    chunk_id
)

		

One Chunk gets one vector per EmbeddingSet.

122. Completeness

If:

			
ChunkSet.chunk_count =
42

then a completed EmbeddingSet should normally contain:

42 ChunkEmbeddings

unless some Chunk types are explicitly excluded by policy.

For the MVP:

			
one Chunk
=
one Embedding

is easiest to reason about.

123. Embedding Validation

Before setting an EmbeddingSet current, verify:

			
embedding_count == chunk_count
all vectors have expected dimensions
all Chunks belong to ChunkSet
all Embeddings belong to tenant
all embedding_input_hash values are present
all required vectors exist

		

124. Duplicate Chunk Test

If two Chunks have identical retrieval text, they may produce identical vectors.

That is valid.

Do not deduplicate the Chunks themselves simply because their vectors match.

They may have different:

			
Documents
Pages
Sections
Provenance

125. Embedding Reuse Versus Retrieval Identity

We may reuse vector computation for identical text.

But each:

ChunkEmbedding

should still point to its own Chunk.

Computation deduplication must not erase provenance.

126. Unit Test: Provider Adapter

Mock:

EmbeddingProvider

Input:

3 texts

Expected:

3 vectors

Verify:

			
model metadata
dimensions
ordering

127. Batch Test

Create:

130 Chunks

with:

batch_size = 64

Expected provider calls:

Verify every Chunk receives exactly one embedding.

128. Dimension Validation Test

Expected:

Provider returns:

Expected:

EMBEDDING_DIMENSION_MISMATCH

No invalid current EmbeddingSet.

129. Cardinality Test

Send:

10 inputs

Provider returns:

9 vectors

Expected:

EMBEDDING_RESPONSE_INVALID

130. Ordering Test

Input:

			
Chunk 1
Chunk 2
Chunk 3

Verify returned vectors map to the correct Chunks.

Never assume unordered provider responses unless explicitly documented and handled.

131. Retry Test

Provider returns temporary:

Expected:

retry with backoff

according to configured policy.

132. Permanent Failure Test

Invalid model.

Expected:

			
FAILED
error_code =
EMBEDDING_MODEL_INVALID

No infinite retry.

133. Partial Batch Recovery Test

100 Chunks.

First 80 persisted.

Worker crashes.

On retry:

remaining 20

are processed.

Do not regenerate all 100 unnecessarily.

134. Idempotency Test

Process the same ChunkSet with identical profile twice.

Expected:

no uncontrolled duplicate current EmbeddingSets

Equivalent work is safely detected.

135. Re-Embedding Test

Existing:

Model A

Then configure:

Model B

Expected:

New EmbeddingSet

without:

			
Re-extraction
Rechunking

136. Failed Re-Embedding Test

New model fails after 60%.

Expected:

Old EmbeddingSet remains current

No semantic-search outage.

137. Content Hash Reuse Test

Two Chunks have identical:

retrieval_text

and identical profile.

Verify the architecture can reuse vector computation if caching is enabled while still creating separate ChunkEmbedding provenance records.

138. Changed Heading Context Test

Source text unchanged.

Retrieval text changes because:

heading_context

changes.

Expected:

embedding_input_hash changes

Therefore a new embedding is generated.

139. Tenant Isolation Test

Tenant B attempts to retrieve EmbeddingSet metadata for Tenant A.

Expected:

not found / forbidden

140. Vector Tenant Filter Test

Create semantically identical Chunks in two tenants.

Run a vector query scoped to Tenant A.

Expected:

Tenant A results only

This test is essential.

141. Deleted Document Test

Create vectors.

Delete Document.

Low-level vectors may remain.

Normal semantic retrieval eligibility should become:

false

immediately.

142. Current EmbeddingSet Test

Create:

			
EmbeddingSet A
current

Then successfully build:

EmbeddingSet B

Expected:

			
A = not current
B = current

atomically.

143. Incomplete EmbeddingSet Test

ChunkSet:

42 chunks

EmbeddingSet:

41 vectors

Expected:

cannot become current

144. pgvector Storage Test

Insert a known vector.

Read it back.

Verify:

			
dimensions
values
chunk relationship
tenant relationship

are preserved.

145. Similarity Smoke Test

Create:

			
Chunk A:
Payment will be made in three milestone-based instalments.
Chunk B:
The application uses PostgreSQL.
Chunk C:
The customer requires high availability.

		

Query:

What are the payment terms?

Expected:

Chunk A

ranks highest or materially above unrelated Chunks.

146. Semantic Paraphrase Test

Chunk:

The implementation period is six months.

Query:

How long will the project take?

Expected:

relevant similarity

This demonstrates the semantic benefit over literal keyword matching.

147. No Cross-Tenant Semantic Test

Tenant A:

The project costs €90,000.

Tenant B:

The project costs €85,000.

Tenant A query:

What is the project price?

Tenant B’s semantically similar Chunk must never be returned.

148. Usage Accounting Test

Embed:

42 Chunks

Verify:

AIUsageEvent

or equivalent records:

			
organization
provider
model
input tokens
operation = embedding

		

correctly.

149. Secrets Test

Verify logs and database records do not contain:

			
API Key
Authorization Header
Provider Secret

150. Provider Failure Does Not Damage Chunks

Disable provider access.

Expected:

			
Documents remain available
Extractions remain available
Chunks remain available
Embedding Job fails

No upstream data is lost.

151. No ChatGPT Dependency Test

Disable ChatGPT integration.

Embedding processing should still work.

Why?

Because:

Knowledge Indexing

belongs to Quorentra’s backend infrastructure.

152. No Retrieval Dependency Test

Part 29 should work without implementing:

semantic_search

yet.

We can validate vector storage with low-level tests.

Production retrieval comes next.

153. Performance Test

Generate:

1,000 Chunks

Measure:

			
Batch Generation Time
Database Insert Time
Total Tokens
Provider Latency
Storage Growth

		

This establishes a baseline.

154. Large Document Test

Process a large document producing:

500+ Chunks

Verify:

			
Batching
Retries
Progress
Memory Usage
Completeness

		

remain controlled.

155. Multi-Tenant Load Test

Create EmbeddingJobs for multiple organizations.

Verify:

			
fair processing
tenant isolation
correct usage accounting

and no cross-tenant vector relationships.

156. Database Migration Test

Start with the Part 28 schema.

Run:

alembic upgrade head

Verify:

			
vector extension
embedding tables
constraints
indexes

exist correctly.

Then test rollback if your development workflow supports migration downgrade testing.

157. Version Update

Part 29 introduces:

			
EmbeddingProvider
Embedding Profiles
Embedding Jobs
Embedding Sets
Chunk Embeddings
Batch Generation
Vector Validation
Embedding Versioning
Content Hash Reuse
Re-Embedding
Token Accounting
AI Usage Accounting
PostgreSQL pgvector
Tenant-Scoped Vector Storage
Vector Index Architecture
Semantic Similarity Infrastructure

		

Update:

app/core/constants.py

from:

APP_VERSION = "0.15.0"

to:

APP_VERSION = "0.16.0"

158. Quorentra 0.16.0

Our modular MVP now looks like:

			
Platform
├── FastAPI                           ✓
├── PostgreSQL                        ✓
├── SQLAlchemy                        ✓
├── Alembic                           ✓
├── pgvector                          ✓
├── Processing Workers                ✓
└── Provider Abstractions             ✓
Identity
├── Organizations                     ✓
├── Users                             ✓
├── Memberships                       ✓
├── Authentication                    ✓
└── JWT                               ✓
Security
├── TenantContext                     ✓
├── Tenant Isolation                  ✓
├── RBAC                              ✓
├── Document Authorization            ✓
├── Chunk Authorization               ✓
└── Vector Tenant Isolation           ✓
CRM
├── Companies                         ✓
├── Contacts                          ✓
├── Opportunities                     ✓
├── Activities                        ✓
├── Tasks                             ✓
├── Meetings                          ✓
└── Documents                         ✓
Document Extraction
├── PDF                               ✓
├── DOCX                              ✓
├── PPTX                              ✓
├── TXT                               ✓
├── CSV                               ✓
├── Normalization                     ✓
└── Provenance                        ✓
Document Chunking
├── Chunking Jobs                     ✓
├── Chunk Sets                        ✓
├── Structure-Aware Chunking          ✓
├── Token Budgets                     ✓
├── Overlap                           ✓
├── Table Preservation                ✓
├── Source Mapping                    ✓
└── Chunk Versioning                  ✓
Embedding Layer
├── EmbeddingProvider                 ✓
├── Embedding Profile                 ✓
├── Embedding Jobs                    ✓
├── Batch Generation                  ✓
├── Dimension Validation              ✓
├── Retry Handling                    ✓
├── Rate-Limit Handling               ✓
├── Embedding Sets                    ✓
├── Chunk Embeddings                  ✓
├── Re-Embedding                      ✓
├── Embedding Input Hash              ✓
├── Usage Accounting                  ✓
└── Cost Observability                ✓
Vector Storage
├── PostgreSQL                        ✓
├── pgvector                          ✓
├── Vector Columns                    ✓
├── Tenant Metadata                   ✓
├── Document Metadata                 ✓
├── Chunk Relationships               ✓
├── Similarity Operators              ✓
└── ANN Index Architecture            ✓
Knowledge Pipeline
├── Document                          ✓
├── Extraction                        ✓
├── Normalization                     ✓
├── Chunking                          ✓
├── Embedding                         ✓
├── Vector Storage                    ✓
├── Semantic Retrieval                -
├── Metadata Filtering                -
├── Hybrid Search                     -
├── Reranking                         -
└── RAG                               -
CRM Context
├── Structured CRM Evidence           ✓
├── Document Metadata                 ✓
├── Extraction Status                 ✓
├── Chunking Status                   ✓
├── Embedding Status                  ✓
└── Semantic Index Status             ✓
ChatGPT
├── CRM Search                        ✓
├── CRM Context                       ✓
├── CRM Mutations                     ✓
├── Document Management               ✓
├── Knowledge Processing Status       ✓
├── Semantic Search                   -
└── Grounded Document Q&A             -

		

159. Acceptance Criteria

Part 29 is complete when:

			
✓ Part 28 regression suite remains green
✓ pgvector is installed
✓ vector extension is enabled
✓ vector schema migrations apply successfully
✓ EmbeddingProvider abstraction exists
✓ provider-specific APIs are isolated behind adapters
✓ application services do not depend directly on provider SDK contracts
✓ approved Embedding Profile exists
✓ provider is explicit
✓ model is explicit
✓ dimensions are explicit
✓ distance metric is explicit
✓ EmbeddingJob exists
✓ EmbeddingSet exists
✓ ChunkEmbedding exists
✓ optional AIUsageEvent exists
✓ PENDING exists
✓ PROCESSING exists
✓ COMPLETED exists
✓ FAILED exists
✓ Embedding requires a valid ChunkSet
✓ Embedding does not require re-extraction
✓ Embedding does not require rechunking
✓ retrieval_text is used as embedding input
✓ source_text remains authoritative evidence
✓ embedding_input_hash identifies actual model input
✓ batching exists
✓ batch item limit exists
✓ batch token limit can be enforced
✓ batch ordering is preserved
✓ response cardinality is validated
✓ vector dimensionality is validated
✓ malformed vectors are rejected
✓ vectors are not silently truncated
✓ vectors are not silently padded
✓ vector column uses pgvector
✓ vectors are not stored as JSON strings
✓ every vector references a Chunk
✓ every vector references an EmbeddingSet
✓ EmbeddingSet records provider
✓ EmbeddingSet records model
✓ EmbeddingSet records model version where available
✓ EmbeddingSet records dimensions
✓ EmbeddingSet records distance metric
✓ ChunkEmbedding records tenant
✓ ChunkEmbedding records Document
✓ ChunkEmbedding records Chunk
✓ ChunkEmbedding records EmbeddingSet
✓ ChunkEmbedding records embedding input hash
✓ one Chunk has one vector per EmbeddingSet
✓ database uniqueness protects this invariant
✓ embedding generation is idempotent
✓ duplicate job delivery is safe
✓ equivalent embedding work can be detected
✓ re-embedding exists
✓ re-embedding can use an existing ChunkSet
✓ new model creates a new EmbeddingSet
✓ old EmbeddingSet remains traceable
✓ incomplete EmbeddingSet never becomes current
✓ failed re-embedding leaves previous EmbeddingSet current
✓ current EmbeddingSet switch is atomic
✓ partial batch progress can be retained safely
✓ retry can continue missing embeddings
✓ successful batches need not be regenerated unnecessarily
✓ retryable provider failures are classified
✓ permanent failures are classified
✓ retry limit exists
✓ exponential backoff exists
✓ jitter exists where appropriate
✓ rate limits are handled
✓ concurrency is bounded
✓ provider timeouts exist
✓ embedding token usage is recorded
✓ usage is tenant-scoped
✓ provider is recorded
✓ model is recorded
✓ operation type is recorded
✓ API keys are stored securely
✓ secrets are never persisted in Embedding records
✓ secrets are not written to logs
✓ external provider data processing is treated as a privacy boundary
✓ approved providers are server-controlled
✓ arbitrary user-supplied providers are not allowed
✓ every ChunkEmbedding contains organization_id
✓ vector data is tenant-scoped
✓ vector queries can filter at database level
✓ architecture never requires global search followed by Python tenant filtering
✓ Document filters are supported by the schema
✓ Chunk relationships remain intact
✓ CRM Document relationships remain available for future retrieval filters
✓ deleted Documents are not semantically retrievable
✓ obsolete EmbeddingSets are not used by normal retrieval
✓ current-state metadata exists
✓ vector indexes are migration-managed if enabled
✓ distance operator matches configured metric
✓ HNSW / ANN strategy is explicit rather than accidental
✓ embedding metrics exist
✓ provider error metrics exist
✓ token metrics exist
✓ cost metrics can be recorded
✓ vector storage growth can be observed
✓ logs include job metadata
✓ logs do not routinely include customer Chunk contents
✓ logs do not include full vectors
✓ logs do not include secrets
✓ low-level semantic similarity smoke test works
✓ semantic paraphrases rank meaningfully
✓ unrelated text ranks lower
✓ cross-tenant vectors are excluded
✓ provider failure does not damage Documents
✓ provider failure does not damage Extractions
✓ provider failure does not damage ChunkSets
✓ ChatGPT is not required for embedding generation
✓ production semantic retrieval is not yet required
✓ RAG is not yet required
✓ CRM Context can report embedding status
✓ CRM Context can report semantic-index readiness
✓ CRM Context does not contain vectors
✓ Quorentra reports version 0.16.0

		

Most importantly:

Quorentra can now transform provenance-rich CRM knowledge chunks into versioned semantic vectors stored securely in PostgreSQL, while keeping Documents and Chunks—not embeddings—as the authoritative source of truth.

160. What We Have Achieved

Our knowledge pipeline has now evolved through four major transformations.

First:

			
Customer File
      ↓
Document

Then:

			
Document
      ↓
Extracted Text

Then:

			
Extracted Text
      ↓
Knowledge Chunks

And now:

			
Knowledge Chunks
      ↓
Semantic Vectors

The complete flow is:

			
Customer Document
      ↓
Secure Storage
      ↓
Text Extraction
      ↓
Normalization
      ↓
Structure-Aware Chunking
      ↓
Retrieval Text
      ↓
EmbeddingProvider
      ↓
Embedding Vector
      ↓
PostgreSQL + pgvector

		

Quorentra now possesses a semantic representation of its document knowledge.

161. But Semantic Storage Is Not Semantic Retrieval

Having vectors does not automatically mean we have a safe production search system.

A naive implementation might do:

			
Question
      ↓
Embed
      ↓
Nearest 5 Vectors
      ↓
Return Them

		

That is not enough.

We need a retrieval architecture.

162. The Retrieval Problem

Suppose the user asks:

What does Adventure Works say about disaster recovery?

Quorentra may contain:

			
Tenant:
Adventure Works workspace
Opportunity:
Azure Migration
Documents:
Requirements.pdf
Proposal.pdf
Architecture.pptx
Chunks:
97
Vectors:
97

		

We need to determine:

			
Which tenant?
Which CRM entity?
Which Documents?
Which EmbeddingSet?
Which Chunks?
How many?
What similarity threshold?
Which metadata filters?
Which permissions?
Which provenance?

		

That is a retrieval problem.

163. Similarity Alone Is Not Enough

Imagine the vector search returns:

			
1. Requirements.pdf
   Disaster Recovery Requirements
   Score: 0.91
2. Proposal.pdf
   Backup Architecture
   Score: 0.87
3. Another Customer's DR Strategy
   Score: 0.95

		

Result 3 may be the most semantically similar.

It must still never be returned.

Security and business context outrank similarity.

164. Retrieval Must Be Policy-Aware

The production retrieval pipeline should look more like:

			
User Question
      ↓
TenantContext
      ↓
Authorization
      ↓
Search Scope
      ↓
Query Embedding
      ↓
Tenant-Filtered Vector Search
      ↓
Document / CRM Filters
      ↓
Current-State Filters
      ↓
Similarity Threshold
      ↓
Top-K
      ↓
Provenance Assembly
      ↓
Retrieval Results

		

This is what Part 30 will build.

165. Search Scope

A user may ask:

What are the payment terms in the proposal?

The context may already identify:

			
Opportunity:
Azure Migration

Therefore search should not necessarily run across every document in the tenant.

It may scope to:

			
Azure Migration
      ↓
Linked Documents
      ↓
Current Chunks

		

This improves both:

			
Precision
Security
Performance

166. Metadata Filtering

Future retrieval may filter by:

			
organization_id
document_id
document_type
CRM entity type
CRM entity ID
section_name
source format
created_at

		

Semantic similarity should operate inside an authorized search space.

167. Top-K

We also need:

top_k

For example:

But blindly returning the top five results can still be poor.

If only one result is genuinely relevant, we should not fill the remaining four slots with weak evidence.

168. Similarity Thresholds

Therefore retrieval needs:

minimum_similarity

or equivalent distance threshold.

Conceptually:

			
Top 10 candidates
      ↓
Remove weak matches
      ↓
Return strongest evidence

		

Thresholds may require empirical tuning.

169. Neighbor Expansion

Suppose Chunk 22 is highly relevant.

The answer may depend on Chunk 23 as well.

Because Part 28 preserved:

			
ChunkSet
+
sequence_number

we can optionally expand:

			
Chunk 21
Chunk 22
Chunk 23

within controlled limits.

This can improve contextual completeness.

170. Deduplication

Overlap from Part 28 means neighboring Chunks may contain repeated source content.

Retrieval must avoid returning:

			
same paragraph
same paragraph
same paragraph

as three separate pieces of evidence.

We will need result deduplication.

171. Provenance Assembly

A retrieval result should not simply contain:

			
text
score

It should contain:

			
Chunk ID
Document ID
Document Name
Section
Page Range
Slide Range
Source Text
Retrieval Score
CRM Relationship

		

This gives ChatGPT grounded evidence.

172. Future Retrieval Result

Conceptually:

			
{
  "document": "Proposal-v3.pdf",
  "section": "Payment Terms",
  "pages": "16-17",
  "source_text": "Payment will be made in three stages...",
  "similarity": 0.91
}

		

This is far more useful than an anonymous vector match.

173. From Vector Search to ChatGPT

Eventually:

			
User:
What are the payment terms?
      ↓
Quorentra Retrieval
      ↓
Proposal-v3.pdf
Pages 16–17
Payment Terms
      ↓
ChatGPT
      ↓
The proposal specifies three payment milestones:
30% at commencement, 40% after migration completion,
and 30% after final acceptance.

		

And the answer can point back to the source.

That is grounded CRM intelligence.

174. Why Part 30 Is Another Separate Module

We could put vector search directly into:

EmbeddingService

But that would mix:

Index Creation

with:

Knowledge Retrieval

These are different responsibilities.

Therefore Part 30 introduces a dedicated:

KnowledgeRetrievalService

175. Next Article

In Part 30, we will build:

Building the Semantic Retrieval Engine — Tenant-Scoped Vector Search, Metadata Filtering, Search Scope, Similarity Thresholds, Top-K Ranking, Neighbor Expansion, Deduplication, and Provenance-Rich Results

We will introduce:

			
KnowledgeRetrievalService
SemanticSearchRequest
SemanticSearchResult
SearchScope
Tenant Scope
CRM Entity Scope
Document Scope
Query Embedding
Current EmbeddingSet Resolution
Vector Similarity Search
Cosine Similarity
Top-K
Candidate Limits
Similarity Thresholds
Document Filters
CRM Relationship Filters
Section Filters
Current-State Filtering
Deleted Document Exclusion
Authorization Filtering
Neighbor Expansion
Chunk Sequence Navigation
Overlap Deduplication
Content Deduplication
Result Ranking
Retrieval Score
Provenance Assembly
Document Name
Page Ranges
Slide Ranges
Section Context
Source Text
Retrieval Text
Retrieval Limits
Search Metrics
Retrieval Logging
Semantic Search API
ChatGPT Semantic Search Tool
Retrieval Tests

		

The pipeline will become:

			
User Question
      ↓
Search Scope
      ↓
EmbeddingProvider
      ↓
Query Vector
      ↓
PostgreSQL + pgvector
      ↓
Tenant-Scoped Similarity Search
      ↓
Authorized Candidate Chunks
      ↓
Metadata Filters
      ↓
Threshold
      ↓
Ranking
      ↓
Deduplication
      ↓
Provenance-Rich Results

		

At that point, Quorentra will be able to answer a fundamentally new question:

			
Which parts of our CRM knowledge are semantically relevant to what the user is asking?

That capability will provide the retrieval foundation required for the stage after it:

			
Semantic Retrieval
      ↓
Evidence Assembly
      ↓
ChatGPT
      ↓
Grounded CRM Answer

		

Part 30 therefore moves Quorentra from semantic indexing to semantic knowledge retrieval—one of the central capabilities of a truly ChatGPT-native CRM.

Quorentra CRM Building the Embedding and Vector Storage Layer: Building from Zero — Part 29

1. Introduction

2. What Is an Embedding?

3. Semantic Similarity

4. Why Embeddings Matter for CRM

5. Embeddings Are an Index

6. Why This Matters

7. Starting Checkpoint

8. Enable pgvector

9. Why PostgreSQL + pgvector?

10. Avoid Premature Vector Infrastructure

11. Embedding Provider Abstraction

12. Why Provider Abstraction?

13. ChatGPT-Native Does Not Mean Embedding-Locked

14. Embedding Model Configuration

15. Example Configuration

16. Embedding Model Identity

17. Why Dimensions Matter

18. MVP Dimensionality Strategy

19. Embedding Profile

20. Distance Metric

21. Consistency Is More Important Than Fashion

22. Embedding Processing Stage

23. EmbeddingJob

24. Embedding Status

25. Pipeline State

26. EmbeddingSet

27. Why an EmbeddingSet?

28. Pipeline Lineage

29. ChunkEmbedding

30. Vector Column

31. SQLAlchemy Model

32. Do Not Store Vectors as JSON

33. What Text Should Be Embedded?

34. But Evidence Still Uses Source Text

35. Embedding Batch

36. EmbeddingBatchResult

37. Validate Batch Cardinality

38. Validate Dimensions

39. Batch Size

40. Batch by Token Budget Too

41. Embedding Workflow

42. Do Not Hold One Huge Database Transaction Across API Calls

43. Staged Persistence

44. Partial Progress

45. Embedding Item State

46. Idempotency

47. Embedding Identity

48. Embedding Reuse

49. Be Careful with Retrieval Text

50. Store Both Hashes

51. Re-Embedding

52. Re-Embedding Does Not Require Re-Extraction

53. Rechunking Versus Re-Embedding

54. Current EmbeddingSet

55. Atomic Current Switch

56. Failed Re-Embedding

57. Retry Policy

58. Permanent Errors

59. Structured Errors

60. Exponential Backoff

61. Rate Limits

62. Concurrency Control

63. Token Accounting

64. Why Token Accounting Matters

65. Cost Accounting

66. Future AI Usage Ledger

67. Why Usage Must Be Tenant-Aware

68. Secrets

69. Never Log API Keys

70. What About the Text Sent to the Provider?

71. Provider Policy Must Be Explicit

72. Future Local Embeddings

73. Tenant Isolation in Vector Storage

74. Never Search Globally Then Filter in Python

75. Correct Conceptual Query

76. Document Filtering

77. CRM Entity Filtering

78. Vector Indexes

79. HNSW