From Upload to Verifiable Answers: A Go Document Q&A RAG Implementation

Posted on 2026-07-17 Edited on 2026-09-05 Views: Word count in article: 31k Reading time ≈ 28 mins.

How a Go document Q&A system turns an uploaded file into verifiable answers: async parsing, OCR, chunking, hybrid retrieval with RRF, reranking, and citation checking.

I have been building a document Q&A project recently. Users upload PDFs, DOCX, plain text, or images; once background processing finishes, they can ask questions against their library. What the system returns is not just an answer — it also includes the file name, the page number, the source snippet, and a content hash, so users can go back to the original and verify.

To keep the prose short, I will call this project “Zhiye” (知页, “knowledge pages”) below.

This article answers exactly one question: after a file is uploaded, how does it go through parsing and indexing to produce an answer with verifiable citations?

The core idea first: RAG is not one model call, it is two pipelines

If you have very little material, you can send the full text and the question to the model in a single prompt. But with many documents this approach hits several limits quickly: the context no longer fits, resending everything repeatedly is expensive, scanned documents have no text layer, answers are hard to trace back to their source, and per-user permission isolation becomes a problem.

So instead of reading every file at question time, Zhiye splits the system into two pipelines:

Document processing pipeline (offline)
File → parse / OCR → chunk by page → write to MySQL → embed → write to Qdrant

Question answering pipeline (online)
Question + filters → keyword / vector recall → RRF fusion
                 → (optional) reranker semantic fine-ranking
                 → LLM generates the answer → backend verifies and assembles citations

The first pipeline turns raw files into retrievable chunks; a given document typically goes through it once. The second pipeline fetches only the small set of chunks most relevant to the current question and hands them to the model.

This is also the most important dividing line for understanding everything below:

The document processing stage solves “how does material become retrievable”;
The online QA stage solves “how do we find the evidence, and constrain the model to answer only from it.”

A concrete example ties it together: a user uploads a 30-page contract in which page 12 says “the remaining payment shall be made within 10 days after acceptance is completed.” The backend turns that sentence into a chunk with a page number, and builds both keyword and vector indexes for it. The user then asks “what is the latest date payment is due?” The system first retrieves that chunk, then has the model answer from it. The model only returns the IDs of the chunks it used; the file name, the page 12 source text, and the hashes that the user finally sees are reassembled by the backend from the retrieval results.

Zhiye document Q&A service architecture

The backend is Go + Gin. MySQL stores libraries, documents, chunks, background tasks, and QA records; original files are managed behind an object-storage abstraction whose current implementation lands on a local directory; Qdrant holds the vector index; Poppler (the open-source PDF parsing and rendering toolkit) extracts PDF text and renders pages; scanned pages go to OCR; and finally the LLM is called.

To avoid mixing up terms later, here is a minimal glossary first:

Term	Meaning in this article
Chunk	A small, independently retrievable piece of text cut out of one page’s body
Embedding	Encoding a question or a chunk into a dense vector, used to compare semantic similarity
Point / Payload	One vector record in Qdrant, plus its attached metadata: user, document, page number, and so on
Recall	First fetching a batch of possibly-relevant candidates out of a large chunk set
RRF	Fusing results based on their ranks across retrieval lists, without comparing raw scores from different systems
Reranker	Reading the question and the candidate snippets together, then doing a finer relevance ranking

Now let us follow one document through processing.

Pipeline 1: turning files into retrievable chunks

1. The upload endpoint only handles reliable intake

The upload endpoint does not parse documents synchronously. It does exactly six things:

Verify library ownership
  ↓
Sanitize the file name, check extension and size
  ↓
Store the file under a UUID object name and compute its SHA-256
  ↓
Deduplicate by user, library, and file hash
  ↓
Create the document record and a document_parse task in one transaction
  ↓
Return the document status immediately

Why not just parse inside the HTTP request?

Plain text might finish in a few hundred milliseconds, but a scanned PDF may need per-page rendering, OCR, and embedding. Put that work inside the upload request and you must simultaneously handle long-held connections, request timeouts, failure retries, progress queries, and task recovery after a service restart.

The boundary here is therefore explicit: the HTTP request reliably receives the file; the background task finishes processing it.

There are two easy-to-miss details at the upload stage.

First, never trust the declared content length alone. The endpoint checks the size the client claims, but when actually saving, it still reads one extra byte with io.LimitReader and re-checks the size reported by object storage after the write, in case Content-Length is missing or forged.

Second, deduplication needs two lines of defense:

The application layer first queries by user_id + case_id + file_hash, where case_id is what this article calls the library ID;
The database then handles concurrent uploads with a unique constraint.

When two identical files are uploaded at the same time, both requests may pass the first query, but only one insert can succeed. The other request detects the unique-key conflict, returns the existing document, and cleans up the extra object it produced. What the database constraint guarantees here is concurrent correctness, not just query performance.

2. Background processing is not just “spawn a goroutine”

Document processing is split into two tasks:

document_parse
  read file → parse / OCR → chunk → transactionally replace MySQL chunks
        ↓
document_embed
  generate embeddings in batches → write to Qdrant → update vector status

The reason for two tasks: text availability and vector availability are not the same thing.

A document may have parsed successfully and already be findable by keyword search, while the embedding service or Qdrant is temporarily down. With a single, vague “processing succeeded/failed” status, a failure on the vector path would make the whole document unsearchable.

So the code tracks status separately:

1
2
3

processing_status  whether text parsing and chunking finished
vector_status      whether document-level vectorization finished
embedding_status   whether one chunk's vector finished

This buys several clean recovery paths:

Once the text is done, MySQL keyword search works immediately;
If vectorization fails, only retry document_embed;
When swapping the embedding model, only rebuild the vectors;
With no vector capability configured at all, the system still works in keyword-only mode.

The background runner polls for tasks every two seconds by default and processes at most three tasks concurrently. Each task records its status, processing stage, progress, start and completion time, latest heartbeat, retry count, and error message.

While running, a task updates its heartbeat every 15 seconds by default. A processing task with no heartbeat for over a minute is considered interrupted and re-enqueued; it is marked failed only after automatic retries are exhausted. A panic inside a task is also converted into a failed state, so a record never sits in processing forever.

The runner also performs a layer of status reconciliation. If a document is pending parsing but has no parse task, or the text is done and vectors are pending but there is no embed task, the runner creates the missing task. This reconciliation does not replace strict idempotency design, but it repairs some of the “state written but task record lost” anomalies.

One more distinction matters here: “tasks can retry” is not the same as “retrying a task cannot corrupt data with duplicate writes.” Heartbeats, timeout reclamation, and reconciliation address the problem of lost tasks; two workers processing the same task, or a stale worker writing after its timeout, need stricter idempotency and concurrency control. In production you should at least consider:

Atomically claiming tasks, with a lease attached to the claim result;
Using a fencing token or a task version to reject writes from expired workers;
Deriving deterministic Qdrant point IDs from stable chunk identifiers;
Using replay-safe upserts against both MySQL and Qdrant;
Recording a generation/version for each index build so old and new data can be told apart;
Periodically repairing orphaned points, missing points, and stale indexes.

What I can confirm about the current implementation is that it has heartbeats, retries, and status reconciliation. The replay-safety items above remain boundaries to verify and harden — their safety must not be assumed merely because a retry mechanism exists.

Finally, cancellation signals and deadlines from the request must propagate all the way down. Every task uses a context.Context with a deadline, and OCR, embedding, Qdrant requests, and Poppler subprocesses are all bound to that context. Otherwise the outer layer may already have canceled or timed out while the underlying HTTP request or subprocess keeps consuming resources.

3. Different formats are normalized to “page number + text” first

The parsers currently support:

txt / md / csv / json
docx
pdf
jpg / jpeg / png / webp

The internal structures of these formats differ wildly, but the parsing layer outputs exactly one structure:

TrialCueParsedPage{
    PageNumber: 1,
    Text:       "extracted text",
}

From this layer onward, chunking, indexing, and retrieval no longer care whether the input was a PDF, Word document, or an image. However, PageNumber is not fully equivalent across formats:

Format	What “page number” currently means
PDF	The physical page index inside the PDF file; usable for going back to that page to verify
Image	One image counts as one page, page number 1
TXT / Markdown / CSV / JSON	The whole file is treated as one logical page, page number 1
DOCX	The whole body is treated as one logical page; it cannot be reliably mapped to Word’s rendered physical pages

DOCX deserves special attention: word/document.xml stores document structure, not a pagination result that stays stable everywhere. Fonts, paper size, margins, and the rendering engine all change Word’s visual pagination. If the business requires citing “which Word page,” the more reliable path is to render or convert the DOCX to PDF in a pinned environment, then parse it as PDF physical pages.

Plain text has its BOM and surrounding whitespace stripped and is treated as a single page. A DOCX is essentially a ZIP; the code reads the text nodes in word/document.xml and inserts newlines at paragraph ends. The current implementation only covers body text — table semantics, headers and footers, comments, and text boxes are not fully handled yet, and there are no strict limits on decompressed size or compression ratio.

The PDF path is longer:

pdfinfo gets the page count
  ↓
pdftotext -layout extracts the text layer
  ↓
page numbers are recovered from form feeds
  ↓
if the whole extracted text is almost empty, fall through to OCR
  ↓
when needed, pdftoppm renders pages to images, which are then recognized

What is Poppler? Poppler is an open-source PDF parsing and rendering toolkit. pdfinfo, pdftotext, and pdftoppm, all used here, are its command-line programs: pdfinfo reads document metadata such as the page count, pdftotext extracts an existing text layer, and pdftoppm renders PDF pages into images for subsequent OCR. Poppler itself does not do OCR.

The project does not run Poppler as a resident service; Go starts these commands on demand through exec.CommandContext. The parser limits PDF concurrency and per-command execution time so that a pathological file cannot hold a worker hostage.

PDF OCR is capped at 20 pages by default, with a bounded number of exponential-backoff retries for retryable failures only — throttling, timeouts, and server errors.

“20 pages max” is not just a performance knob; it also affects result completeness. If only the first 20 pages of an 85-page scanned PDF get recognized, the system should not still mark that document as “processing complete,” or users will wrongly believe all 85 pages are searchable. From the current code I can only confirm that the OCR page cap exists; I cannot confirm whether the product layer warns users about incompletely processed content. The safer design picks one of two options: fail outright past the cap and tell the user to split the file, or explicitly return a partially-complete status such as:

1
2
3

processing_status = partial
processed_pages   = 20
total_pages       = 85

Whichever is chosen, trailing pages must never be silently dropped.

There is still one clear gap: the decision to OCR is currently made per whole PDF rather than per page. If most pages have a text layer and only a few pages are scans, those scanned pages can slip through unnoticed. Per-page detection with mixed parsing would be more sensible.

4. Chunking: preserve page numbers first

The parse result is then cut into smaller chunks. Current parameters:

1 2	target_runes = 800 overlap_runes = 100

Chunks are counted in runes rather than bytes, which avoids cutting through the middle of a multi-byte UTF-8 character; breaks prefer newlines, periods, question marks, semicolons, or whitespace; adjacent chunks overlap by 100 runes to reduce the chance that a key fact lands exactly on a boundary.

But rune count is not token count. Different embedding and LLM models use different tokenizers, so the same 800 runes can correspond to different numbers of tokens. Runes are therefore good for a language-agnostic first cut, but they cannot guarantee staying under a model’s input limit. The safer approach: after chunking, check length with the actual model’s tokenizer, enforce the token budget separately when generating embeddings and when assembling the LLM context, and truncate when necessary.

The more important constraint: chunks never span pages.

Not spanning pages may cost some recall on cross-page semantics, but it keeps citation page numbers stable. This project requires users to be able to go back to the original and verify, so between “possibly higher recall” and “more reliable page numbers,” the current implementation picks the latter.

Every chunk stores:

page_start / page_end
chunk_index
content
content_hash
embedding_status / embedding_model / embedding_dimension

Once chunks are written to MySQL, keyword retrieval is already usable. If embedding is enabled, the background then generates vectors in batches and writes the user, library, document, chunk, page number, material type, evidence number, content hash, and model name into the Qdrant payload.

That completes the first pipeline: raw files have become chunks with page numbers, permission boundaries, and keyword or vector searchability.

Pipeline 2: from question to evidence, then to an answer

A question does not go straight into the LLM. It first passes through filtering, recall, fusion, ranking, and citation verification:

Permission checks and filter normalization
        ↓
MySQL keyword candidates (default 40)
        +
Qdrant candidates (default 40)
        ↓
RRF fusion, dedup (keep 20)
        ↓
Reranker semantic fine-ranking (optional)
        ↓
Final context (default 8)
        ↓
LLM generates a structured answer
        ↓
Backend verifies citation IDs and assembles the source text

The three numbers that most commonly confuse people here: recall 40, 20 kept after fusion, and 8 finally handed to the model. They are not the same Top K: the front end widens recall, and the context is then progressively compressed.

1. First decide which material this question is allowed to search

Besides question, the QA endpoint supports filters on documents, material type, evidence numbers, and page ranges:

{
  "question": "What payment deadline does this material mention?",
  "filter": {
    "document_ids": [12, 18],
    "document_types": ["contract", "payment receipt"],
    "evidence_numbers": ["evidence-3"],
    "page_from": 2,
    "page_to": 20
  }
}

The code first bounds the number of filter entries, deduplicates them, and validates the page range, then applies the same filter set to both MySQL and Qdrant. Page numbers use interval-intersection semantics:

1 2	chunk.page_end >= page_from chunk.page_start <= page_to

Compared to requiring chunks to fall entirely within the query range, this better matches the intuition of “search pages 2 through 20.”

Permission conditions must likewise enter every recall path. Both keyword and vector retrieval carry the current user and library; after Qdrant returns chunk IDs, the service goes back to MySQL and reloads the chunks by user, library, filters, and document status.

In other words, the vector store only finds candidates; final access permissions and data state are still validated by the business database. IDs returned by an external index must never bypass permission and status checks.

2. What keywords and vectors each handle

The current pipeline involves three base retrieval capabilities: MySQL full-text search, Qdrant dense-vector retrieval, and optional Qdrant dense-plus-sparse hybrid retrieval. These form two fusion layers:

Qdrant internal hybrid retrieval: fusing dense and sparse/BM25 inside Qdrant;
Service-layer multi-list fusion: fusing MySQL candidates with the candidate list returned by Qdrant, inside the Go service.

I will take each capability and both fusion layers in turn.

MySQL handles keyword recall

The MySQL path prefers a full-text index. The core expression in the code:

1	MATCH(content) AGAINST (? IN NATURAL LANGUAGE MODE)

It helps to break this into three parts:

MATCH(content): specifies that the search happens in the content column. That column needs a FULLTEXT index matching the query;
AGAINST (?): specifies the query content. The ? here is not a question mark to search for — it is the parameter placeholder the database driver uses when executing the prepared statement, bound at runtime to the user’s question;
IN NATURAL LANGUAGE MODE: uses natural language mode. MySQL tokenizes the query and the documents, computes relevance, and returns a floating-point score where higher usually means more relevant. Natural language mode is also MySQL’s default full-text mode, so this clause mostly makes the intent explicit in the code.

For example, when a user asks “what is the latest date payment is due,” after parameter binding it can be read approximately as:

1	MATCH(content) AGAINST ('what is the latest date payment is due' IN NATURAL LANGUAGE MODE)

The following query is simplified to illustrate the syntax; it does not represent the project’s real schema or permission conditions:

SELECT
    id,
    content,
    MATCH(content) AGAINST (? IN NATURAL LANGUAGE MODE) AS score
FROM document_chunks
WHERE MATCH(content) AGAINST (? IN NATURAL LANGUAGE MODE) > 0
ORDER BY score DESC
LIMIT 40;

There are two ? here, so the same question parameter must be bound twice: once to compute score in the SELECT, and once to filter out irrelevant rows in the WHERE. A representative Chinese full-text index looks like:

1 2	ALTER TABLE document_chunks ADD FULLTEXT INDEX idx_content (content) WITH PARSER ngram;

This differs from LIKE '%payment deadline%': LIKE mainly tests whether the raw string contains a given fragment, while full-text search computes relevance from tokenization and term statistics, which is far better for selecting candidates out of a large chunk set. This project uses the ngram tokenizer for Chinese; it cuts continuous text into overlapping short tokens so that Chinese — which has no word spaces — can enter the full-text index at all. No separate Elasticsearch deployment is used, mainly to keep the VPS’s resident footprint and operational complexity down.

MySQL version note: Full-text search and MATCH ... AGAINST capability date back to MySQL 3.23.23 and originally supported MyISAM only; the explicit IN NATURAL LANGUAGE MODE modifier arrived in MySQL 5.1.7; InnoDB full-text indexes came with MySQL 5.6; the built-in ngram full-text parser shipped in MySQL 5.7.6. So the explicit syntax alone requires MySQL 5.1.7 or newer, while the InnoDB + ngram + MATCH ... AGAINST combination actually used here requires at least MySQL 5.7.6.

How full-text search degrades on failure

When full-text search errors out or finds no hits, the code falls back to simple token search: English and digits are split into runs, Chinese generates 2–4 character fragments, LIKE fetches up to 200 candidates, and Go re-scores and re-ranks them.

This fallback is decent for names, IDs, and dates, but it relies on repeated LIKE queries and application-level scoring; as data grows, the scan range widens and query latency climbs, so it is not fit as a long-term primary strategy.

Qdrant handles semantic recall

The Qdrant path converts the question into an embedding, then runs dense-vector similarity retrieval. It mainly covers paraphrases and high-level questions that keywords handle poorly.

Two Qdrant terms first. A point can be understood as one record in the vector store: in this design, one document chunk maps to one point. Besides storing one or more vectors, a point carries a payload — business metadata such as chunk ID, user, library, document, and page number.

How Qdrant’s internal hybrid retrieval works

The code also offers optional Qdrant-internal hybrid retrieval. When enabled, every point stores two representations:

Representation	How it is generated	What it matches best
Dense vector	The embedding model encodes the whole passage into a fixed-length float array	Semantically similar but literally different phrasings, e.g. “when is the final payment due” vs. “the deadline for the remaining payment”
Sparse vector	BM25 encodes the terms in the text, stored in sparse form	Exact names, IDs, dates, and verbatim source terms

Dense vectors typically have a value in every dimension and are suited to overall meaning; sparse vectors are mostly zeros whose non-zero dimensions correspond to terms that actually occurred, which puts them closer to traditional keyword search.

At query time, one question runs as two sub-queries:

Question
├─ Embedding → dense vector search → take top N
└─ BM25      → sparse vector search → take top N
                               ↓
                          Qdrant internal RRF
                               ↓
                     Qdrant internal hybrid candidates

Qdrant’s Query API calls these sub-queries prefetch. “Prefetch” here does not mean preloading data into memory; it means running several candidate queries first, then handing the candidates to the main query for fusion. RRF merges purely by rank across the two lists, never comparing dense similarity against BM25 scores directly, because the two live on different scales.

Back to the contract example. For the question “what is the latest date payment is due,” dense retrieval may find the page-12 chunk “the remaining payment shall be made within 10 days after acceptance is completed,” while sparse retrieval more readily grabs literal words such as “payment.” If the same chunk ranks highly in both lists, it earns a higher fusion score and is more likely to make the final candidates.

There are two fusion layers in the full pipeline:

Qdrant internal RRF: fuses Qdrant’s dense results with its BM25 sparse results;
Service-layer RRF: fuses the MySQL full-text list with the entire Qdrant candidate list.

So with Qdrant-internal hybrid enabled, the “Qdrant candidates” the service receives have already been fused once; the service-layer RRF still has to merge them with the independent MySQL keyword recall.

Qdrant hybrid retrieval is off by default. The existing collection stored only a single dense vector, while this mode needs named dense and sparse vectors; even after altering the collection’s vector schema, existing points do not auto-generate the missing BM25 sparse vectors — they still have to be recomputed and backfilled. To avoid mixing old and new point structures and to keep a fast rollback path, this project took the safer migration: create a new collection, rebuild the whole index, and switch over after verification.

Version note: Qdrant 1.18.0 added support for dynamically adding or removing named vector configurations on an existing collection, but that only changes the schema — it does not generate vectors for old data. “The structure can be changed” and “the data does not need rebuilding” are two different things; on older versions, or when you want a migration that is easy to verify and roll back, creating a new collection is still the simpler plan.

3. RRF fuses ranks, not raw scores

MySQL full-text scores, Qdrant dense similarity, and Qdrant’s internal hybrid fusion scores are not on the same scale; adding them together means nothing.

So the service layer uses RRF (Reciprocal Rank Fusion). Informally, think of it as “letting multiple leaderboards vote”: the higher a result ranks, the more weight its vote carries; if it appears on several leaderboards, the contributions stack, and it typically ends up above results favored by only one list.

Say keyword search’s top three are A, B, C and vector search’s top three are C, A, D. A and C are endorsed by both lists while B and D appear in only one, so after fusion A and C generally win. RRF never has to decide whether a keyword score of 8 equals a vector score of 0.8 — it only compares each result’s rank within its own list.

The name decomposes nicely:

Rank: look only at where a chunk places in each list;
Reciprocal: turn the rank into an inverse score — the better the rank, the smaller the denominator, the higher the score;
Fusion: if a chunk appears in multiple lists, sum the scores it earned in each.

It never compares raw scores, only each chunk’s rank in each list:

1	score += 1 / (rrf_k + rank + 1)

The rank in the code is 0-based, hence the trailing +1. Written the everyday way — first place = 1, second place = 2 — the formula reads more intuitively:

1 2	per-list contribution = 1 / (rrf_k + rank) final RRF score = sum of a chunk's contributions across all recall lists

With rrf_k = 60 and two chunks:

Chunk	MySQL keyword rank	Qdrant rank	RRF score (approx.)
A	2nd	1st	`1/62 + 1/61 = 0.0325`
B	1st	no hit	`1/61 = 0.0164`

B ranks first in MySQL, but A ranks highly on both paths, so after fusion A’s score is higher. RRF is saying exactly this: a result endorsed by multiple retrieval methods is generally more reliable than one that only ranks well on a single path.

rrf_k is a smoothing constant. The larger it is, the flatter the rank-decay curve and the smaller the contribution gap between adjacent ranks; the smaller it is, the more the top ranks dominate. Current configuration: 60.

After fusion, results are deduplicated by chunk_id, optionally filtered below a threshold, then sorted with the top 20 kept.

Current configuration:

trial_cue_search:
  keyword_candidate_limit: 40
  vector_candidate_limit: 40
  fusion_candidate_limit: 20
  result_limit: 8
  rrf_k: 60
  min_fusion_score: 0

These numbers are engineering defaults, not proven-optimal parameters. Tuning them properly requires a fixed evaluation set, not repeated tweaks based on how production feels.

4. The reranker only does the final semantic pass

Both RRF and the reranker affect ordering, but they solve different problems:

RRF fuses multiple rankings. It scores chunks by their rank in the keyword and vector lists; it never re-reads chunk content;
The reranker judges semantic relevance. It reads “question + candidate snippets” together and re-evaluates how relevant each snippet is to the question.

Suppose the user asks “what is the latest date payment is due.” One snippet may mention “payment” several times and thus rank high in keyword search without ever stating a deadline; another mentions it once but explicitly says “the remaining payment shall be made within 10 days after acceptance is completed.” The reranker, reading the full question together with the snippets’ semantics, ranks the latter first.

In the current flow, RRF first fuses keyword and vector results into up to 20 candidates; the reranker then scores those candidates and picks the final 8 for the LLM. Its only job is narrowing and reordering candidates — it never generates the answer itself.

“Optional” here means the reranker is not a hard dependency of the QA pipeline:

When configured and enabled, it runs semantic fine-ranking;
When unconfigured, the RRF ordering is used directly;
When the call fails or the response is invalid, the pipeline degrades to the RRF ordering — the QA request does not fail over it.

The reranker is currently invoked as an HTTP service returning candidate indexes and relevance scores. The backend also checks:

Indexes must not be out of bounds or duplicated;
Scores must not be NaN or infinite;
The returned count must not exceed the requested top_k.

In one sentence: RRF fuses multi-list retrieval rankings; the reranker further judges which snippets are most relevant to the question.

5. The LLM only sees the final candidates

After all the filtering and ranking, only a handful of chunks enter the LLM context. Each chunk contains:

chunk_id
file name
start/end page
body text

The prompt instructs the model to answer only from this material; when the material is insufficient it must say so explicitly, returning structured JSON:

{
  "suggested_reply": "The remaining payment shall be made within 10 days after acceptance is completed.",
  "basis_chunk_ids": [12],
  "risks": "The acceptance completion date needs to be confirmed.",
  "confidence": "high",
  "risk_level": "medium"
}

Allowed values for both confidence and risk_level are low, medium, and high. confidence expresses how well the current material supports the answer; risk_level expresses the business risk of adopting the answer as-is, or whether a key premise is still missing. They are not opposites: the example answer is directly grounded in the source text, so confidence is high; but computing an actual deadline still requires the “acceptance completion date,” so risk stays medium.

When there are no retrieval results at all, the service does not call the LLM — it returns “insufficient material” directly. If the LLM is unconfigured, the call fails, or the output is invalid, it also returns only a degradation notice, never fabricating a plausible-looking answer.

6. The model only picks citation IDs; the backend assembles the details

basis_chunk_ids merely state which snippets the model claims to have used; they cannot be treated as trusted citations directly.

The backend builds a whitelist from this request’s final retrieval results, then:

Accepts only chunk_ids on the whitelist;
Removes duplicates;
Reassembles file name, document ID, page numbers, and source text from the chunks the backend itself holds;
Caps each individual citation at 500 runes;
Returns both the file’s SHA-256 and the chunk’s content SHA-256, so material can be identified and changes to the file or the chunk detected.

If no ID passes validation, the endpoint forces:

basis = []
basis_verified = false
confidence = low
risk_level at least medium

The system never passes off the first retrieval result as the model’s basis just to make the page look complete.

One distinction matters here, though:

basis_verified=true means the citation IDs genuinely came from this request’s whitelist;
It does not mean those citations adequately support every fact in the answer.

What the current implementation solves is “citations cannot be fabricated from nothing.” Whether the answer and its citations agree claim by claim is not yet solved.

Degradation vs. hard stop: what can fail, what must terminate

Vector retrieval and reranking are enhancement capabilities, not hard dependencies of the QA endpoint. What you degrade to depends on which layer failed:

If embedding and Qdrant are not both enabled, question embedding fails, the Qdrant query fails, or vector hits cannot be safely re-fetched from MySQL, fall back to MySQL keyword retrieval;
If the reranker call fails or returns an invalid result, keep the RRF-fused ordering.

But context.Canceled and context.DeadlineExceeded must not be dressed up as a “successful degradation.” When the caller has canceled or an upstream deadline has expired, the whole pipeline should terminate as fast as possible.

Both surface as errors, but their semantics differ:

Optional dependency failed: keep existing capability and continue in degraded mode;
Request lifecycle over: stop all work and propagate the cancellation or timeout upward.

Log the retrieval process, or you cannot tell how an answer was produced

Each QA request, besides the question, answer, risk, and citations, currently also records:

retrieval_mode        keyword or hybrid
embedding_model       the embedding model used for the query
keyword_hit_count     number of keyword candidates
vector_hit_count      number of Qdrant candidates
retrieval_latency_ms  retrieval time

Here retrieval_mode=hybrid means the service layer used both the MySQL and Qdrant candidate lists; it does not imply “Qdrant definitely ran dense + sparse/BM25.” If both layers need observability, they should be separate fields — say retrieval_mode and qdrant_query_mode — so that a single hybrid never has to mean two things at once.

The existing fields already answer: did the request degrade to keyword retrieval, did the vector path return candidates, was total retrieval time abnormal, and which embedding model the query used. But to diagnose a slow request or reproduce an answer you need finer fields:

degraded                 whether degradation occurred
degradation_reason       why it degraded
keyword_latency_ms       MySQL keyword recall time
embedding_latency_ms     question embedding time
vector_latency_ms        Qdrant query time
rerank_latency_ms        reranking time
llm_latency_ms           answer generation time
embedding_model_version  embedding model version
reranker_model_version   reranker model version
prompt_version           prompt version
index_generation         index generation
qdrant_query_mode        dense or dense_sparse

These belong to the next round of observability work and should not be presented as shipped capabilities. They would decompose “this answer was bad” into recall, fusion, ranking, model, or degradation problems, and support comparing chunking parameters, RRF thresholds, and model versions over time.

Looking back: seven system trade-offs that actually mattered

Trade-off	Current choice	Main goal
Upload vs. processing	Decouple HTTP intake from background tasks	Avoid long-lived requests; enable retry and recovery
Text vs. vector status	Tracked separately	Keep keyword search alive when the vector service is down
Multi-format parsing	Normalize to “page number + text” first	Decouple chunking and indexing from the original format
Permission boundary	Carried through every recall path, re-validated on MySQL re-fetch	Never let an external vector index decide data access
Retrieval strategy	Widen recall first, then fuse and re-rank	Minimize missed recall while controlling reranking cost and model context length
Citation generation	The model picks IDs, the backend assembles the source	Prevent the model from fabricating file names, pages, and content
Page numbers vs. recall	Chunks currently never span pages	Citations stay verifiable

All of these trade-offs point at one goal: when an external service fails or the model’s output is unreliable, the system keeps as much existing capability as possible — while recording which material an answer used, and progressively filling in observability for degradation reasons and processing stages.

What is still not solved

The code forms a complete pipeline, but “runs” is still a long way from “answers you can trust.”

1. No reproducible evaluation set

I need a batch of de-identified “question – correct document – correct page – correct chunk” samples, with at least these metrics:

Recall@3, Recall@5, Recall@8;
MRR or nDCG;
Citation accuracy and citation completeness;
Refusal rate on unanswerable questions;
Per-stage latency for keyword, vector, RRF, and reranker;
Degradation rate and no-valid-citation rate.

Without this data, 40 / 40 / 20 / 8, rrf_k=60, and min_fusion_score=0 are all just initial configuration.

2. A citation whitelist is not factual consistency

The next step is decomposing answers into independent claims, checking each claim against its citations, and distinguishing:

Facts directly recorded in the material;
Model summaries synthesized across multiple passages;
Content with no basis that needs human confirmation.

3. Document parsing still has uncovered edge cases

Mixed PDFs should decide OCR per page;
OCR confidence and text bounding boxes should be stored, to support highlighting on the original page;
DOCX needs limits on decompressed size, compression ratio, and XML complexity;
File-type checks should use magic numbers, not just extensions.

4. Task idempotency and multi-store consistency are not stable enough

MySQL, object storage, and Qdrant do not share a database transaction. Beyond the task leases, fencing tokens, deterministic point IDs, and index generations mentioned earlier, periodic reconciliation is needed: repair chunks that exist in MySQL but are missing from Qdrant, and clean up orphaned Qdrant points whose business records are gone.

The deletion path has the same problem. Currently deletion cleans up Qdrant first, then removes the original file, and finally deletes the MySQL record. A failure at any step can leave orphaned resources or inconsistent state. Deletion also needs to be task-driven and idempotent, with compensating retries allowed.

5. Qdrant filter fields need index planning

Writing user, library, document, page, and material-type conditions into the payload does not guarantee efficient filtering. As data grows, high-frequency filter fields should get Qdrant payload indexes, with the effect validated against field cardinality and query patterns. I have not confirmed that the current deployment indexes these fields, so this is listed as productionization work, not an existing capability.

6. Model input/output boundaries still need hardening

Document content is itself untrusted input. Prompt injection, strict JSON schemas, field-length limits, sensitive-data redaction, and model providers’ data-retention policies all still need work.

References

Code location index

The project I have been calling “Zhiye” uses the internal module name trial_cue, so some file names and config keys keep that prefix.

To read along with the code, the main modules are:

Responsibility	Primary code location
Document upload, status, retry, and delete	`internal/service/trial_cue_document.go`
Parsing PDF, DOCX, text, and images	`pkg/document/`
OCR and the PDF-to-image fallback	`pkg/ocr/`
Chunking	`pkg/document/trial_cue_chunker.go`
Background task scheduling	`internal/task/trial_cue_runner.go`
MySQL keyword retrieval	`internal/dao/trial_cue_search.go`
Embedding	`pkg/embedding/`
Qdrant indexing and retrieval	`pkg/vectorstore/`
RRF, filtering, degradation, and answer orchestration	`internal/service/trial_cue_question.go`
Optional reranker	`pkg/reranker/`
LLM context and structured output	`pkg/llm/trial_cue.go`

Summary

The core of this document Q&A pipeline is not “turn documents into vectors.” It is progressively turning raw files into material that is retrievable, filterable, traceable, and verifiable, and then constraining the model to answer only from that material.

Three boundaries are the ones worth defending: whether the parse result is complete, whether every recall path stays under permission control at all times, and whether the final citations genuinely come from this request’s candidates. Embeddings, Qdrant internal hybrid retrieval, and the reranker can all improve results — but task idempotency, degradation semantics, citation verification, and offline evaluation are what decide whether this system can reliably move from demo to real-world use.

This article is the English edition of a post originally published in Chinese.