The same fine-tune, now with a hybrid retriever (MiniLM + BM25, top-5 over 42,412 chunks) feeding it the actual passages before it answers. This is where the facts arrive — correctness jumps and it becomes the decisive winner of the experiment.
Ask it and read its answer after a hybrid retriever pulls the top-5 passages from 42,412 chunks and feeds them in, so the answer is grounded in real card text. Every answer lists the passages it used.
Before answering, it fetches the five most relevant pages and reads them, so the facts come from real card text rather than memory.
Where this system lands against the other two on the 60 held-out biased questions, scored blind by a reference-grounded judge out of 10. Green is this system; tan are the other two.
Measured mean judge /10 on the 60 held-out (biased set), scored blind by a reference-grounded judge (gemini-3.1-flash-lite). This system in green; the other two in tan.
| System | Biased 60 | Unbiased 60 |
|---|---|---|
| A · base | 3.98 | 1.93 |
| B · fine-tune | 5.25 | 2.62 |
| C · +retrieval | 8.05 | 8.25 |
Mean /10 on the biased and the equal-split unbiased 60. Paired deltas — fine-tuning A→B: +1.27 (p=0.007 biased; +0.68 marginal, p=0.041 unbiased). Retrieval B→C: +2.80 (p<0.001 biased; +5.63 unbiased).
Each rung keeps everything below it and adds one thing — but the big jump in real facts only comes at the top, with retrieval.
All three systems are the same Gemma 2 2B — 26 layers, grouped-query attention (8 heads sharing 4 KV), a sliding 4,096-token window, and a GeGLU feed-forward. Run 4-bit at inference.
| Class | Gemma2ForCausalLM |
| Layers | 26 |
| Hidden size | 2,304 |
| Attention | 8 heads / 4 KV · dim 256 · GQA |
| Feed-forward | GeGLU · inner 9,216 |
| Attention window | sliding 4,096 · logit soft-cap |
| Norm | RMSNorm |
| Context | 8,192 tokens |
| Vocabulary | 256,128 |
| Embeddings | tied input/output |
| Inference | 4-bit NF4 |
The Gemma 2 2B configuration shared by all three systems.
The prompt is turned into numbers, passed through the same small stack of maths 26 times, then turned back into the next word.
How this system was built: a teacher wrote QA pairs, a blind judge filtered them, a QLoRA adapter was trained on what survived, and a FAISS index keeps every passage searchable at answer time.
| Init from | gemma-2-2b-it · 4-bit NF4 |
| Method | QLoRA · rank-16 / α-32 |
| Targets | all linear modules |
| Trainable params | 20.8M LoRA (~0.8%) |
| Schedule | LR 2e-4 cosine · seq 512 · bf16 |
| Data | 2,683 QA pairs · early-stopped ~1 epoch |
| Val perplexity | 3.87 (ref 4.26) |
| Hardware | Modal L4 |
| Cost | $3.26 · Modal $0.26 + Gemini $3.00 |
| Retriever | MiniLM-L6 + BM25 · RRF |
| Index | FAISS flat · 42,412 chunks |
| Top-k | 5 · recall@5 0.93 |
Same QLoRA fine-tune as System B, plus a hybrid retriever over a FAISS index of 42,412 chunks (recall@5 0.93).
Measured corpus composition by source · 27.5 MB total, 78% free-licensed Yugipedia prose + 22% YGOPRODeck card facts.
Free web text becomes practice questions, a strict judge keeps the best, they train the model, and every passage is indexed so it can be looked up while answering.