Local RAG on your own documents

I wanted to be able to ask questions about my own documents – invoices, contracts and technical notes – without anything ever leaving my computer. The result is a simple RAG pipeline: documents are split into chunks, embedded with nomic-embed-text and stored in ChromaDB. A question is embedded with the same model, the closest chunks are retrieved and llama3 composes the answer from them.

Tuning the chunk size ate most of the time. Too small and they lost context, too large and they diluted relevance. I ended up with 500 tokens and an overlap of 50 – it works surprisingly well on my Slovak documents.