Pi Local RAG: Offline Edge Vector Search & LLM Pipeline on Raspberry Pi 5

Overview

Deploying Retrieval Augmented Generation pipelines to resource constrained single board computers requires overcoming strict memory bandwidth, storage access, and ARM CPU execution limits. This project establishes a self contained local RAG system running on an 80 dollar Raspberry Pi 5 single board computer with 8GB RAM, eliminating external API calls, subscription costs, and data privacy risks. The ingestion pipeline processes unstructured robotics textbooks and technical documentation. Documents are parsed into clean text blocks, split into fixed size semantic chunks, and embedded using the all MiniLM L6 v2 sentence transformer model. The resulting 384 dimensional dense vector representations are indexed into a local FAISS flat L2 vector store, enabling sub millisecond similarity searches across tens of thousands of document chunks directly from local storage. When a query is submitted through the local web interface, the system executes an automated retrieval and generation workflow. The user query is converted into a vector embedding, matched against the indexed document vectors in FAISS, and the top relevant context passages are injected into an optimized prompt template. Local inference is powered by Llama 3.2 3B Instruct running on the Raspberry Pi 5 Quad core ARM Cortex A76 processor. Token generation is handled natively via Ollama local runtime, using NDJSON streaming to pipe generated tokens directly to the frontend interface with minimal time to first token latency. The frontend consists of a lightweight Flask web application featuring a ChatGPT style responsive chat interface. It includes an AbortController trigger for instant response cancellation, side by side toggles to switch between RAG context augmented generation and raw LLM inference, and an interactive context inspection panel showing exact source document references for every generated answer.