Ask questions about any document by voice and get spoken answers grounded in the document's content. Local embeddings, local LLM inference, speech-to-text, and text-to-speech, running entirely on-device with no cloud dependency.
An on-device AI pipeline loads a document, chunks it, and generates local embeddings using picoLLM. When the user asks a question by voice, Cheetah transcribes the question in real time. picoLLM generates an embedding for the question, retrieves the most relevant document chunks by cosine similarity, and generates an answer grounded in those chunks. Orca speaks the answer back. The entire pipeline runs on-device: No document content, no queries, and no audio data is transmitted externally.
Cheetah transcribes the user's question in real time as they speak, streaming partial results with automatic punctuation and endpoint detection. Cheetah detects when the user has finished speaking and immediately hands the complete question to the retrieval step. Running speech-to-text on-device means the user's spoken queries are never transmitted to a cloud transcription service.
picoLLM serves two roles in this pipeline. First, it generates document embeddings for chunked text, converting each chunk into a vector for similarity search. Second, it generates answers grounded in retrieved chunks using a system prompt that constrains responses to the provided document excerpts. picoCompression quantizes models while preserving task accuracy, making it possible to run capable models on laptops, desktops, and Raspberry Pi. picoLLM Inference runs any language models (Llama, Gemma, Phi, Mistral) locally on CPU or GPU. No document content or queries are sent to any external service.
Orca speaks the LLM's answer as tokens stream in, so the user hears the response before generation is complete. Unlike traditional TTS engines that wait for complete text, Orca processes raw LLM tokens as they arrive, achieving 130 ms first-token-to-speech. Orca uses 29 MB peak memory and a 7 MB model size, making it deployable alongside Cheetah and picoLLM on the same device. No audio is transmitted externally.
Legal teams, compliance officers, and auditors can query policy documents, contracts, and regulations by voice. "What are the termination clauses in this agreement?" The document and all queries stay on-device, so privileged or regulated content is never transmitted externally.
Technicians can ask questions about equipment manuals, wiring diagrams, and repair procedures by voice while their hands are occupied. "What's the reset sequence for fault code E47?" The RAG pipeline runs offline on the technician's device, so it works in facilities with no connectivity.
Clinicians can query drug reference guides, formularies, or protocol documents by voice during patient encounters. "What is the recommended dosage of amoxicillin for pediatric patients?" All document content and queries stay on-device, keeping clinical information off third-party servers.
New employees can ask questions about handbooks, SOPs, and training materials by voice instead of searching through documents. "What's the return policy for opened items?" The voice interface makes internal knowledge accessible without navigation skills or document familiarity.
A complete working recipe in Python. Open-source on GitHub. Runs 100% on-device.