A Local RAG on a RasPi

To begin, this will likely be the first in a thread of articles on a broader topic. My goal today is to set the context and explain why I am learning these tools, the progress I've made, what I've learned, and my next steps.

Spoiler: the tools available and approach to building a personal RAG have changed since I started thinking about this in January.

Some Context About Why?

Like many others, I've been taking the time to learn about various generative AI tools, especially textual ones like ChatGPT, Claude, and Gemini. My interest has been in how to make them more useful in education. My general skepticism of the privacy of these tools is relatively high. Much like the search and social giants, they are likely motivated to monetize me, not just by selling me services.

Earlier this year, after Deepseek R1 was released, I learned how to run it locally using Ollama. I eventually ran it on my uConsole, a cyberdeck-like device that utilizes compute modules (specifically Raspberry Pi CM3, CM4, and CM5 Single Board Computers [SBCs]). Once I got the LLM running, I had one of those moments where I finally got it. As someone who specialized in AI for their CS degree, I have been hesitant to join the generative AI hype cycle. Though having a portable queryable (and private!) knowledge base always available, running on a hobby computer that was slightly smaller than the netbook computers of the mid to late 2000s, caused that lightbulb (💡) in me to flicker on.

A challenge, though, with many of these models in general, has been the correctness and accuracy of the responses, commonly referred to as hallucinations. To better understand this, I conducted a few experiments around the same time my daughter's class was studying the Revolutionary War. My experiment showed me that even the medium sized models I could run on my desktop (which has a 12GB GPU and can comfortably load and run models that are 8-9 GB in size ) could accurately answer questions like "What were the events that contributed and lead to the revolutionary war?" or "Who is credited for the Great Compromise?" Unfortunately, the smaller models that run on the CM4 (which can support at most a model that requires less than 4GB of memory space) got basic facts wrong. Unfortunately, for most learners, I doubt they would question it, because it comes from a computer, the thing that is always right... right?

This led me to ask and learn how one controls the responses drawn from generative AI and how to ensure that the source content is highly trustworthy and expertly curated.

The three approaches that came up are:

1. Prompt Engineering: Loading source content into the context window of Ollama and telling the model to only use content provided to give answers to questions. The prompt engineering approach.
2. Retrieval Augmented Generation (RAG): This generally works by processing a question and converting it into a set of vectors. Using those vectors to generate a query against a database that has content in it, which is similarly organized (as a bunch of vectors) and can be retrieved based on similarity, to then be leveraged as the context for an LLM to generate a response only based on the authoritative content. Sort of like having an American historian at my kid’s fingertips.
3. Refine/retrain: Rebuilding or retraining an existing model to focus on content from authoritative sources only, excluding content that may have originated from unreliable sources, like a debate on Reddit or comments on Facebook. Note: This approach often requires renting really expensive GPUs or buying a big, beefy new computer. I will likely try this, it's reasonably achievable with a GPU, though it may have a very long runtime, in the future.

I decided to get my toes wet and start with using a RAG approach. So, I embarked on a naive approach to implement a RAG system. Taking this approach, while limiting the scope of the demo, I set out to learn as much of the tools and techniques as I needed to get something working and demoable.

The Objective

Create a self-contained and private knowledge base that can be queried using an LLM and a generative AI interface on a "low" powered computer. Something that I would feel comfortable handing to my daughter to use, to help her learn, particularly with subjects I or my wife are less knowledgeable on like the Revolutionary War or for me to get even deeper into nerding out about the StarTrek universe.

Building That First RAG

Because I like starting with the punch line, and then backing out the story, here's the fruit of this effort, which is accessible on Github. (Feel free to let me know your thoughts - it's how I learn, do be nice though, I'm a learner.)

Tools Of Choice

• Dev Environment: Cursor and VS Code (with Copilot). I did most of the development on my desktop, though I did do some on my uConsole, not just on my desktop (Yep, I coded some of this in bed). Cursor helped create unit and end-to-end tests and assisted me in documenting and debugging the code I wrote. I also used it to help with refactoring and other tasks. Unfortunately, it wasn't particularly stable this go around on my uConsole (as its agentic behavior to update files kept failing).
• Programming Language: Python with FastAPI. I chose languages and tools with which I'm comfortable. Python also offers good support for tools like LangChain.
• Vector DB: ChromaDB. No particular reason for its choice other than it was used in an example from someone who wrote an article I could understand and follow with some of that sweet, sweet Claude and Copilot coding help. There are other vector DBs for sure, and I know more now...
• Source Code Management & CI/CD: GitHub with Github Actions. Skilled individuals would point out that I wasn't practicing continuous delivery, as I didn't have a deployment environment. What I would say, though, is that having the build tested with lightweight functional testing using GitHub Actions saved my bacon and made testing on the uConsole way easier (because I knew the code would build, and I could always pull the container down with Docker when I needed to).
• LLMs running locally: Ollama with LLama3.2 (3b parameter model, the default) and DeepSeek R1 (1.5b parameter model). I used the two as I was experimenting... I prefer the responses generated by DeepSeek R1, but LLama3.2 was faster on the Raspberry Pi. The codebase is currently set up to use DeepSeek R1, look at this PR to see where updates need to be made.

How Does Retrieval-Augmented Generation Work?

And now for the super-duper high-level walk-through of how RAGs work.

This workflow is generally broken up into two parts: loading in a bunch of content into a vector DB, and then querying that DB for the relevant content to recompose into a response to a question.

1. Loading Content into a Vector DB (ChromeDB)

Break up your content into chunks in a way that is "smart", create an embedding (a numerical vector) for each chunk, and put those embeddings into the vector database so that they can be searched for by how similar they are to the query.
- A vector is simply a way to describe a set of numerical values, which is a set of coordinates. If that doesn't make sense, think of it as a pin on a map (a very big map).

2. Query the database of embeddings (content chunks) based on the question asked (also called a query)

Once queried, retrieved chunks from the vector db are fed back into an LLM to be composed as more understandable prose.
- What is returned from the database are the closest pieces of content that are related to the provided query. Put differently, if I ask a question about the Enterprise spaceship, chunks with the word "enterprise" will be returned as related chunks. Because this is a semantic search based on meaning, not just keywords, a query about "the flagship of the Federation" could also retrieve the same content about the Enterprise.
- Many systems will process any question you ask to reduce it to just the pertinent parts of the query (removing articles, punctuation, etc) so that the meaning of the query is being searched (LLMs help with this as well).

How The Repo Is Laid Out

From the readme, the folder structure:

startrek-rag-llm/
├── startrek-rag/            # Main RAG application
│   ├── app.py               # Application factory and main entry point
│   ├── config.py            # Centralized configuration management
│   ├── embed.py             # Embedding generation utilities
│   ├── db_config.py         # ChromaDB configuration
│   ├── routes/              # API route blueprints
│   │   ├── __init__.py
│   │   └── api.py           # REST API endpoints
│   ├── services/            # Business logic layer
│   │   ├── __init__.py
│   │   └── rag_service.py   # RAG operations service
│   ├── requirements.txt     # Python dependencies
│   └── Dockerfile           # Container configuration
├── content_loader/          # Content processing service
│   ├── process_content.py   # Original text content processing script
│   ├── enhanced_processor.py # Enhanced processor supporting HTML and URLs
│   ├── html_processor.py    # HTML parsing and text extraction utilities
│   ├── requirements.txt     # Dependencies
│   └── Dockerfile           # Container configuration
├── test_content/            # Sample content for testing
│   ├── startrek_original_series.txt
│   ├── star_trek_wikipedia.html
│   └── star_trek_urls.txt
├── docker-compose.yml       # Service orchestration
├── Makefile                 # Build and run commands
└── README.md                # The ReadMe

As a starting point, I strongly suggest reading the `README.md` file and then the `Makefile` in the root directory, as this will give you the best idea of how everything works. I also experimented with LLM-generated architecture documents; if you're interested, you can find them under `docs` as PlantUML files.

The `ReadMe.md` file also includes dependencies, as you will need to have Ollama running with the models you want to use for embedding, as well as for generating the responses.

In the repository, look in the `content_loader` folder if you are curious about how to create embeddings from content. For this particular project, I targeted three types of content: text files, HTML files, and URLs (for obtaining content). Many systems will go further and crawl websites or mine content data sources, such as databases of current research on a given topic.

The content loader can be run as a container, and it will look in the defined folders to determine where to retrieve content from. Setting this up is relatively simple, as there's a `test_content` folder that contains content I grabbed from Wikipedia to provide examples.

For the query side, look at the `startrek_rag` folder. Running the container launches a web service using FastAPI, accessible from your local machine at `localhost:8080`.

To keep things simple, I decided to use the same LLM model for all scenarios, generating the embeddings, understanding the query, and generating the response. This is not the norm; many embedding-specific models tuned for their purpose perform better.

Getting This To Work On Two Machines

Once I got it working on my desktop, the next step was to get the same simple RAG running on my uConsole, which has a lot fewer resources (and is an ARM64 machine, not AMD64). I'm fortunate in that I have a desktop with a good CPU and a modest GPU, but that’s not the case for most people. Also, my desktop is not portable at all. The uConsole, on the other hand, is very portable; mine lives by my bedside and in my sling when I'm out and about. Also, it's not particularly expensive (making it ideal to give to my kiddo, as she can break it, and I can fix it or replace it. All in, it (the device, CM4 module, 256GB microSD card, and batteries) runs about $250 shipped, still expensive for most, but in the world of similarly sized devices, it's on the cheaper end. It also comes with none of the related concerns that come with handing your kid a phone and the open web.

Getting the repo running on the CM4 was surprisingly straightforward. The primary change was to move from using Llama 3.2 3b parameters, which would freeze on the device (as ~4GBs of memory space is used when running, and it is sloooow), to Deepseek R1 1.5b parameters (~3GBs when running and a bit peppier, being a smaller model). My CM4 has 8GB of built-in RAM and features four cores clocked at 1.5GHz that can be boosted to 2GHz.

How Did It Work Out?

I tried LLama 3.2 and DeepSeek R1 on both my desktop and on the uConsole. I didn't do this scientifically, as I was just happy that everything worked! The desktop, of course, ran everything much faster. Generally, my queries were responded to in 1-2 seconds, regardless of the model I was using.

On the uConsole was a different story. The Llama 3.2 model would load, and I could ask it direct questions, but when I started ChromaDB, the app, and the model, all three of those, the device would freeze. In one instance, when I had Thunderbird running in the background, it was worse, and the device shut off (which could have been due to overcurrent protection or some seg fault). Llama with 3B parameters requires more CPU resources when running the model than DeepSeek, not just RAM. And this created enough resource contention to cause the containers and other running processes to effectively halt. DeepSeek R1, on the other hand, was able to handle queries (as pictured below). cURL calls to the `/query` endpoint would respond after a few minutes.

In the picture above (if you squint hard enough), on the uConsole, on the left is the response to a query of “What is the USS Enterprise” and the response coming from DeepSeek, using the RAG documentation that was loaded in at the beginning of the test.

I view this as a win! What that means is that it is possible to run a local LLM with RAG to respond to questions given a known knowledge set on the device, meeting the state goal above.

So, What Did You Learn?

First and foremost, it requires a fair bit of elbow grease and help from coding tools to build AI tools (as a noob). Second, learning through doing is a great way to gain knowledge for folks who enjoy experimenting and tinkering.

More importantly, the initial inertia that I started with, to solve the problem of how to support learning through inquiry, is here. The idea that everyone can have an off-grid encyclopedia, like The Guide in The Hitchhiker's Guide to the Galaxy, in their hand is now.

Growing up, my parents spent thousands of dollars on encyclopedias (both Britannica and New World) for my brother and me, as this allowed us to have a general library at home to help us learn the basics before visiting the library to delve into a subject. Now, with a lot of privacy in mind, kids can not only read the content but also interrogate that knowledge in ways that didn’t exist previously. Imagine being able ask a question like “Did Washington want to be president a third term?” I know the answer is no and that he longed to go home and be with his family in Virginia. Though I’ve had the fortune to read his letters home to Martha and to John Adams. Being able to do this on any portable device, accessible to anyone for a few hundred dollars, is just amazing!

To accomplish this, I learned how RAGs work, but unfortunately, the tools I used were outdated from the day I started. As I hinted at the beginning, my naive way is already old news. I started this journey initially in January, and the practical and accessible content and writing on this technique I could find is generally 8–10 months old. As of a couple of months ago, there are projects that, out of the box, allow one to build a full RAG pipeline with appropriate tooling (much more sophisticated than what I did here) in days (this project for me has been a couple of months in execution as it’s been nights and weekends).

I’m specifically talking about tools like n8n, which allows the workflow designer much more user-friendly tooling to get the work done, and in a way that is much more DevEx minded. Additionally, ChromaDB has gradually lost ground to alternative vector databases like Qdrant, which offer better performance with reduced CPU overhead and provide more out-of-the-box tooling for management.

A main takeaway is that this technical space is still a very evolving corner, and it’s exciting. These technologies have the power and opportunity to reshape how we learn about the world around us and how we acquire new knowledge. I’m excited about where this can go.

What Next?!

From this experience, I have a much longer to-do list than when I started (shocking!). Next up:

1. Rebuild the current solution to use n8n for both generating and storing the embeddings.
2. Migrate from ChromaDB to Qdrant
3. Build a better site crawler to see if I can scale the amount of content I can load into the uConsole
4. Switch my uConsole from the CM4 to a CM5 16GB model (for the performance and memory headroom).

After that, I’ll want to build a front-end so that I can try this with my kiddo. I think she’ll love it, though I’m sure she’ll use it for D&D or to learn more about Zelda or Naruto (both universes have depth and complexity) rather than the Revolutionary War or Star Trek (as she put it, “nerd alert!”)