How to run local AI on tiny hardware like ESP32 and Pi

TL;DR

Tiny AI models run on ESP32S3 and Arduino chips, and more capable models run on Raspberry Pi and iPhone 16. The video demonstrates hardware classes, model types, and basic workflows for running local AI, plus ideas for projects and limitations around multimodal or conversational capabilities.

Transcript

Hello. Today we're going to be running Every size AI locally with [music] the hardware that we have here and see what we can do with it. Hopefully give you guys some ideas for cool things that you can build with local AI too. Let's go. A portion of this video is sponsored by Cruso. So I'm going to be grouping the hardware that I have here into diff... Read More

Key Insights

• Chips lack operating systems and memory sticks, and cannot host AI models due to RAM constraints.
• ESP32S3 has 8 MGB of RAM and can run Tiny Stories 1MB and 3MB models comfortably.
• Arduino R4 Uno has 32 kilob of RAM, which is too small to fit any AI models.
• Mini computers like Raspberry Pi 5 and iPhone 16 run Linux and iOS respectively and can host multiple models.
• Raspberry Pi can perform speech to text with Whisper, and run a small TT model and Piper for text to speech.
• Moonream and Piper enable visual language tasks on the Raspberry Pi with a webcam.
• iPhone 16 can run small plus visual language models and image generation with Draw Things, within RAM limits imposed by Apple.
• GPUs are a bottleneck for image, music, and video generation on small devices, limiting multimodal capabilities.

Questions & Answers

Q: How does the video categorize AI hardware and why is it important for running models locally?

The video groups hardware into chips, mini computers, and phones, explaining that chips have no OS or memory and cannot host AI models due to RAM constraints, while mini computers like the Raspberry Pi and iPhone provide OS environments, memory management, and more capable ecosystems for running local AI. This categorization helps viewers pick suitable models for each device and understand practical limitations, such as RAM ceilings and the absence of dedicated GPUs on chips.

Q: What Tiny Stories models can run on the ESP32S3, and what does this imply about tiny devices?

The ESP32S3 with 8 MGB RAM can run Tiny Stories 1 megabyte and Tiny Stories 3 megabyte models comfortably, showing that extremely small devices can execute basic local AI tasks. This demonstrates that meaningful AI experiences can be pushed to low-cost hardware, albeit with limited capabilities like simple word prediction rather than full conversations or multimodal processing.

Q: Why is the Arduino Uno R4 not suitable for AI models, and is it entirely useless for local AI?

The Arduino Uno R4 has only 32 kilob of RAM, which means it cannot fit any AI models, so it cannot run local AI. However, it is not useless, as it can still connect via Wi-Fi to bigger devices, enabling networked AI workflows or acting as a sensor front end in a larger local AI system.

Q: What capabilities does the Raspberry Pi demonstrate in running local AI, and what are its limitations?

The Raspberry Pi demonstrates running speech to text with Whisper, a QA text model with a tiny text to speech component like Piper, and a visual language model with a webcam using Moonream. It can run micro to small language models and some visual tasks, but struggles with midsize LLMs and cannot efficiently generate images, music, or video due to lack of GPUs.

Q: How does the iPhone 16 affect local AI model choices compared to the Raspberry Pi?

The iPhone 16 has 8 GB of RAM but Apple caps per-app memory usage to around 4 to 5 GB, limiting the ability to run midsize 7 to 14B models that the Raspberry Pi could potentially handle. It supports small plus visual language models and image generation with Draw Things, but larger models and multimodal capabilities are constrained by platform restrictions.

Q: What are some practical project ideas mentioned for tiny devices?

Possible projects include turning ESP32 into a little game like Scrambled versus Not Scrambled to learn English grammar, adding LED mood indicators to stories, and creating modular AI setups via cameras and small language models. The video invites viewers to imagine scenarios by combining tiny models with inexpensive hardware for creative standalone or networked projects.

Q: What models are highlighted as enabling visual understanding on the Raspberry Pi, and how do they interact?

Moonream enables a visual language task by describing what a camera sees, while Piper handles text to speech to vocalize descriptions. Whisper performs speech to text, enabling a conversation flow, while Quen 3 or similar models can interpret text and generate responses. The combination demonstrates a basic but functional local AI stack on a Raspberry Pi.

Q: What is the overall takeaway about running AI locally on consumer hardware from the video?

Local AI on consumer hardware is feasible across a spectrum of devices, from tiny ESP32S3 boards to full mini computers like Raspberry Pi and iPhone 16. The key takeaway is that model size, RAM, and platform constraints dictate what you can run, but clever chaining of multiple lightweight models can achieve practical, interactive AI experiences without cloud access.

Summary & Key Takeaways

• The video categorizes hardware into chips, mini computers, and phones, showing what AI models fit each class and how to chain multiple models for simple tasks. It emphasizes RAM limits and the absence of GPUs on chips, which restricts larger models and multimodal capabilities. Practical project ideas and quick-start guides are highlighted.

• On ESP32 based devices, Tiny Stories models of 1MB and 3MB run comfortably, demonstrating that extremely small devices can host basic local AI. The Raspberry Pi and iPhone 16 can handle more complex workloads, including speech to text, small language models, and some visual language tasks with webcams.

• The presenter uses a kitchen analogy to explain how hardware components and models cooperate, comparing the prep counter, head chef, and machines to RAM, model size, and specialized hardware, illustrating modular design and the limits of single devices for AI workloads.