Local AI for Home Assistant’s Voice Assistant… on a Raspberry Pi (Worksheet)
THE TUTORIAL IS MEANT TO BE USED WITH THIS VIDEO!
YouTube tutorials can be a pain to actually follow, so here is the worksheet that accompanies the video - for your convenience.
I recommend both printing this out, to use as a checklist, and keeping the page up, so you can copy-and-paste the entries.
This build keeps everything local. Your voice never leaves the house: an ESP32-S3-BOX-3 does the listening, and a Raspberry Pi 5 running Gemma 4 E2B does the thinking. No Amazon, no Google, no cloud.
What you need:
A Raspberry Pi 5, 16GB(active cooling and an official 27W USB-C supply; NVMe via the M.2 HAT+ strongly recommended; wired Ethernet, right? RIGHT?)
An ESP32-S3-BOX-3(or the BOX-3B; voice works identically, the difference is just the sensor dock)
A USB-C data cable(a real data cable, not a charge-only one)
Another computer to flash the satellite(Windows/macOS/Linux, running desktop Chrome or Edge)
Home Assistant OS(Step 1 covers this if you do not have it running yet)
2.4 GHz WiFi(the ESP32 has no 5 GHz radio)
STEP 1: Get Home Assistant OS running on the Pi 5
This worksheet assumes Home Assistant OS, the full appliance image, not Core or a container. HA OS is what gives you the one-click add-on store, which is the whole reason this is approachable.
Flash Home Assistant OS to the Pi with Raspberry Pi Imager, boot it, and finish onboarding
Open the UI at http://homeassistant.local:8123
Record your Pi's IP address HERE: __________________
(Already running HA OS? Skip to Step 2.)
STEP 2: Stand up Ollama and pull Gemma 4 E2B
Read this first: there is NO Ollama add-on in the Home Assistant store. Do not go looking for one. What HA ships is the Ollama INTEGRATION (a conversation agent, Step 4), and it talks to an Ollama SERVER you run. This step gets that server running.
HA OS is a locked appliance, so you add Ollama through a community add-on repository. In HA, go to Settings > Add-ons > Add-on Store, open the three-dot menu (top right), choose Repositories, and add:
[add under Add-on Store > Repositories >] https://github.com/SirUli/homeassistant-ollama-addonThen install the Ollama add-on it provides, and start it. This is a third-party community add-on, not an official one, so give it a look before you install it.
Security: this exposes the Ollama API to your whole LAN with no password. Fine on a trusted home network. Do NOT forward port 11434 through your router, ever.
Record your Ollama server URL HERE: __________________ (for example, http://192.168.0.38:11434 for the same Pi, or http://YOUR-BOX-IP:11434 for a separate machine)
Troubleshooting - the flash_attn 500 error:if ollama run returns "500 Internal Server Error: V cache quantization requires flash_attn", the server has a quantized KV cache turned on and Gemma 4 E2B cannot use flash attention. Force an unquantized cache. Set OLLAMA_KV_CACHE_TYPE=f16 on the server (in the add-on Configuration tab on the Pi, or as a user environment variable on Windows), then fully restart Ollama, and run it again.
STOP HERE - YOUR LLM BRAIN IS RUNNING LOCALLY!
STEP 3: Install the three Wyoming add-ons
In HA, go to Settings > Add-ons > Add-on Store and install these three. Each starts a small service that HA discovers automatically over the Wyoming protocol.
Whisper (faster-whisper) - speech to text. Model: base.en(or tiny.en for roughly 2x speed and slightly lower accuracy)
Piper - text to speech. Voice: en_US-lessac-medium
openWakeWord - wake word (server side). Preload the word you want.
After installing each one, Start it, then go to Settings > Devices & Services, find each under Discovered, click Configure, and Submit. That is the whole wiring step. The protocol does the rest.
STEP 4: Add the Ollama integration and build the Assist pipeline
Add the integration:
Settings > Devices & Services > Add Integration > Ollama
Point it at your server URL from Step 2, including the port:
[for example >] http://homeassistant.local:11434If HA cannot see any models, the URL is almost always missing the :11434 port.
Select gemma4:e2b as the model
Build the pipeline:
Go to Settings > Voice assistants and create a new assistant. Set each stage:
Conversation agent: the Ollama integration, model gemma4:e2b
Speech to text: Whisper
Text to speech: Piper, your chosen voice
Wake word: openWakeWord, your chosen word
Set the preference to be the built-in assistant. This is the single most important toggle. With it on, "turn off the kitchen lights" resolves in a few hundred milliseconds and never touches Gemma. Only phrasing the intent engine cannot parse falls through to the model. Leave it off and every "lights off" waits on the LLM.
Record your pipeline name HERE: __________________
STEP 5: Test the brain BEFORE you touch the satellite
Prove the whole pipeline works with zero satellite hardware. If something breaks after you flash the BOX-3, you will already know the brain is good.
Text test: open Assist in the HA web interface (the chat icon), set the assistant picker to your pipeline, and type to it. Try one command ("turn off the office lamp") and one open question ("what is a good silence timeout for a voice assistant"). This tests HA to Ollama to response, no audio involved.
Voice test with your phone: install the Home Assistant Companion app, open its Assist, set it to your pipeline, and use the mic button. This runs the full chain - phone mic to Whisper to Gemma to Piper - and is the most valuable pre-flash test, because it exercises STT and TTS, the two pieces the text test skips.
When a stage misbehaves: open your pipeline under Settings > Voice assistants and use its debug view. It shows each stage's output: exactly what Whisper heard, what handled the intent, and what Piper was told to say.
STOP HERE - YOU HAVE A WORKING LOCAL VOICE ASSISTANT (just no dedicated satellite yet)!
STEP 6: Flash the ESP32-S3-BOX-3 voice firmware
Desktop Chrome or Edge ONLY. The web flasher uses Web Serial, which does not exist on Safari, Firefox, phones, or tablets.
Plug the USB-C cable into the BOX-3 itself, not into the dock
In desktop Chrome or Edge, open the official installer page:
[open in Chrome/Edge >] https://www.home-assistant.io/voice_control/s3_box_voice_assistant/
Click Connect, pick the BOX-3's serial port, then Install Voice Assistant > Install, then Next
When prompted, join your 2.4 GHz WiFi
Select Add to Home Assistant, drop it in an area, and Finish
First-time firmware compiles on the server need about 2GB of free RAM. On the 16GB Pi this is a non-issue.
STEP 7: Choose where the wake word runs
This is the BOX-3's headline advantage. It can detect the wake word ON the device, so it only streams audio to the Pi after it hears you.
On-device (microWakeWord): lowest network use, best privacy, limited to the trained words. ESPHome device page > Wake word engine location > On device. Default word is Okay Nabu.
Server-side (openWakeWord): lets you train a custom word, streams a little more audio to the Pi.
To change the on-device word, edit the device's ESPHome config and set the substitution:
[in the ESPHome config substitutions: block >] micro_wake_word_model: hey_jarvis
Trained options include okay_nabu, hey_jarvis, and alexa. Save and Install, and HA pushes it over the air - no cable needed for re-flashes once it is on WiFi.
STEP 8: Point the BOX-3 at your Gemma pipeline
Settings > Devices & Services > ESPHome > the ESP32-S3-BOX-3 device
Set its Assistant / pipeline to the assistant you built in Step 4
Confirm Use wake word is enabled
Test: say your wake word, wait for the screen and LED, then ask something the local intent engine will not catch, like "what is a good rule of thumb for resistor wattage?" You are listening for Gemma, not the built-in intents.
STOP HERE - YOU HAVE A FULL LOCAL VOICE SATELLITE RUNNING GEMMA 4 E2B!
STEP 9: Tune the BOX-3
It cuts you off mid-sentence: raise the Finished speaking detection / silence timeout in the device's ESPHome configuration. It thinks you stopped talking too early.
It wakes at the TV: on server-side openWakeWord, raise the sensitivity threshold. Start around 0.5 and push toward 0.7 in rooms with a TV.
Screen is distracting at night: dim or disable the screen from the ESPHome device controls. It keeps working with the screen off.
Speaker too quiet or too loud: media volume is exposed as an entity. Set it per room.
STEP 10: Prove no audio leaves your network
Verify the privacy claim rather than trusting it. From a machine on your LAN, watch the Pi's traffic while you talk to the satellite:
[copy/paste >] sudo tcpdump -i any -n host 192.168.0.38 and not net 192.168.0.0/16
During a normal command you should see effectively nothing leaving the LAN. If packets head to the open internet, something in your pipeline fell back to a cloud service. Find it and fix it. (Use your real Pi IP in place of 192.168.0.38.)
Maintenance
Set a recurring monthly reminder.
Update HA and the add-ons from Settings > Add-ons and the HA update prompts
Update Ollama periodically. Gemma 4 tool-call handling and other fixes land often, so keep the server current
Refresh the model when a new build ships:
