Home Assistant Voice Agent (LiveKit)

A voice + text assistant for Home Assistant, built on

LiveKit Agents. This repo is the worker (the agent

brain): it connects to Home Assistant's native

MCP Server through the framework's

built-in MCPToolset, and serves STT / LLM / TTS via LiveKit Inference.

The dashboard UI is a Home Assistant custom integration + Lovelace card that runs the

session inside HA:

**ha-livekit-agent-frontend**. Run this

worker, install that integration, point both at the same LiveKit project, and you're set.

• Native MCP — Home Assistant tools (HassTurnOn, HassLightSet, GetLiveContext, …)

are exposed to the LLM directly through LiveKit's MCPToolset. No custom MCP client code.

• LiveKit Inference — STT, LLM, and TTS are served by LiveKit's inference gateway; no

per-provider API keys required.

• Cost-aware — STT is billed continuously, so it follows the mic: enabled while audio

input is live and torn down STT_IDLE_TIMEOUT seconds (default 120) after the mic is

gated. The worker boots dormant (STT off, no TTS), so an idle card on a dashboard costs

nothing beyond the worker connection, and text chat works without ever enabling STT.

• Turn modes — manual push-to-talk (start_turn / end_turn / cancel_turn) or auto

turn detection; switchable at runtime via set_turn_mode.

• Helper tools — a few function tools (get_areas, get_device_domains, get_devices,

get_environment_info) pre-process GetLiveContext into compact, area/domain-filtered

views so the LLM isn't flooded with the full home state.

Structure

.
├── agent/            # Python LiveKit worker (uv)
│   ├── src/          # main.py (entrypoint), agent.py, config.py, ha.py, scheduler_client.py, …
│   ├── prompt.yaml   # system prompt (bind-mounted, edit without a rebuild)
│   ├── pyproject.toml
│   └── Dockerfile
├── scheduler/        # task scheduler service (FastAPI + APScheduler)
├── docker-compose.yml
├── .env.example      # agent + scheduler env
└── README.md

Prerequisites

• Home Assistant with the MCP Server integration enabled, and a long-lived access token.
• A LiveKit server (self-hosted or LiveKit Cloud) for the realtime transport.
• LiveKit Inference credentials for STT/LLM/TTS (LiveKit Cloud).
• The **HA integration** installed for

the dashboard UI.

Run the worker

Uses uv.

cd agent
cp ../.env.example ../.env    # then fill in the values (see below)
uv sync
uv run src/main.py dev        # or `console` to test in the terminal

Or with Docker Compose (builds and runs the worker):

docker compose up --build

Environment (.env in the repo root)

The agent calls load_dotenv(), which walks up to the repo-root .env.

Models default to Chinese-friendly choices and can be overridden via

STT_MODEL, STT_LANGUAGE, LLM_MODEL, TTS_MODEL, TTS_VOICE, TTS_LANGUAGE

(see .env.example).

Turn modes & session control

The card configures the session at runtime over RPCs the worker registers:

• **set_turn_mode** (manual | auto) — manual is push-to-talk: the mic stays gated

until start_turn, then end_turn commits (agent replies) or cancel_turn discards.

auto keeps the mic live and lets the model detect turn ends.

• **start_turn / end_turn / cancel_turn** — manual turn lifecycle. start_turn opens

the mic (bringing STT back); end_turn/cancel_turn gate it (STT tears down after the

idle timeout). STT simply follows the mic.

• **set_audio_output** (on | off) — toggles the agent's spoken (TTS) replies; off means

text-only. Off by default, so text chat always works without any STT/TTS cost.

The worker broadcasts its live STT + audio-output state on the ha.speech_state data channel

so the card can mirror it (e.g. a "sleeping" indicator when STT is torn down), and the

tool-execution lifecycle on ha.tool_call.

Frontend / dashboard UI

Use the companion Home Assistant integration:

**ha-livekit-agent-frontend** — a HACS

custom integration + Lovelace card with native device tiles (tap to control), voice, text

chat, transcript, and tool-call status. It mints LiveKit tokens for logged-in HA users and

dispatches this worker by AGENT_NAME.