Audio and Vision

Zeph supports audio transcription and image input across all channels.

Audio Input

Pipeline: Audio attachment → STT provider → Transcribed text → Agent loop

Configuration

Enable the stt feature flag:

cargo build --release --features stt

[llm.stt]
provider = "whisper"
model = "whisper-1"

When base_url is omitted, the provider uses the OpenAI API key from the openai [[llm.providers]] entry or ZEPH_OPENAI_API_KEY. Set base_url to point at any OpenAI-compatible server (no API key required for local servers). The language field accepts an ISO-639-1 code (e.g. ru, en, de) or auto for automatic detection.

Environment variable overrides: ZEPH_STT_PROVIDER, ZEPH_STT_MODEL, ZEPH_STT_LANGUAGE, ZEPH_STT_BASE_URL.

Backends

Local Whisper Server (whisper.cpp)

The recommended setup for local speech-to-text. Uses Metal acceleration on Apple Silicon and handles all audio formats (including Telegram OGG/Opus) server-side.

Install and run:

brew install whisper-cpp

# Download a model
curl -L -o ~/.cache/whisper/ggml-large-v3.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin

# Start the server
whisper-server \
  --model ~/.cache/whisper/ggml-large-v3.bin \
  --host 127.0.0.1 --port 8080 \
  --inference-path "/v1/audio/transcriptions" \
  --convert

Configure Zeph:

[llm.stt]
provider = "whisper"
model = "large-v3"
base_url = "http://127.0.0.1:8080/v1"
language = "en"   # ISO-639-1 code or "auto"

Local Whisper (Candle)

cargo build --release --features candle   # CPU
cargo build --release --features metal    # macOS Metal GPU
cargo build --release --features cuda     # NVIDIA GPU

[llm.stt]
provider = "candle-whisper"
model = "openai/whisper-tiny"

Models are downloaded from HuggingFace on first use. Device auto-detection: Metal → CUDA → CPU.

Channel Support

• Telegram: voice notes and audio files downloaded automatically
• Slack: audio uploads detected, downloaded via url_private_download (25 MB limit, .slack.com host validation). Requires files:read OAuth scope
• CLI/TUI: no audio input mechanism

Limits

• 5-minute audio duration guard (candle backend)
• 25 MB file size limit
• No streaming transcription — entire file processed in one pass
• One audio attachment per message

Image Input

Pipeline: Image attachment → MessagePart::Image → LLM provider (base64) → Response

Provider Support

Ollama Vision Model

Ollama models vary widely in vision support — a text-only model (e.g. qwen3:8b) cannot process images, so Zeph never assumes vision support for Ollama: it checks the configured model’s capabilities via /api/show at startup, and drops incoming images (with a warning) for any model that isn’t confirmed vision-capable.

Route image requests to a dedicated vision model while keeping a smaller text model for regular queries:

[llm]
model = "mistral:7b"
vision_model = "llava:13b"

Alternatively, set model directly to a vision-capable model (e.g. llava:13b, qwen2.5vl, llama3.2-vision) and Zeph detects the capability automatically — no vision_model override needed.

OpenAI / Compatible Vision Override

type = "openai" and type = "compatible" providers auto-detect vision support from a built-in model-name prefix table (gpt-4o, gpt-4-turbo, gpt-4.1, gpt-5, o+digit reasoning models — all match; gpt-3.5* and anything unrecognised do not). Unlike Ollama, there is no standardized capability-discovery endpoint for arbitrary OpenAI-compatible servers, so this table is the only automatic signal available, and it fails safe to false for any model name it cannot identify — which is the common case for third-party compatible endpoints (Together AI, local vLLM, etc.), since their model names carry no OpenAI naming convention.

Set vision explicitly to override the auto-detected value in either direction:

[[llm.providers]]
name = "local-vlm"
type = "compatible"
base_url = "http://localhost:8000/v1"
model = "llava-onevision"
vision = true   # the prefix table cannot recognise this model name

[[llm.providers]]
type = "openai"
model = "gpt-4o"
vision = false  # force images off even for a recognised vision model

Sending Images

• CLI/TUI: /image /path/to/screenshot.png What is shown in this image?
• Telegram: send a photo directly; the caption becomes the prompt

Limits

• 20 MB maximum image size
• One image per message
• No image generation (input only)