AI Agents

Voice and chat AI on a Twilio-compatible API.

VoiceTel AI Agents implements the Twilio AI Assistants surface so application teams can ship voice and chat agents — with tool calls, knowledge bases, and per-session memory — without rewriting client code for a new vendor.

AI Assistants API reference Sign up

voice-agent.xmlConversationRelay

<Response>
  <Connect>
    <ConversationRelay url="wss://your-app.example/conversation"
        language="en-US"/>
  </Connect>
</Response>

Anatomy

Assistants you configure, not train.

Each assistant points at an LLM provider and model, carries a system prompt, and attaches the tools and knowledge bases it can use. Tool calls run against your application; knowledge is retrieved from what you ingested; every session keeps its history.

Endpoint map

What an AI Agents assistant connects to. Assistant (model + instructions) connects to Tools (function-call schemas; outputs return as context), Knowledge (web pages, files, raw text → indexed chunks), Sessions (per-session message history, read back), Feedback (scores assistant turns over time).

What you get

Assistants you configure, not train

Each assistant points at an LLM provider and model, carries a system prompt, and attaches the tools and knowledge bases it can use. No fine-tune cycle — change the prompt, change the behaviour.
Tool calls into your application

Define function-call schemas as Tools, attach them to assistants, and the runtime invokes them when the model decides to. Outputs feed back into the conversation as model context.
Retrieval-augmented knowledge

Knowledge resources ingest web pages, files, or raw text. Status and per-chunk inspection endpoints surface what's in the index. Attach knowledge to specific assistants for grounded answers.
Sessions, messages, and feedback

Per-session message history is read-back-able for auditing. Feedback events score assistant turns so quality can be tracked over time without bolt-on logging.

Voice agents on VoiceML

Put an assistant on a live call with ConversationRelay.

Pair an assistant with <Connect><ConversationRelay> to put it on a live call. VoiceML handles call control, media, STT, and TTS; the assistant handles turn-by-turn reasoning, tool calls, and knowledge retrieval. Existing voice-agent clients re-host with a hostname swap.

These open-source projects were built for Twilio's ConversationRelay and, by our review of their source, run on VoiceML as-is. Bring your own model key. Limits, setup notes, and the ConversationRelay messages VoiceML supports are on the ConversationRelay projects page. Reviewed 2026-10-04; not tested on live calls.

Project	Built with	From
twilio-samples/conversationrelay-elevenlabs-openai	Node.js · OpenAI	Twilio sample
twilio-samples/conversationrelay-ai-demo	Next.js · OpenAI	Twilio sample
langflow-ai/langflow-twilio-voice	Node.js · Langflow	Langflow
pkamp3/cr-anthropic-demo	Node.js · Anthropic Claude	Community
rishabkumar7/twilio-cr-gemini-python	Python · Google Gemini	Community
manbearshark/ConversationRelay-LiveChat-Bedrock-Fargate	Node.js · Amazon Bedrock on AWS Fargate	Community

Voice AI frameworks

Open-source frameworks for building your own voice agent pipeline reach the phone network through VoiceML's Media Streams (<Connect><Stream>) or through VoiceTel SIP trunking. How each one connects:

Framework	Connects through	What to configure
PipecatBSD-2-Clause	VoiceML `<Connect><Stream>` (Twilio Media Streams protocol)	Use Pipecat's Twilio serializer and set its `base_url` to `https://voiceml.voicetel.com` so automatic hang-up goes to VoiceML.
LiveKit AgentsApache-2.0	VoiceTel SIP trunking to LiveKit SIP	Create LiveKit inbound and outbound SIP trunks for your VoiceTel numbers. Its Twilio-specific transfer helper calls Twilio's API directly and doesn't apply.
DograhBSD-2-Clause	Its Asterisk ARI provider, with Asterisk on a VoiceTel SIP trunk	Its Twilio provider is fixed to Twilio's API host for number setup, outbound calls, and transfers, so use the Asterisk provider or change that host.
TEN FrameworkApache-2.0 with additional conditions	VoiceML `<Connect><Stream>`, with its Twilio voice-assistant example	The example places and ends calls with the Twilio Python SDK; point the SDK at VoiceML with a custom HTTP client.
BolnaMIT	VoiceML `<Connect><Stream>` (Twilio provider), or Asterisk on a VoiceTel SIP trunk (sip-trunk provider)	Inbound calls use its `<Connect><Stream>` response as-is; for outbound calls through its Twilio API server, point the Twilio Python SDK at VoiceML.

Reviewed 2026-10-04 by reading each framework's telephony code; not tested on live calls. Project names belong to their owners; listing a project does not imply endorsement by, or affiliation with, its authors.

API surface

Assistants, tools, knowledge, sessions, feedback, and policies.

The v1 paths, field names, and SID shapes match the upstream Twilio surface. Point your client at voiceml.voicetel.com, swap credentials, and the existing SDK works.

Assistantsplus POST /Assistants/{id}/Messages for inference CRUD
Toolsglobal catalog and per-assistant attachments CRUD
Knowledgechunk and indexing-status inspection CRUD
Sessionssession and per-turn message history Read-only
Feedbackper-assistant feedback events Submit · list
Policiesguardrail policies available to assistants Read-only

FAQ

Questions about AI Agents.

Which LLM providers can an assistant point at?

Provider and model are configured per assistant. Available providers depend on which integrations are enabled on your account — contact support to confirm the model you want to use is wired up.

Do Twilio AI Assistants client SDKs work against VoiceTel?

Yes — the v1 paths, field names, and SID shapes match the upstream Twilio surface. Point your client at voiceml.voicetel.com, swap credentials, and the existing SDK works.

How do I put an assistant on a live phone call?

Use VoiceML's <Connect><ConversationRelay> verb in your call-control response. VoiceML handles call control, media, STT, and TTS; the assistant handles model reasoning, tool calls, and retrieval. See the VoiceML docs for the wiring.

Where do tool calls actually execute?

Tools are function-call schemas the assistant invokes. The runtime serialises the call to your endpoint or webhook — your application performs the work and returns the result, which feeds back into the model's next turn.

How are knowledge bases indexed?

Knowledge resources ingest web URLs, uploaded files, or raw text. The indexing pipeline chunks and embeds content; GET /Knowledge/{id}/Status and GET /Knowledge/{id}/Chunks let you inspect ingestion progress and what's in the index.

Can I run multiple assistants for different use cases?

Yes — assistants are per-account resources. Spin up one per use case (support triage, sales qualifier, voice IVR replacement, etc.), attach the tools and knowledge each needs, and route inbound conversations to the right one from your application code.

Ship your first agent.

Open an account, create an assistant, attach a tool and a knowledge base, and put it on a call with <Connect><ConversationRelay>. The Twilio AI Assistants SDK works as-is against voiceml.voicetel.com.