Architecture overview

How PolyAI agents process conversations from start to finish.

How conversations flow

When a user connects to your PolyAI agent, the conversation passes through several key stages. The exact path depends on the channel–voice (telephony), webchat, or SMS.

1. Connection layer

The connection layer handles how users reach your agent. For voice, this is telephony; for webchat, it’s HTTP/WebSocket connections. PolyAI’s infrastructure supports automatic failover across all channels.

Voice (Telephony)

PolyAI’s voice infrastructure is designed for enterprise reliability with automatic failover and redundancy. Calls are distributed across multiple media servers for continuous availability. If the PolyAI service goes down, calls automatically transfer back to your contact center – designed to minimize dropped calls.

Supported telephony providers:

• Twilio
• Amazon Connect
• Genesys
• SIP-based systems
• Custom telephony integrations

Webchat

For webchat interactions, users connect via HTTP/WebSocket:

• Instant connection: No telephony latency–conversations begin immediately
• Persistent sessions: Maintains conversation state across page reloads
• Customizable widget: Embed directly in your website or application

See also: Webchat integration

SMS

SMS interactions are handled through integrated messaging providers, allowing agents to send and receive text messages.

See also: SMS integration, Voice integrations

2. Input processing

How user input reaches the agent depends on the channel:

• Voice: Speech is converted to text using automatic speech recognition (ASR)
• Webchat/SMS: Text input is received directly–no ASR needed

Speech recognition (ASR) – Voice only

For voice interactions, the user’s speech is converted to text using automatic speech recognition (ASR). PolyAI’s platform integrates with multiple ASR providers for accuracy and coverage across use cases and languages.

Supported ASR providers:

• Google Cloud Speech-to-Text
• Google Gemini
• Amazon Transcribe
• Deepgram
• OpenAI
• Mistral
• NVIDIA Riva
• NVIDIA NeMo
• Custom ASR integrations

The platform routes requests to the best-fit provider based on language, domain, and availability, with automatic fallback if a provider is unavailable.

Key capabilities:

• Multiple languages and accents
• Industry-specific vocabulary
• Real-time transcription with low latency
• ASR biasing and keyphrase boosting for domain-specific terms
• Automatic provider failover for high availability

See also: ASR, ASR biasing, Global ASR configuration

3. Agent service

The agent service is the core of the system, powered by PolyAI’s LLM-native architecture. It receives the transcribed user input and coordinates:

• Language understanding: Uses large language models (LLMs) to interpret what the user said, their intent, and extract entities in a conversational, context-aware manner
• Decision making (Policy engine): Determines the appropriate response based on your configured FAQs, flows, and rules by executing nodes in priority order
• Knowledge retrieval: Uses RAG (Retrieval-Augmented Generation) to pull relevant information from both tabs of the Knowledge area – FAQs and Connected
• Action execution: Triggers any necessary tool calls or API integrations
• Context management: Maintains dialogue context and turn history throughout the conversation

Unlike intent-based NLU, the LLM processes the full conversation history each turn rather than classifying into fixed intents.

See also: LLM, Policy engine, Node, RAG

4. Response generation

Based on the decision engine’s output, the system generates an appropriate response using your agent’s configured voice, tone, and knowledge. This may involve:

• Retrieving relevant information using RAG (Retrieval-Augmented Generation)
• Applying global rules and response control filters
• Generating contextually appropriate responses via the LLM

See also: RAG, LLM, Response control

5. Response delivery

How responses reach the user depends on the channel:

• Voice: Text is converted to speech using TTS and streamed to the user
• Webchat/SMS: Text responses are delivered directly

Text-to-speech (TTS) – Voice only

For voice interactions, the generated response is converted to natural-sounding speech and played back to the user. PolyAI integrates with multiple TTS providers to deliver high-quality, natural-sounding voices across languages and use cases.

Supported TTS providers:

• ElevenLabs
• Amazon Polly
• Azure Speech
• Cartesia
• Google Cloud Text-to-Speech
• Hume
• MiniMax
• Neuphonic
• OpenAI
• PlayHT
• Rime
• Custom TTS integrations

Audio management and caching:

PolyAI’s audio management system optimizes user experience and reduces latency through intelligent caching:

• Audio cache: Frequently used phrases (greetings, confirmations, transfer messages) are cached for instant playback — the same audio plays every time, with no TTS latency
• Cache requirements: Audio is cached when the same utterance is generated at least twice within a 24-hour window
• Regeneration control: Edit cached audio directly in Agent Studio to adjust stability, clarity, and pronunciation
• UX optimization: Fine-tune voice quality for critical phrases without regenerating audio on every call

Additional capabilities:

• SSML markup for fine-grained control over pronunciation, pauses, and emphasis
• Custom pronunciations using IPA notation
• Multiple voice options and custom voice cloning
• Real-time audio streaming for low-latency responses

See also: TTS, SSML, Pronunciations, Audio Management

Data storage and synchronization

During and after a conversation, PolyAI captures, stores, and synchronizes several types of data to support analytics, compliance, and operational workflows.

Data types and retention

Data synchronization and access

PolyAI provides multiple methods to access and synchronize conversation data with your systems:

• Studio transcripts: Review transcripts and recordings directly in the PolyAI platform
• Conversations API: Programmatically retrieve conversation metadata, transcripts, and recordings
• AWS S3 integration: Automatically sync call data to your AWS S3 bucket for long-term storage and compliance
• Handoff metadata: Share real-time conversation state with live agents during transfers

All data handling follows enterprise security standards and can be configured to meet compliance requirements such as HIPAA, GDPR, and PCI-DSS.

See also: Dialogue context, Turn, Conversation metadata, Call data documentation

Key components you configure

As a builder in Agent Studio, you control how the agent behaves through:

• Knowledge: The Knowledge area under Build contains two tabs:
  • FAQs: Curated knowledge with fine-grained control over utterances and actions. Use for structured, stable information that requires precise agent behavior. Can trigger functions, flows, and other agentic actions.
  • Connected: Fast integration of external knowledge sources (URLs, files, Zendesk, Gladly, ServiceNow). Ideal for FAQ-style content and large volumes of continuously updated information. Cannot trigger actions or flows.
• Flows: Structured conversation paths for complex tasks
• Tools: Custom logic and external integrations
• Behavior: Global behavior constraints
• Voice settings: How the agent sounds

FAQs tab vs. Connected tab

Both tabs within the Knowledge area expose information to your agent, but serve different purposes:

Supported Sources integrations:

• Zendesk
• Gladly
• ServiceNow
• Additional integrations are in development – contact your PolyAI representative for the latest availability

If both tabs contain conflicting information, FAQs always takes priority.

See also: FAQs overview, Sources overview

Processing a single turn

Each turn in a conversation follows this sequence:

See also: Turn, Policy engine

Related resources