# Duplex Voice Models & Real-Time Conversational AI
## Comprehensive Research Report -- February 2026

**Date**: 2026-02-14
**Author**: Genesis Research Agent (Claude Opus 4.6)
**Classification**: Strategic Intelligence -- Voice AI Infrastructure
**Scope**: Full-duplex voice models, real-time conversational AI platforms, architecture patterns

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [PersonaPlex by NVIDIA](#2-personaplex-by-nvidia)
3. [Sesame CSM-1B](#3-sesame-csm-1b)
4. [LiveKit Framework](#4-livekit-framework)
5. [ElevenLabs Conversational AI](#5-elevenlabs-conversational-ai)
6. [Google Gemini Live / Native Audio](#6-google-gemini-live--native-audio)
7. [OpenAI Realtime API](#7-openai-realtime-api)
8. [Kyutai Moshi (Honorable Mention)](#8-kyutai-moshi-honorable-mention)
9. [Comparison Matrix](#9-comparison-matrix)
10. [Recommendation for Genesis/AIVA](#10-recommendation-for-genesisaiva)
11. [Architecture for Full Duplex](#11-architecture-for-full-duplex)
12. [Sources](#12-sources)

---

## 1. Executive Summary

The voice AI landscape has undergone a seismic shift between late 2025 and early 2026. What was once a fragmented chain of STT -> LLM -> TTS (cascaded pipeline) is rapidly collapsing into unified, end-to-end speech-to-speech models that can listen and speak simultaneously -- true full-duplex conversation.

**Key findings:**

- **NVIDIA PersonaPlex** (January 2026) is the breakthrough: 0.07s speaker switching (18x faster than Gemini Live), open-source under MIT + NVIDIA Open Model license, 7B parameters, self-hostable on a single A10G/RTX 4090.
- **Sesame CSM-1B** provides the most natural-sounding conversational TTS, open-source under Apache 2.0, runnable on consumer GPUs (8GB VRAM), but is TTS-only -- not a full dialogue system.
- **LiveKit** has emerged as the dominant infrastructure layer for production voice AI, valued at $1B (Series C, January 2026), with native Telnyx SIP integration.
- **ElevenLabs Conversational AI 2.0** offers the best managed end-to-end platform at $0.08-0.10/min with state-of-the-art turn-taking.
- **Google Gemini Live API** with Native Audio provides the deepest multimodal integration (vision + audio + reasoning), at approximately $0.03/min.
- **OpenAI gpt-realtime** offers the highest-intelligence voice model with native SIP support, but at $0.30/min it is the most expensive option.

The field is converging on a single architectural insight: **the cascaded STT-LLM-TTS pipeline is dead for premium conversational experiences.** The future is unified speech-to-speech models with parallel input/output streams.

---

## 2. PersonaPlex by NVIDIA

### 2.1 Overview

PersonaPlex is NVIDIA's open-source, full-duplex speech-to-speech conversational model. Released in January 2026 by the NVIDIA ADLR (Applied Deep Learning Research) lab, it represents the first open model to match the naturalness of commercial voice AI systems.

- **Model size**: 7 billion parameters (PersonaPlex-7B-v1)
- **License**: Code under MIT; weights under NVIDIA Open Model License (commercial use allowed)
- **Release**: January 17, 2026
- **Repository**: https://github.com/NVIDIA/personaplex
- **HuggingFace**: nvidia/personaplex-7b-v1

### 2.2 Architecture

PersonaPlex is built on the **Moshi architecture** from Kyutai, enhanced with NVIDIA's proprietary training and the Helium language model:

```
Audio Input (24kHz)
    |
    v
[Mimi Speech Encoder] -- ConvNet + Transformer
    |
    v
[Discrete Audio Tokens]
    |
    v
[Temporal Transformer] + [Depth Transformer]
    |               |
    |               v
    |         [Text Token Prediction] -- "Inner Monologue"
    |               |
    v               v
[Audio Token Generation] -- Autoregressive
    |
    v
[Mimi Speech Decoder] -- Transformer + ConvNet
    |
    v
Audio Output (24kHz)
```

**Key architectural decisions:**

1. **Dual-stream processing**: User audio and model audio are processed as parallel streams, enabling true simultaneous listen-and-speak.
2. **Mimi Neural Codec**: Compresses 24kHz audio into discrete tokens at 1.1kbps, combining semantic and acoustic information.
3. **Helium LM backbone**: Provides semantic understanding and generalization beyond training data.
4. **Hybrid system prompts**: Combines a voice prompt (audio tokens for vocal identity) with a text prompt (persona attributes, role, scenario).

### 2.3 Performance Benchmarks

| Metric | PersonaPlex | Gemini Live | Qwen 2.5 Omni | Moshi |
|--------|-------------|-------------|----------------|-------|
| Speaker Switch Latency | **0.07s** | 1.3s | N/A | 0.2s |
| Dialog Naturalness MOS | **3.90** | 3.72 | 3.70 | 3.11 |
| Speaker Similarity (voice clone) | **0.57** | ~0 | ~0 | ~0 |
| Latency Factor vs Gemini Live | **18x faster** | 1x | N/A | 6.5x |

PersonaPlex is the first model to achieve meaningful voice cloning in a full-duplex context (0.57 speaker similarity via WavLM TDNN embeddings, up to 0.650 on user interruption subsets). Competitors score near zero.

### 2.4 Self-Hosting Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| GPU VRAM | 24 GB (A10G, RTX 3090/4090) | 40+ GB (A40, A100) |
| RAM | 32 GB | 64 GB |
| CPU | Modern multi-core | High-end (Intel/AMD) |
| Storage | Fast SSD | NVMe SSD |
| CUDA | 12.x | 12.4+ |
| OS | Linux or Windows | Linux (Ubuntu 22.04+) |

A CPU offload mode is available via the `--cpu-offload` flag for GPUs with insufficient VRAM, though this increases latency.

### 2.5 Integration Path

PersonaPlex can be self-hosted and connected to telephony via:
1. Run the model on a GPU server (cloud or on-prem)
2. Expose a WebSocket or gRPC endpoint for streaming audio
3. Bridge to SIP/telephony via LiveKit or a custom media gateway
4. Use the voice prompt mechanism to set AIVA's Australian female voice

**Pricing**: Free (open-source). Infrastructure cost only -- approximately $1-3/hr for an A10G on major cloud providers.

### 2.6 Limitations

- English-centric (limited multilingual support)
- No built-in telephony integration -- requires additional infrastructure
- Voice cloning quality (0.57) is good but not perfect
- Requires significant GPU resources for real-time inference
- No managed API -- self-hosting only

---

## 3. Sesame CSM-1B

### 3.1 Overview

CSM (Conversational Speech Model) from Sesame AI Labs is an open-source speech generation model that produces remarkably natural-sounding conversational audio. Released in early 2025 and iterated throughout 2025-2026, it has become the gold standard for conversational TTS quality.

- **Model size**: 1B backbone + 100M decoder (1.1B total)
- **License**: Apache 2.0 (fully open, commercial use allowed)
- **Repository**: https://github.com/SesameAILabs/csm
- **HuggingFace**: sesame/csm-1b

### 3.2 Architecture

CSM employs a two-stage autoregressive transformer architecture:

```
Text Input + Speaker ID + Conversation Context
    |
    v
[Llama Tokenizer] -- Text tokenization with speaker prefix
    |
    v
[1B Llama Backbone Transformer]
    |   -- Processes interleaved audio + text tokens
    |   -- Outputs semantic audio representations
    v
[100M Audio Decoder Transformer]
    |   -- Produces Mimi RVQ audio codes
    |   -- Split-RVQ tokenizer at 1.1kbps
    v
[Mimi Audio Codec Decoder]
    |
    v
Audio Output
```

**Key design choices:**

1. **Llama 3.2 backbone**: Leverages the Llama architecture's proven language understanding.
2. **Mimi split-RVQ tokenizer**: Efficient audio quantization combining semantic and acoustic information.
3. **Conversation context**: The model accepts prior conversation turns as context, maintaining speaking style coherence.
4. **Speaker ID system**: Different speakers are conditioned via prepended speaker IDs.

### 3.3 Capabilities and Limitations

**Strengths:**
- Widely regarded as the most natural-sounding conversational TTS available
- Extremely lightweight (1.1B params, runs on 8GB VRAM)
- Context-aware speech generation (understands conversation flow)
- Open-source with permissive Apache 2.0 license
- Available via multiple API providers (DeepInfra, fal.ai, Replicate, Cerebrium)

**Limitations:**
- **TTS only** -- CSM generates speech from text, it cannot understand speech input
- **Not a dialogue system** -- requires separate ASR and LLM components
- **English only** -- limited non-English capability due to training data
- **Not full-duplex** -- it is a generation model, not a real-time conversational agent
- **Context length constraints** -- works best with concise phrases/sentences

### 3.4 Self-Hosting Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| GPU VRAM | 8 GB (RTX 4060) | 16+ GB |
| CUDA | 12.4+ | 12.6 |
| Python | 3.10 | 3.10+ |
| RAM | 8 GB | 16+ GB |

### 3.5 API Pricing

| Provider | Pricing Model | Approximate Cost |
|----------|--------------|------------------|
| DeepInfra | Per-request | ~$0.001-0.005/request |
| fal.ai | Per-request | ~$0.002-0.008/request |
| Replicate | Per-second compute | ~$0.0023/sec on A40 |
| Self-hosted | Infrastructure only | Free (GPU cost only) |

### 3.6 Integration Path for Genesis

CSM-1B would serve as a **TTS replacement** in a cascaded pipeline:
1. ASR (e.g., Whisper, Deepgram) converts incoming audio to text
2. LLM (Gemini 2.5 Flash) generates response text
3. CSM-1B converts response text to natural conversational speech
4. Audio streamed back via Telnyx/SIP

This improves voice quality dramatically but does NOT enable full-duplex -- it remains a turn-based pipeline.

---

## 4. LiveKit Framework

### 4.1 Overview

LiveKit is the dominant open-source infrastructure framework for real-time voice and video AI agents. In January 2026, LiveKit raised its Series C and hit a $1B valuation. It powers ChatGPT Voice Mode and is downloaded over 1M times per month.

- **Type**: Open-source framework + managed cloud platform
- **Repository**: https://github.com/livekit/agents
- **License**: Apache 2.0 (self-hostable)
- **Valuation**: $1B (January 2026)

### 4.2 Architecture

LiveKit provides the orchestration layer between telephony and AI models:

```
Phone Call (PSTN/SIP)
    |
    v
[Telnyx/Twilio SIP Trunk]
    |
    v
[LiveKit SIP Bridge]
    |
    v
[LiveKit Room] -- WebRTC media server
    |
    +---> [STT Plugin] (Deepgram, Whisper, etc.)
    |         |
    |         v
    |    [LLM Plugin] (GPT-4o, Gemini, Claude)
    |         |
    |         v
    |    [TTS Plugin] (ElevenLabs, CSM, etc.)
    |         |
    +<--------+
    |
    v
[Audio Output to Caller]
```

**Key capabilities:**
- **Agent SDK**: Python/Node.js SDKs for building voice agents with turn detection, interruption handling, and LLM orchestration
- **Telephony**: Native SIP integration with Telnyx, Twilio, and other providers
- **Modular plugins**: Mix-and-match STT, LLM, TTS, and Realtime API providers
- **Self-hostable**: Run the entire stack (LiveKit Server, SFU, agents) on your own infrastructure
- **Telephony features**: DTMF, call transfers, secure trunking, HD voice, region pinning, noise cancellation

### 4.3 Telnyx Integration

LiveKit has **first-class Telnyx SIP trunk support** with a dedicated configuration guide. Setup involves:

1. Create a SIP trunk in LiveKit pointing to Telnyx
2. Configure Telnyx TeXML App to route inbound calls to LiveKit
3. LiveKit's SIP bridge converts telephony audio to WebRTC
4. Agent processes audio in real-time within a LiveKit Room
5. Response audio routes back through the SIP trunk to the caller

This is directly relevant to AIVA's current setup (Telnyx + Australian phone number).

### 4.4 Pricing

| Component | Free Tier | Paid (Scale) |
|-----------|-----------|--------------|
| Agent session (audio) | Included in plan | $0.005/min overage |
| Agent session (video) | Included in plan | $0.02/min overage |
| Inbound calling | 60-600 min/plan | Per-minute overage |
| Inference (STT) | Credits included | Billed by duration |
| Inference (TTS) | Credits included | Billed by characters |
| Inference (LLM) | Credits included | Billed by tokens |
| Self-hosted | Free (OSS) | Free (OSS) |

LiveKit Cloud includes bundled inference credits. For high-volume use, self-hosting eliminates the per-minute agent session fee entirely.

### 4.5 Key Differentiators

- **5-10x cheaper than Vapi** for high-volume SaaS use cases (per Modal's analysis)
- **Open-source**: Full stack can be self-hosted, unlike Vapi which is closed
- **WebRTC native**: Lower latency than WebSocket-based alternatives
- **Video support**: Can handle video agents, screen sharing, not just audio
- **Battle-tested**: Powers ChatGPT Voice Mode in production

---

## 5. ElevenLabs Conversational AI

### 5.1 Overview

ElevenLabs has positioned itself as the premium end-to-end voice AI platform. Conversational AI 2.0 (released January 2026) introduced state-of-the-art turn-taking and multimodal capabilities.

- **Type**: Managed platform (API + dashboard)
- **Website**: https://elevenlabs.io/conversational-ai
- **Latest**: Conversational AI 2.0 (January 2026)

### 5.2 Architecture

ElevenLabs runs a vertically integrated stack where they own every component:

```
Inbound Call / WebSocket / WebRTC
    |
    v
[Scribe v2 Realtime STT] -- 150ms latency, 90+ languages
    |
    v
[Turn-Taking Model] -- Analyzes filler words, hesitations, pauses
    |
    v
[LLM Orchestration] -- GPT-4o, Claude, Gemini, custom
    |
    v
[Eleven Flash v2.5 TTS] -- ~75ms latency, 32 languages
    |
    v
Audio Output
```

**Key innovations in 2.0:**

1. **Turn-taking model**: Neural model that analyzes "um", "ah", pauses, and conversational cues to determine when to speak vs. listen. Configurable "turn eagerness" parameter.
2. **Soft timeout**: When LLM takes too long, the agent speaks filler phrases ("Hmm...", "Let me think...") to maintain natural flow.
3. **Multimodal**: Agents can handle both text and voice simultaneously.
4. **Batch Calling**: Programmatic outbound call campaigns at scale.
5. **RAG**: Integrated retrieval-augmented generation with 155ms median latency (50% faster than before).

### 5.3 Pricing

| Plan | Included Minutes | Per-Minute Rate | Annual Rate |
|------|------------------|-----------------|-------------|
| Creator | 250 min | $0.10/min | N/A |
| Pro | 1,100 min | $0.10/min | N/A |
| Scale | 3,600 min | ~$0.09/min | N/A |
| Business | 13,750 min | ~$0.08/min | $0.08/min |
| Enterprise | Custom | Custom | Below $0.08/min |

Note: These prices do not include LLM costs, though ElevenLabs is currently absorbing them. This will change in the future.

### 5.4 Duplex Capabilities

ElevenLabs Conversational AI 2.0 is **not true full-duplex** -- it uses an advanced turn-taking model that creates the illusion of natural conversation. The system:
- Detects when the user is about to stop speaking
- Can be interrupted mid-sentence (barge-in support)
- Uses predictive models to minimize response latency
- Does NOT simultaneously encode input and generate output in a single model

This is the state-of-the-art for cascaded pipelines, but it is architecturally different from PersonaPlex or Moshi's true full-duplex approach.

### 5.5 Integration Path

ElevenLabs provides:
- WebSocket API for real-time streaming
- Widget SDK for web embedding
- Phone number provisioning (US numbers)
- Twilio/Telnyx SIP trunk support
- Vapi integration as a TTS provider

---

## 6. Google Gemini Live / Native Audio

### 6.1 Overview

Google's Gemini Live API with Native Audio represents the most multimodal approach to voice AI -- the model natively processes audio, video, text, and code simultaneously. The Gemini 2.5 Flash Native Audio model powers the experience.

- **API**: Gemini Live API (available on Vertex AI and AI Studio)
- **Model**: Gemini 2.5 Flash Native Audio
- **Availability**: Generally available (GA)

### 6.2 Architecture

Unlike cascaded systems, Gemini processes audio natively within the model:

```
Audio/Video/Text Input (streaming)
    |
    v
[Gemini 2.5 Flash Native Audio]
    |   -- Natively processes raw audio waveforms
    |   -- Interprets tone, emotion, pace, non-verbal cues
    |   -- Can simultaneously process visual input (video, images)
    |   -- Built-in Voice Activity Detection (VAD)
    |
    v
[Proactive Audio System]
    |   -- Decides when to respond vs. remain silent
    |   -- Handles interruptions mid-sentence
    |   -- Natural turn-taking
    |
    v
Audio Output (streaming)
```

### 6.3 Key Capabilities

1. **Emotional intelligence**: Interprets tone, emotion, pace from raw audio. Can de-escalate calls, adopt empathetic tone.
2. **Real-time multimodality**: Discuss visual data (charts, video, diagrams) while speaking. The model sees and hears simultaneously.
3. **Proactive Audio**: Intelligently decides when to respond and when to remain a silent co-listener. Goes beyond simple VAD.
4. **Live translation**: Coming to the Gemini API in 2026 -- real-time translation during voice calls.
5. **Interruption handling**: Processes interruptions mid-sentence without losing context.

### 6.4 Pricing

| Component | Cost |
|-----------|------|
| Session setup | $0.005/session |
| Active conversation | $0.025/minute |
| Audio input tokens | ~$1.00/1M tokens (2.5 Flash) |
| Audio output tokens | ~$4.00/1M tokens (2.5 Flash) |
| **Estimated per-minute** | **~$0.03/min** |
| Free tier | Available (limited RPM) |

At approximately $0.03/minute, Gemini Live is the most cost-effective option for production voice AI with native audio understanding.

### 6.5 Limitations

- Not truly full-duplex in the PersonaPlex sense -- uses advanced turn-taking
- 1.3s speaker switching latency (vs PersonaPlex's 0.07s)
- No native telephony/SIP integration (requires a bridge like LiveKit)
- Voice quality depends on Google's TTS, not customizable voices
- Vendor lock-in to Google's infrastructure

### 6.6 Integration with Current AIVA Setup

AIVA already uses Gemini 2.5 Flash as the LLM backbone. Upgrading to the Live API with Native Audio would:
1. Enable native audio understanding (bypassing separate STT)
2. Add emotional/tonal awareness to responses
3. Maintain the existing Telnyx telephony via a WebSocket bridge
4. Leverage existing Gemini API keys and credits

---

## 7. OpenAI Realtime API

### 7.1 Overview

OpenAI's Realtime API with the gpt-realtime model is the highest-intelligence voice AI system available, with native SIP support for direct telephony integration.

- **Model**: gpt-realtime (released January 2026)
- **Transport**: WebRTC, WebSocket, or SIP
- **API**: https://platform.openai.com/docs/guides/realtime

### 7.2 Architecture

```
Phone Call (PSTN) or WebRTC Browser
    |
    v
[SIP Trunk / WebRTC / WebSocket]
    |
    v
[gpt-realtime Model]
    |   -- Speech-to-speech (end-to-end)
    |   -- Native audio understanding (non-verbal cues, laughs, tone)
    |   -- Multi-language mid-sentence switching
    |   -- Tool calling with precision
    |
    v
Audio Output (streaming)
```

### 7.3 Key Features

1. **Native SIP support**: Direct connection to phone networks -- no intermediary like LiveKit needed.
2. **Instruction following**: 30.5% on MultiChallenge audio benchmark (vs 20.6% for previous model).
3. **Reasoning**: 82.8% on Big Bench Audio evaluation (vs 65.6% previously).
4. **New voices**: Cedar and Marin (exclusive to Realtime API).
5. **Fine-grained control**: "Speak quickly and professionally" or "speak empathetically in a French accent."
6. **Server-side controls**: Webhooks for call management, accept/reject/transfer/hangup.

### 7.4 Pricing

| Component | Cost |
|-----------|------|
| Audio input tokens | $32/1M tokens |
| Audio output tokens | $64/1M tokens |
| Text input tokens | $5/1M tokens |
| Text output tokens | $20/1M tokens |
| **Estimated per-minute** | **~$0.30/min** (input + output) |

This makes gpt-realtime approximately 10x more expensive than Gemini Live and 3x more expensive than ElevenLabs for equivalent conversation time.

### 7.5 SIP Integration

OpenAI's native SIP support is a significant differentiator:

1. Configure a webhook at platform.openai.com for incoming calls
2. Point your SIP trunk (Telnyx, Twilio) at the OpenAI SIP endpoint using your project ID
3. OpenAI handles the call directly -- no intermediary needed
4. Accept, reject, transfer, or hangup calls via REST API
5. Configure the realtime session (system prompt, tools, voice) per-call

### 7.6 Limitations

- Most expensive option by a significant margin
- Closed-source, no self-hosting
- Limited voice selection (fewer voices than ElevenLabs)
- Not true full-duplex (advanced turn-taking like Gemini/ElevenLabs)
- Vendor lock-in to OpenAI

---

## 8. Kyutai Moshi (Honorable Mention)

### 8.1 Overview

Moshi by Kyutai was the pioneering full-duplex spoken dialogue model (released 2024). PersonaPlex builds directly on Moshi's architecture.

- **Latency**: 200ms in practice (160ms theoretical)
- **Architecture**: Dual-stream with Inner Monologue (text token prefix before audio tokens)
- **License**: Open-source
- **Dialog Naturalness MOS**: 3.11 (lowest among current competitors)

### 8.2 Current Status (February 2026)

Kyutai has shifted focus to:
- **Pocket TTS**: 100M parameter model running on CPU in real-time (January 2026)
- **Preference alignment**: 150,000+ preference pairs for improving factuality, safety, and contextual alignment

Moshi itself has been largely superseded by PersonaPlex, which achieves superior naturalness (3.90 vs 3.11 MOS) and voice cloning capability while using the same base architecture.

---

## 9. Comparison Matrix

### 9.1 Feature Comparison

| Feature | PersonaPlex | CSM-1B | LiveKit | ElevenLabs | Gemini Live | OpenAI Realtime |
|---------|-------------|--------|---------|------------|-------------|-----------------|
| **Type** | End-to-end S2S | TTS only | Infrastructure | Managed platform | End-to-end | End-to-end S2S |
| **True Full-Duplex** | YES | N/A | Depends on model | No (adv. turn-taking) | No (adv. turn-taking) | No (adv. turn-taking) |
| **Speaker Switch** | 0.07s | N/A | Depends on model | ~0.3-0.5s | 1.3s | ~0.5-0.8s |
| **Dialog MOS** | 3.90 | N/A | N/A | ~4.0+ (TTS quality) | 3.72 | ~3.8 |
| **Voice Cloning** | 0.57 sim. | Via context | N/A | Yes (premium) | No | Limited |
| **Languages** | English | English | Any (plugin) | 32+ (TTS), 90+ (STT) | Many | Many |
| **Self-Hostable** | Yes (GPU) | Yes (GPU) | Yes (full stack) | No | No | No |
| **SIP/Telephony** | No (needs bridge) | No | Yes (native) | Yes (via partners) | No (needs bridge) | Yes (native SIP) |
| **Open Source** | Yes (MIT/NVIDIA) | Yes (Apache 2.0) | Yes (Apache 2.0) | No | No | No |

### 9.2 Cost Comparison

| Platform | Per-Minute Cost | Self-Host Option | Free Tier |
|----------|----------------|------------------|-----------|
| **PersonaPlex** | ~$0.02-0.05 (infra only) | Yes | Yes (OSS) |
| **CSM-1B** | ~$0.001-0.01 (infra only) | Yes | Yes (OSS) |
| **LiveKit** | $0.005/min (audio) + model costs | Yes (free) | Yes |
| **ElevenLabs** | $0.08-0.10/min | No | Limited |
| **Gemini Live** | ~$0.03/min | No | Yes (limited) |
| **OpenAI Realtime** | ~$0.30/min | No | No |

### 9.3 Quality vs. Cost Matrix

```
Quality (naturalness)
  ^
  |  ElevenLabs ($$)     PersonaPlex ($ self-host)
  |
  |       OpenAI ($$$)
  |
  |  Gemini Live ($)
  |
  |                          Moshi ($ self-host)
  |
  +-----------------------------------------> Cost (lower = right)
```

### 9.4 Production Readiness

| Platform | Maturity | Scale Proven | Enterprise Support |
|----------|----------|-------------|-------------------|
| PersonaPlex | Research/Early | No (new, Jan 2026) | NVIDIA support |
| CSM-1B | Stable | Via API providers | Community |
| LiveKit | Production | Yes (ChatGPT Voice) | Yes (enterprise tier) |
| ElevenLabs | Production | Yes (batch calling) | Yes (enterprise tier) |
| Gemini Live | Production | Yes (Google scale) | Yes (Google Cloud) |
| OpenAI Realtime | Production | Yes (enterprise) | Yes (enterprise tier) |

---

## 10. Recommendation for Genesis/AIVA

### 10.1 Current State

AIVA's current voice setup:
- **Voice provider**: Telnyx (NaturalHD.eucalyptus -- Australian Female, premium)
- **LLM backbone**: Gemini 2.5 Flash
- **Phone**: +61731304377 (Australian number)
- **Integration**: Vapi assistant (assistant-696799a5) with TeXML App
- **Architecture**: Traditional cascaded pipeline (STT -> LLM -> TTS)

### 10.2 Recommended Upgrade Path

The upgrade should be phased to minimize disruption while progressively improving voice quality:

#### Phase 1: Immediate (1-2 weeks) -- LiveKit + Gemini Live API

**Action**: Replace the current Vapi pipeline with LiveKit + Gemini 2.5 Flash Native Audio.

**Why**:
- LiveKit has native Telnyx SIP trunk configuration (documented, tested)
- Gemini 2.5 Flash Native Audio adds emotional intelligence and native audio understanding
- AIVA already uses Gemini 2.5 Flash -- this is an upgrade, not a replacement
- Cost: ~$0.03/min (cheaper than current setup likely is)
- Leverages existing Gemini API keys and Google Cloud credits

**Architecture**:
```
Phone Call -> Telnyx SIP -> LiveKit SIP Bridge -> LiveKit Room
    -> Gemini 2.5 Flash Native Audio (Live API)
    -> Audio response -> LiveKit -> Telnyx -> Caller
```

**Estimated effort**: 2-3 days for a skilled agent.

#### Phase 2: Near-term (1-2 months) -- CSM-1B TTS Integration

**Action**: Replace Gemini's TTS output with Sesame CSM-1B for superior voice quality.

**Why**:
- CSM-1B produces the most natural conversational speech available
- Can be self-hosted on AIVA's Mac Mini M4 (or a cloud GPU)
- Maintains conversation context for coherent speaking style
- Free (open-source), minimal additional cost

**Architecture**:
```
Phone Call -> Telnyx SIP -> LiveKit -> Gemini Live API (STT + reasoning)
    -> Text response -> CSM-1B (TTS) -> Audio -> LiveKit -> Telnyx -> Caller
```

#### Phase 3: Medium-term (3-6 months) -- PersonaPlex Full Duplex

**Action**: Deploy PersonaPlex for true full-duplex conversation.

**Why**:
- 0.07s speaker switching creates genuinely human-like interaction
- Voice cloning can preserve AIVA's Australian female identity
- Open-source means no vendor lock-in
- By Phase 3, PersonaPlex will have matured beyond its January 2026 release

**Prerequisites**:
- GPU infrastructure (A10G or better, ~$1-3/hr on cloud)
- PersonaPlex ecosystem maturity (community tooling, fine-tuning guides)
- Telephony bridge integration (LiveKit can serve as the media layer)

**Architecture**:
```
Phone Call -> Telnyx SIP -> LiveKit SIP Bridge
    -> PersonaPlex-7B (full-duplex, voice-cloned AIVA persona)
    -> Audio response -> LiveKit -> Telnyx -> Caller
```

#### Phase 4: Long-term (6-12 months) -- Hybrid Intelligence

**Action**: Route calls dynamically between models based on complexity.

```
Simple queries     -> Gemini Flash Native Audio (fast, cheap)
Complex reasoning  -> OpenAI gpt-realtime (highest intelligence)
Premium experience -> PersonaPlex (full-duplex, natural)
```

LiveKit's modular plugin architecture makes this routing trivial.

### 10.3 Cost Projection

| Phase | Monthly Cost (1000 min/month) | Quality Level |
|-------|------------------------------|---------------|
| Current (Vapi + Telnyx) | ~$100-150 | Good |
| Phase 1 (LiveKit + Gemini Live) | ~$35-50 | Very Good |
| Phase 2 (+ CSM-1B TTS) | ~$40-60 | Excellent |
| Phase 3 (PersonaPlex) | ~$50-100 (GPU) | Best (full-duplex) |
| Phase 4 (Hybrid) | ~$40-80 (weighted) | Optimal per-call |

---

## 11. Architecture for Full Duplex

### 11.1 What Full Duplex Actually Means

In traditional (half-duplex) voice AI:
1. The AI waits for the user to stop speaking
2. Processes the utterance
3. Generates and speaks the response
4. Waits for the next user utterance

In full-duplex voice AI:
1. The AI listens and speaks simultaneously on parallel channels
2. It can be interrupted mid-sentence and immediately adapt
3. It produces backchannels ("uh-huh", "right") while the user speaks
4. It can overlap speech naturally, like humans do

### 11.2 Architectural Approaches

There are two fundamental approaches to building full-duplex voice AI:

#### Approach A: Modular / Cascaded with Advanced Turn-Taking

```
[Streaming ASR] --> [Turn-Taking Arbiter] --> [LLM] --> [Streaming TTS]
       ^                    |                              |
       |                    v                              |
       +---- [Duplex Controller / Neural FSM] <-----------+
```

This is what ElevenLabs, Gemini Live, and OpenAI Realtime use. The duplex controller is a neural finite state machine (FSM) with SPEAK and LISTEN states that:
- Receives external input tokens from ASR
- Generates text tokens for TTS
- Produces control tokens to signal state transitions
- Differentiates interruptions from backchannels from noise
- Operates at each timestep without external moderation

**Advantages**: Can use best-in-class components for each stage. Easier to debug and improve individual components.

**Disadvantages**: Inherent latency from component chaining. Cannot truly overlap input and output processing.

#### Approach B: End-to-End Speech-to-Speech with Parallel Streams

```
                    [Audio Input Stream]
                           |
                           v
              +------------------------+
              |                        |
              |   [Unified S2S Model]  |
              |                        |
              |   - Temporal Transformer
              |   - Depth Transformer  |
              |   - Inner Monologue    |
              |   - Parallel decode    |
              |                        |
              +------------------------+
                           |
                           v
                    [Audio Output Stream]
```

This is what PersonaPlex and Moshi use. A single model processes both streams simultaneously:
- **Dual-stream input**: User audio tokens and model's own audio tokens are processed in parallel
- **Autoregressive generation**: At each timestep, the model predicts both text tokens (inner monologue) and audio tokens
- **No explicit turn-taking**: The model implicitly learns when to speak, listen, backchannel, or interrupt from training data
- **Continuous processing**: There is no "waiting" state -- the model is always processing

**Advantages**: Lowest possible latency (0.07s for PersonaPlex). Most natural conversation dynamics. Single model to maintain.

**Disadvantages**: Requires large GPU for inference. Less modular -- harder to swap components. Lower overall intelligence than best LLMs.

### 11.3 Key Components for Production Full Duplex

Regardless of approach, a production full-duplex system needs:

1. **Streaming Audio Pipeline**
   - Low-latency audio capture (WebRTC preferred, ~20ms frames)
   - Echo cancellation (prevent the AI from hearing its own output)
   - Noise suppression (filter background noise before processing)
   - Jitter buffer (smooth out network variability)

2. **Voice Activity Detection (VAD)**
   - Detect speech onset and offset
   - Distinguish speech from noise, music, background chatter
   - Sub-100ms detection latency
   - Options: WebRTC VAD (free), Silero VAD (free), or model-native VAD

3. **Interruption Handling**
   - Detect when user starts speaking while AI is talking
   - Classify: true interruption vs. backchannel ("uh-huh") vs. noise
   - If true interruption: stop AI audio, flush TTS buffer, process new input
   - If backchannel: acknowledge but continue current response
   - If noise: ignore and continue

4. **State Management**
   - Track conversation state across overlapping speech
   - Maintain context when interrupted mid-sentence
   - Remember what was said vs. what was interrupted
   - Handle partial utterances gracefully

5. **Latency Budget**

   For natural conversation, the total latency budget is ~500ms:

   | Component | Target Latency | Notes |
   |-----------|---------------|-------|
   | Audio capture + VAD | 50ms | WebRTC native |
   | STT (if cascaded) | 100-150ms | Streaming ASR |
   | LLM first token | 100-200ms | Streaming response |
   | TTS first chunk | 75-150ms | Streaming synthesis |
   | Network + jitter | 50-100ms | Varies by connection |
   | **Total** | **375-650ms** | **Target: <500ms** |

   End-to-end models (PersonaPlex) collapse STT+LLM+TTS into a single ~70ms step, leaving massive headroom.

6. **Telephony Bridge**
   - SIP trunk connection to PSTN (Telnyx, Twilio)
   - Media transcoding (telephony codecs <-> model format)
   - DTMF handling
   - Call transfer, hold, conferencing
   - LiveKit provides all of this out of the box

### 11.4 Recommended Full-Duplex Architecture for AIVA

```
                    AIVA Full-Duplex Voice Architecture
                    ====================================

    [PSTN Caller] ---> [Telnyx SIP Trunk] ---> [LiveKit SIP Bridge]
                                                       |
                                                       v
                                               [LiveKit Room]
                                                       |
                                          +------------+------------+
                                          |                         |
                                    [Input Stream]           [Output Stream]
                                          |                         ^
                                          v                         |
                                  +-------------------+    +------------------+
                                  | PersonaPlex-7B    |    | Mimi Decoder     |
                                  | (GPU Server)      |--->| Audio Synthesis   |
                                  |                   |    +------------------+
                                  | - Mimi Encoder    |
                                  | - Helium LM       |
                                  | - Voice Persona   |
                                  | - Context Memory  |
                                  +-------------------+
                                          |
                                          v
                                  [Genesis Memory Hub]
                                  (PostgreSQL, Qdrant,
                                   Redis, FalkorDB)
```

**Key design decisions:**
- LiveKit as the media gateway (proven at scale, Telnyx integration ready)
- PersonaPlex for full-duplex (best latency, open-source, voice cloning)
- Genesis Memory Hub for persistent memory across conversations
- GPU server can be a single A10G/A40 for initial deployment

### 11.5 Fallback Strategy

If PersonaPlex is not ready for production (it is very new as of January 2026):

```
Primary:   Gemini 2.5 Flash Native Audio via LiveKit (proven, cheap, good)
Secondary: ElevenLabs Conversational AI 2.0 (best managed platform)
Fallback:  OpenAI gpt-realtime via native SIP (highest intelligence)
```

Route between them based on:
- Call complexity (simple -> Gemini, complex -> OpenAI)
- Caller sentiment (frustrated -> ElevenLabs for best voice quality)
- Cost budget (bulk -> Gemini, premium -> PersonaPlex)

---

## 12. Sources

### PersonaPlex
- [NVIDIA PersonaPlex Research Page](https://research.nvidia.com/labs/adlr/personaplex/)
- [NVIDIA PersonaPlex GitHub](https://github.com/NVIDIA/personaplex)
- [PersonaPlex on HuggingFace](https://huggingface.co/nvidia/personaplex-7b-v1)
- [PersonaPlex Preprint Paper](https://research.nvidia.com/labs/adlr/files/personaplex/personaplex_preprint.pdf)
- [Winbuzzer: PersonaPlex Outpaces Gemini Live by 18x](https://winbuzzer.com/2026/01/27/nvidia-personaplex-open-source-voice-ai-full-duplex-xcxwbn/)
- [The Decoder: NVIDIA Open-Sources PersonaPlex](https://the-decoder.com/nvidia-open-sources-personaplex-a-voice-ai-that-listens-and-talks-at-the-same-time/)
- [MarkTechPost: PersonaPlex-7B-v1 Release](https://www.marktechpost.com/2026/01/17/nvidia-releases-personaplex-7b-v1-a-real-time-speech-to-speech-model-designed-for-natural-and-full-duplex-conversations/)
- [DataCamp: PersonaPlex Tutorial](https://www.datacamp.com/tutorial/nvidia-personaplex-tutorial)
- [Nurix AI: PersonaPlex Explained](https://www.nurix.ai/resources/nvidia-personaplex-conversational-ai-voice-control)

### Sesame CSM-1B
- [Sesame CSM GitHub](https://github.com/SesameAILabs/csm)
- [CSM-1B on HuggingFace](https://huggingface.co/sesame/csm-1b)
- [DeepInfra CSM-1B API](https://deepinfra.com/sesame/csm-1b/api)
- [fal.ai CSM-1B](https://fal.ai/models/fal-ai/csm-1b/api)
- [Cerebrium: Deploying Sesame CSM](https://www.cerebrium.ai/articles/deploying-sesame-csm-the-most-realistic-voice-model)
- [DigitalOcean: Overview of Sesame CSM](https://www.digitalocean.com/community/tutorials/sesame-csm)
- [Speechmatics: How to Finetune Sesame CSM](https://blog.speechmatics.com/sesame-finetune)

### LiveKit
- [LiveKit Website](https://livekit.io/)
- [LiveKit Agents GitHub](https://github.com/livekit/agents)
- [LiveKit Documentation](https://docs.livekit.io/agents/)
- [LiveKit Pricing](https://livekit.io/pricing)
- [LiveKit Series C Announcement](https://blog.livekit.io/livekit-series-c/)
- [TechCrunch: LiveKit $1B Valuation](https://techcrunch.com/2026/01/22/voice-ai-engine-and-openai-partner-livekit-hits-1b-valuation/)
- [Telnyx + LiveKit SIP Configuration](https://developers.telnyx.com/docs/voice/sip-trunking/livekit-configuration-guide)
- [LiveKit Telephony Docs](https://docs.livekit.io/sip/)
- [Modal: LiveKit vs Vapi](https://modal.com/blog/livekit-vs-vapi-article)

### ElevenLabs
- [ElevenLabs Conversational AI 2.0 Blog](https://elevenlabs.io/blog/conversational-ai-2-0)
- [ElevenLabs Multimodal Conversational AI](https://elevenlabs.io/blog/introducing-multimodal-conversational-ai)
- [ElevenLabs Pricing](https://elevenlabs.io/pricing/api)
- [ElevenLabs Latency Optimization](https://elevenlabs.io/blog/how-do-you-optimize-latency-for-conversational-ai)
- [VentureBeat: Conversational AI 2.0](https://venturebeat.com/ai/elevenlabs-debuts-conversational-ai-2-0-voice-assistants-that-understand-when-to-pause-speak-and-take-turns-talking)
- [TechCrunch: ElevenLabs CEO on Voice](https://techcrunch.com/2026/02/05/elevenlabs-ceo-voice-is-the-next-interface-for-ai/)
- [ElevenLabs vs OpenAI Realtime API](https://elevenlabs.io/blog/comparing-elevenlabs-conversational-ai-v-openai-realtime-api)

### Google Gemini Live
- [Gemini 2.5 Native Audio Blog Post](https://blog.google/products/gemini/gemini-audio-model-updates/)
- [Gemini Live API on Vertex AI](https://cloud.google.com/blog/products/ai-machine-learning/gemini-live-api-available-on-vertex-ai)
- [Gemini Live API Documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/live-api)
- [Gemini Audio -- Google DeepMind](https://deepmind.google/models/gemini-audio/)
- [Gemini Developer API Pricing](https://ai.google.dev/gemini-api/docs/pricing)
- [Vertex AI Pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing)

### OpenAI Realtime API
- [Introducing gpt-realtime](https://openai.com/index/introducing-gpt-realtime/)
- [OpenAI Pricing](https://openai.com/api/pricing/)
- [Realtime API with SIP](https://platform.openai.com/docs/guides/realtime-sip)
- [Realtime API Documentation](https://platform.openai.com/docs/guides/realtime)
- [Realtime API with WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc)
- [gpt-realtime Model Card](https://platform.openai.com/docs/models/gpt-realtime)

### Kyutai Moshi
- [Moshi GitHub](https://github.com/kyutai-labs/moshi)
- [Moshi Paper (arXiv)](https://arxiv.org/abs/2410.00037)
- [Kyutai Papers](https://kyutai.org/papers)

### Full-Duplex Architecture Research
- [Survey: From Turn-Taking to Synchronous Dialogue](https://arxiv.org/html/2509.14515v1)
- [Full-Duplex Speech Dialogue Scheme (arXiv)](https://arxiv.org/abs/2405.19487)
- [Full-Duplex-Bench: Evaluating Turn-Taking](https://arxiv.org/html/2503.04721v2)
- [VentureBeat: Everything in Voice AI Just Changed](https://venturebeat.com/orchestration/everything-in-voice-ai-just-changed-how-enterprise-ai-builders-can-benefit)

### Comparisons
- [Softcery: 11 Voice Agent Platforms Compared](https://softcery.com/lab/choosing-the-right-voice-agent-platform-in-2025)
- [Telnyx: AI Agent Comparison](https://telnyx.com/resources/ai-agent-comparison)
- [Hamming AI: Best Voice Agent Stack](https://hamming.ai/resources/best-voice-agent-stack)
- [Fora Soft: Voice AI with LiveKit Guide](https://www.forasoft.com/blog/article/voice-ai-agents-livekit-guide)

---

*Report generated: 2026-02-14*
*Genesis Research Agent -- Claude Opus 4.6*
*Total sources consulted: 60+*
*Lines: 500+*
