BenchmarkIntroducing The Converse-Diarization Benchmark

The Converse-STT Benchmark

The Conversational Speech-to-Text Benchmark measures how accurately frontier AI models transcribe real-world, two-person American English conversations. 15 models, scored in collaboration with Cekura.

View Leaderboard15 Read Blog Dataset

Model

Score

01
Reson8 (Reson8)
2.93%
Pipecat 2.10% · rank 3 → 1
02
Cartesia Ink 2 (Cartesia)
3.09%
Pipecat 2.39% · rank 5 → 2
03
Smallest Pulse (Smallest)
3.46%
Pipecat 3.66% · rank 11 → 3
04
GPT Realtime Whisper (OpenAI)
3.53%
Pipecat 2.16% · rank 4 → 4
05
Google Chirp 2 (Google)
3.80%
Pipecat 2.59% · rank 7 → 5
06
Speechmatics Linden (Speechmatics)
3.93%
Pipecat 2.46% · rank 6 → 6
07
Deepgram Flux English (Deepgram)
4.18%
Pipecat 3.90% · rank 13 → 7
08
Google Chirp 3 (Google)
4.19%
Pipecat 2.00% · rank 2 → 8
09
AssemblyAI Universal 3.5 Pro (AssemblyAI)
4.23%
Pipecat 1.93% · rank 1 → 9
10
Deepgram Nova-3 (Deepgram)
4.39%
Pipecat 3.46% · rank 10 → 10
11
GPT-4o Mini Transcribe (OpenAI)
4.45%
Pipecat 3.35% · rank 9 → 11
12
Gradium (Gradium)
4.45%
Pipecat 6.48% · rank 15 → 12
13
Deepgram Flux Multilingual (Deepgram)
4.53%
Pipecat 4.96% · rank 14 → 13
14
Inworld STT-1 (Inworld)
10.60%
Pipecat 2.72% · rank 8 → 14
15
GPT-4o Transcribe (OpenAI)
12.45%
Pipecat 3.90% · rank 12 → 15

Encoding Human Expertise into Machines.

Human voice, vision, reasoning, & expertise — encoded into training data by our Expert Network and Data Foundry.

Get Data

The Converse-Diarization Benchmark

The Conversational Speaker Diarization Benchmark measures how accurately frontier AI models attribute speech to the right speaker in real-world, two-person American English conversations. 10 models, scored on diarization error rate.

View Leaderboard11 Technical Report Dataset

Metric

Model

Score

01
Pyannote Community-1 (pyannote)
27.98%
02
DiariZen WavLM Large s80 MD v2 (BUT)
28.16%
03
Nemotron 3 Diarization (NVIDIA)
28.21%
04
Scribe v2 (ElevenLabs)
30.34%
05
Voxtral Mini Transcribe 2 (Mistral)
31.22%
06
Grok Voice Transcribe 2.0 (xAI)
31.33%
07
GPT-4o Transcribe Diarize (OpenAI)
31.57%
08
MAI Transcribe 2 (direct) (Microsoft)
32.36%
09
Gemini 3.5 Transcribe (Google)
32.50%
10
Grok STT 1.0 (xAI)
34.28%
11
Muse Voice Transcribe 1.0 (Meta)
35.84%

Fueling the World's Top Frontier AI Labs
and Fortune 100 Enterprises
with Frontier AI Data

Built By Engineers From

Backed By

Research & Evals

Converse Benchmarks

We believe voice will be the natural interface between people and AI, but human speech is uniquely hard to get right. Unlike code or math, there is rarely one right answer: a good conversation depends on the setting, and every utterance carries emotion, tone, pace, accent, and language on top of its words. Converse tests whether voice-native frontier models can understand, speak, and act on real-world speech the way people do.

View all

Speaker Diarization

Converse-Diarization

Evaluates frontier AI and speaker diarization models on Real World Conversational Data, measuring diarization error rate (DER): missed, false, and mis-attributed speech.

Diarization Error Rate · lower is better

Show 6 more

Speech-to-Text (STT)

Converse-STT

Evaluates frontier AI and speech-to-text models on Real World Conversational Data and on agent-generated Pipecat audio, measuring word error rate (WER) on each.

Word Error Rate · lower is better

Show 10 more

In collaboration with

View all

The Problem

AIcansolveolympiadproblems.Itstilllackshumannuance.

We'reinaTechnologicalRenaissance.Modelshavememorizedtheinternet.Theycanwriteessays,passbarexams,andprovetheorems.Butaskonetonegotiateadeal,comfortagrievingpatient,orspeakwiththewarmthandtimingofarealhumanvoice—andtheillusionbreaks.

Humanexpertiseisstaggeringlycomplex.Itspanseverymodalityandeveryculture—howasurgeonseestheoneshadowonascanthatchangeseverything,howatraderhearsriskinapausebetweenwords,howatherapistreadsafacebeforeasinglesentenceisspoken,howmeaningshiftsbetweenlanguages,accents,anddialectsthatnomodelwastrainedtounderstand.

Noneofthiswaseverinthetrainingdata.Scalingcomputewon'tconjureit.Syntheticdatawon'tapproximateit.Thebottleneckwasneverintelligence—it'stherichnessoflivedhumanexperience.

Our Solution

Weencodehumanexpertiseintomodelsthatworkfortherealworld.

We'reanappliedresearchlabbuildingthedatainfrastructureandhumanexpertisenetworktoencodereal-worldknowledgeintofrontiermodels—acrosseverymodality,language,anddomain.

Wepartnerwitheliteprofessionalstocapturewhattheyactuallydo.Thereasoningbehindadiagnosis.Theinstinctinanegotiation.Thecadenceofanativespeaker.Theengineeringinsightinadesigndecision.Themicro-expressionsamachinehasneverbeentaughttosee.

ThisflowsthroughourDataFoundry—apurpose-builtenginethattransformsrawexpertiseintostructuredtrainingdata,alignmentsignals,andrigorousevaluationsatscale.FromPhDmathematiciansandvoiceactorstoconstitutionallawyersandlinguists,everydiscipline,accent,anddialectgetsitsownpipeline.

Expert-Level Training Data

Datasets

Alignments

Evals

Benchmarks

Data Foundry

Tasks, Tools, RL Environments, & Rubrics

Elite Expert Network

Domain Experts

Linguists

Researchers

Global Workforce

Conversational AI

Models can speak. Teaching them how to sound human is the real work.

Closing the gaps our benchmarks expose takes data recorded the way people actually talk: full-duplex captures, emotional tagging, prosodic markers, scenario-anchored conversations, human preference data, and the evaluation loops that turn raw audio into training signal. This is the catalogue we ship to the labs building the next generation of conversational AI.

Full-Duplex Conversational Datasets

Two-speaker conversations captured at 48 kHz with isolated channels, overlap, backchannels, and barge-in preserved verbatim — the training audio behind real-time, conversational voice agents.

Listening…

Domain-Specific Speech Datasets

Task-anchored sessions across medical intake, customer support, technical interviews, and emergency calls — tagged by scenario, role, and intent for vertical voice agents.

The odor of spring makes young hearts jump.

Scripted Voice Datasets

Single-speaker performance reads from voice actors and trained narrators, phonetically balanced with controlled emotion ranges and multiple takes per line — production-grade material for TTS, voice cloning, and speech-to-speech.

TranscriptionConversation 04 · 2 speakers

05:02.4A

Because there's a lot of implications there. You know, you can't do that. <breath/>

05:03.2B

Right.

05:06.3A

And you can't do a rigorous... well, I can't <breath/> do a rigorous workout, like, in the evening and then go to bed, you know?

05:09.6B

Right.

Annotation & Evaluation Datasets

Word-level transcripts, diarization, prosodic markers, scenario and role labels, continuous emotional tagging, and human preference scores — the training signal that turns raw audio into controllable, evaluable speech.

Available in 40+ languages