Independent benchmark · October 2026

Six decision models, one inbox

We gave the same 1,565 business emails to six decision models from four companies. Each model had to put each email into one of ten categories and say how sure it was.

1,565emails

6models, 4 companies

9,390calls, zero errors

10categories

The short version

Perplexity's new model wins almost everything

Perplexity v1.1 is the most accurate and the cheapest. It beats Jev clearly on the real emails and on the emails where we know the answer for certain.
Jev gives the most "certain" answers that are always right: 759 answers at 99% or higher, zero wrong.
OpenAI is strong on clean emails but weak on messy real ones. Overall it ties Jev.
Clef is accurate but always too modest about it. Clef-flash is the weakest.
One trick email fooled all six models.

How to read this

Four numbers, in plain words

Accuracy: How often the model picked the right category.
Auto-routed share: Sort the answers from most sure to least sure. Go down the list until 1 answer in 100 is wrong. The share you reached is the mail you could let the model handle alone.
Calibration error: When a model says "80% sure", is it right 80% of the time? This is the average gap. 0 is perfect. Lower is better.
Latency: How long one answer takes. "Typical" is the middle answer. "Slowest 1 in 100" is the bad case.

Scoreboard

All six, side by side

Bold marks the best value in each row.

	Pplx v1.1	Pplx v1	Clef	Jev	OpenAI	Clef-flash
Accuracy	98.4%	98.0%	97.7%	96.4%	96.2%	95.7%
Accuracy, known answers	99.5%	98.1%	96.7%	95.9%	98.9%	95.3%
Auto-routed share	98.5%	97.0%	96.2%	94.4%	90.9%	83.5%
Calibration error	0.026	0.031	0.131	0.025	0.031	0.251
Typical latency	0.26s	0.26s	0.64s	0.29s	0.22s	0.34s
Cost per 1,000 emails	$0.03	$0.06	$0.41	$0.08	$0.15	$0.15

01 · Accuracy

Perplexity v1.1 is the most accurate

How often each model was right

Filled dot: all 1,565 emails. Open dot: the 364 emails where we know the answer for certain.

Numbers on the right: all emails · known answers

For the 1,201 real emails, another AI model (Claude Opus 5) decided the right answer. It never saw any model's output. For the 364 written emails, we wrote each email to fit a category, so the answer is certain.

Where Perplexity v1.1 and Jev disagree on those certain answers, Perplexity v1.1 is right 14 times and Jev once. That is statistically significant (p = 0.001). OpenAI also beats Jev there, 11 to 0, but it loses to Jev on the real emails.

02 · Automation

How much mail can each model handle alone?

Share of mail handled automatically at 99% precision

Most sure answers first. Stop before 1 answer in 100 is wrong.

Perplexity v1.1 can handle 98.5% of the inbox with 99 answers in 100 right. Jev can handle 94.4%. Clef-flash can handle 83.5%.

03 · Honesty

Does "80% sure" mean right 80% of the time?

Said vs. actual

Dashed line: a perfectly honest model. Above the line: the model is better than it says.

Perplexity v1.1

calibration error 0.026

Perplexity v1

calibration error 0.031

Clef

calibration error 0.131

Jev

calibration error 0.025

OpenAI

calibration error 0.031

Clef-flash

calibration error 0.251

Each dot is a group of answers · bigger dots hold more emails · groups under 10 emails hidden

Jev, both Perplexity models and OpenAI sit close to the line. Their numbers mean what they say.

Clef and Clef-flash sit far above the line. When Clef says 75%, it is right 98% of the time. When Clef-flash says 65%, it is right 99% of the time. You can still rank their answers, but you cannot read their numbers as real chances.

Answers at 99% sure or higher

Out of 1,565. Red: how many of those were wrong.

Jev said "99% or more" 759 times and was right every time. OpenAI said it 643 times and was wrong once. Perplexity v1.1 said it 323 times and was wrong once. The old Perplexity v1 almost never went that high. The two Clef models never did.

04 · Speed and cost

All fast, all cheap. Perplexity v1.1 is the cheapest.

Typical time per email

The middle answer of 1,565, from the same server.

Slowest 1 in 100

The bad case. This is what a waiting user feels.

OpenAI is the fastest in the typical case (0.22s) but its slow cases take twice as long as Jev's or Perplexity's. Clef is the slowest model in both views.

Cost per 1,000 emails

From each company's own token counts. None of them charge for output.

For comparison: two Gemini chat models cost $0.80 and $1.79 per 1,000 of these same emails in September.

05 · The trick email

One email fooled all six

One of the written emails is a public tender for office cleaning. It mentions a "Leistungsverzeichnis" (a bill of quantities), so it looks like a real tender. But the supplier does not clean offices, so the right answer is "not relevant". We wrote it to be hard on purpose.

How sure each model was of its wrong answer

All six picked "bill of quantities". The right answer was "not relevant".

Perplexity v1.1 was the most sure, at 99.3%. This is the one mistake that a "99% or higher" rule would have let through for Perplexity v1.1.

06 · Model by model

What stood out for each model

Perplexity v1.1

pplx-decider-v1.1-27b · released Oct 5

Best overall

Most accurate, most mail handled alone, and the cheapest at $0.03 per 1,000 emails. The price is half of v1.

It is much bolder than v1. v1 never went above 99.0% sure. v1.1 goes up to 99.9%. That made it more useful, and it also let it be confidently wrong once.

Perplexity v1

pplx-decider-v1-27b · Oct 1

Still available

Very good, but careful to a fault. Its highest answer in 1,565 emails was 99.0%. So a "99% or higher" rule would almost never fire.

Its open-source code is named autojev and accepts jev-1.13.0 as a model name.

Jev

TypeSafe · jev-1.13.0

Most trustworthy "certain"

The only model that said "99% or more" hundreds of times and was never wrong. Best calibration score, by a small margin.

Now behind on accuracy. Very stable: only 4 of 1,565 answers changed in two weeks. Its model name carries a full version number, so you know what you are calling.

OpenAI

gpt-6-luna · Decisions API beta

Clean vs. messy

Second best on the clean written emails (98.9%). Worst on real emails (95.4%). It called 28 real orders and quote requests "not relevant". Several we checked were forwarded emails full of signatures and logos.

It was unsure on all 28 (30% to 82%), so a threshold would send them to a person. It is a general chat model behind a decisions endpoint, not a dedicated decision model.

Clef

Cloudflare · 27B

Too modest

Accurate (97.7%), but it never says it is sure. Its highest answer was 97.2%. When it says 75%, it is right 98% of the time.

The slowest and most expensive of the six. Cloudflare says it is 2.5x faster than Jev. From our server it was 2.2x slower.

Clef-flash

Cloudflare · 9B

Weakest

The least accurate and the most modest. It said "90% sure" on only 3% of emails, but it was right 96% of the time.

At 99% precision it can handle 83.5% of the inbox. Every other model handles more than 90%.

Limits

What this does not prove

One task. Sorting business email into ten categories. Other tasks can rank the models differently.
No person checked most answers. For the 1,201 real emails, an AI model decided what was right.
Text only. 84% of the real emails had attachments. No model saw them.
The written emails are cleaner than real mail. OpenAI shows the biggest gap between the two sets.
Models can change. Jev and Perplexity name their versions; OpenAI and Cloudflare do not. Perplexity v1.1 and OpenAI were run on October 7, 2026; the other four on October 2, 2026.

In one sentence

Perplexity v1.1 is the best and cheapest decision model on this task, Jev is still the one whose "certain" you can trust without testing, and OpenAI does well on clean email but stumbles on real inboxes.

Method. 1,565 emails: 1,201 real ones from four company mailboxes and 364 written to cover rare categories. Every model got the same text and one "pick one of 10" question with the same category definitions. Models: jev-1.13.0, pplx-decider-v1-27b, pplx-decider-v1.1-27b, gpt-6-luna (OpenAI Decisions API), clef, clef-flash, all called from one US cloud server. Confidence is each model's highest probability; the companies' own "confidence" fields were ignored because each one computes them differently. Significance by McNemar's exact test on paired answers.