> Perplexity v1 and v1.1, OpenAI, Jev, Cloudflare Clef and Clef-flash on the same 1,565 business emails. **Nikhil Mudholkar** · Published 2026-10-07 · https://nikhilmudholkar.com/experiments/six-decision-models/ Thread on X: https://x.com/nikhilmudholkar/status/2107779079361434022 --- Independent benchmark · October 2026 # Six decision models, one inbox We gave the same 1,565 business emails to six decision models from four companies. Each model had to put each email into one of ten categories and say how sure it was. **1,565** emails **6** models, 4 companies **9,390** calls, zero errors **10** categories The short version ## Perplexity's new model wins almost everything - **Perplexity v1.1** is the most accurate and the cheapest. It beats Jev clearly on the real emails and on the emails where we know the answer for certain. - **Jev** gives the most "certain" answers that are always right: 759 answers at 99% or higher, zero wrong. - **OpenAI** is strong on clean emails but weak on messy real ones. Overall it ties Jev. - **Clef** is accurate but always too modest about it. **Clef-flash** is the weakest. - One trick email fooled **all six** models. How to read this ## Four numbers, in plain words - Accuracy How often the model picked the right category. - Auto-routed share Sort the answers from most sure to least sure. Go down the list until 1 answer in 100 is wrong. The share you reached is the mail you could let the model handle alone. - Calibration error When a model says "80% sure", is it right 80% of the time? This is the average gap. 0 is perfect. Lower is better. - Latency How long one answer takes. "Typical" is the middle answer. "Slowest 1 in 100" is the bad case. Scoreboard ## All six, side by side Bold marks the best value in each row. | | Pplx v1.1 | Pplx v1 | Clef | Jev | OpenAI | Clef-flash | | ----------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------ | | Accuracy | 98.4% | 98.0% | 97.7% | 96.4% | 96.2% | 95.7% | | Accuracy, known answers | 99.5% | 98.1% | 96.7% | 95.9% | 98.9% | 95.3% | | Auto-routed share | 98.5% | 97.0% | 96.2% | 94.4% | 90.9% | 83.5% | | Calibration error | 0.026 | 0.031 | 0.131 | 0.025 | 0.031 | 0.251 | | Typical latency | 0.26s | 0.26s | 0.64s | 0.29s | 0.22s | 0.34s | | Cost per 1,000 emails | $0.03 | $0.06 | $0.41 | $0.08 | $0.15 | $0.15 | 01 · Accuracy ## Perplexity v1.1 is the most accurate How often each model was right Filled dot: all 1,565 emails. Open dot: the 364 emails where we know the answer for certain. Numbers on the right: all emails · known answers For the 1,201 real emails, another AI model (Claude Opus 5) decided the right answer. It never saw any model's output. For the 364 written emails, we wrote each email to fit a category, so the answer is certain. Where Perplexity v1.1 and Jev disagree on those certain answers, Perplexity v1.1 is right 14 times and Jev once. That is statistically significant (p = 0.001). OpenAI also beats Jev there, 11 to 0, but it loses to Jev on the real emails. 02 · Automation ## How much mail can each model handle alone? Share of mail handled automatically at 99% precision Most sure answers first. Stop before 1 answer in 100 is wrong. Perplexity v1.1 can handle 98.5% of the inbox with 99 answers in 100 right. Jev can handle 94.4%. Clef-flash can handle 83.5%. 03 · Honesty ## Does "80% sure" mean right 80% of the time? Said vs. actual Dashed line: a perfectly honest model. Above the line: the model is better than it says. Perplexity v1.1 calibration error 0.026 Perplexity v1 calibration error 0.031 Clef calibration error 0.131 Jev calibration error 0.025 OpenAI calibration error 0.031 Clef-flash calibration error 0.251 Each dot is a group of answers · bigger dots hold more emails · groups under 10 emails hidden Jev, both Perplexity models and OpenAI sit close to the line. Their numbers mean what they say. Clef and Clef-flash sit far above the line. When Clef says 75%, it is right 98% of the time. When Clef-flash says 65%, it is right 99% of the time. You can still rank their answers, but you cannot read their numbers as real chances. Answers at 99% sure or higher Out of 1,565. Red: how many of those were wrong. Jev said "99% or more" 759 times and was right every time. OpenAI said it 643 times and was wrong once. Perplexity v1.1 said it 323 times and was wrong once. The old Perplexity v1 almost never went that high. The two Clef models never did. 04 · Speed and cost ## All fast, all cheap. Perplexity v1.1 is the cheapest. Typical time per email The middle answer of 1,565, from the same server. Slowest 1 in 100 The bad case. This is what a waiting user feels. OpenAI is the fastest in the typical case (0.22s) but its slow cases take twice as long as Jev's or Perplexity's. Clef is the slowest model in both views. Cost per 1,000 emails From each company's own token counts. None of them charge for output. For comparison: two Gemini chat models cost $0.80 and $1.79 per 1,000 of these same emails in September. 05 · The trick email ## One email fooled all six One of the written emails is a public tender for office cleaning. It mentions a "Leistungsverzeichnis" (a bill of quantities), so it looks like a real tender. But the supplier does not clean offices, so the right answer is "not relevant". We wrote it to be hard on purpose. How sure each model was of its wrong answer All six picked "bill of quantities". The right answer was "not relevant". Perplexity v1.1 was the most sure, at 99.3%. This is the one mistake that a "99% or higher" rule would have let through for Perplexity v1.1. 06 · Model by model ## What stood out for each model ### Perplexity v1.1 pplx-decider-v1.1-27b · released Oct 5 Best overall Most accurate, most mail handled alone, and the cheapest at $0.03 per 1,000 emails. The price is half of v1. It is much bolder than v1. v1 never went above 99.0% sure. v1.1 goes up to 99.9%. That made it more useful, and it also let it be confidently wrong once. ### Perplexity v1 pplx-decider-v1-27b · Oct 1 Still available Very good, but careful to a fault. Its highest answer in 1,565 emails was 99.0%. So a "99% or higher" rule would almost never fire. Its open-source code is named `autojev` and accepts `jev-1.13.0` as a model name. ### Jev TypeSafe · jev-1.13.0 Most trustworthy "certain" The only model that said "99% or more" hundreds of times and was never wrong. Best calibration score, by a small margin. Now behind on accuracy. Very stable: only 4 of 1,565 answers changed in two weeks. Its model name carries a full version number, so you know what you are calling. ### OpenAI gpt-6-luna · Decisions API beta Clean vs. messy Second best on the clean written emails (98.9%). Worst on real emails (95.4%). It called 28 real orders and quote requests "not relevant". Several we checked were forwarded emails full of signatures and logos. It was unsure on all 28 (30% to 82%), so a threshold would send them to a person. It is a general chat model behind a decisions endpoint, not a dedicated decision model. ### Clef Cloudflare · 27B Too modest Accurate (97.7%), but it never says it is sure. Its highest answer was 97.2%. When it says 75%, it is right 98% of the time. The slowest and most expensive of the six. Cloudflare says it is 2.5x faster than Jev. From our server it was 2.2x slower. ### Clef-flash Cloudflare · 9B Weakest The least accurate and the most modest. It said "90% sure" on only 3% of emails, but it was right 96% of the time. At 99% precision it can handle 83.5% of the inbox. Every other model handles more than 90%. Limits ## What this does not prove - **One task.** Sorting business email into ten categories. Other tasks can rank the models differently. - **No person checked most answers.** For the 1,201 real emails, an AI model decided what was right. - **Text only.** 84% of the real emails had attachments. No model saw them. - **The written emails are cleaner than real mail.** OpenAI shows the biggest gap between the two sets. - **Models can change.** Jev and Perplexity name their versions; OpenAI and Cloudflare do not. Perplexity v1.1 and OpenAI were run on October 7, 2026; the other four on October 2, 2026. In one sentence Perplexity v1.1 is the best and cheapest decision model on this task, Jev is still the one whose "certain" you can trust without testing, and OpenAI does well on clean email but stumbles on real inboxes. **Method.** 1,565 emails: 1,201 real ones from four company mailboxes and 364 written to cover rare categories. Every model got the same text and one "pick one of 10" question with the same category definitions. Models: `jev-1.13.0`, `pplx-decider-v1-27b`, `pplx-decider-v1.1-27b`, `gpt-6-luna` (OpenAI Decisions API), `clef`, `clef-flash`, all called from one US cloud server. Confidence is each model's highest probability; the companies' own "confidence" fields were ignored because each one computes them differently. Significance by McNemar's exact test on paired answers.