Last Updated August 2024

What changed with the most recent set of benchmarks?

• We made sure to test the most recent models from OpenAI, Anthropic, Google, and Mistral.
• We removed the summarization benchmark. Providing a fair comparison of how well models perform on summarization tasks requires significant nuance and is difficult to assess with a human evaluator. We recommend looking into resources like: Summarization Leaderboard and the Hallucination Leaderboard.
• For one of our extraction tasks (Charities), LLM-based solutions have higher precision but significantly lower recall than the equivalent Discriminative solution.
• For the NDA task, some LLMs outperform the discriminative solution for the first time. Gemini 1.5 Pro, Gemini 1.5 Flash lead the pack, followed by GPT-4-1106-preview and GPT-4-turbo.
• There is still a large gap between Discriminative AI and Gen AI on the CORD receipt extraction task, which includes quite granular categories. Even after breaking up the extraction task into smaller pieces for the GenAI models (with separate prompts for line items, summary information, and overall receipt metadata) there is still a gap of nearly 40 F1 points on CORD.
• Gemini 1.5 Flash and GPT4o-mini both have impressive showings for their weight class, trailing their larger counterparts by less than 5 F1 points on the extraction tasks. Gemini’s new pricing comes in at less than 1/100th the cost of the most expensive models in our benchmark, GPT4o-mini at less than 1/50th.
• While progress seems to have slowed down at the upper end of model sizes, small models continue to become more competitive and cost continues to drop, meaning “intelligence per dollar” is accelerating rapidly. These smaller models are also lower latency, so we expect these smaller models (not GPT4) to become the workhorses of the enterprise.
• Claude 3.5 Sonnet is the best overall performer if price is no object, coming in 1st on the classification benchmark and a close 3rd on the extraction benchmark.
• All major providers are competitive, with the lead OpenAI once held having all but dissipated.
• We identified an issue with the last round of benchmarks that meant LLM performance was underestimated, but have corrected the issue in this round of benchmarks.
• Providers are starting to differentiate based on feature/function rather than pure response quality. OpenAI is doubling down on “function calling,” Anthropic on better chat UX through their Projects and Artifacts features, and Gemini on cost-savings via context caching and aggressive pricing.

What hasn’t changed?

• In aggregate, discriminative models still lead the pack for Extraction tasks, and the gap is still significant: around 20 F1 points, due largely to the poor performance of LLMs.