The summary in four points.

Key numbers

Four goals.

1. Understand which LLMs can actually handle generating Qlik Set Analysis.
2. Compare models on accuracy, cost, speed, and stability.
3. Test the hypothesis: can prompt engineering bring a cheap model up to the level of an expensive one?
4. Form data-driven recommendations for a possible integration of LLMs into the product.

A two-phase scheme, a dual judge.

The tasks come from the QATA training platform. They're open: anyone can solve them and check themselves against the reference with automated grading. Real cases — no tasks invented by the researcher.

Source of the tasks

31 verified Set Analysis tasks from three domains: Sports, HR, Sales. We used the QATA platform to automatically check results against references. Access platform: OpenRouter (a single API to 300+ models), budget $20.

Phase 1 + Phase 2

Two independent judges

Every model answer was run through two LLM judges. One looked at what came out, the other at how it was written. When they diverge, a "logic gap" appears.

13 models · 4 categories.

We didn't include outdated versions (Llama 2, GPT-3.5), variant fine-tunes (for roleplay/medicine), or small models (≤8B parameters).

13 models, ranked by number match.

One standard prompt × 31 tasks. The Coincidental column shows how many times the model "guessed" the number with an expression that differs from the reference.

* DeepSeek Coder V3 excluded — API broken (0/31).

Who holds up when the prompt varies.

Top 5 models × 31 tasks × 3 prompt levels = 93 answers per model. Ranked by logic match.

Six technical findings.

count(distinct {<Sex={"M"}>} Name)
/ count(distinct Name)

Count({<Sex={'M'}>} DISTINCT ID)
/ Count(DISTINCT ID)

{<Year = {'1996'},
   Season = {'Summer'}>}

{<Games = {'1996 Summer'}>}

$17.35 for the whole benchmark.

~4,300 requests, ~2.7M tokens. 70% of the budget was eaten by the LLM-as-judge (Claude Opus in Phase 1) — on a repeat with Sonnet the cost was 14× lower for the same number of answers.

The hypothesis "use Sonnet/Haiku as the judge" was confirmed — savings of 5–14× with no loss in grading quality.

If an LLM goes into the product.

Three integration scenarios with realistic accuracy (with mandatory human review) and cost per 1,000 requests.

* With mandatory human review.

Four rules you don't go to production without.

1. Never without review. Never use it without human review or Qlik runtime validation. The best model (Gemini 2.5 Pro) — 47% strict logic; roughly every other answer needs checking.
2. Configure the reasoning models. GPT-5, Gemini 2.5 Pro require max_tokens=4000 + reasoning_effort=low. Otherwise, systematically understated results.
3. Don't overload few-shot. For most models the enriched prompt lowers accuracy. A simple prompt + strict validation works better.
4. Sonnet/Haiku as the judge. Not Opus. Savings of 5–14× with no loss in grading quality — verified on 868 answers.

Which open-source model to deploy locally?

A separate question: if a cloud LLM is off-limits under your security policy — what to run on-prem.

What we learned.

The research confirms: LLMs can generate correct Qlik Set Analysis — but with a serious caveat about the strictness of the evaluation. By the number — up to 68% for the top models; by strict equivalence to the reference — up to 47%. A separate takeaway: in roughly 18% of correct answers the model's formula is more correct than the reference — on complete datasets, counting by the key is more reliable than counting by the displayed field.

The main recommendation is to use it only in "assistant for a human" mode, not in automatic generation mode without validation. The main technical insight — about configuring reasoning models — is critically important for any team that's going to integrate GPT-5 / Gemini Pro / o1 / o3 into production.

The main methodological insight — about the dual check (number + logic) — should become the standard for any future LLM benchmarks on the team.

Quick model summary