Skip to content

Before You Buy AI for Your Warehouse

The decision

You have a warehouse. Someone wants you to switch on its AI features, or to buy a tool that sits on top of it. Both pitches quote an accuracy number, and the numbers in circulation run from about ten percent to over ninety.

The decision is not which product to buy. It is whether to spend two weeks finding out what happens on your own data first.

Two weeks is enough, because the tests that matter are cheap and you can run them yourself.

Does this apply to you?

This is for you if your company already has a data warehouse and someone is proposing AI on top of it.

It is not a product comparison. It names one vendor, purely as a worked example of how to read an accuracy claim. Rankings go stale at the next release. The method does not.

Your problem may be definitions rather than technology. If three teams each hold a defensible meaning for the same word, start with Why Your AI Gives Three Different Answers instead.

The options

Turn on what the warehouse already offers. The feature comes bundled with infrastructure you already pay for. No procurement, no new vendor.

Buy a separate tool that sits on the warehouse. A new line item, usually sold to a business function rather than to the data team.

Do neither yet, and spend two weeks measuring. The option nobody is selling you, and the only one that produces evidence about your own data.

How the options compare

Start with a wide gap in the published evidence, because it is instructive rather than scandalous.

The headline arrives with a condition already attached. Snowflake describes "an agentic AI system, coupled with a semantic model to achieve an impressive 90%+ SQL accuracy on real-world use cases" (Snowflake, 2024). SQL is the query language every report is ultimately built from.

Snowflake then tells you exactly how they measured that, which is what makes it a good worked example rather than a target. They "developed an internal benchmark suite of 150 questions" (Snowflake, 2024). They curated the tables first, into what the post calls "a flattened, wide table" shape (Snowflake, 2024). And they loosened the scoring deliberately, "setting a more lenient threshold to judge accuracy" (Snowflake, 2024).

None of that is hidden. It is all in the post. The problem is that "90%+ accuracy" travels into a board deck and the four qualifiers do not. The first is the one that matters most here: the figure was measured with a semantic model already in place.

Now the other end. BEAVER is a benchmark built from real private enterprise warehouses and real query logs, not from public sample databases. It covers 9,128 questions paired with the SQL that answers them, over 812 tables in 19 domains (Chen et al., 2026). A state-of-the-art system reached 10.8% execution accuracy, and the best unassisted result in the paper is 11.4% (Chen et al., 2026).

Do not average those two figures. They are not two estimates of one quantity, and a blended number would be wrong about both.

What the low numbers actually say

Set two unassisted measurements side by side, and the gap between them is the finding.

Snowflake set a baseline on its own internal data, on questions "requiring cross-table joins, metric formula lookups and filter convention knowledge" (Snowflake, 2026). The result: "without any context layer, AI had around 25% accuracy" (Snowflake, 2026).

Averaged across models, BEAVER's strongest framework scored 9.5% unassisted. It reached 18.9% once a human supplied the schema links, and 25.9% once a human supplied the answers to all five sub-steps (Chen et al., 2026).

Do not pair Snowflake's 25% with BEAVER's 25.9%. They look alike and they are not. One is a starting point. The other is the top rung of an annotation ladder, reached only after a human supplied the answers to every sub-step.

The closer comparison is the unassisted pair, 25% against 9.5%.

Both sit far below the 90%+ headline, and they differ by more than a factor of two.

That spread is the useful part, and so is the reason for it. These two are not strictly comparable either. They used different systems, different questions and different scoring. The 9.5% is also an average across seven models, where the 25% is a single measurement.

That is the point rather than a caveat on it. Once the system, the questions and the scoring all move, no published figure describes your situation. The only measurement that settles it is one taken on your own tables.

Treat both figures with the same suspicion, including the low one. A vendor benefits from a low no-context baseline, because the baseline is what sizes the improvement it then sells. And a benchmark's product is difficulty, since nobody publishes a test that everything passes. BEAVER is academic work, but its author list includes people at Intel and AWS. Neither number is neutral.

What the lift is actually worth

The same Snowflake post reports what their managed context layer did: "Cortex Sense improved accuracy from 24.1% to 86.3% on our benchmark" (Snowflake, 2026). Read the 24.1% carefully, because it is not the same measurement as the 25% above. It comes from a second, harder question set, and that arm already had semantic views available to retrieve. They disclose it as "Based on internal test results and cost calculations in June 2026" (Snowflake, 2026).

Treat the size of that jump as unverified, and note what the two arms were. Its direction matches the annotation ladder above, which is the nearest thing here to an independent check. That ladder measures human-supplied annotations rather than a managed layer.

The direction is what matters for your decision. Across every measurement on this page, what moves the number is the context supplied rather than the model chosen. How much context your own tables already carry is the part only you can measure.

The claim worth testing

Older versions of this promise needed someone to hand-author the context, and that authoring is where similar programmes have historically gone to die. The 2026 pitch is that this no longer applies. Snowflake reports that standing the system up "took a single day rather than the months these projects usually run" (Snowflake, 2026). The reason given is that "there are no new semantic views to author, you just connect to sources and go".

That is the claim your two weeks should test. Not whether AI can write SQL, which is settled. Not which platform is fastest, which changes quarterly. Whether the context layer builds itself well enough on your tables.

The claim is reasonable and it may well hold. It is also new, and the party selling it did the measuring. The product was announced in June 2026, and this research found no independent test of it. Those three facts together describe something to check, rather than something to believe or dismiss.

When each one wins

Turning on the warehouse feature wins when your first use case touches tables someone has already modelled, and the questions recur. It needs one more condition: a technical person reads the generated SQL before anyone acts on it.

A separate tool wins when what you need is not query generation at all. You may really be buying an interface, a workflow, or a set of connectors.

Waiting two weeks wins whenever you cannot yet answer one question. On our tables, across our twenty most common questions, how often is it right, and who would notice if it were not?

How to decide

Five steps across two weeks. Write down step one before you run anything, because a score with no pre-agreed meaning gets interpreted afterwards by whoever wants the outcome most.

1. Set the pass mark, in writing, before you test. Half a day.

Name three numbers now. Out of twenty questions, how many correct would make you buy? How many would make you wait and fix the data first? How many would make you stop?

Have the person who will read the results sign that note. This is the whole safeguard, and it costs one meeting.

There is no published threshold to borrow, because nobody has established one. The number depends entirely on step five.

2. Pick your twenty questions and get access. Days one to three.

Write down the twenty questions your team actually asks. Real ones, in the words people use.

Then ask the vendor, in writing, for a two-week trial on your own data rather than a demo on theirs. A pitch that cannot survive that request has answered your question already.

Note which tables those questions touch. If nobody can tell you, that is a finding, and it is the first thing to fix.

3. Run the stranger test on those tables. Half a day.

Read the column names of the tables from step two as a new analyst would. Count how many a newcomer could explain unaided. Record the count. That number is your starting condition.

The payoff from documenting your data depends on where you start. A warehouse vendor ran the same 500 questions with and without column descriptions, under their own scoring rather than the benchmark's (Matson, 2026). The effect was not uniform. Against its own no-comment baseline, adding comments gained 8.7 points on the most cryptically named schema and lost 2.9 points on the most intuitive one (Matson, 2026). On the clean schema they added noise to a picture the model already had right.

One more caveat, from the author. They call the harness "heavily vibe-coded, so YMMV, but the data is real" (Matson, 2026).

Read the stake carefully, because it does not point the way it first appears. The author sells a warehouse and argues for data modelling over metadata platforms. A finding that column descriptions barely help is a finding that supports the argument already being made. Treat it as a direction to check on your own tables, not a result to lean on.

4. Run the twenty questions twice. Days four to eight.

First run them as they are. Score each answer right or wrong against what the person who knows the data says is true.

Split the twenty before you start. Ten guide the work, ten are held back and not looked at.

Then spend one afternoon writing definitions, using only the first ten and what they got wrong. Have an analyst write down what your core measures mean, the conventions your filters follow, and the terms people confuse. Supply that document to the tool the way it accepts context, whether that is a pasted prompt, an uploaded file, or a configuration field.

Now run all twenty again and score them the same way. The held-back ten are the honest measurement. If you write the definitions after seeing every failure and then re-score the same questions, the improvement you get is partly just a fitted answer.

The second run is worth the afternoon because it repeats a real experiment. A semantic-layer vendor reported gains of 17.2, 22.2 and 23.2 percentage points over its own no-document baseline, across three frontier models (Rumiantsau, 2026). What produced those gains is the interesting part. It was not their platform. It was a hand-written document of measures and conventions. "No code, no runtime, no tool calls" (Rumiantsau, 2026), costing "an afternoon of an analyst's time" (Rumiantsau, 2026). The same sentence adds the condition: "But it has to capture what your metrics actually mean."

A semantic-layer vendor's best published evidence is for something an analyst could write in an afternoon, provided the analyst genuinely knows what the measures mean. Take that bargain before signing anything. The difference between the two runs on the held-back ten is the honest signal about how much of your problem is definitions rather than software.

One caveat, and it is the reason to run this yourself: 100 questions on a single retail dataset says nothing about your warehouse.

5. Price a wrong answer, then read your scores. Days nine to ten.

For your first use case, write down what it costs when the number is wrong and somebody acts on it. A wrong figure in a board pack. A wrong renewal calculation. A wrong regulatory total.

Then compare both scores against the pass mark from step one.

The cost of a wrong answer is what makes a given score acceptable. It is also why no outside number can set your threshold. Where a wrong answer is cheap and immediately visible, a modest score may still be worth having. Where a wrong answer is expensive and quiet, the accuracy you need sits above what the published benchmarks report.

Two things matter as much as the score.

Twenty questions is a small sample, so treat it as coarse. One question moves the result by five points. Around the middle of the range, the true figure could sit roughly twenty points either side of what you measured. If your score lands near a pass mark rather than clearly above or below it, run the set again before deciding, or widen it.

And whoever checked the twenty answers is the cost nobody quotes. Their time is the real unit price of a correct answer.

Watch out for

Two numbers averaged into one. The ten percent and the ninety percent measure different questions, on different tables, under different scoring rules. Blend them into a single reliability figure and you have a number that describes nothing.

Benchmark scores read as deployment forecasts. An industry paper puts it plainly: "state-of-the-art systems now exceed 90% execution accuracy on controlled benchmarks" (Serrao et al., 2026), and yet practitioners report "a persistent gap between benchmark scores and real-world reliability" (Serrao et al., 2026). Note the stake. These authors work for a consultancy selling analytics implementation, so a gap best closed by implementation is a congenial finding.

A low score read as proof the technology is useless. Strict benchmark scoring rejects correct answers too. Across most of six evaluation frameworks tested, "these frameworks reject roughly half of queries that human experts consider correct" (Serrao et al., 2026). The same study found one framework erring the other way, passing almost everything.

So read any single score as an artefact of its scorer as much as of the system, and note that scoring misleads in both directions. Neither benchmark quoted on this page was among the six tested. Strict scoring also gives no credit to an assistant that drafts most of a query for an analyst to finish. That is a common way to use one.

Comparison articles that hide their method. The cross-platform AI benchmarks that surfaced in this research were published, almost without exception, by companies selling into this market. Check each one as you would a vendor deck. One example, ranking five warehouses on AI workloads, is published by Estuary, which sells data pipelines into those same warehouses. Read on 29 September 2026, it named its dataset, then gave only qualitative verdicts such as top performer and struggler. It stated no warehouse size, no run count and no pricing tier, and placed the underlying results behind a download form. It did link a public methodology repository. Named, and dated, so you can check that description rather than take it on trust.

Concluding that nothing independent exists. That is untrue, and the shape of what exists is useful. A real academic literature tests whether the capability works. What this research did not find was any independent, non-vendor evaluation comparing one platform against another. The capability is studied. The contest appears to be scored only by the players.

A pilot that quietly changes its own success metric. The common ending is not a cancelled project. It is a pilot narrowed to curated tables and a pre-approved question list, where accuracy genuinely is high. The headline stays the same while the measure of success drifts from correctness to adoption. Fix your twenty questions and your scorer in writing beforehand, and that ending becomes visible.

What nobody can tell you yet

Three things are simply unknown, and any guide that supplies them is making them up.

What a correct answer costs. Vendors do publish compute costs, and the Snowflake post above prices a query. What no figure found here includes is the salary of the person verifying the output, which is the unit a budget actually needs.

Whether documentation work transfers. The payoff depends on the schema you start from. The only published candidate for what predicts it is how cryptic the column names are, and that rests on one vendor's eleven databases. Your own ten tables remain the better guide.

What leaving costs. Searching in September 2026 turned up no documented case of an enterprise building context assets inside one platform's AI layer and then moving them elsewhere. Any confident claim about lock-in, in either direction, is a forecast rather than a finding.

These three gaps are why this page tells you how to measure rather than what to buy. The measurement is available to you this month. The answers were not available to anyone when this was written.

A note on readiness

Our data is not ready is true more often than not, and it is close to unusable as a sentence. It sounds like an accusation aimed at whoever ran the last data programme. It sounds like stalling to whoever asked for the AI.

The fix is to make the question the subject, rather than the organization, which is what the evidence actually supports. BEAVER's ladder describes how hard a task is, not how competent a team is.

So say the longer version instead. This question cannot currently be answered from these tables. Here is what would have to be true, here is what that costs, and here is who owns it. That is a plan. Not ready on its own is not.

The same principle appears elsewhere here. Readiness is a property of the specific data path a use case touches, not a company-wide condition to reach first. See Readiness Is a Use-Case Property.

Sources

  • Snowflake (Renee Huang) — Snowflake Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI, Snowflake engineering blog, 29 August 2024, 2024. an agentic AI system, coupled with a semantic model to achieve an impressive 90%+ SQL accuracy on real-world use cases. View source · verified 2026-09-29 · primary
  • Snowflake (Renee Huang) — Snowflake Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI, Snowflake engineering blog, 29 August 2024, 2024. we developed an internal benchmark suite of 150 questions that mirror the real-world tasks business users encounter. View source · verified 2026-09-29 · primary
  • Snowflake (Renee Huang) — Snowflake Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI, Snowflake engineering blog, 29 August 2024, 2024. curated BI tables are typically structured with clear distinctions between dimensions, time dimensions and measures. This often results in a flattened, wide table. View source · verified 2026-09-29 · primary
  • Snowflake (Renee Huang) — Snowflake Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI, Snowflake engineering blog, 29 August 2024, 2024. Instead of requiring an exact match with the gold query, we calculate precision and recall for column matching, setting a more lenient threshold to judge accuracy. View source · verified 2026-09-29 · primary
  • Chen, Yang, Li, Wenz, Zhang, Tatbul, Cafarella, Demiralp, Stonebraker (MIT, Harvard, TU Munich, Greenshoe, Intel, AWS AI) — BEAVER: An Enterprise Benchmark for Text-to-SQL, arXiv:2409.02038v3, 13 May 2026. It comprises 9128 question-SQL pairs sourced from real-world query logs and 812 tables across 19 diverse domains. View source · verified 2026-09-17 · primary
  • Chen, Yang, Li, Wenz, Zhang, Tatbul, Cafarella, Demiralp, Stonebraker (MIT, Harvard, TU Munich, Greenshoe, Intel, AWS AI) — BEAVER: An Enterprise Benchmark for Text-to-SQL, arXiv:2409.02038v3, 13 May 2026, abstract. These are the abstract's GPT-5.2 figures. The paper's own strongest end-to-end result is 11.4% (ReFoRCE with Claude-4.5-Sonnet, section 5.3 and Table 4), so 10.8% is not the maximum observed. Our evaluation reveals a significant performance gap compared to existing benchmarks: SOTA agentic frameworks using the advanced model GPT-5.2 achieve only 10.8% accuracy. When provided with all subtask annotations as oracle hints, accuracy increases to 30.1%, confirming that a major bottleneck lies in correctly resolving these subtasks. View source · verified 2026-09-17 · primary
  • Snowflake (Agarwal, Vadrevu, Samdani and Richards) — Introducing Cortex Sense: Grounded Context for the Data You Never Modeled, Snowflake blog, 30 June 2026, 2026. we evaluated agents on product analytics questions on Snowflake's internal data requiring cross-table joins, metric formula lookups and filter convention knowledge. View source · verified 2026-09-29 · primary
  • Snowflake (Agarwal, Vadrevu, Samdani and Richards) — Introducing Cortex Sense: Grounded Context for the Data You Never Modeled, Snowflake blog, 30 June 2026, 2026. Our data team found that without any context layer, AI had around 25% accuracy. View source · verified 2026-09-29 · primary
  • Chen, Yang, Li, Wenz, Zhang, Tatbul, Cafarella, Demiralp, Stonebraker (MIT, Harvard, TU Munich, Greenshoe, Intel, AWS AI) — BEAVER: An Enterprise Benchmark for Text-to-SQL, arXiv:2409.02038v3, 13 May 2026, Table 6 of the PDF, Execution Accuracy column, ReFoRCE row of each setting. Other frameworks are lower in every setting (Setting 2: Few-shot 23.1, DIN-SQL 13.9, DAIL-SQL 11.2). Table 6: Avg. execution accuracy and subtask performance in end-to-end setting and with subtask annotations across all models. View source · verified 2026-09-17 · primary
  • Snowflake (Agarwal, Vadrevu, Samdani and Richards) — Introducing Cortex Sense: Grounded Context for the Data You Never Modeled, Snowflake blog, 30 June 2026. NOTE: this figure comes from a SECOND, harder benchmark (a set of hard questions), not the baseline evaluation that produced the around-25% figure; the 24.1% arm is vanilla CoCo WITH semantic-view retrieval available, not a no-context-layer condition, 2026. Cortex Sense improved accuracy from 24.1% to 86.3% on our benchmark. View source · verified 2026-09-29 · primary
  • Snowflake (Agarwal, Vadrevu, Samdani and Richards) — Introducing Cortex Sense: Grounded Context for the Data You Never Modeled, Snowflake blog, 30 June 2026, 2026. Based on internal test results and cost calculations in June 2026. View source · verified 2026-09-29 · primary
  • Snowflake (Agarwal, Vadrevu, Samdani and Richards) — Introducing Cortex Sense: Grounded Context for the Data You Never Modeled, Snowflake blog, 30 June 2026, 2026. Standing up Cortex Sense took a single day rather than the months these projects usually run, since there is no consulting engagement or manual build to wait on. Because there are no new semantic views to author, you just connect to sources and go without having to spend days on curation. View source · verified 2026-09-29 · primary
  • Jacob Matson, MotherDuck (sells a data warehouse; argues for data modeling over metadata platforms) — Does "AI-Ready Data" simply mean "Good Data Modeling"?, 2026-02-06; BIRD Mini-Dev, 500 questions, 11 databases, three frontier models; accuracy under the author's own LLM-judge scoring, not strict BIRD execution accuracy. In BIRD benchmark testing, adding column comments to schemas improved accuracy by just 1.1 percentage points on the training set and 0.2 on the test set. Comments helped only when schemas were genuinely confusing (like a database called "debit_card_specializing"), but actually hurt performance on schemas with intuitive names. View source · verified 2026-09-17 · primary
  • Jacob Matson, MotherDuck (sells a data warehouse; argues for data modeling over metadata platforms) — Does "AI-Ready Data" simply mean "Good Data Modeling"?, 2026-02-06; per-database breakdown blended across train and test. Comments help when the schema is genuinely confusing. debit_card_specializing (try to guess what that schema looks like) got the biggest boost. But schemas with intuitive names and obvious relationships? Comments made things worse. The models had already formed a correct mental model, and the comments introduced noise. View source · verified 2026-09-17 · primary
  • Jacob Matson, MotherDuck (sells a data warehouse; argues for data modeling over metadata platforms) — Does "AI-Ready Data" simply mean "Good Data Modeling"?, 2026-02-06, methodology footer. heavily vibe-coded, so YMMV, but the data is real and I've looked at all of it. View source · verified 2026-09-17 · primary
  • Michael Rumiantsau, VP of Product at Cube (sells a semantic layer) — Why semantic layers make LLM analytics reliable: a paired benchmark across three frontier models, 2026-04-28; 100 natural-language questions over the Cleaned Contoso Retail Dataset in ClickHouse; paper arXiv:2604.25149. Adding the semantic layer improves accuracy by +17 to +23 percentage points across all three models. View source · verified 2026-09-17 · primary
  • Michael Rumiantsau, VP of Product at Cube (sells a semantic layer) — Why semantic layers make LLM analytics reliable: a paired benchmark across three frontier models, 2026-04-28, description of the treatment condition. hand-authored markdown document describing the dataset's measures, conventions, and disambiguation rules. No code, no runtime, no tool calls. Just the document, pasted into the prompt. View source · verified 2026-09-17 · primary
  • Michael Rumiantsau, VP of Product at Cube (sells a semantic layer) — Why semantic layers make LLM analytics reliable: a paired benchmark across three frontier models, 2026-04-28, section "Why this matters". The markdown we used was 4 KB — an afternoon of an analyst's time. But it has to capture what your metrics actually mean. View source · verified 2026-09-17 · primary
  • Serrao, Chatterjee, Strori, Sharma and Miller (Merkle Analytics) — BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning, 2026. state-of-the-art systems now exceed 90% execution accuracy on controlled benchmarks. View source · verified 2026-09-29 · primary
  • Serrao, Chatterjee, Strori, Sharma and Miller (Merkle Analytics) — BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning, 2026. Yet practitioners deploying these systems report a persistent gap between benchmark scores and real-world reliability. View source · verified 2026-09-29 · primary
  • Serrao, Chatterjee, Strori, Sharma and Miller (Merkle Analytics) — BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning, section 6, 2026. False negative rates are high for most baselines (42.4-45.6%): these frameworks reject roughly half of queries that human experts consider correct. View source · verified 2026-09-29 · primary