◆ QUEST LOG / ASKDATA

AskData — the agent that shows its work

“A wrong number that looks right is worse than an error message.”

01The problem

Text-to-SQL agents demo beautifully. You type “what was our average order value last quarter?”, the model writes SQL, and a number appears. The problem is what happens on question forty-seven, when the model quietly joins the wrong table or filters the wrong date column — and still hands you a confident number. Nobody notices, because the answer looks right.

For analytics, that failure mode is worse than crashing. A stakeholder who gets an error message asks for help; a stakeholder who gets a wrong number makes a wrong decision.

02What I built

AskData is a complete analytics stack on the public Olist dataset — about 100k real Brazilian e-commerce orders — built in three layers:

• A dimensional warehouse. Raw CSVs modeled into a Kimball-style star schema in DuckDB, so every metric has one defensible definition.
• A review-risk model. A scikit-learn classifier that predicts which orders are likely to end in a bad review, trained on delivery timing, price, and product features.
• A text-to-SQL agent that checks itself. Written from scratch — no framework — so every step of the loop is inspectable.

03How the verification works

The agent never just returns its first answer. Each question runs through a loop: plan → generate SQL → execute → verify → tag. The verifier re-derives the answer through independent checks — column sanity, row counts, unit checks, and a second pass that asks whether the SQL actually answers the question that was asked, not a lookalike.

Every answer ships with one of three tags: trusted (it passed verification), repaired (the first attempt failed, the agent found and fixed the problem, and the fix passed), or abstained (the agent could not convince itself and says so instead of guessing). Abstaining is a feature, not a failure.

04Results — measured, not vibes

Building the verifier was the easy half. The real work was finding out whether it helps: 150 golden questions across six tiers (from simple lookups to unanswerable traps), run against five models, three runs each, with exact McNemar tests on paired confidently-wrong outcomes.

• Verification cut confidently-wrong answers by up to 69%: from 21.8% to 6.7% on the mid-tier model (p = 1.7e-15).
• The effect peaks mid-capability. The strongest model only improved 17%, and on the weakest local model the change didn’t reach significance (p = 0.053). Verification isn’t a free lunch everywhere, and the eval says exactly where it isn’t.
• The judge matters more than the generator: keeping the weak 3B local generator but giving it a stronger judge cut its confidently-wrong rate from 10.9% to 2.2%.
• The review-risk model reached 0.756 ROC-AUC; flagging the riskiest 10% of orders catches 4.1× more bad reviews than random.
• The harness is repeatable: 29 deterministic tests run without an API key, and the full study is written up as a paper draft in the repo.

05What I learned

Building the agent without a framework forced me to own every failure mode. The biggest lesson: evaluation design is the hard part. Deciding what “wrong” means, building a question set that can detect it, and measuring honestly taught me more than any amount of prompt tuning. The second lesson: users forgive an agent that says “I’m not sure” far more easily than one that lies confidently — in analytics, humility is a product feature.