𝗔 𝘁𝗶𝗻𝘆 𝘀𝘁𝗼𝗿𝘆
I asked ChatGPT to write a SQL query.
It took about eight seconds.
The query looked perfect.
Clean CTEs.
Good column names.
No obvious mistakes.
I ran it.
The result was 14.8% higher than my expected number.
So I asked AI to fix it.
It confidently changed the wrong thing.
𝗧𝗵𝗲 𝗮𝗹𝗺𝗼𝘀𝘁 𝗿𝗶𝗴𝗵𝘁 𝘁𝗿𝗮𝗽
This is becoming one of the biggest problems with AI-assisted analysis.
Not completely wrong answers.
Almost-right answers.
Stack Overflow's 2025 developer survey found that 66% of respondents' biggest AI frustration was solutions that were "almost right, but not quite."
And 45% said debugging AI-generated code can take more time.
AI adoption is rising.
Trust is not keeping up.
I call this the almost-right trap.
The answer looks good enough to skip verification.
That's exactly why it is dangerous.
𝗪𝗵𝘆 𝗶𝘁 𝗸𝗲𝗲𝗽𝘀 𝗵𝗮𝗽𝗽𝗲𝗻𝗶𝗻𝗴
AI is extremely good at producing plausible work.
Data analysis needs something slightly different.
It needs work that is correct for your data, your definitions, and your business rules.
AI doesn't know that your "customer" excludes test accounts.
It doesn't know that "revenue" means recognised revenue in your company.
It doesn't know that yesterday's query had a special exclusion.
Unless you tell it.
𝗧𝗵𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸
𝘙𝘶𝘭𝘦 1: 𝘎𝘪𝘷𝘦 𝘈𝘐 𝘵𝘩𝘦 𝘣𝘶𝘴𝘪𝘯𝘦𝘴𝘴 𝘳𝘶𝘭𝘦𝘴
"Before writing the analysis, list the business definitions and exclusions you must follow."
Don't start with:
"Write SQL for monthly revenue."
Start with:
"Revenue excludes refunds, test accounts, and cancelled orders. Month means calendar month based on order date."
Now AI has something real to work with.
𝘙𝘶𝘭𝘦 2: 𝘍𝘰𝘳𝘤𝘦 𝘪𝘵 𝘵𝘰 𝘤𝘩𝘦𝘤𝘬
"Before giving the final answer, identify three ways your analysis could be wrong and show how to test each one."
This changes the workflow.
AI stops being only the answer generator.
It becomes a reviewer of its own assumptions.
You still need to verify the tests.
But now you're looking in the right places.
𝘙𝘶𝘭𝘦 3: 𝘎𝘪𝘷𝘦 𝘪𝘵 𝘢 𝘬𝘯𝘰𝘸𝘯 𝘳𝘦𝘴𝘶𝘭𝘵
"Here is a result I already trust. Reproduce it first, then explain your logic."
This is one of the easiest ways to catch subtle errors.
If your trusted number is 102,431 and AI produces 109,882, don't debate the query.
Stop.
Find the difference.
𝘙𝘶𝘭𝘦 4: 𝘈𝘴𝘬 𝘧𝘰𝘳 𝘢𝘶𝘥𝘪𝘵𝘢𝘣𝘭𝘦 𝘰𝘶𝘵𝘱𝘶𝘵
"Show the assumptions, filters, joins, and calculations used to produce this result."
A final number without its path is difficult to trust.
Make AI show its work in a form you can actually inspect.
Not hidden reasoning.
Just the analytical steps that matter.
𝘙𝘶𝘭𝘦 5: 𝘕𝘦𝘷𝘦𝘳 𝘷𝘦𝘳𝘪𝘧𝘺 𝘰𝘯 𝘵𝘩𝘦 𝘴𝘢𝘮𝘦 𝘭𝘢𝘺𝘦𝘳
"Validate the output using an independent calculation, query, sample, or source."
Don't ask AI to check whether AI is correct.
Use a second path.
SQL against Excel.
Power BI against a source query.
Sample rows against the aggregate.
𝗢𝗻𝗲 𝘄𝗮𝗿𝗻𝗶𝗻𝗴
Don't turn every five-line SQL query into a 40-step verification ritual.
The goal isn't distrust.
The goal is proportional verification.
Fix: verify hardest where the business impact is highest.
𝗙𝗼𝘂𝗿 𝗮𝗰𝘁𝗶𝗼𝗻𝘀 𝘁𝗵𝗶𝘀 𝘄𝗲𝗲𝗸
✅ Take one AI-generated SQL query and compare it with a trusted result.
✅ Add your business definitions and exclusions to your next AI prompt.
✅ Ask AI to identify three ways its own answer could be wrong.
✅ Create one independent check for your most important KPI.
𝗖𝗹𝗼𝘀𝗶𝗻𝗴
That 14.8% difference wasn't dramatic.
That's what made it dangerous.
If the query had crashed, I would have caught it immediately.
It didn't.
It looked professional.
That's the new analyst skill.
Not knowing how to get AI to produce an answer.
Knowing how to decide whether the answer deserves to survive.
The future analyst won't be the person who uses AI the most.
It will be the person who knows when not to trust it.
𝗥𝗲𝗽𝗹𝘆 𝘄𝗶𝘁𝗵 𝗼𝗻𝗲 𝘄𝗼𝗿𝗱:
AI if you're using it every day.
TRUST if you've caught an almost-right answer.
VERIFY if you want more AI workflows for analysts.
I reply to every single one.
