In-CRM AI assistant
The client's sales team wanted to ask the CRM questions in plain English: what's in the pipeline, who at this company have we spoken to, which deals are stuck. The first version answered confidently and was wrong almost half the time. The rebuild is what this case study is about.
- Model
- Claude Opus in a tool loop
- Data
- 23 CRM tables, read only
- Eval
- 13 of 13, including traps
- Cost
- about a dollar a month
Why it exists
The first version ran on a small model with one read tool that returned 100 rows. Asked for the pipeline total it summed one page and invented the rest, off by $600k. Asked about a live Tier 1 account it said the company wasn't in the system. Asked about a field that shipped the week before it said the field didn't exist. All three answers sounded certain.
How it runs
Stronger model, own setting
The assistant runs on Claude Opus with its own model constant. Reply classification and social drafts stay on the cheap model. The two are deliberately separate.
Search that finds things
A search tool that tries the full phrase, the phrase with spaces removed and the longest words, across name, domain and ticker. That's what finds a company whose row is named differently from how people say it.
Totals in Python
Sums, counts and breakdowns page through every row in Python and hand the model a number. The model never adds rows itself.
Facts from the data
The prompt used to hardcode thresholds and a column list, both stale. Now it reads the config table and asks the database for the live schema.
Prompt caching
Everything per request, the user, the date, the open record, rides on the user turn instead of the system prompt, so about 20,000 tokens a turn come from cache.
Eval
Thirteen real questions with answers checked against paged database queries, including a fake company that must come back as not found.
What came out of it
Thirteen of thirteen on the eval, including the adversarial cases. The pipeline total matches the database to the dollar. The whole thing costs about a dollar a month to run because of the caching.
Notes
My own first set of ground truth answers was wrong for the same reason the bot was: Supabase's REST layer caps a response at 1,000 rows and says nothing. The fix is the same in both places, page it or read the count off the header. Two tables that hold credentials are excluded from the assistant on purpose.
Stack
- Python
- Claude API (Opus)
- Supabase (Postgres)
- Docker
- Prompt caching
Want something like this built?
Tell me what you're trying to automate. Most things can be built in a week or two.