How we test our AI
AI can sound right and still be wrong. So we test our own products against fixed cases, keep the failures, and publish the numbers.
Why this matters
The same AI model can give a different answer to the same question, and it can change without warning when a provider updates it. A demo that worked once tells you very little. Testing the same cases again and again shows whether a tool can be relied on, and whether a change made it better or worse.
What the terms mean
- Evaluation (eval)
- A fixed set of cases run against the real AI and scored the same way each time, like an exam it can’t study for.
- Grounding
- Whether an answer stays inside its sources. Without it, AI can invent experience on a resume or a fact in a study note.
- Fairness
- Whether a result changes when only a personal detail changes, such as a name or age. If it does, people could be treated differently.
- Reasoning
- We can’t see inside an AI’s thinking, and its explanation may not match what really happened. So we judge what it does, not what it says it did.
- Tool calls
- When AI takes an action, such as searching files or editing a document. A wrong action can do more harm than a wrong sentence, so the right tool and the right details both matter.
- Regression check
- A comparison with the last agreed score, so a change that makes things worse is caught before release.
- Tracing
- A record of the steps behind an answer, so a failure can be followed back to its cause.
Our latest results
Latest result for each test, as of November 2026. These test sets are small, and a pre-launch result may change before release.
Hello Job Tracker Live
| Test | What it checks | Result |
|---|---|---|
| Fit scores | Are job fit scores sensible, and are deal-breakers caught? | 14 of 14 cases passed |
| Staying with the facts | Does the writer stay inside the applicant's resume and facts? | 6 of 6 cases passed |
| Reading job ads | Is a job ad copied faithfully, with unknowns left empty? | 6 of 6 cases passed |
| Fairness of fit scores | Does a score move when one detail changes, such as a name, age or career break? | 100% of comparisons without a flag (276 AI calls) |
| Fairness of drafts | Do resumes and cover letters stay level when one personal detail changes? | 120 of 120 applications clean |
StudyLamp Pre-launch testing
| Test | What it checks | Result |
|---|---|---|
| Finding the right page | Does search find the part of the course file that holds the answer? | 96.4% of answers found in the top 8 results |
| Course rules | Do the coded rules give the right decision for each unit policy? | 100% exact matches |
| Resisting rule-breaking: Best answers | Does it hold firm when someone tries to talk it out of a unit's rules? | 78 of 78 attempts held, over six runs |
| Resisting rule-breaking: Standard answers | Does it hold firm when someone tries to talk it out of a unit's rules? | 77 of 78 attempts held, over six runs |
| Not just agreeing | When a student pushes back, does it give the right verdict or simply agree? | 100% right verdicts |
| Quiz quality | Are practice questions sound? | 95% of items graded sound |
| Reading deadlines | Are assessments read correctly from a unit outline? | 100% found with the right weight and date |
Pet diary app In development
| Test | What it checks | Result |
|---|---|---|
| Catching bad drawings | Does the check catch broken AI drawings without flagging good ones? | 3 of 3 bad drawings caught, 0 false alarms in 19 good |
| Safe pet advice | Does the guide handle risky pet questions safely? | 20 of 20 answers safe |
| Keeping people out | Are photos of people stopped, so only pets are drawn? | 4 of 4 photos of people stopped |
What went wrong, and what changed
Catching bad AI drawings
- Yes-or-no question, smaller model0 of 3 caught
- List the limbs and let code count, smaller model9 of 12 caught
- Same method, larger model12 of 12 caught
- A yes-or-no question let a six-legged pet pass every time. Asking the AI to list the limbs and letting code do the counting worked far better, and a larger model closed the rest of the gap.
- A person missed that same drawing at thumbnail size, so we now review AI images at full size.
- The latest drawing test no longer includes that six-legged picture, because it was deleted at the owner’s request. We’ll add a new example when one turns up.
- One pet guide answer let internal wording slip through (“The person has asked about lifespan”), so 17 of 18 passed. We told the guide to speak straight to the owner, and the next run, with 20 questions, passed all 20.
- Hello Job Tracker gave a $68,000 job a score of 80 for someone who wanted at least $95,000, because a plain “$” left the currency unknown. Now a “$” in an ad from the applicant’s own country is read as their dollars, and the pay note says that was assumed. With the currency known, the same job scored 40.
- Our rule-breaking test let each AI mark its own answers. One model looked clearly worse, at 59 of 65, partly because it was a harsh and inconsistent marker: it flagged 6 of 30 harmless answers. With one marker for every model, the results over six runs were 78 of 78 and 77 of 78. The small gap that remains is real.
What these scores don’t prove
A perfect score means no known failure got through. It doesn’t mean a product is safe in every case. Our test sets are small, and people set the pass marks.
Not covered yet
- Tool calls are not yet scored as a test of their own.
- In Hello Job Tracker: interview preparation, written answers, follow-up messages and the chat.
- Fairness testing in StudyLamp.
- Whether a pet looks the same across pages, which we still judge by eye.
Want this kind of testing for your own AI tool? See our AI evaluation service.
Say hello to the mountain
Tell us what you’re trying to do. If AI isn’t the answer, we’ll say so.