Jev from TypeSafe AI went into early access on September 15. It doesn’t write text. It answers typed questions (pick an option, give a score, or true/false) and returns probabilities.
That makes it a natural fit for test management calls like triaging failed runs, routing defects, or scoring severity. But a few things give us pause:
The benchmarks so far are vendor-run.
It can’t return malformed output, but it can still pick the wrong answer with confidence.
There’s no reasoning trace, which is hard to defend in a release review.
Over to you:
Would you let AI label a failure without a person checking it? What confidence would you need?
What would you need logged to trust it in an audit?
If you piloted it, which decision would you start with?
We’re writing about Jev for QA and want to hear what practitioners think first.
No, I would not trust an AI model specially Jev to handle failure triage completely without human oversight right out of the box—especially in a strict release review environment.
agree with arvind, not fully without a human. but i would happily let it do the first pass. the triage i actually spend time on is “is this a flaky env thing, a locator change, or a real bug”. if a model sorts failures into those three buckets with a score and a human still clicks confirm, that already saves most of the morning.
confidence i would want depends on the bucket. for “flaky, just rerun” i am fine with 0.8 because the cost of being wrong is one extra rerun. for “real bug, open a ticket” i would want it very high and still have someone look.
for audit the minimum is: the exact question asked, the options given, the score for every option not just the winner, the model version, and who confirmed it. no reasoning trace is a real problem in a release review, so the human confirm step is what you defend, not the model.
and i would start with flaky detection. lowest risk, highest volume, and we already have months of labeled reruns to check it against.