How should banks measure AI errors in advisor workflows?
Agree before go-live on what counts as an interaction and as an error, count user mistakes too, and state the period and the population. In a live deployment of Streetbeat's AI Suite for Advisors at a European bank with 3,000 advisors, 99.64% of interactions had no errors over three months, and the 0.36% with errors includes user mistakes. A success rate can be compared with another only when its method is stated.
What was measured in the live deployment?
Streetbeat measured the deployment over three months at a European bank with 3,000 financial advisors. The same figures appear in our overview of the benefits of AI for wealth managers.
| Measure | Result | Definition |
|---|---|---|
| Interactions without errors | 99.64% | Share of interactions completed without errors |
| Interactions with errors | 0.36% | About 1 interaction in 280, user mistakes included |
| Active advisors | 85% | More than 2,500 of 3,000 advisors |
| Period | 3 months | The most recent months of the live deployment |
These are one institution's results over one period. They do not predict another bank's results.
Why should user mistakes count as errors?
Because the advisor and the client experience them in the same way. A badly phrased request, a value typed in the wrong field or a question outside scope produces an answer that has to be redone, whoever caused it. Leaving user mistakes out makes a success rate higher; keeping them in measures the system as people use it every day, as the 0.36% above does. It also points to what can change: the prompt, the interface, the training or the process.
Which questions should a bank ask about any AI success rate?
The same five, every time, because each exclusion that goes unstated can lift the number.
| Question | Why it matters |
|---|---|
| What is an interaction? | A message, a conversation or a completed task can give different rates on the same platform. |
| What is an error? | A narrow definition lifts the rate; a broad one lowers it. |
| Who is in the count? | Leaving out users, integrations or data improves the rate without improving the system. |
| Which period and which population? | A few pilot weeks with volunteers are not months of use across a network. |
| Who classified the errors, and how? | An automatic rule, a manual review or a sample: the method is part of the result. |
The deployment figure states its period, 3 months, its population of 3,000 advisors and the inclusion of user mistakes. The definitions of interaction and error, and the method used to classify errors, are questions to put to every vendor, Streetbeat included.
What do US supervisors expect?
That the bank writes its own controls. The Federal Reserve, the OCC and the FDIC issued revised guidance on model risk management on April 17, 2026, replacing the 2011 version. It describes outcomes analysis, which compares model outputs with real-world results, but it leaves generative and agentic AI outside its scope: for those tools, the bank's own risk management and governance should set the controls. For broker-dealers, FINRA's Regulatory Notice 24-09, issued on June 27, 2024, reminds member firms that their rules, including supervision, apply when they use generative AI. A control function cannot sign off on an error rate it cannot reconstruct.
| Control | What to check |
|---|---|
| Traceability | Can each interaction be reconstructed: request, answer, data used? |
| Written error definition | Was it agreed before go-live, not derived after the numbers were known? |
| Continuous measurement | Is the rate measured again after every change of model, data or process? |
| Human review | Can review be required before an output reaches a client? |
How can a bank check the error rate in its own pilot?
Measure every cause, not only the model, and follow the trend over a period at least as long as the 3 months above.
| Measure | What it shows |
|---|---|
| Share of interactions with errors, all causes | The error rate as advisors experience it |
| Share of errors by cause: model, data, integration, user | Where to act first |
| Errors caught in advisor review before reaching a client | Whether human review works |
| Error rate before and after each change | Whether quality holds over time |
What does the advisor still decide?
Everything that reaches a client, at the bank above for each of its more than 2,500 active advisors. Recommendations and messages are drafts for advisor review, and Streetbeat offers configurable human review for designated critical decisions. The institution remains responsible for configuration, oversight and final decisions.
About Streetbeat
Streetbeat helps organizations use AI to grow revenue, not costs. Streetbeat OS, the AI operating system for companies, is the foundation of Streetbeat for Finance, Streetbeat for Impact and Brain50 by Streetbeat.
Streetbeat for Finance
Streetbeat for Finance brings AI to banks, wealth and asset managers, insurers and real estate investment firms. It includes the AI Suite for Advisors, the Portfolio Engine, the AI Suite for Retail and the AI Suite for Operations, deployed on-premise or in the client's cloud and configured around the institution's workflows, data and controls. See Streetbeat for Finance.