_edited.png){kind=logo}
What Our Customers Look For When Assessing AI Systems
- 2 days ago
- 4 min read
We support around 150 healthcare, charity and tech customers as their DPO or AI Compliance Lead and perform AI governance assessments routinely.
Over the last few years we've worked on both sides of the procurement journey;
with organisations developing AI products and trying to bring them to market
and with organisations assessing, scrutinising and governing AI systems they are considering adopting.
We wanted to share some of the most common things we've learned, and what we wish more suppliers knew before they start engaging with potential customers.
Customers Are Rarely Looking for a Perfect AI System
Many suppliers spend effort trying to prove their AI is accurate, powerful or innovative but in our experience, customers understand that no system is perfect.
What they're looking for is evidence that you understand the risks and have thought about how to manage them.
The suppliers that inspire confidence are not necessarily those claiming the highest performance.
They're often the ones who can clearly explain:
Where the technology works well
Where it may struggle
What safeguards are in place
What users should be aware of
Transparency builds trust.
Overclaiming usually does the opposite.
"It's Proprietary" Is Rarely a Helpful Answer
We understand why suppliers want to protect intellectual property.
However, customers still need to understand enough about a system to make informed decisions. Transparency is a legal obligation.
One of the most common frustrations we encounter is when basic questions about how a system operates are met with:
"That's proprietary."
Customers are not asking for source code.
They're trying to understand:
What data is used
What the AI is doing
What influences outputs
What assumptions are built into the system
The suppliers that succeed tend to distinguish between protecting trade secrets and providing meaningful transparency.
Most Customers Want Plain English, Not Technical Detail
Many AI suppliers have invested heavily in technical documentation.
Unfortunately, much of it is written for technical audiences.
The people reviewing AI systems are often:
Service managers
Governance leads
Procurement teams
Risk managers
Information governance professionals
They need explanations they can understand and defend.
A clear two-page explanation will often achieve more than a fifty-page technical document.
Limitations Are One of the First Things We Look For
When we assess AI systems, one of the first questions we ask is:
"What are the known limitations?"
Surprisingly, this information is often difficult to find and suppliers can become defensive here.
My experience working with suppliers who openly documents limitations are more productive, as this demonstrates maturity and inspires confidence.
A supplier who appears to have no limitations raises concerns as every system has constraints.
Your Response to Scrutiny Matters More Than You Think
One of the more surprising lessons we've learned is that the assessment process is not only about the technology.
It is also an opportunity for customers to understand what it might be like to work with you as a tech partner.
During assessments, we sometimes ask challenging questions about governance, oversight, limitations, risk management or data handling. Occasionally, the response is defensive: "Nobody else has asked us that before." or "We've never been asked to provide that."
While understandable, these responses can create concern rather than confidence. Customers are not simply assessing whether a product works.
They are considering how a supplier will respond when something goes wrong.
For example:
How will they react if a potential data breach occurs?
How open will they be about limitations or incidents?
Will they engage constructively with concerns?Can they work collaboratively to resolve issues?
In many cases, the assessment process becomes an early indication of the future supplier relationship. The suppliers that leave the strongest impression are not necessarily those with all the answers.
They are often the ones willing to engage openly, acknowledge challenges and work through difficult questions collaboratively.
Human Oversight Matters More Than Many Suppliers Realise
One of the strongest indicators of a well-governed AI system is clarity around human oversight.
Customers want to understand:
Who reviews outputs
When intervention occurs
What happens if the AI is wrong
How concerns are escalated
In many assessments, these questions are more important than the technical architecture itself. We have seen pilots struggle, or fail altogether, because suppliers underestimated the practical burden of human oversight.
What We Wish More Suppliers Would Do
If we could offer three pieces of advice to suppliers developing AI products, they would be:
Explain your system in plain English
Assume your audience is intelligent but not technical.
Document limitations early
Don't wait until procurement asks.
Make trust as much of a priority as functionality
Customers need confidence in both the technology and the organisation behind it.
As AI adoption continues to grow, that ability to build trust may become one of the most important competitive advantages a supplier can have.
💡 Need help getting your AI product ready for customer scrutiny?
Our team can help you prepare the AI governance, DPIA, risk assessment, transparency and assurance evidence customers increasingly expect before adoption.
💡 Need help assessing an AI product before you adopt it?
Our team can help you evaluate AI systems, identify risks, review supplier assurances and build the governance evidence needed to support informed adoption decisions.
Comments