We spent several years working on trustworthy AI for critical systems, within industrial and academic consortia. The goal: to be able to show that a system behaves as intended, and to understand why when it does not.
In a business, the issue is simpler to state. A decision produced or prepared by AI always ends up in front of someone: an engineer signing a report, a manager approving a file, a customer disputing a refusal, an auditor. Each needs a different explanation.
Explain to whom?
- To the user, so they know when to trust the result and when to check it.
- To the person in charge, so they can defend the decision to their leadership.
- To the person affected, when the decision concerns them directly.
- To the auditor, who will want to trace what was decided, by which system, and on what data.
Three simple practices
- Keep a person who signs. In a technical reporting system we deployed, AI drafts and the engineer reviews and signs. Responsibility stays where it was.
- Show the sources. A result that points to the data or documents it used can be checked in seconds.
- Plan a no-AI mode. If the model is unavailable or wrong, the work must be able to continue by hand.
If no one can explain a decision to the person it affects, the system is not ready for production.
What we take away
Explainability is not a technical option added at the end. It is decided at scoping, together with the problem to solve and the person who owns the system.
Want to scope an AI system your teams can stand behind? Let’s talk.
Book a callMore notes