Trust, explanation and failure
The operator is a safeguard only while their judgement stays independent of the system they are safeguarding. Calibration is the discipline that keeps it so.
Automation bias and anchoring
The operator is a safeguard only while their judgement remains independent of the system they are safeguarding.
Automation bias
Operators systematically over-weight machine-generated recommendations, both by accepting incorrect output (commission) and by failing to act on a problem the system did not flag (omission). Bias grows with time pressure, workload and the perceived sophistication of the interface.
Operationalising explainable AI
Explanation is an operational instrument, not a compliance artefact. It has to be readable inside the decision window it belongs to.
Reading saliency and attribution
Visual saliency maps and feature attributions indicate where a model's output was driven from. They tell you what the model attended to, not whether attending to it was correct. A saliency map centred on a legitimate feature is weak evidence; a map centred on an artefact of capture is strong evidence against the output.
Designing against over-reliance
Failure-mode safeguards are procedural as much as technical, and they are what makes an SLO's output defensible.