Putting AI in a credit decision without losing the argument later

There is a conversation that happens about eighteen months after an AI system goes into a lending decision. Somebody — a supervisor, an ombudsman, a claimant's lawyer, an acquirer's diligence team — asks why a specific applicant was declined in a specific month.

The answer needs to be a reconstruction, not a recollection. Which model version, which inputs, which policy rules, which thresholds, what the human reviewer saw, and what they did.

Most teams find out they cannot do this at the point they are asked. This piece is about building so you can.

What the law actually asks for, in engineering terms#

If you use AI to evaluate the creditworthiness of natural persons in the EU, you are almost certainly operating an Annex III high-risk system under the AI Act. The obligations for Annex III systems started applying in August 2026. Creditworthiness assessment is named explicitly; fraud detection is carved out in some readings and not others, and pricing depends on how the decision is constructed — which is itself a reason to get an opinion rather than assume.

Strip out the legal language and the obligations land on an engineering team as roughly six things:

A risk management system that is continuous. Not a document produced at launch. Something that runs, produces evidence, and gets reviewed.

Data governance you can evidence. Where training data came from, what it represents, what is known about its gaps and biases, and what you did about them.

Technical documentation. A model card that a competent outsider could read and understand what the system does, on what inputs, with what known limitations.

Automatic logging. Records of operation, retained, sufficient to reconstruct a decision. In practice: inputs, model version, score, thresholds applied, policy rules fired, override, outcome — per decision, immutable.

Human oversight that is real. A named person with the authority and the information to override, and evidence they were in a position to do so. A reviewer shown a score and a button is not oversight; a reviewer shown the score, the drivers and the option to disagree, with their disagreements tracked, is.

Accuracy, robustness and post-market monitoring. You measure it after launch, not just before, and you act on what you find.

None of that is unusual for a lender that already runs models under model risk management. What is new is that it applies to systems whose behaviour can change when somebody edits a prompt.

The bit teams get wrong: generative components in the path#

Classical scorecards are well understood. The failure we see is a language model quietly acquiring influence over a decision without anyone classifying it as part of the decision.

Common versions:

• A model summarises bank statement transactions into affordability categories, and the categories feed the score
• A model extracts employment details from uploaded documents
• A model drafts the adverse action reason sent to the applicant
• An agent triages which applications go to manual review

Every one of those is inside the decision. The third one is inside the applicant's ability to appeal. If the reason given does not correspond to the actual drivers of the decline, you have a problem that is not primarily technical.

Treat any generative component in the path with the same seriousness as the scorecard. Version it. Log its input and output. Evaluate it. Include it in the model inventory.

What we build, concretely#

Decision logging that is immutable and complete. One record per decision containing: application identifier, timestamp, model name and version, feature values as scored, score, policy rules evaluated and which fired, threshold in force, generative outputs used and their prompt versions, reviewer identity and action, final outcome. Written once, never updated. This is the single highest-value artefact and it is cheap if you do it from the start.

Reason codes that trace to drivers. Whatever is sent to the applicant must be derivable from the record. If a decline reason is generated, it is generated from the logged drivers and then checked against them, not written freehand.

Evals on every generative component. A golden set with correct outputs, scored on every prompt, model or data change, with a release gate. For extraction tasks this is straightforward — you have ground truth. For summarisation you need a rubric and a judge, plus a human review sample.

A grounding check before anything is used. If a summary contains a figure that is not in the source transactions, it fails and falls back to manual review rather than proceeding.

Human oversight with teeth. A confidence threshold above which the case goes to a person; the person sees the drivers, not just the score; overrides are logged with a reason and their outcomes are tracked. Override rates and override performance become a monitored metric, because an unmonitored override rate is where a good model quietly stops being used.

Monitoring that runs monthly, not annually. PSI on the score distribution and on each significant input. Characteristic analysis to explain a shift. Predicted against actual bad rate by band for seasoned cohorts. Eval scores for every generative component over time. If any of that only exists in an annual validation pack, you will find out about degradation up to eleven months late.

Fairness testing you can show. Outcome rates across protected characteristics where you are permitted to hold them, proxy analysis where you are not, and a documented view of what you tested and what you found. The honest version of this is not a certificate; it is a record of having looked.

Two things worth saying plainly#

Explainability is not optional and it is not free. If you cannot explain a decline, do not automate the decline. That does not rule out complex models — it rules out complex models without the surrounding machinery. The machinery is most of the work and it is where most of the budget should go.

Being in scope is not a reason to avoid AI. It is a reason to build differently. The lenders getting value here are not the ones who avoided regulated use cases; they are the ones who accepted from the first week that every decision would need to be reconstructable, and designed for it instead of retrofitting.

Retrofitting logging onto a live decision system is one of the most miserable projects in this industry. Doing it up front costs a fortnight.