AI Governance: Automatically Checking AI Systems Against the EU AI Act and NIST Rules

Abstract

Financial institutions deploying models for credit evaluation, fraud detection, and anti-money laundering must demonstrate that their internal documentation satisfies frameworks such as the European Union's Artificial Intelligence Act (EU AI Act) and the NIST AI Risk Management Framework (AI RMF). Automating this documentation review is attractive, but a checker that approves a control merely because a document mentions the right topic keywords is dangerous: it converts empty corporate assurance into apparent compliance. We present a system design and evaluation for evidence-linked compliance checking, with three contributions. First, a structured control catalog of twelve documentation-checkable controls mapped to EU AI Act articles and NIST AI RMF functions, each with explicit substance criteria. Second, an evidence-linked checker architecture whose every "satisfied" decision is tied to an exact supporting sentence, supported by a tiered decision pipeline, calibrated abstention, and a reviewer workflow. Third, a controlled synthetic benchmark of financial AI documentation in which each control is substantive, boilerplate-only, or absent, with disjoint training and unseen phrasing sets. In our baseline experiments, keyword rules falsely approve 85% of boilerplate-only controls. Trained sparse lexical classifiers reach high precision but, under unseen phrasing, recall falls to between 0.12 and 0.22. Adding 25 labeled documents in the new phrasing restores recall above 0.99, an estimate we caution is optimistic given the synthetic setting. Removal-based deletion tests show that evidence-faithfulness metrics must be interpreted against the redundancy of evidence within a document. The benchmark is a measurement tool for checker reliability, not a legal determination of conformity.