AI Audit

Audit AI features in YOUR product against responsible-AI policies. Pre-deployment review for governance, bias, prompt-injection resistance, data leakage.

What AI Audit is for

Most AI security focuses on attackers exploiting your AI features. AI Audit asks the inverse: is YOUR AI feature behaving responsibly? Does it leak training data, treat users unequally, expose system prompts, fail to refuse harmful requests, or violate your own policies? This is what governance/legal/risk teams want to verify before AI ships.

Audit dimensions

Prompt-injection resistance
System-prompt leakage
Refusal of harmful requests
PII handling (redact or refuse)
Bias across demographic groups
Hallucination on factual questions
Source citation accuracy
Output format / schema adherence
Cost / latency regression
Toxic / unsafe output filtering

How it differs from AI Security

AI Security is offense — attacking your AI to find vulnerabilities. AI Audit is governance — measuring AI behavior against your stated policies. Both run against the same endpoints but with different goals + reports.

Running an audit

Configure at AI Audit → New audit:

  • Target endpoint — the LLM-backed API you're auditing.
  • Policy bundle — pick from defaults (OWASP LLM Top 10, EU AI Act conformance, OECD AI Principles, NIST AI RMF, your-org-custom).
  • Test corpus — your domain-specific examples (optional but recommended for accurate measurement).
  • Demographics — for bias checks, the demographic groups to test across.

Output report

Each audit produces a multi-section report:

  • Executive summary — pass/fail per policy dimension
  • Detailed findings per dimension — pass rate %, examples of failures
  • Bias analysis — performance variance across groups (statistical significance flagged)
  • Compliance mapping — which clauses of EU AI Act / NIST AI RMF this audit verifies
  • Remediation suggestions — prompt-engineering fixes, fine-tuning recommendations, guardrail patterns

Continuous audit

Schedule audits to re-run weekly. A model swap or prompt change can silently regress an audit dimension — continuous audit catches that within days.

What's next

  • AI Security — adversarial testing of the same endpoints.
  • Compliance — wire AI Audit results into compliance reports.