◎ OS PUB Apache 2.0 ← All specifications

P172 — AIEP — Evidence Quality Metrics and Corpus Health Protocol

Publication Date: 2026-03-27 Status: Open Source Prior Art Disclosure Licence: Apache License 2.0 Author/Organisation: Phatfella Ltd Schema: AIEP_OS_SPEC_TEMPLATE v1.0.1 — https://aiep.dev/schemas/aiep-os-spec-template/v1.0.1

Framework Context

[0001] This disclosure operates within an Architected Instruction and Evidence Protocol (AIEP) environment as defined in United Kingdom patent application number GB2519711.2, filed 20 November 2025, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure defines a protocol for computing, reporting, and governing a set of corpus-level evidence quality metrics that characterise the overall health of an AIEP evidence corpus — its coverage, freshness distribution, trust tier distribution, conflict rate, language diversity, and completeness against defined coverage targets — enabling node operators to identify quality gaps and prioritise evidence ingestion investments.

Field of the Disclosure

[0003] This disclosure relates to evidence quality metrics and corpus health monitoring protocols for governed artificial intelligence reasoning systems.

[0004] More particularly, the disclosure concerns a CorpusHealthReport schema, a CorpusHealthMonitor component computing metrics at configurable intervals, a set of standard quality dimensions, a HealthScore aggregate index, alert thresholds generating automated operator notifications, and a CoverageTarget mechanism enabling operators to specify desired corpus composition and measure deviation against it.

Background

[0005] AIEP evidence corpora are assembled continuously from diverse sources via ingestion pipelines, streaming subscriptions, and federated provider manifests. Without systematic monitoring of corpus quality metrics, operators cannot determine whether their corpus is becoming stale (freshness degrading), unbalanced (over-represented in certain subject domains), or deteriorating (increasing conflict rate, declining trust tier distribution). These quality dimensions directly affect the reliability of reasoning outputs.

[0006] Individual artefact metrics — freshness scores (P147), trust scores (P124), classification vectors (P160) — are computed and stored at artefact level. Corpus health is an aggregate property that emerges from the distribution of these per-artefact metrics across the entire corpus. No existing AIEP specification defines how to aggregate these distributions into corpus-level health metrics or how to set and measure coverage targets.

[0007] Corpus health monitoring provides value at two timescales: real-time alerting when a critical metric degrades below an operator-defined threshold (e.g. freshness compliance rate drops below 90%), and periodic reporting for governance purposes (monthly CorpusHealthReports submitted to information governance boards).

Summary of the Disclosure

[0008] Quality Dimensions: The CorpusHealthMonitor computes the following metrics at each evaluation interval:

Coverage Metrics:

total_active_artefacts — count of Active tier artefacts
taxonomy_coverage_distribution — count and fraction of artefacts per Subject Domain code (P160), identifying domains with sparse coverage
language_distribution — count and fraction of artefacts by source language (BCP 47)

Freshness Metrics:

freshness_compliance_rate — fraction of Active artefacts whose current confidence score (P147) is above a configurable threshold (default: 0.70)
stale_artefact_count — count of artefacts below the freshness threshold
overdue_refetch_count — count of artefacts past their scheduled re-fetch timestamp (P163)

Trust and Quality Tier Metrics:

quality_tier_distribution — count and fraction of artefacts per Evidence Quality Tier (EQ code, P160)
average_trust_score — mean trust score (P124) across Active artefacts
low_trust_artefact_count — count of artefacts with trust score below configurable threshold (default: 0.50)

Conflict and Integrity Metrics:

open_conflict_count — count of ConflictRecords (P161) with resolution_status = PENDING
conflict_rate_per_thousand — conflicts raised per 1000 artefact ingestions in the reporting window
provenance_integrity_pass_rate — percentage of ProvenanceChain verifications (P175) that passed in the reporting window

Classification Metrics:

unclassified_artefact_count — artefacts with no ClassificationVector assigned
uncertain_classification_count — artefacts with classification_source = UNCERTAIN

[0009] HealthScore: An aggregate HealthScore is computed from the quality dimension metrics as a weighted sum:

HealthScore =
  0.25 × freshness_compliance_rate
+ 0.20 × (1 - low_trust_fraction)
+ 0.20 × (1 - open_conflict_fraction)
+ 0.15 × (classification_complete_fraction)
+ 0.10 × provenance_integrity_pass_rate
+ 0.10 × language_diversity_index

where language_diversity_index is the Shannon entropy of the language distribution normalised to [0, 1], and the conflict and low-trust fractions are computed as the respective counts divided by total_active_artefacts. HealthScore is a float [0, 1]. The weights are operator-configurable.

[0010] CorpusHealthReport Schema:

report_id — SHA-256 of canonical serialisation of all other fields
node_id — the reporting node’s fingerprint
generated_at — ISO 8601
reporting_window — {start, end} ISO 8601
total_active_artefacts — integer
quality_metrics — map of metric name → value for all metrics in [0008]
health_score — float [0, 1]
coverage_target_deviations — list of CoverageTargetDeviation records (see [0011])
alerts_triggered — list of AlertRecord (see [0012])

[0011] Coverage Targets: Operators define CoverageTargets specifying the desired proportion of the corpus that should be classified to a given taxonomy code:

target_taxonomy_code — e.g. SD.MED.ONCO
minimum_fraction — minimum desired fraction of corpus
minimum_absolute_count — minimum desired absolute artefact count

A CoverageTargetDeviation record is generated when either threshold is not met, identifying the shortfall and recommending ingestion actions.

[0012] Alerts: Operators configure AlertThresholds; when a metric crosses a threshold, an AlertRecord is created with metric_name, threshold, actual_value, and severity (WARNING or CRITICAL). Alerts are delivered to configured operator notification channels and stored in the ledger.

ASCII Architecture

Ledger Partitions + EvidenceNode metadata
        │
        ▼
┌─────────────────────────────┐
│  CorpusHealthMonitor        │ (runs at configurable interval)
│  - freshness distribution   │
│  - trust score aggregation  │
│  - conflict rate            │
│  - taxonomy coverage        │
│  - language entropy         │
│  - provenance pass rate     │
└──────────┬──────────────────┘
           │ metric values
           ▼
┌─────────────────────────────┐    ┌───────────────────────┐
│  HealthScore computation    │    │  CoverageTarget       │
│  (weighted sum)             │───▶│  deviation analysis   │
└──────────┬──────────────────┘    └───────────────────────┘
           │
           ▼
┌─────────────────────────────┐
│  CorpusHealthReport         │──▶ Ledger + Operator channel
│  (metrics + score + alerts) │
└─────────────────────────────┘

Operational Detail

[0013] Incremental Computing: For large corpora, HealthScore metrics are computed incrementally rather than by full-corpus scan at each interval. Artefact-level metric changes (freshness score decays, new conflict raised, new artefact ingested) emit metric delta events; the CorpusHealthMonitor accumulates deltas between reporting intervals, producing a report by applying deltas to the previous report’s baseline counts.

[0014] Trend Reporting: CorpusHealthReports are retained in the ledger, enabling trend analysis over rolling windows. The CorpusHealthMonitor may optionally include a trend_metrics section comparing current metrics against the reports from the previous 30 days, 90 days, and 12 months.

[0015] Federation Aggregation: For federated deployments, a designated aggregation node may collect CorpusHealthReports from multiple subordinate nodes and produce a federated CorpusHealthReport summarising the health of the combined corpus. Federated reports carry counts and distributions aggregated across all contributing nodes, with per-node breakdowns in an appendix section.

Claims-Exclusion Notice

This specification is published as open-source prior art. No patent claims are asserted by the author in respect of the mechanisms described. Any third party seeking to patent mechanisms substantially equivalent to those described herein is placed on notice of this prior art disclosure.