Violation Detection & Rule Engine Logic

Regulatory compliance in a modern water utility has shifted from retrospective, spreadsheet-driven reporting toward a continuous, telemetry-driven discipline, and this domain defines the automation layer that makes that shift defensible. It sits downstream of the SCADA Data Ingestion & Time-Series Sync pipeline and consumes the regulatory vocabulary established by the Core Architecture & SDWA Compliance Taxonomy; together those three domains form the compliance reference cataloged across this site. At the center of this domain is a production-grade rule engine: a deterministic framework that translates 40 CFR Part 141 definitions, laboratory reporting limits, and operational constraints into executable, version-controlled logic. For utility operators, environmental compliance teams, municipal developers, and Python automation engineers, building this engine demands strict adherence to regulatory definitions, rigorous data-integrity controls, and resilient pipeline design — the sections below trace a single reading through the full evaluation path, from validated ingestion to regulatory reporting.

Foundational Pipeline Architecture

A violation detection system is a staged pipeline, not a single evaluation function. Each stage has one responsibility and hands a well-typed artifact to the next, so that any compliance determination can be reconstructed from its inputs during an audit. Telemetry from SCADA historians, PLC/RTU streams, and LIMS exports enters an ingestion stage that enforces schema, aligns time, and normalizes units; contiguous data then flows into temporal and CFR-threshold evaluation; exceedances are scored for severity; and every outcome — compliant or not — is written to an immutable audit trail. Decoupling these stages is what lets the engine re-run a determination deterministically when historical data is corrected or a rule set is revised, without disturbing the stages on either side.

The engine cannot evaluate raw, unvalidated telemetry. Historians, PLC/RTU streams, and laboratory exports arrive at heterogeneous frequencies, with varying precision, units, and quality flags. Before any regulatory logic executes, a preprocessing stage must enforce strict schema validation, temporal alignment, unit normalization, and outlier flagging. Stateful stream-processing frameworks provide the windowed aggregations, rolling calculations, and idempotent state persistence this requires. Applying pandas time-series alignment and resampling techniques, or the equivalent Polars and Spark streaming patterns, synchronizes asynchronous sensor feeds to regulatory evaluation windows without introducing interpolation artifacts that could skew a compliance calculation — a concern handled in depth by the upstream Time-Series Alignment Strategies module. Every data point should carry immutable metadata: source identifier, acquisition timestamp, processing timestamp, quality flag, and rule-set version, so the lineage that anchors the entire domain begins at the first stage.

Regulatory & Standards Foundation

The evaluation logic is only as correct as its alignment with federal rule text. Many EPA compliance determinations are not based on a single instantaneous reading. Depending on the contaminant, Maximum Contaminant Levels (MCLs) and Maximum Residual Disinfectant Levels (MRDLs) are evaluated against single samples, running annual averages (RAA), or locational running annual averages (LRAA). Total trihalomethanes and haloacetic acids, for example, are assessed as an LRAA under the Stage 2 Disinfectants and Disinfection Byproducts Rule, while the chlorine and chloramine MRDLs are assessed as an RAA. The engine must implement precise temporal windows, correctly handle overlapping evaluation periods, and apply the rounding rules codified in 40 CFR Part 141.

An LRAA is the mean of the four most recent quarterly averages at a single monitoring location, evaluated each quarter on a rolling basis:

where $\bar{C}_i$ is the arithmetic mean of all samples collected at that location during quarter $i$. Encoding this correctly means the four-quarter window, the per-location grouping, and the rounding step are all explicit — a single misplaced grouping key silently converts an LRAA into a system-wide RAA and can mask a localized exceedance.

Because the authoritative thresholds themselves live in the compliance taxonomy, the rule engine does not hard-code limit values; it resolves them through the SDWA MCL Reference Mapping, which supplies the contaminant identifier, the applicable MCL/MRDL or treatment-technique requirement, and the averaging basis. Rule definitions must be version-controlled artifacts: when a revised MCL takes effect or a monitoring frequency changes, the engine loads a new rule-set version rather than mutating logic in place, and each determination records which version produced it. A representative slice of the reference data the engine consumes:

State primacy adds a second layer of constraint: states authorized to implement the SDWA may adopt limits more stringent than the federal baseline and may prescribe their own reporting formats and monitoring frequencies. The engine treats these as configuration, not code, so that a primacy update propagates without a redeploy — a pattern developed fully under Jurisdictional & Primacy Variations below.

Component Architecture & Data Contracts

The engine decomposes into four cooperating subsystems, each with its own detailed treatment, connected by strict data contracts so that a value crossing a boundary is always well-typed and fully attributed.

The threshold-evaluation subsystem applies CFR-aligned logic to contiguous, normalized data. Implementing MCL Exceedance Logic Implementation requires careful handling of non-detect values, laboratory method reporting limits (MRLs), and sample weighting. A misconfigured temporal window, an incorrect averaging methodology, or improper treatment of results below the MRL can trigger false violations or, more dangerously, mask actual exceedances. This subsystem must be decoupled from ingestion so that determinations can be recalculated deterministically when historical data is corrected or a primacy agency issues updated guidance.

The monitoring-completeness subsystem guards the assumption that continuous monitoring actually occurred. Operational reality introduces data voids through sensor degradation, calibration cycles, communication outages, and maintenance windows, and a failure to collect a required sample is itself a monitoring and reporting violation under 40 CFR Part 141. Deploying Monitoring Gap Detection Algorithms lets the system classify missing data by cause, duration, and regulatory impact, distinguishing planned maintenance documented in a work-order system from unplanned telemetry loss. Imputed values must never satisfy a regulatory determination; the engine applies only the substitution conventions defined in the applicable rule (such as treating non-detects as zero or as half the reporting limit where permitted) or escalates to manual review when completeness falls below the required threshold.

The operational-tuning subsystem separates tunable operational boundaries from the fixed regulatory logic. Operational alarm limits drift in usefulness as source-water characteristics shift and treatment processes are optimized. Integrating Threshold Tuning Frameworks lets operators establish warning bands below regulatory MCLs, prompting proactive treatment adjustments before compliance boundaries are approached, while guaranteeing that tuning never alters a CFR-mandated threshold.

The severity-scoring subsystem ranks confirmed violations for response. When an exceedance is confirmed, the Severity Scoring Models prioritize it based on contaminant health risk, population exposed, exceedance duration, and the system’s compliance history; the resulting score drives automated routing to incident management, public-notification workflows, and regulatory reporting queues. Downstream, those routed outcomes are reconciled against the EPA and state code sets defined in Violation Code Classification.

The contract that binds these subsystems is a single validated reading record. Enforcing it with a schema library means malformed or under-attributed data is rejected at the boundary rather than silently propagating into a compliance calculation:

from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field


class QualityFlag(str, Enum):
    GOOD = "GOOD"
    ESTIMATED = "ESTIMATED"
    CALIBRATION = "CALIBRATION"
    BAD = "BAD"
    MISSING = "MISSING"


class ComplianceReading(BaseModel):
    location_id: str = Field(..., description="Monitoring point / entry-point identifier")
    parameter_code: str = Field(..., description="EPA contaminant / analyte code")
    value: float = Field(..., ge=0)
    unit: str = Field(..., description="Reporting unit, e.g. 'mg/L' or 'NTU'")
    method_code: str = Field(..., description="EPA-approved analytical method code")
    acquired_at: datetime = Field(..., description="Sample / read timestamp (UTC)")
    processed_at: datetime = Field(..., description="Pipeline ingestion timestamp (UTC)")
    quality_flag: QualityFlag = QualityFlag.GOOD
    ruleset_version: str = Field(..., description="Semantic version of the active rule set")

The quality flag is not decorative: each value dictates how the evaluation subsystems treat the reading, and the codes must be defined once and honored everywhere.

Security & Operational Technology (OT) Boundaries

The rule engine straddles the OT and IT worlds: its inputs originate on control networks and its outputs feed enterprise reporting and public-facing notification systems. That crossing must be one-directional and tightly scoped. Compliance telemetry should flow from control systems toward reporting environments and never the reverse, so that a compromise of the reporting tier cannot reach into SCADA. This is the discipline formalized in the cross-domain Security Boundary Design work, and the rule engine is a primary consumer of it.

Concretely, the engine runs in the IT/DMZ tier, not on the OT network. It receives data through a controlled conduit — a historian replica, a message broker, or a data diode — rather than by opening sessions back into PLCs or RTUs. Its service accounts follow least privilege: read-only access to the ingestion buffer, append-only access to the audit store, and no write path whatsoever into control systems. Network segmentation between the OT cell, the DMZ, and the enterprise zone limits lateral movement and query injection. Because the engine parses external-origin data, every field is treated as untrusted until it passes schema validation, which is exactly why the ComplianceReading contract rejects malformed input at the boundary. OT/IT convergence should follow recognized operational-technology security guidance so that the same engine that unlocks compliance automation does not widen the attack surface of critical infrastructure.

Audit Trail & Data Lineage Requirements

Audit readiness is the property that makes an automated determination defensible in an enforcement proceeding, and it is designed in from the first stage rather than bolted on. Every rule execution should emit a structured record containing the input parameters, the evaluation result, the rule-set version, and a deterministic hash signature. These records must be retained in tamper-evident, append-only storage for the period mandated by state primacy agencies and EPA recordkeeping requirements, preserving an unbroken chain of custody from raw acquisition through final reporting.

Immutability and idempotency are the two load-bearing guarantees. When a rule definition changes or a data correction lands, the engine must be able to back-apply the current rule to historical data for the open compliance period without producing duplicate violation records. Keying each determination on the combination of monitoring location, parameter, evaluation-window start, and rule-set version makes re-evaluation safe: reprocessing the same window with the same rule set resolves to the same record identity instead of a second row.

import hashlib


def evaluation_key(location_id: str, parameter_code: str,
                   window_start: str, ruleset_version: str) -> str:
    """Deterministic identity for one compliance determination.

    Keying on (location, parameter, window, rule-set version) lets the engine
    re-evaluate corrected or back-applied data without emitting duplicate
    violation records: the same inputs always resolve to the same key.
    """
    raw = f"{location_id}|{parameter_code}|{window_start}|{ruleset_version}"
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

Regular pipeline audits should confirm that the loaded rule logic matches current CFR text, that lineage metadata remains unbroken from acquisition to report, and that exception-handling protocols behave as designed under simulated outage scenarios. This requirement belongs in the engine’s acceptance criteria, not discovered during the first mid-year rulemaking after deployment.

Implementation Standards & Tooling

Building a production compliance rule engine demands software-engineering practices tuned to a regulated environment. Rule definitions live in version-controlled repositories, compile into immutable evaluation artifacts, and deploy through CI/CD pipelines with mandatory peer review. Schema enforcement at the ingestion boundary is handled by a library such as Pydantic, which turns under-attributed or mistyped data into an explicit validation error rather than a downstream miscalculation. Time-series handling leans on pandas or Polars for windowed aggregation, and heavy backfill or multi-location workloads are offloaded to an asynchronous task layer via the Async Batch Processing Setup patterns rather than blocking the ingestion path.

Testing is where regulatory correctness is proven. Unit and integration tests must cover the edge cases that silently corrupt averaging math: leap-year temporal windows, timezone and daylight-saving transitions, mixed-unit LIMS imports, partial evaluation windows, and concurrent determinations across multiple monitoring locations. Golden-file regression tests that replay historical datasets and assert on known compliance outcomes catch drift whenever the rule set or evaluation code changes.

Predictive analytics extends the deterministic core without ever overriding it. By integrating historical telemetry, treatment-process variables, and seasonal hydrological patterns, utilities can anticipate threshold approaches and optimize chemical dosing, filtration cycles, and distribution flushing. Machine-learning models introduce probabilistic risk scoring that complements CFR evaluation, but their outputs are strictly decision-support: they may trigger early operational interventions or reshape monitoring schedules, and they must be trained on validated data and monitored for concept drift, but they never substitute for a deterministic regulatory determination.

Jurisdictional & Primacy Variations

The final architectural requirement is that jurisdictional differences never fork the codebase. States with primacy may set limits more stringent than the federal floor, approve different treatment techniques, mandate distinct monitoring frequencies, and require their own reporting formats. Encoding any of these as if state == ... branches turns every regulatory update into a code change and a redeploy.

Instead, primacy parameters belong in version-controlled configuration tables that the engine resolves at runtime for each service area’s governing agency. A determination looks up the effective rule for its location, parameter, and date, applies it, and records both the resolved values and the rule-set version in the audit trail. This keeps the core evaluation logic unchanged when a primacy agency tightens a limit without an accompanying federal rulemaking, and it makes multi-state operation a matter of loading additional configuration rather than maintaining parallel code paths. Coupling this runtime resolution with the Monitoring Frequency Scheduling module ensures that the sampling calendar and the evaluation logic always reference the same jurisdictional rule set, closing the gap where a state-specific frequency and a federal-baseline threshold could otherwise disagree.

Related

• Core Architecture & SDWA Compliance Taxonomy — the regulatory taxonomy and MCL reference this engine resolves against
• SCADA Data Ingestion & Time-Series Sync — the upstream pipeline that feeds validated telemetry into the engine
• MCL Exceedance Logic Implementation — CFR-aligned threshold evaluation in detail
• Monitoring Gap Detection Algorithms — completeness checks and data-void resolution
• Severity Scoring Models — ranking confirmed violations for response and reporting
• Threshold Tuning Frameworks — operational warning bands beneath fixed regulatory limits