Monitoring Frequency Scheduling for SDWA Compliance Calendars

Monitoring frequency scheduling is the service that converts static Safe Drinking Water Act (SDWA) sampling mandates into executable, per-system compliance calendars — and then proves, on demand, that every required sample landed inside its window. This topic covers how to resolve a contaminant’s monitoring cadence from population, source type, and tier; how to open and close monitoring windows deterministically; and how to dispatch, track, and reconcile sampling tasks so that a missed sample is caught before the reporting deadline rather than during an audit. It is written for water utility operations staff who own the sampling program, environmental compliance teams who sign the reports, and the municipal Python developers who automate the calendar. It sits inside the Core Architecture & SDWA Compliance Taxonomy and consumes the limits published by the SDWA MCL Reference Mapping while feeding sampling-completeness signals downstream to the Violation Detection Rule Engine.

Deterministic ingestion-to-validation scheduling pipeline with fallback routing.

Regulatory Foundation

Monitoring frequency is not a single number; it is a function of contaminant family, system population, source water type, and the system’s recent compliance history. The governing rules live in 40 CFR Part 141, and each rule family defines its own cadence logic. The scheduler exists to encode those rules as data so a regulatory change is a table edit, not a code deployment.

The families that drive most automated scheduling are:

• Revised Total Coliform Rule (RTCR): routine sample count scales with population served — from 1 sample/month for the smallest systems to hundreds per month for large systems — and a positive result triggers same-month or next-month repeat and additional sampling.
• Surface Water Treatment Rules (SWTR/IESWTR/LT2): continuous or four-hour turbidity and disinfectant-residual monitoring, driving the real-time NTU capture handled in Modbus TCP Parsing Workflows.
• Stage 1 & Stage 2 Disinfectants and Disinfection Byproducts Rules (DBPR): quarterly TTHM/HAA5 monitoring at locational sites, with an eligibility path to reduced (annual) monitoring.
• Phase II/V inorganic and organic chemicals: standard three-year or nine-year cycles with waiver-based reductions.
• Lead and Copper Rule (LCR): six-month monitoring periods that can step down to annual or triennial after consecutive rounds below the action level.

Two structural mechanics matter more than any individual number. First, primacy variation: states with delegated authority may impose cadences stricter than the federal floor, so the frequency table must be keyed on jurisdiction, not hard-coded to EPA defaults. The official Safe Drinking Water Act guidance defines the federal baseline that state overrides layer on top of. Second, tiering: routine monitoring can be increased (after a detection or violation) or reduced (after a demonstrated record of compliance), and both transitions must recalibrate the calendar immediately. A reduced-monitoring waiver, a tier escalation, and a source-water change are each schedule-mutating events.

For DBP systems, the reduced-monitoring decision is a threshold test on an average, which is why it belongs in the scheduler rather than in ad hoc review. A system qualifies to step from quarterly to annual monitoring when its locational running annual average stays at or below half the maximum contaminant level:

where $\bar{C}_{\ell,i}$ is the mean of all samples at location $\ell$ in quarter $i$. Encoding the window (four quarters), the grouping (per location), and the coefficient (0.5) as configuration lets one evaluator drive every reduction decision without branching code.

Architecture & Design Decisions

The scheduler is a rule-driven service with three responsibilities that must stay separate: resolve the applicable frequency for a monitoring obligation, materialize that frequency into concrete open/close windows on a calendar, and reconcile collected samples against those windows to emit a completeness verdict. Collapsing resolution and reconciliation into one pass is the design mistake that produces silent gaps — a window can only be judged complete by a component that knows the rule independently of the data.

The data contract entering the scheduler is a monitoring obligation: a system identifier, a contaminant/analyte code, the applicable limit resolved from the SDWA MCL Reference Mapping, the population and source-type facets, and the current tier. The contract leaving the scheduler is a stream of monitoring windows, each with an explicit open and close instant, a required sample count, and — once the window closes — a completeness verdict. Timestamps entering the reconciliation stage must already be normalized to UTC by Time-Series Alignment Strategies; the scheduler assumes aligned input and never re-guesses a sample’s collection instant.

A key trade-off is periodic cron expressions versus rule-materialized windows. Cron is adequate only where a cadence is strictly periodic and calendar-agnostic (e.g., “poll turbidity every 4 hours”). It is the wrong tool for regulatory windows, because SDWA periods are anchored to calendar quarters, fiscal boundaries, and population-dependent counts that cron cannot express. The scheduler therefore materializes windows from the rule table and uses a periodic runner only to drive the reconciliation sweep, not to define when samples are due. Gap signals produced at window close feed the monitoring gap detection algorithms so that an unmet count becomes a candidate monitoring violation.

The core resolution table is small and explicit. A representative slice:

Phase-by-Phase Implementation

The reference implementation splits cleanly along the three responsibilities above. Each phase is a pure, testable function so the same logic runs in a scheduled worker and in a backfill job.

Phase 1 — Resolve the applicable frequency

Resolution binds a monitoring obligation to a cadence by looking up the rule table with the system’s facets and current tier. The lookup is deterministic and jurisdiction-aware; primacy overrides win over federal defaults.

1. Load the versioned rule table for the obligation’s jurisdiction.
2. Select the base cadence by rule family and facet key.
3. Apply the tier modifier (increased / routine / reduced).

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum


class Tier(str, Enum):
    INCREASED = "increased"
    ROUTINE = "routine"
    REDUCED = "reduced"


@dataclass(frozen=True)
class MonitoringObligation:
    system_id: str
    analyte: str
    jurisdiction: str
    population: int
    source_type: str          # "surface" | "groundwater"
    tier: Tier = Tier.ROUTINE


@dataclass(frozen=True)
class Cadence:
    period: str               # "monthly" | "quarterly" | "annual"
    required_count: int       # samples required per period


def resolve_cadence(obl: MonitoringObligation, rule_table: dict) -> Cadence:
    """Bind an obligation to a cadence using the jurisdiction rule table.

    rule_table is keyed (jurisdiction, analyte) -> list of facet rules,
    each rule a dict with match predicates and a base Cadence. State
    (primacy) tables are merged over the federal baseline before lookup.
    """
    rules = rule_table.get((obl.jurisdiction, obl.analyte)) \
        or rule_table.get(("US", obl.analyte))
    if not rules:
        raise KeyError(f"No monitoring rule for {obl.analyte} in {obl.jurisdiction}")

    for rule in rules:
        if _facets_match(rule["match"], obl):
            base = Cadence(rule["period"], rule["required_count"])
            return _apply_tier(base, obl.tier)
    raise KeyError(f"No facet match for {obl.system_id} / {obl.analyte}")


def _facets_match(match: dict, obl: MonitoringObligation) -> bool:
    if "source_type" in match and match["source_type"] != obl.source_type:
        return False
    lo, hi = match.get("pop_min", 0), match.get("pop_max", float("inf"))
    return lo <= obl.population <= hi


def _apply_tier(base: Cadence, tier: Tier) -> Cadence:
    if tier is Tier.INCREASED:
        # Escalation doubles the count within the same period, minimum 2.
        return Cadence(base.period, max(2, base.required_count * 2))
    if tier is Tier.REDUCED:
        return Cadence("annual", 1)
    return base

Phase 2 — Materialize monitoring windows

A cadence is not yet a calendar. Materialization projects the cadence onto anchored, timezone-correct windows with explicit open and close instants. Windows are half-open intervals [open, close) so that an instant on a boundary belongs to exactly one period.

1. Choose the anchor (calendar month, calendar quarter, or year start) for the cadence period.
2. Generate windows across the reporting horizon.
3. Stamp each window with the required sample count from the cadence.

from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo


@dataclass(frozen=True)
class MonitoringWindow:
    analyte: str
    open_utc: datetime
    close_utc: datetime
    required_count: int


def materialize_windows(
    obl: MonitoringObligation,
    cadence: Cadence,
    horizon_start: datetime,
    horizon_end: datetime,
    local_tz: str = "America/New_York",
) -> list[MonitoringWindow]:
    """Project a cadence onto UTC half-open windows anchored to local
    calendar boundaries (regulatory periods are local, storage is UTC)."""
    tz = ZoneInfo(local_tz)
    step = {
        "monthly": lambda d: _add_months(d, 1),
        "quarterly": lambda d: _add_months(d, 3),
        "annual": lambda d: _add_months(d, 12),
    }[cadence.period]

    windows: list[MonitoringWindow] = []
    cursor = _period_start(horizon_start.astimezone(tz), cadence.period)
    while cursor < horizon_end.astimezone(tz):
        nxt = step(cursor)
        windows.append(
            MonitoringWindow(
                analyte=obl.analyte,
                open_utc=cursor.astimezone(timezone.utc),
                close_utc=nxt.astimezone(timezone.utc),
                required_count=cadence.required_count,
            )
        )
        cursor = nxt
    return windows


def _add_months(d: datetime, months: int) -> datetime:
    total = (d.year * 12 + (d.month - 1)) + months
    year, month = divmod(total, 12)
    # Day is always 1 for period starts, so no clamping is required.
    return d.replace(year=year, month=month + 1, day=1)


def _period_start(d: datetime, period: str) -> datetime:
    base = d.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
    if period == "quarterly":
        first_month = ((d.month - 1) // 3) * 3 + 1
        return base.replace(month=first_month)
    if period == "annual":
        return base.replace(month=1)
    return base

Phase 3 — Reconcile samples against windows

Reconciliation is where completeness is decided. For each closed window, count the valid samples whose collection instant falls inside [open, close); if the count is short, emit a gap record. Crucially, the verdict is driven from the rule (required_count) outward, never inferred from whatever data happens to exist.

1. Bucket collected samples into windows by UTC collection instant.
2. At or after each window’s close, compare observed count to required count.
3. Emit COMPLETE or a GAP record carrying the shortfall.

@dataclass(frozen=True)
class Sample:
    analyte: str
    collected_utc: datetime
    valid: bool               # excludes rejected / QC-failed results


@dataclass(frozen=True)
class Reconciliation:
    window: MonitoringWindow
    observed: int
    verdict: str              # "COMPLETE" | "GAP" | "OPEN"


def reconcile(window: MonitoringWindow, samples: list[Sample],
              now_utc: datetime) -> Reconciliation:
    observed = sum(
        1 for s in samples
        if s.valid and s.analyte == window.analyte
        and window.open_utc <= s.collected_utc < window.close_utc
    )
    if now_utc < window.close_utc:
        return Reconciliation(window, observed, "OPEN")
    verdict = "COMPLETE" if observed >= window.required_count else "GAP"
    return Reconciliation(window, observed, verdict)

The distinction between monthly and quarterly materialization — and the tier transitions that flip a system between them — is developed in depth in Automating Monthly vs. Quarterly SDWA Monitoring Schedules.

Validation, Quality Flags & Edge Cases

Calendar arithmetic is where schedulers quietly fail. The following cases must each have an explicit test, because every one of them can shift a sample into the wrong reporting period.

• Timezone anchoring. Regulatory periods are local-calendar constructs; storage and comparison are UTC. A sample collected at 11:30 PM local on the last day of a quarter can land in the next UTC quarter if boundaries are computed in UTC. Always derive period boundaries in the system’s local zone, then convert to UTC — the materialize_windows function does exactly this.
• DST transitions. A local “month start” is still midnight local, but the UTC offset shifts across spring/fall transitions. Using zoneinfo (rather than a fixed offset) keeps boundaries correct through the changeover.
• Leap years and month-length variation. Never add 30 days to advance a month. Advance by calendar month (_add_months) so February, 31-day months, and leap days are handled by construction.
• Half-open intervals. Using [open, close) prevents a boundary sample from being double-counted into two adjacent windows — the single most common source of phantom duplicate compliance records.
• Partial-window handling. A window that opens mid-period (a new source coming online) must pro-rate or explicitly waive its required count; treating it as a full window fabricates a gap.
• Idempotent recompute. Re-running reconciliation must be safe. Keyed upserts ({system_id}:{analyte}:{window_open}) ensure a replay overwrites its own prior verdict instead of creating a second record.

A quality-flag state machine keeps window status auditable as samples arrive, results are invalidated, or a window closes short:

Every state transition is written to an append-only ledger capturing the triggering rule, the input parameters, and the resulting instant — the provenance a primacy agency expects when a missed or delayed sample is later mapped to the appropriate Violation Code Classification.

Deployment & Integration Patterns

The scheduler runs as a small, stateless worker over durable state. Two dispatch layers are typical: a persistent job scheduler such as APScheduler or Celery beat drives the periodic reconciliation sweep, following the persistent-jobstore patterns documented for async batch processing setup. Configure a database-backed job store (Postgres or Redis) so scheduled sweeps survive a service restart — an in-memory store silently drops the calendar on every deploy.

Design guidance for a production deployment:

• Materialize windows ahead of time, sweep frequently. Generate windows for the full reporting horizon at rule-change time; run reconciliation on a short interval (e.g., hourly) so an approaching close is caught early. The sweep is cheap and idempotent.
• Separate the runner from the rule. The periodic runner only triggers reconciliation; window boundaries always come from the materialized calendar, never from the cron expression.
• Read-only rule tables at runtime. Mount the versioned rule set read-only and pin its version hash into every ledger entry, so a schedule can be reproduced exactly as it was evaluated.
• Backpressure on late data. Reconciliation must tolerate late-arriving lab results; keep windows re-openable and process a bounded lookback each sweep rather than the entire history.
• Alert on approaching close, not just on gap. Wire a warning when an OPEN window is within its lead time and still short, giving field crews time to collect before the deadline.

Production Validation Checklist

• Rule table is versioned and keyed on jurisdiction so primacy overrides win over federal defaults.
• Period boundaries are computed in the system’s local timezone, then converted to UTC.
• Month/quarter advancement uses calendar-month arithmetic, not fixed day counts (leap-year safe).
• Windows are half-open [open, close) and boundary samples land in exactly one period.
• Tier transitions (increased / routine / reduced) recalibrate the calendar immediately on the triggering event.
• DBP reduced-monitoring eligibility is evaluated from the LRAA rule, not set manually.
• Reconciliation is idempotent via keyed upserts and safe to replay during backfill.
• The persistent job store is database-backed and survives service restarts.
• Every schedule mutation and window verdict is written to the append-only ledger with the rule version hash.
• A warning fires while a window is still OPEN and short, ahead of the reporting deadline.

Failure Modes & Gotchas

The highest-consequence failure is not a missed sample but an undetected missed sample — one where the system believed a window was satisfied when it was not. It happens when completeness is inferred from the data (“we have readings, so we’re fine”) instead of asserted from the rule (“this window required 5 samples and received 4”). The fix is structural: each monitoring window is explicitly opened and closed by the rule engine, and a formal GAP record is written whenever the required count is unconfirmed at close. That verdict must be produced by a component that knows the requirement independently of the collected data, so the absence of samples is itself an observable event. Caught at window close, a shortfall becomes a same-period corrective action; caught at audit, it becomes a documented monitoring violation. The entire three-stage design above — resolve, materialize, reconcile — exists to move that detection to the earliest possible moment.

Related

• Core Architecture & SDWA Compliance Taxonomy — parent architecture and shared vocabulary
• SDWA MCL Reference Mapping — the limit and averaging-rule source the scheduler resolves
• Violation Code Classification — where missed-window gaps become standardized codes
• Automating Monthly vs. Quarterly SDWA Monitoring Schedules — tier transitions and period materialization in depth
• Monitoring Gap Detection Algorithms — downstream consumer of window-completeness signals