DMR Generation & SDWIS Submission Workflows

A compliance determination that never reaches a regulator is worthless, and this section closes that last gap for the Core Architecture & SDWA Compliance Taxonomy it belongs to. It covers how a validated measurement — already carrying its quality flag, method code, and rule-set version — is rolled up into a reporting period, cast into the exact record layout a regulator expects, and transmitted as a signed, acknowledged filing. Three filing families dominate a water utility’s calendar: NPDES Discharge Monitoring Reports (DMRs) submitted electronically through NetDMR to EPA’s Central Data Exchange (CDX); annual Consumer Confidence Reports (CCRs) prepared under 40 CFR Part 141 Subpart O; and compliance-sample results delivered to a state’s Safe Drinking Water Information System (SDWIS) as flat files or XML. This material is written for the compliance officers who sign the certifications, the municipal developers who own the submission service, and the Python automation engineers who must make a determination reproducible from raw sample to receipt. The determinations themselves arrive from the Violation Detection & Rule Engine Logic domain; this section is the reporting boundary where those outcomes become defensible regulatory filings.

Regulatory / Protocol Foundation

The three filing families answer to distinct legal authorities, and conflating them is the first mistake an automation project makes. NPDES Discharge Monitoring Reports are effluent reports: a wastewater or stormwater permit issued under the Clean Water Act specifies limits on parameters such as biochemical oxygen demand, total suspended solids, ammonia, and residual chlorine at named outfalls, and the permittee reports the actual monitored values each monitoring period. Since 2016 the NPDES Electronic Reporting Rule has made electronic submission through NetDMR the default, and NetDMR forwards signed reports into EPA’s ICIS-NPDES system by way of the Central Data Exchange. Consumer Confidence Reports, by contrast, are drinking-water documents mandated by the Safe Drinking Water Act: every community water system must deliver an annual report to its customers describing detected contaminants against their Maximum Contaminant Levels, with the content and format prescribed by 40 CFR Part 141 Subpart O. SDWIS submissions are the compliance-monitoring backbone underneath both — the sample results, monitoring schedules, and violation records a state primacy agency maintains and periodically synchronizes with EPA’s federal SDWIS.

What unifies them is a signature requirement. Electronic environmental reports submitted to EPA fall under the Cross-Media Electronic Reporting Rule (CROMERR), which demands identity-proofing, a valid electronic signature, and a tamper-evident copy of record for each submission. An automated pipeline therefore cannot simply POST a file; it must reach the point where a certifying official’s signature is bound to an immutable snapshot of exactly what was sent. The parameters being reported and the limits they are judged against are resolved through the SDWA MCL Reference Mapping, so the reporting layer never hard-codes a numeric limit; it looks up the applicable value, its unit, and its averaging basis, then reports the observed statistic beside it.

Because state primacy agencies may prescribe their own field layouts, code lists, and delivery portals, the SDWIS path is the least standardized of the three. Some states accept a fixed-width flat file keyed by a nine-character PWSID and a four-character analyte code; others require an XML instance validated against a state schema derived from the federal SDWIS data model. The reporting layer treats both the layout and the code translations as configuration resolved at runtime, exactly as the Violation Code Classification work resolves EPA and state code sets rather than embedding them in logic.

Architecture & Design Decisions

The submission service is designed as a deterministic transformer with an immutable memory, and three decisions shape everything downstream.

A report is a projection of the audit trail, never a new source of truth. Every number that appears on a DMR or CCR must already exist as a validated determination with its own lineage. The report builder is forbidden from computing a compliance value; it may only select, aggregate, round, and format values that the rule engine already produced. This is what keeps a filing reconstructible: given the same determinations and the same rule-set version, the builder must emit a byte-identical record. The lineage that makes this possible lives in the Audit Trail & Data Lineage Storage Patterns section, and every submission writes back a copy of record plus the regulator’s acknowledgment into that same append-only store.

Reporting-period boundaries are computed, not assumed. A monthly DMR period is not “the last thirty days”; it is a calendar month in the permit’s stated time zone, closed at a specific instant. A CCR reporting year is the prior calendar year for most parameters but reaches back further for contaminants monitored less frequently, and a locational running annual average carried into a CCR spans four quarters that may not align with the report year at all. The builder resolves each period explicitly from the permit or rule configuration and refuses to aggregate across a boundary it cannot name.

Submission is idempotent and acknowledgment-driven. Networks fail mid-transmission, and a regulator’s node may accept a report while the client times out waiting for the receipt. The service therefore keys each submission on the tuple of permit or system identifier, report type, and period end, so that a retry resolves to the same logical submission rather than a duplicate filing. A report is not “done” when it is transmitted; it is done when a CROMERR receipt or a node acceptance acknowledgment has been captured and reconciled against the submitted copy of record. Anything transmitted but unacknowledged sits in a pending state and is retried under the same idempotency key.

Structurally the service is a small pipeline: a period-close trigger, an aggregation stage that reads validated determinations, a family of report builders (one per filing type), a serialization and schema-validation stage, a signing and transport adapter per regulator, and an acknowledgment reconciler. Each stage hands a well-typed artifact to the next so that a stuck submission can be resumed from the last durable artifact rather than recomputed from scratch.

Phase-by-Phase Implementation

The build proceeds in four phases, each yielding an artifact the next consumes: a validated report record model, a set of reporting-period statistics, a rendered human-readable report, and a machine-readable state submission.

Phase 1 — Modeling the report record

The first artifact is a strict record model. A DMR cell is deceptively simple — a parameter, a statistical base, a value, and a unit — but it carries a rule that trips up naive implementations: a cell holds either a reported quantity or a No Data Indicator (NODI) code explaining its absence, never both and never neither. Encoding that as a validated invariant stops a malformed report at construction rather than at the regulator’s node.

Implementation steps:

1. Define the statistical-base and NODI vocabularies as enumerations keyed to the codes NetDMR and ICIS-NPDES expect, so an invalid code is impossible to construct.
2. Pin the permit identifier format and the five-character parameter code so a transposed field is rejected at the boundary.
3. Enforce the value-or-NODI exclusivity with a model-level validator, and require the monitoring-period end date so every record self-describes its reporting window.

from datetime import date
from decimal import Decimal
from enum import Enum

from pydantic import BaseModel, Field, model_validator


class StatBase(str, Enum):
    MONTHLY_AVERAGE = "MK"   # NetDMR statistical-base codes
    DAILY_MAXIMUM = "DD"
    DAILY_MINIMUM = "NL"
    MONTHLY_MAXIMUM = "MX"


class Nodi(str, Enum):
    NO_DISCHARGE = "C"       # NODI codes recognised by ICIS-NPDES
    CONDITIONAL_MONITORING = "7"
    NOT_REQUIRED = "9"


class DmrResult(BaseModel):
    permit_id: str = Field(..., pattern=r"^[A-Z]{2}\d{7}$")
    outfall: str = Field(..., min_length=3, max_length=3)
    parameter_code: str = Field(..., min_length=5, max_length=5)
    period_end: date
    stat_base: StatBase
    unit_code: str
    value: Decimal | None = Field(default=None, ge=0)
    nodi: Nodi | None = None

    @model_validator(mode="after")
    def value_xor_nodi(self) -> "DmrResult":
        if (self.value is None) == (self.nodi is None):
            raise ValueError("a DMR cell needs exactly one of value or nodi")
        return self

Phase 2 — Aggregating monitoring data to reporting-period statistics

The second artifact is the set of period statistics the record model will carry. Validated daily readings are grouped into calendar periods and reduced to the exact statistics the permit demands — most commonly a monthly average and a daily maximum — while samples whose quality flag disqualifies them from a regulatory determination are excluded before any arithmetic runs. For effluent parameters the report often needs a mass loading in pounds per day alongside the concentration, which combines the average concentration with the average daily flow:

where $\bar{C}_{\text{mo}}$ is the monthly average concentration in mg/L, $n$ the number of qualifying samples, $Q$ the average daily flow in million gallons per day (MGD), $L$ the mass loading in lb/day, and $8.34$ the pounds-per-gallon conversion for water that turns mg/L·MGD into lb/day.

Implementation steps:

1. Filter to qualifying samples on the quality flag before grouping, so an excluded reading can never enter an average.
2. Group by parameter and calendar period and compute each required statistic in one pass, retaining the sample count for the completeness check.
3. Derive mass loading from the aggregated concentration and the period’s average flow rather than averaging per-sample loadings, which would bias the result.

import pandas as pd


def summarize_period(readings: pd.DataFrame) -> pd.DataFrame:
    """Reduce validated daily readings to NPDES monthly statistics.

    `readings` columns: sample_date, parameter_code, value, quality_flag.
    Only GOOD-flagged samples are eligible for a regulatory statistic.
    """
    good = readings[readings["quality_flag"] == "GOOD"].copy()
    good["period"] = good["sample_date"].dt.to_period("M")
    grouped = good.groupby(["parameter_code", "period"])["value"]
    return grouped.agg(
        monthly_average="mean",
        daily_maximum="max",
        sample_count="size",
    ).reset_index()


def mass_loading_lb_per_day(concentration_mg_l: float, flow_mgd: float) -> float:
    """Effluent mass loading: concentration (mg/L) x flow (MGD) x 8.34."""
    return concentration_mg_l * flow_mgd * 8.34

Phase 3 — Rendering the Consumer Confidence Report

The CCR is the one filing meant for human readers, so its builder renders a document rather than a code-laden record. The content is nonetheless tightly regulated: Subpart O enumerates the required elements — the detected level, the range of detections, the MCL and its health goal (MCLG), the likely source, and any violation — and the report must present them for every contaminant detected during the reporting year. Templating keeps the layout in a reviewable artifact separate from the data, and rendering from validated rows guarantees the printed table matches the determinations.

Implementation steps:

1. Assemble one row per detected contaminant from the aggregated statistics and the resolved MCL reference, marking whether the year carried a violation.
2. Render the required Subpart O elements through a template so the wording and column order are version-controlled, not embedded in Python.
3. Emit the table in a portable format the delivery channel can wrap, keeping the same rows available for the mailed and web-posted copies.

from jinja2 import Environment, select_autoescape

CCR_TABLE = """\
| Contaminant | MCL | MCLG | Detected (avg) | Range (low–high) | Violation |
|---|---|---|---|---|---|
"""


def render_ccr_table(rows: list[dict]) -> str:
    """Render the Subpart O detected-contaminants table from validated rows."""
    env = Environment(
        autoescape=select_autoescape(default=False),
        trim_blocks=True,
        lstrip_blocks=True,
    )
    return env.from_string(CCR_TABLE).render(rows=rows)

Phase 4 — Serializing a SDWIS-compatible state submission

The final artifact is a machine-readable submission for the state’s SDWIS. Because states diverge, the serializer targets both common shapes: a fixed-width flat file where field position is significant, and an XML instance validated against a state schema. Fixed-width layouts are unforgiving — a field one character too long silently shifts every field after it — so each field is padded and truncated to its declared width explicitly.

Implementation steps:

1. Model the sample result once with the fields SDWIS keys on: the PWSID, the analyte code, the sample date, the numeric result, and the reporting unit.
2. Serialize to fixed width by padding and truncating each field to its exact column span, formatting the date and numeric result to the state’s specification.
3. Serialize the same records to XML for states that require it, emitting a declaration and a stable element order so the instance validates against the published schema.

import xml.etree.ElementTree as ET
from dataclasses import dataclass
from datetime import date


@dataclass(frozen=True)
class SampleResult:
    pwsid: str          # 9-char public water system id, e.g. "CA1910067"
    analyte_code: str   # 4-char SDWIS analyte code
    sample_date: date
    result: float
    unit: str


def to_fixed_width(rec: SampleResult) -> str:
    """Serialize one result to a common fixed-width SDWIS flat-file layout."""
    fields = [
        rec.pwsid.ljust(9)[:9],
        rec.analyte_code.rjust(4, "0")[:4],
        rec.sample_date.strftime("%Y%m%d"),
        f"{rec.result:012.4f}",
        rec.unit.ljust(4)[:4],
    ]
    return "".join(fields)


def to_sdwis_xml(results: list[SampleResult]) -> bytes:
    """Serialize results to a SDWIS-compatible XML instance."""
    root = ET.Element("SampleResults", attrib={"schemaVersion": "3.0"})
    for r in results:
        node = ET.SubElement(root, "Result")
        ET.SubElement(node, "PWSID").text = r.pwsid
        ET.SubElement(node, "AnalyteCode").text = r.analyte_code
        ET.SubElement(node, "SampleDate").text = r.sample_date.isoformat()
        ET.SubElement(node, "Value").text = f"{r.result:.4f}"
        ET.SubElement(node, "Unit").text = r.unit
    return ET.tostring(root, encoding="utf-8", xml_declaration=True)

Validation, Quality Flags & Edge Cases

A submission moves through a small state machine, and treating each transition as explicit is what prevents the two worst outcomes: a duplicate filing and a report the regulator silently rejected. A record is drafted, validated against the layout and range rules, transmitted, and then either accepted against a captured receipt or rejected with a reason that routes it back to a corrected draft.

Every serialized field carries a validation status that governs whether it may leave the service, and keeping this vocabulary consistent lets the acknowledgment reconciler reason about a whole submission from its parts.

Several edge cases recur and each needs a deterministic rule. No-discharge periods are the classic one: an outfall that did not discharge in a monitoring period must still be reported, using the no-discharge NODI code rather than a zero or a blank cell — a zero asserts a measurement that was never taken, and a blank reads as a missing report. Reporting-period boundaries bite when a sample taken minutes before or after midnight on a month boundary lands in the wrong period; resolving the period from an explicit, time-zone-aware close instant, not a naive date comparison, keeps the sample where the permit intends. Resubmissions must overwrite, not append: when a corrected DMR replaces an accepted one, it references the original submission so the regulator’s system supersedes rather than double-counts, and the audit trail records both the original and the correction with the reason. Incomplete monitoring — fewer qualifying samples than the schedule required — is not a reporting choice but a monitoring violation in its own right, so the builder flags the shortfall to the rule engine rather than quietly averaging whatever it has. Non-detects must be reported per the rule’s substitution convention, typically as a less-than value at the method reporting limit rather than as a zero, because a zero understates the average.

Deployment & Integration Patterns

The submission service runs in the enterprise or DMZ tier, well away from the control network, and consumes validated determinations through the same one-directional conduit the rest of the compliance stack uses — it reads from the audit store and the determination stream and writes only receipts back. It is packaged as a scheduled worker rather than a long-lived server, because reporting is inherently periodic: a period-close event or a cron-driven trigger wakes the aggregation stage, the builders run, and the transport adapters transmit.

Deployment guidance:

• Isolate the signing credential. The CROMERR electronic signature binds a real certifying official to a filing, so the signing key or delegated credential lives in a secrets manager, is used only by the transport adapter, and is never available to the builders. Signing happens on an immutable copy of record, so the signature always covers exactly what was sent.
• Make transport adapters pluggable. NetDMR, a state SDWIS portal, and a CCR delivery channel differ in protocol, authentication, and acknowledgment shape; isolating each behind a common submit-and-await-receipt interface keeps the builders regulator-agnostic and lets a new state be added as configuration and one adapter rather than a fork.
• Reconcile acknowledgments asynchronously. Some regulators acknowledge synchronously with a CROMERR receipt; others accept the payload and validate it on a delayed batch, returning a rejection hours later. The reconciler polls or receives those delayed results and advances each pending submission to accepted or rejected, so a late rejection is never mistaken for success.
• Persist the copy of record before transmitting. The exact bytes sent, the signature, and the acknowledgment all belong in the Audit Trail & Data Lineage Storage Patterns store, written before the network call so a crash mid-submission leaves a recoverable pending record rather than an untraceable gap.

When a large multi-facility utility closes dozens of permits and hundreds of monitoring points on the same day, the aggregation fan-out is offloaded to the asynchronous task layer rather than run inline, and the individual filings are still keyed idempotently so a retried batch never produces a duplicate. The determinations feeding all of this originate in the Violation Detection & Rule Engine Logic domain, and the codes that classify any reported violations are resolved through Violation Code Classification, so the reporting layer stays a formatter and transporter, never a second place where compliance is decided.

Production Validation Checklist

• Reporting-period boundaries resolved from explicit, time-zone-aware close instants and unit-tested across month and year boundaries
• Value-or-NODI exclusivity enforced at the record boundary, with no-discharge periods reported by NODI code rather than zero or blank
• Monthly averages and mass loadings computed only from qualifying quality-flagged samples, with sample counts retained for completeness checks
• Fixed-width SDWIS fields verified against the state’s column spec, including padding, truncation, and date and numeric formatting
• XML instances validated against the current state schema before transmission
• CCR content covers every Subpart O required element for all detected contaminants, with the MCL reference resolved at render time
• Every submission keyed idempotently on system, report type, and period end so retries cannot duplicate a filing
• Copy of record and CROMERR signature persisted to the append-only audit store before the network call
• Acknowledgment reconciler advances pending submissions and surfaces late rejections rather than assuming success
• Resubmissions reference and supersede the original filing, with both retained and the correction reason recorded

Failure Modes & Gotchas

The most damaging failure in a reporting pipeline is a silent rejection treated as a success. A report can pass every local validation, transmit cleanly, and still be rejected hours later by the regulator’s node for a schema mismatch or a code the state retired — and if the service marked the filing complete at transmission, no one learns of the gap until an enforcement letter arrives. The guard is to define “submitted” and “accepted” as distinct states and to advance to accepted only when a receipt or acceptance acknowledgment has been captured and matched to the copy of record; anything transmitted but unacknowledged stays visibly pending until reconciled. Build the reconciler to poll for delayed results and to alarm on submissions that remain pending past their acknowledgment window, because a regulator’s silence is not consent.

Three quieter traps compound this. A duplicate filing from a naive retry is nearly as bad as a missing one — it forces a manual correction and muddies the compliance record — which is why the idempotency key is not optional. A NODI omission, where a no-discharge period is left blank instead of coded, reads to the regulator as a report that was never filed, converting an operational non-event into a reporting violation. And a fixed-width drift, where one field is a character too long, shifts every downstream field so the analyte code lands in the date column and the whole record is misread; because such a file often still parses without error, only a length assertion per field and a validation pass against the state spec catches it before submission. Re-run the schema and layout checks whenever a state publishes a new code list or schema version, since a filing that validated last quarter can fail this quarter against an unchanged payload.

Related

• Core Architecture & SDWA Compliance Taxonomy — the parent section whose vocabulary and rule-set versions every filing carries
• SDWA MCL Reference Mapping — resolves the limits, units, and averaging bases a report presents beside each value
• Violation Code Classification — the EPA and state code sets used when a filing reports a violation
• Audit Trail & Data Lineage Storage Patterns — the append-only store holding each copy of record and acknowledgment
• Generating NPDES Discharge Monitoring Reports in Python — the DMR record set and NetDMR path in depth
• Building Consumer Confidence Reports from Compliance Data — assembling the Subpart O annual report
• Formatting SDWIS State Submission Files — flat-file and XML serialization against state schemas