Cannabis Regulatory Data Normalization Spec

At a glance. One append-only fact row — metric_observation — is exactly one observed number with its full dimensional context (metric · period · place · market segment · product category · value · unit) and its provenance. You can compare "flower sales" across states and recover that Massachusetts called it Buds; values keep their native units (conversion is a query-layer concern, never at write); and every cross-state crosswalk carries a confidence label so the data never silently lies. The spec and its JSON Schemas are licensed CC BY 4.0 — free to implement, adapt, and cite, even commercially, with attribution (§0); the public government data they normalize is owned by no one. Stability and versioning are in §2; conformance in §8.

Example — one Florida weekly medical THC-dispensed observation:

{
  "jurisdiction_code": "FL",
  "period_start": "2026-06-06", "period_end": "2026-06-12", "cadence": "weekly",
  "geo_level": "state", "is_aggregate": false, "operator_name_raw": "Trulieve",
  "market_segment": "medical", "product_category": "all",
  "metric": "thc_dispensed", "value": 184500000, "unit": "mg",
  "suppressed": false, "is_latest": true, "restated": false
}

0. License

The specification — this document (SPEC.md) and the JSON Schema files served at /static/spec/*.json — is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0). You may freely use, implement, adapt, and redistribute it — including in commercial or competing products — as long as you give appropriate credit to DevCannabis and link back to this spec.

That open license covers the standard only. The DevCannabis platform — its code, normalization pipeline, compiled datasets, editorial analysis, and brand — is source-available but all rights reserved (not open-source). In short: the method is open; the implementation and the dataset are not. The underlying government records are public facts owned by no one.

1. Purpose & positioning

Every U.S. cannabis regulator publishes market data in its own shape: Florida reports trailing weekly milligrams of THC dispensed per medical operator; Massachusetts publishes daily sales per product category; Colorado and Oregon publish monthly dollars by county; Illinois splits resident vs non-resident; Washington reports fiscal years. The measures, cadences, geographies, market channels, and even the word for "the licensed business" all disagree.

This spec defines one append-only, long/narrow fact record — metric_observation — where one row is exactly one observed number with its full dimensional context and provenance. It normalizes the shared concepts (a metric, a period, a place, a market segment, a product category, a value, a unit) into a small canonical vocabulary, while preserving every jurisdiction's local terminology and verbatim labels alongside the normalized values. A consumer can compare "flower sales" across states and recover that Massachusetts called it Buds and Florida called it Marijuana in a Form for Smoking.

Two design commitments make this honest rather than merely tidy:

• Native units, no forced conversion at storage. value/unit store exactly what the source published. Conversions live in the query layer.
• Crosswalks are per-jurisdiction and confidence-labeled. The system can say "MA vape vs OR vape is not comparable" instead of silently lying.

The underlying regulator records are public government data and are not owned by DevCannabis; this specification covers only the normalization schema and vocabulary. See LICENSE.

Positioning relative to existing docs

This spec is the schema + normalization reference. It does not replace the deeper operational docs; it ties them together and points to them:

Where this spec and a doc disagree on adapter field rules, docs/adapter-contract.md and the test module win. This spec restates concepts for portability and does not redefine them.

2. Status, stability & versioning

• Status: v0.1 draft. The record shape, enums, and metric/category taxonomies below are implemented and live in the codebase (Alembic revision 015), but the spec itself is not yet stable.
• Coexistence note: the canonical forward model is metric_observations. Florida's legacy weekly_reports table (app/models/report.py) still coexists under a dual-write parity gate; retirement is gated on multiple clean cycles and has not occurred. A spec-conformant consumer reads metric_observations.
• Versioning policy: SemVer-style MAJOR.MINOR.
  • Open vocabularies — metric, product_category, unit — grow additively (MINOR) and are forward-compatible without a schema bump: the JSON Schema constrains them to type: string plus a non-validating x-known-values list, so a consumer validating today's schema against a future dataset that adds a metric/category/unit does not break. Tolerate unseen members of these three.
  • Closed enums — cadence, geo_level, market_segment, buyer_residency — and the field set are bounded. Adding a member or a new optional field is still MINOR, but the schema does constrain these, so a consumer must adopt the newer schema version to validate the new data. Pin the $id schema version you build against.
  • Breaking changes (removing/renaming a field or value, changing nullability or the unique key) are MAJOR.
  • Enum values are persisted as their lowercase string .value (e.g. cadence="weekly", unit="usd"), never the Python member name. The string values are the contract.
• Minimum conformance revision: Alembic 015. An environment stuck at 012 has a narrower unique key (it lacks the product_category_raw term — see §3.4) and is not conformant.

2.1 Single source of truth

The Python enums in app/models/metric_observation.py and the metric catalog (METRIC_DEFINITION_SEEDS) are the source of truth. The published JSON Schemas and the §3.1 / §4 / §5.1 / §6.4 vocabularies are kept in lockstep by app/tests/test_spec_taxonomy_drift.py, which fails CI on any divergence — so this document and the schemas cannot silently drift from the running platform.

3. The normalized record: metric_observation

Table metric_observations. Append-only: published numbers are never UPDATEd in place; corrections append a new row (see §3.3). Source: app/models/metric_observation.py.

3.1 Field table

value is fixed-precision Numeric(20, 4) (20 total digits, 4 decimal places) — model it as a fixed decimal, never an integer or arbitrary-precision float. Enum columns persist the lowercase string value.

* value nullability is conditional: enforced by CHECK ck_observation_value_or_suppressed: value IS NOT NULL OR suppressed.

Three distinct time concepts must never be conflated: period_start/period_end (coverage window), published_at (source publication date), created_at (DB insert time).

Enumerated value sets. The closed enums (changed only by a versioned bump — see §2):

• cadence — daily · weekly · monthly · quarterly · annual
• geo_level — state · county · province · municipality
• market_segment — medical · adult_use · combined · delivery · wholesale · unknown
• buyer_residency — resident · non_resident · all

The open vocabularies (§2 — grow without a schema bump) are: metric (§4), product_category (§5.1), and unit — usd · count · receipts · mg · g · oz · lb · usd_per_g · usd_per_lb. MetricDefinition.value_kind — flow · stock · price · ratio — governs aggregation (§3.5).

3.2 The is_latest restatement rule (current vintage)

Current-vintage uniqueness is enforced by a partial unique index uq_observation_dimensions_latest, WHERE is_latest (postgresql_where and sqlite_where), over the full dimensional key (§3.4). Superseded history rows (is_latest=false) are exempt, so a key's full revision history lives alongside its current vintage.

3.3 Restatement semantics (append-only)

A revision is never an in-place UPDATE. It is one new row plus two flag flips:

1. Append the new observation with restates_observation_id = the old row's id (forward pointer).
2. On the old row, set is_latest = false.
3. On the old row, set restated = true.

3.4 The dimensional key & the coalesce invariant

The unique key, in order: jurisdiction_code, metric, period_start, period_end, geo_level, coalesce(geo_code,''), coalesce(operator_name_raw,''), is_aggregate, market_segment, buyer_residency, product_category, coalesce(product_category_raw,'').

Two invariants are load-bearing and must not be "simplified":

• Verbatim keying. The key uses operator_name_raw (+ is_aggregate) — not operator_id — because every alias-resolution failure carries operator_id = NULL and would otherwise collide. It likewise uses product_category_raw — not just canonical product_category — so native sub-grain rows (MA Concentrate vs Concentrate (Bulk) vs Concentrate (Each), all canonically concentrate) coexist without forced aggregation.
• The coalesce invariant. The three nullable key parts (geo_code, operator_name_raw, product_category_raw) are coalesced to '' inside the index so a NULL participates as a concrete value (Postgres NULLs are otherwise distinct and defeat uniqueness).

Schema-source drift gotcha: tests build the schema via SQLAlchemy create_all (reads the ORM __table_args__); production uses Alembic migrations 012 + 015. The product_category_raw coalesce term in the unique key exists only because migration 015 recreated the index. An environment stuck at 012 has a narrower unique key. The ORM and migration 015 are defined identically and are authoritative.

3.5 is_aggregate and aggregation safety

is_aggregate=true marks a state/market rollup with no single operator. Whether a value may be summed is NOT stored on the observation — it lives on MetricDefinition.value_kind (flow/stock/price/ratio), reached via the metric FK. Flows sum; stocks and prices never do. Consumers MUST join to metric_definitions and check value_kind before aggregating.

3.6 Worked examples

A suppressed Colorado county row — the source withheld a small-county figure; suppression is data, not a missing row (value is NULL only because suppressed is true):

{
  "jurisdiction_code": "CO", "period_start": "2026-03-01", "period_end": "2026-03-31", "cadence": "monthly",
  "geo_level": "county", "geo_code": "08097", "geo_name_raw": "Pitkin", "is_aggregate": false,
  "market_segment": "medical", "product_category": "all", "metric": "sales",
  "value": null, "unit": "usd", "suppressed": true, "suppression_reason": "confidentiality_nr",
  "is_latest": true, "restated": false
}

A statewide aggregate rollup (is_aggregate: true, no operator dimension):

{
  "jurisdiction_code": "CO", "period_start": "2026-03-01", "period_end": "2026-03-31", "cadence": "monthly",
  "geo_level": "state", "is_aggregate": true, "market_segment": "adult_use", "product_category": "all",
  "metric": "sales", "value": 99145201, "unit": "usd", "suppressed": false,
  "is_latest": true, "restated": false
}

A restatement pair (§3.3) — the regulator revised a prior figure; the old row flips to is_latest: false / restated: true, and the replacement points back via restates_observation_id:

[
  { "id": 5012, "metric": "sales", "value": 42000000, "unit": "usd", "is_latest": false, "restated": true },
  { "id": 6240, "metric": "sales", "value": 41880000, "unit": "usd", "is_latest": true, "restated": false, "restates_observation_id": 5012 }
]

A per-jurisdiction crosswalk row (product_category_map, §5.2) mapping a verbatim source label to a canonical category with a confidence — lossy flags that the rollup loses detail:

{ "jurisdiction_code": "MA", "native_category": "Concentrate (Each)", "canonical_category": "concentrate", "confidence": "lossy" }

4. Metric taxonomy (canonical metric codes)

The canonical catalog is METRIC_DEFINITION_SEEDS in app/services/metric_observations.py (the live source of truth, 14 codes). Migration 012 froze a 13-code subset; the only delta, avg_price, lives only in the service list and reaches migrated databases at boot via ensure_metric_definitions(). Each row is (code, label, default_unit, value_kind).

value_kind is the aggregation contract, not decoration. ratio exists in the enum but no current seed uses it (reserved). metric_observations.metric is an FK to metric_definitions.code; adding a metric means adding a MetricDefinition row, not a free string.

5. Product-category taxonomy & the fail-loud rule

5.1 Canonical categories (product_category, 18 values)

all, flower, shake_trim, kief, pre_roll, infused_pre_roll, concentrate, vape, edible, beverage, tincture, topical, capsule, suppository, seed_clone, biomass, other, unknown.

unknown and other are distinct sinks: unknown = unmapped / needs-review (the fail-loud target); other = a deliberately mapped misc bucket (e.g. MA Waste → other).

5.2 Per-jurisdiction crosswalk (product_category_map)

Native source category → canonical, keyed per jurisdiction. Each row carries a confidence (CrosswalkConfidence: exact / approximate / lossy) so the UI can badge lossy comparisons rather than present them as exact. Columns: jurisdiction_code, native_category (verbatim), canonical_category, includes_notes, confidence, effective_from/effective_to, citation_id. Unique on (jurisdiction_code, native_category, effective_from). Seeding via ensure_product_category_map() is non-overwriting: a shipped mapping is only corrected via a new effective window or explicit migration. Coverage today: FL and MA ship crosswalk seeds; CO and OR do not populate the table (their statewide series use product_category=all).

5.3 The unmapped-category-fails-loud rule

An unmapped native string MUST be handled loud, never silent: store the observation as canonical product_category=unknown, preserve the verbatim string in product_category_raw, AND emit a loud alert (logger.error + stdout). Never drop, never guess. Fail-loud does NOT mean crash. The reference implementation is the Massachusetts loader (app/services/ma_open_data.py), which collects unmapped strings and logs the sorted set: [ma_open_data] UNMAPPED categories (stored as 'unknown'): ….

6. The adapter contract

A jurisdiction adapter is a Python package under app/states/<jurisdiction>/ (config.py, operators.py, regulatory.py, __init__.py). Its __init__.py assembles a single top-level ADAPTER dict — the only object the rest of the platform reads (via states/registry.py).

Authority: the executable contract is app/tests/test_state_adapters.py (the REQUIRED_*_KEYS sets); the prose spec is docs/adapter-contract.md. scripts/validate_state_data.py is a secondary, looser CI check over operators/aliases/lineage/STATE_INFO only — not the source of truth.

6.1 Required top-level ADAPTER keys (exactly 12)

code, name, status, state_info, jurisdiction_profile, data_access_profile, program_model, research_profile, capabilities, config, entities, regulatory. Extra keys are allowed; missing any fails. adapter['code'] == state_info['code'] and adapter['name'] == state_info['name'] are also asserted.

Profile dicts are serialized from frozen dataclasses (states/adapter.py), so every dataclass field is a required key by construction. See app/tests/test_state_adapters.py for the exact REQUIRED_*_KEYS sets — reference them, do not re-list.

6.2 Status lifecycle

status is exactly one of two values:

Keep planned until a maintainer validates and flips to active.

6.3 Normalization invariants

• jurisdiction_profile.normalized_operator_term must be operator (hard-asserted by the contract test). state_info['canonical_entity'] must also be operator, but that one is enforced only by the looser scripts/validate_state_data.py, not the contract test. The local term (MMTC, Licensee, ATC, Dispensary) lives in operator_term / local_operator_term / terminology['operator'].
• Canonical operator records must use neutral field names; state-specific column names (mmtc_name, mmtc_count, licensee_name, atc_name) are banned.
• active_years must be ascending-sorted and every listed year must have a data_sources entry.
• module_gates AND research_profile['modules'] must each include the five module keys: timeline, licensing_rounds, claim_ledger, jurisdiction_blueprint, source_workbench.
• Alias vs lineage are different: an alias maps a raw-name variant to a canonical operator; lineage records corporate continuity. Lineage records require {predecessor, operator, change_type, transition_date} where operator is the successor and exists in OPERATORS, and change_type ∈ {acquisition, rebrand, acquisition_rebrand, merger, spinoff, license_transfer}. (The _template LINEAGE comment is stale — follow the validator and Florida.)

6.4 capabilities

capabilities is a machine-readable feature map: top-level flags are booleans (or, where a feature has sub-facets, a small dict). The contract test requires it to be a dict and capabilities['sales'] to be a dict; an adapter may omit any flag.

The allowed top-level flag names are a canonical catalog — CANONICAL_CAPABILITY_FLAGS in states/adapter.py, enforced by tests/test_state_adapters.py. An adapter must not invent new top-level flags ad hoc; adding a genuinely new capability means extending the catalog first, so the vocabulary stays documented and consistent across jurisdictions. The catalog (v0.1):

Nested sub-keys under sales are likewise catalogued (CANONICAL_SALES_FLAGS). Cataloguing the sub-keys of the other nested dicts (reports, legal, price_data, channels) is deferred to a later version (§11).

6.5 Registration

Three edits in states/__init__.py: the from . import … line; an AVAILABLE_STATES entry; a STATE_CODE_TO_MODULE entry. The JURISDICTION_* names are read-only = aliases — do not edit them. Registration only makes metadata resolvable; public liveness is governed by status. See app/docs/NEW_STATE_GUIDE.md for the full build order.

7. Provenance & confidence model

A published number carries two provenance pointers, both nullable: source_snapshot_id (what was fetched) and citation_id (the claim's canonical source).

7.1 Source snapshots (content-addressed capture)

SourceSnapshot is an immutable capture of one raw payload, content-addressed by content_sha256; refetching identical bytes records nothing new (idempotent, deduped by (source_key, collection_key, jurisdiction_code, content_sha256)).

content_format is the stored PAYLOAD enum — what the bytes are: json / html / pdf / csv / text / binary / other (default other).

Two content_format namespaces must not be confused. SourceSnapshot.content_format is the payload enum above. A separate fetcher dispatch key (SourceRef.content_format: csv/json/socrata/xlsx/tableau_csv/pdf/html) describes how bytes are retrieved and is a free string, not the enum. They are bridged by _PAYLOAD_FORMAT, which maps the cross-namespace cases (socrata→json, tableau_csv→csv, xlsx→binary, unknown→other) and the identity cases (csv→csv, json→json, pdf→pdf, html→html). Writing a fetcher key into the snapshot column is an out-of-enum value Postgres rejects and SQLite silently accepts — a documented prod-only bug class.

7.2 Citations (the durable claim layer)

The canonical model is Citation + CitationLink (see docs/source-citation-standard.md). A Citation is a reusable source record; a CitationLink ties one citation to one claim/target. Citation.confidence is the machine enum CitationStatus: source_confirmed / parser_validated / needs_review / inferred / rejected (default needs_review).

7.3 The four human-facing confidence labels

The public, prose confidence taxonomy (INTEGRITY.md, app/DATA_NOTES.md):

These four prose labels do NOT map 1:1 onto the 5-value CitationStatus enum. Interpretation has no enum equivalent; Unverified/TODO maps loosely to needs_review; inferred and rejected have no prose counterpart. Do not present them as a single canonical enum. (There are also distinct verification fields elsewhere — OfficialSource.verification_status, SourceChange.confidence — that must not be conflated with citation confidence. A canonical reconciliation is an open question, §11.)

7.4 The integrity gate (PDF reconciliation)

For sources with a summed-detail-plus-total shape (FL weekly reports), validate_report_rows (app/services/report_integrity.py) reconciles the sum of parsed detail rows against the source total row per metric (MetricCheck, within tolerance). A total_mismatch is severity error and blocks import unless a pre-reviewed entry matches in data/reference/report_integrity_exceptions.json (exact numeric equality on both detail and total values), which downgrades it to info. See INTEGRITY.md.

8. Conformance

8.1 A spec-conformant dataset

• Every record is a metric_observation with all required fields present and enum values drawn from this spec's value sets (lowercase strings).
• value is NULL only when suppressed=true.
• metric references a catalogued MetricDefinition; consumers honor value_kind before summing.
• The dimensional uniqueness + coalesce invariant holds (at Alembic 015 or later); restatements are append-only via the is_latest / restated / restates_observation_id mechanism, never in-place updates.
• Verbatim labels (operator_name_raw, geo_name_raw, product_category_raw) are preserved alongside their normalized columns; unmapped categories are stored as unknown with the verbatim label kept and an alert emitted.

8.2 A spec-conformant adapter

• Exposes a top-level ADAPTER dict with all 12 required keys and the required inner key sets, passing app/tests/test_state_adapters.py.
• status ∈ {planned, active}; an active adapter has a refresh profile or a registered ingestor.
• Honors the normalization invariants in §6.3 (canonical operator entity, neutral field names, sorted active_years with matching data sources, the five required module gates/modules).
• Ships a confidence-labeled product-category crosswalk for any category-split source, and fails loud on unmapped categories.

9. How to cite this spec

DevCannabis Cannabis Regulatory Data Normalization Specification, Version 0.1 (draft). Canonical record: metric_observation. Reference implementation: Alembic revision 015.

Pair the spec version with the codebase revision when citing a dataset produced under it.

10. Pointers to deeper docs

• docs/adapter-contract.md — the adapter field-rule spec (required vs optional keys, refresh fields, source-registry, market-event overlay, public API/route policy).
• app/docs/NEW_STATE_GUIDE.md — build-order tutorial for adding a jurisdiction.
• app/docs/ARCHITECTURE.md — gateway+adapter model, raw-vs-normalized, alias-vs-lineage, route compatibility policy.
• docs/source-citation-standard.md — Citation/CitationLink standard.
• INTEGRITY.md (repo root) and app/DATA_NOTES.md — the four confidence labels and the PDF integrity gate.
• app/tests/test_state_adapters.py — the executable REQUIRED_*_KEYS contract.

11. Open questions for v0.2

These are known gaps the draft does not yet resolve; flagged honestly rather than papered over:

• Generalized refresh shape. config.refresh's required keys are PDF/Florida-shaped (report_type == "weekly_reports", an {date} archive template). Modern adapters (MA/CO/OR/MI) use @register_ingestor with refresh=None instead. A generalized, non-PDF refresh descriptor is the likely v0.2 direction.
• Capability catalog — top level done; nested keys remaining. The top-level capabilities flags and the sales channels are now a canonical, enforced catalog (§6.4, states/adapter.py). Still open: cataloguing the sub-keys of the other nested capability dicts (reports, legal, price_data, channels) so the whole feature map — not just its top level — is a closed vocabulary.
• Confidence-taxonomy reconciliation. The four prose labels, CitationStatus (5 values), OfficialSource.verification_status, and SourceChange.confidence coexist and do not map 1:1 (§7.3). A canonical crosswalk is unresolved.
• data_sources[*].format vocabulary. The contract test does not validate it; shipped adapters use csv, json, socrata, xlsx, tableau_csv, tableau_twbx, pdf, html, google_sheets, html_index_pdf. The JSON Schema treats format as an unconstrained string by design.
• ratio value_kind is reserved in the enum but unused by any catalogued metric.