An open-source national policy data platform ingesting census, health, housing, environmental, and education data at the census tract level for all 85,000+ US tracts.
PDI ingests public data from
multiple sources,
produces
validated variables,
and runs a 5-layer statistical engine grounded in peer-reviewed methodology. From raw indicators to query-time composites, every computation is documented, reproducible, and open.
The platform runs on a six-stage DAG pipeline — fetch → process → enrich → analyze → synthesize → deliver — with stages executing in concurrent waves via topological sort. Raw indicators are stored in PostGIS, validated features are derived mathematically, and composites exist only at query time with mandatory sensitivity analysis.
Stack: Go 1.24+, PostgreSQL 16 + PostGIS 3.4, Python 3.11+. All code is open source under Apache-2.0.
Architecture
Five-Layer Statistical Architecture
Each layer builds on the one below. Composites exist only at the top — computed at query time, never stored as truth.
Layer 5
Composite Views
Query-time only. Geometric mean aggregation with automated sensitivity analysis (±20% weight perturbation). Never persisted. Multiple composite specifications can coexist for different policy questions.
Layer 4
Spatial Analysis
LISA local Moran's I clusters (HH/LL/HL/LH), Geographically Weighted Regression coefficients, multilevel variance decomposition across geographic scales, SKATER regionalization.
Layer 3
Factor Scores
Exploratory Factor Analysis with oblimin rotation. Dimensions named by loading profile; variable loadings documented. Per-tract factor scores + national percentiles.
Layer 2
Validated Features
ICE (Index of Concentration at the Extremes), Dissimilarity Index, z-scores, percentile ranks, and coefficient of variation (CV) reliability flags. Bootstrap confidence intervals for every derived statistic.
Layer 1
Raw Indicators
Stored in PostGIS with vintage tracking. Every value carries margin of error, coefficient of variation flag, and reliability tier (high / moderate / low). Geo-enabled for spatial queries at all levels.
📹 How the Ingest Pipeline Runs
The same dependency graph runs every source adapter through fetch, normalize, upsert, and validate — in topological order.
Data Sources
Data Sources
Live data fetched from the PDI API at page load. Searchable by name, type, description, or geographic level.
Loading sources…
Source
Type
Geographic Levels
Description
APIFILE
Methodology
Statistical Methodology
Six core methods. Each grounded in peer-reviewed research.
ICE — Index of Concentration at the Extremes
Krieger et al., Health & Place, 2016
Measures spatial polarization without collapsing race and income. Formula: (privileged − deprived) ÷ total. Captures both ends of the distribution simultaneously.
Dissimilarity Index
Massey & Denton, 1988
D = 0.5 × ∑ |bi/B − wi/W|. Canonical measure of residential segregation between two groups.
LISA — Local Indicators of Spatial Association
Anselin, Geographical Analysis, 1995
Identifies statistically significant spatial clusters (HH, LL, HL, LH) from local Moran's I decomposition.
Factor Analysis
Kolak et al., JAMA Network Open, 2020
Exploratory Factor Analysis with oblimin rotation. 71% variance explained across 71,901 US tracts. Factors named by loading profile.
Bootstrap Confidence Intervals
Efron & Tibshirani, 1993
Non-parametric resampling for any statistic. Parallelized via Go goroutines for tract-level computation at national scale.
Query-Time Composites
Stiglitz-Sen-Fitoussi, 2009; PMC, 2022–2025
Geometric mean aggregation with mandatory sensitivity analysis. Weight perturbation (±20%) and alternative normalization documented per query.
Additional methods include OLS regression, Blinder-Oaxaca decomposition for equity gaps, segmented regression for threshold detection, and interaction effects via automated term construction. The full methodology is documented in the project repository.
Variables
Variable Catalog
Validated variables across multiple sources. Searchable and grouped by data source.
Loading variable catalog…
Animations
Manim Video Gallery
Statistical concept animations explaining the methodology.
z-score Normalization
ICE Polarization
Bootstrap Sampling
Quantile Transformation
LISA Spatial Clusters
Pipeline DAG Flow
References
Research References
30 peer-reviewed and technical sources that ground the statistical architecture.
Loading references…
#
Title
Authors
Year
Type
Key Finding
Acknowledgments
Built by TresPies
PDI is designed, built, and maintained by Cruz Romero Morales and TresPies — a bilingual lived-experience design and data studio based in Madison, Wisconsin.
Cruz Romero Morales is a systems thinker, design engineer, and community data advocate. His work bridges the gap between lived experience and technical infrastructure — building open-source tools that make policy data accessible, legible, and actionable for the communities that everyone else overlooks.