For The Greater Good

The problem

Food-assistance data is public, and useless. Every state, county, and food bank publishes pantry listings in its own shape: some as HTML tables, some as JSON endpoints nobody documented, some as PDFs. A family trying to find dinner does not care whose schema won.

For The Greater Good normalizes all of it into one standard, HSDS v3.1.1, and publishes the result: the largest database of charitable food locations in the country.

How it works

Scrapers collect raw content and hand it to a content-addressed store keyed by SHA-256. That store is the reason the economics work: content that has been seen before never reaches the model again. A full from-scratch rebuild costs on the order of $1,600 in inference; incremental monthly runs cost tens of dollars.

New content only goes to the queue. Workers run it through an LLM extraction layer that maps arbitrary source shapes onto the standard schema. A validator scores the result on a build-up model: start at 60, earn points for completeness and corroboration, with a hard cap of 90 for anything scraped. Scores of 91 and above are reserved for human verification, so no amount of machine confidence can impersonate a person having checked.

The reconciler is the exclusive write path to the canonical database. Nothing else writes. That single constraint is what makes the dataset auditable.

What it costs and why that changed

The pipeline ran locally until a single GPU was outscaled by processing volume. Inference moved to hosted APIs, producing the November 2025 published dataset. It was then rebuilt as an AWS-native v2 (Bedrock batch inference, Step Functions, autoscaling) and launched in April 2026 with materially better coverage. The personally-funded v1 was retired at that point, superseded rather than abandoned.

This is the honest version of “can you operate at scale”: prototype locally, measure the ceiling, productionize on cloud, and hand off the cost structure.

The practice

943 commits and 207 merged pull requests, with no third-party human contributor anywhere in the graph. Roughly 84% of commits carry an AI co-author trailer.

That is only interesting because of what governs it. The repository runs on a versioned engineering constitution that has been amended seven times, and every amendment traces to a specific shipped defect. Coverage is enforced by a ratchet (a baseline that can only rise) rather than a threshold anyone can argue down, with a stricter per-file floor on the code that handles cryptographic verification.

What it feeds

For The Greater Good is the upstream source behind a national food-assistance directory, merged with that platform’s own provider feed and deduplicated against it. The directory rebuilds hourly and serves all 50 states in nine languages.

Around that directory sits a set of plugin services, each a separate repository with its own infrastructure stack, discovered and deployed by the core pipeline:

• The page builder that generates and hosts the directory itself (a static-site generator turning verified records into an SEO-optimized, nine-language network of pages).
• A backend for operators, where a pantry can claim its listing and start managing its own information (hours, services, contact details) instead of waiting for a scraper to notice a change.
• A write API, owned by the platform rather than by me, which is the only sanctioned path for writing into a running instance of the pipeline.
• A Slack bot so platform staff are notified when someone looking for food reports that a listing is wrong, and can respond without leaving Slack.
• A natural-language experiment: an AI assistant for browsing the directory conversationally, built to find out whether “is there somewhere open near me tonight” beats a map for the people this is for.

The plugins are what turn the dataset into something a pantry operator and a person looking for dinner can each actually touch.