A legal-tech directory, built from its own scrape
My role
Built inside a team. The product direction, the editorial and the ranking methodology are not mine; the data pipeline and the application are.
The problem
The directories in this market are pay-to-play, and none of them publish where their data came from. Building one that could be trusted meant starting from data you could stand behind, field by field.
What I did
- Scraped a competitor's full catalogue across four endpoints, storing every raw response verbatim next to a receipt: source URL, timestamp, response status, the full response headers, and a SHA256 hash of the body.
- Caught the source lying about itself. Each full pass returned a random subset while reporting a fixed total, so an ordinary scrape finished looking complete and was not. The fix was a saturation scraper that kept going until it stopped finding new records.
- Designed the relational model: products, vendors, categories, buyer types, integrations, plans, sources, and a per-field provenance table recording where every value came from and when it was checked.
- Wrote the builder that turns the scraped folders into the database, with an atomic swap and fail-loud assertions that stop the build when the data is wrong rather than shipping it.
- Built the application on top: route templates that generate the whole site statically from the database.
- Ran a verification pass over the products where every accepted fact carries a verbatim quote and the URL it came from, held in a file the build reads so a rebuild cannot lose a verified correction.
- Wired a build gate that fails the build when a published figure stops matching the database, so a number on a page cannot drift from its source.
The result
A directory that runs entirely off its own audited dataset, where every field traces back to a hashed raw response. Never launched publicly.
The judgment call: what the AI couldn't do
The scrape looked finished and was not. The source returned a random slice on every full pass while reporting a fixed total, so a normal run completed, matched the number it was told to expect, and silently missed a third of the catalogue. I only caught it because the unique count moved between runs. Trusting the total a source reports about itself is the mistake; the fix was to keep scraping until new records stopped appearing and let the data say when it was done.
See it
Proof
What it is: A trimmed static build of the real thing: a few products per category, running off the same database and the same page templates. Not indexed.
On request: The schema, the provenance format, and a walk-through of the build.