A legal-tech directory, built from its own scrape

My role

Built inside a team. The product direction, the editorial and the ranking methodology are not mine; the data pipeline and the application are.

The problem

The directories in this market are pay-to-play, and none of them publish where their data came from. Building one that could be trusted meant starting from data you could stand behind, field by field.

What I did

Scraped a competitor's full catalogue across four endpoints, storing every raw response verbatim next to a receipt: source URL, timestamp, response status, the full response headers, and a SHA256 hash of the body.
Caught the source lying about itself. Each full pass returned a random subset while reporting a fixed total, so an ordinary scrape finished looking complete and was not. The fix was a saturation scraper that kept going until it stopped finding new records.
Designed the relational model: products, vendors, categories, buyer types, integrations, plans, sources, and a per-field provenance table recording where every value came from and when it was checked.
Wrote the builder that turns the scraped folders into the database, with an atomic swap and fail-loud assertions that stop the build when the data is wrong rather than shipping it.
Built the application on top: route templates that generate the whole site statically from the database.
Ran a verification pass over the products where every accepted fact carries a verbatim quote and the URL it came from, held in a file the build reads so a rebuild cannot lose a verified correction.
Wired a build gate that fails the build when a published figure stops matching the database, so a number on a page cannot drift from its source.

The result

A directory that runs entirely off its own audited dataset, where every field traces back to a hashed raw response. Never launched publicly.

The judgment call: what the AI couldn't do

The scrape looked finished and was not. The source returned a random slice on every full pass while reporting a fixed total, so a normal run completed, matched the number it was told to expect, and silently missed a third of the catalogue. I only caught it because the unique count moved between runs. Trusting the total a source reports about itself is the mistake; the fix was to keep scraping until new records stopped appearing and let the data say when it was done.

Proof

What it is: A trimmed static build of the real thing: a few products per category, running off the same database and the same page templates. Not indexed.

On request: The schema, the provenance format, and a walk-through of the build.

Loading…

Data + Product · all work

A legal-tech directory, built from its own scrape

My role

Built inside a team. The product direction, the editorial and the ranking methodology are not mine; the data pipeline and the application are.

The problem

What I did

Scraped a competitor's full catalogue across four endpoints, storing every raw response verbatim next to a receipt: source URL, timestamp, response status, the full response headers, and a SHA256 hash of the body.
Caught the source lying about itself. Each full pass returned a random subset while reporting a fixed total, so an ordinary scrape finished looking complete and was not. The fix was a saturation scraper that kept going until it stopped finding new records.
Designed the relational model: products, vendors, categories, buyer types, integrations, plans, sources, and a per-field provenance table recording where every value came from and when it was checked.
Wrote the builder that turns the scraped folders into the database, with an atomic swap and fail-loud assertions that stop the build when the data is wrong rather than shipping it.
Built the application on top: route templates that generate the whole site statically from the database.
Ran a verification pass over the products where every accepted fact carries a verbatim quote and the URL it came from, held in a file the build reads so a rebuild cannot lose a verified correction.
Wired a build gate that fails the build when a published figure stops matching the database, so a number on a page cannot drift from its source.

The result

A directory that runs entirely off its own audited dataset, where every field traces back to a hashed raw response. Never launched publicly.

The judgment call: what the AI couldn't do

Proof

What it is: A trimmed static build of the real thing: a few products per category, running off the same database and the same page templates. Not indexed.

On request: The schema, the provenance format, and a walk-through of the build.