Sep 17, 202610 min read

Building a Product on Third-Party Data You Don't Control

ArchitecturePythonDjango

I run a small site called autos.cr. It answers exactly one question: you type a Costa Rican licence plate, and it gives you back the vehicle — make, model, year, engine, fiscal value. That is the whole product.

None of that data is mine. It comes from a public registry I have no control over, no contract with, and no notice from when it changes. Building a product on third-party data means accepting a permanent condition: the thing your product is made of can change shape on a Tuesday afternoon, and nobody will tell you.

The first version of the scraper was a 650-line class that did everything — fetching, parsing, persisting, computing derived values. It was impossible to test, so nobody tested it, so nobody noticed that one of its integrations had been silently failing for two years. The rewrite was not about better code. It was about which decisions get made where, so that the next time the source moves, the blast radius is one file.

These are the five decisions that actually mattered.

1. The parser must not know your framework exists

The single highest-leverage rule. parse_vehicle_page is a pure function from bytes to a dataclass:

def parse_vehicle_page(html: bytes) -> VehicleRecord:
    """Registry HTML -> VehicleRecord.

    No network, no Django, no globals.
    """

No database. No settings module. No HTTP client. It takes bytes and returns a value.

Two things fall out of that, and both are worth more than they sound.

The first is that you can test it against saved HTML in milliseconds. My parser test suite runs against fixtures captured from the real source — actual pages, saved to tests/fixtures/, warts included. There is no network in the test path, no mocking framework, no fake HTTP layer to keep in sync with reality. When the source changes, I save the new HTML next to the old HTML and both must pass.

The second is deployment flexibility you did not plan for. Because the parser imports nothing, the scraper runs as an AWS Lambda that has neither a database nor a settings module — it just imports the parser. That was not a design goal. It became free the moment the parser stopped knowing about the rest of the application.

The general rule: the layer that understands the foreign format should be the layer that understands nothing else. Every extra thing it knows is a reason it cannot be run, tested, or moved somewhere else.

2. Never discard a field you don't recognize

The naive parser hard-codes positions: the VIN is the third cell in the second row. That breaks the first time the source reorders anything.

The slightly better parser maps known labels to known fields and drops the rest. That does not break — it does something worse. It silently loses data, and you find out eighteen months later when you need a field that has been on the page the whole time.

What the parser does instead is collect every label/value pair on the page, then map the ones it knows onto real fields and put the remainder in a JSON column called extra:

extra = models.JSONField(default=dict, blank=True)

The source can add a field tomorrow and I lose nothing. When I eventually want it, widening the model is a migration and a backfill from data I already have — not a re-scrape of every record I have ever collected.

That asymmetry is the point. Re-fetching is the expensive, rate-limited, fragile operation. Reprocessing what you already stored is a loop over a database table.

3. Store the raw response, always — including the failures

Every scrape writes a ScrapeSnapshot, successful or not, with the complete response body kept as raw bytes and a parser_version stamped alongside the parsed output.

class ScrapeSnapshot(TimeStampedModel):
    plate = models.CharField(max_length=10, db_index=True)
    status = models.CharField(max_length=20, choices=ScrapeStatus.choices)
    raw_html = models.BinaryField(null=True, blank=True, editable=False)
    parsed = models.JSONField(default=dict, blank=True)
    parser_version = models.CharField(max_length=20, blank=True)
    duration_ms = models.PositiveIntegerField(null=True, blank=True)
    error = models.TextField(blank=True)

Storage is the cheapest thing in this entire system. A page of HTML compresses to a few kilobytes; a thousand of them cost less per month than one hour of my time. Meanwhile the thing that raw HTML buys you is the ability to fix a parser bug and reprocess history for free.

Keeping the failures matters more than keeping the successes. A parse that succeeded is already in your database as structured data. A parse that failed is the only evidence you have of what the source actually sent you — and it is exactly the case you need to debug. If you only snapshot the happy path, you have built a system that records everything except the interesting part.

4. Storing, publishing, and indexing are three different decisions

This is the one most teams collapse into one, and it is the one with real consequences.

The source returns more than vehicle specifications. It also returns information about the person who owns the vehicle. I store what I receive — but what gets rendered on a page, and what gets into Google's index, are separate decisions with separate risk profiles:

Decision	Question it answers	Default
Store	Do we keep this at all?	Yes — reprocessing needs it
Publish	Does this render on a page?	Gated per record
Index	May a crawler keep it?	No, unless deliberately enabled

Looking up one plate on an official service and publishing a crawlable directory of plate → person are not the same act, and the second one is what data-protection law is actually about. Costa Rica's Ley 8968 draws that line; most jurisdictions draw something like it. The technical expression of that distinction is two booleans:

is_published = models.BooleanField(default=True, db_index=True)
is_indexable = models.BooleanField(default=True)

The important part is not the flags. It is that there is exactly one read path, and it owns the gate:

def lookup(plate: Plate, *, force: bool = False, ...) -> LookupResult:
    """The single entry point for reading a vehicle.

    It owns the freshness rule, the publication gate and the lookup log,
    so no caller can accidentally bypass one of them.
    """
    stored = Vehicle.objects.filter(
        plate=plate.canonical, is_published=True
    ).first()

Views call lookup. The JSON API calls lookup. Nothing talks to the scraper directly. This means the API physically cannot expose a record that a page would refuse to render — not by convention, not by code review, but because there is no other function to call.

If your publication policy lives in the template, you do not have a policy. You have four of them, and one is wrong.

5. Model the identity the source uses, not the one you'd prefer

In Costa Rica a licence plate is not a number. It is a class and a number, in separate fields: CL + 123456. The same 123456 is a car, a light truck, or a motorcycle depending on the class. The class is not a filter you apply afterwards — it is part of the key.

I resisted this at first, because CL123456 is so much nicer to pass around as a string. Resisting it was wrong and cost me a round of rework. The source's identity model wins, because the source is the thing you have to ask.

Once the domain model matched, the URL design followed without argument:

/placa/carro/byl117/           canonical page
/placa/moto/12345/             class is part of the address
/placa/byl117/                 301 to the URL with the class

One vehicle, one address. When someone pastes a bare number, the page offers the other classes it could be. When they paste a modern plate that carries its own letters, it doesn't — that plate cannot also be a motorcycle, and offering the choice would be noise.

The general form: when you build on someone else's data, their identifiers are your identifiers. Translating between their model and a prettier one of your own is a layer of bugs you volunteered for.

What to return when the source is down

This one is short and almost nobody gets it right.

When the upstream registry fails, the tempting response is 404. The plate is not in my database, the page has nothing to show, 404 means "not found" — done.

It is wrong, and it is wrong in a way that costs you real traffic. A 404 tells a crawler the page does not exist and should be dropped. But the vehicle may well exist; I simply could not ask. The failure is mine, not the visitor's.

Situation	Status	Why
Source answered, no such record	`404`	An answer about their vehicle
Source unreachable or erroring	`503`	An answer about us — come back
Lookup not permitted for this client	`451`	The page exists; access is restricted
Source answered, record is private	`200` + `noindex`	Real page, not for the index

if result.unavailable:
    # 503, not 404: the plate may exist and we simply could not ask, so a
    # crawler should come back rather than drop the page.
    return render(request, "vehicles/unavailable.html", {...}, status=503)

The LookupResult carries that distinction explicitly rather than making the view infer it from an empty value:

@property
def unavailable(self) -> bool:
    """True when we failed to ask, so the plate may well exist."""
    return self.vehicle is None and self.outcome is not None and self.outcome.is_our_fault

is_our_fault is the whole design in three words. Telling someone "no results" when the problem is on your side is lying to them, and telling a crawler the same thing is lying with compounding interest.

The freshness rule nobody warns you about

Last one, because it is a trap I walked into.

The obvious caching rule is a TTL: if the stored record is older than N days, re-fetch it before answering. I built that. It is wrong for this shape of data.

Registry data barely moves. A vehicle's year of manufacture is not going to change. Meanwhile a live fetch costs a couple of seconds of someone staring at a spinner, and it consumes a rate-limited resource I do not own. Making a visitor wait to refresh a record that was almost certainly already correct is a bad trade in both directions.

So the rule is: serve whatever is stored, whatever its age. Only fetch when there is nothing at all. Staleness gets fixed out of band, by a management command that refreshes old records on a schedule, where the latency costs nobody anything.

The TTL still exists — is_stale() is a real function — but it drives the background refresh, not the request path. Freshness is a background job's problem. A visitor's problem is getting an answer.

The short version

Make the parser pure. bytes in, dataclass out, no framework imports. You get millisecond tests against real saved HTML, and the freedom to run it anywhere.
Keep unknown fields. Put anything you don't recognize in a JSON column. Re-fetching is expensive; reprocessing is free.
Snapshot every response, especially failures. Storage is cheap and the failed responses are the ones you'll need.
Separate store, publish, and index. Three decisions, three defaults, one read path that enforces all of them. If the policy lives in your templates, you have several policies.
Adopt the source's identity model. Their key is your key. A prettier internal representation is a bug factory.
Return 503, not 404, when the source is down. "We couldn't ask" is not "it doesn't exist," and crawlers act on the difference.
Don't refresh on the request path. For slow-moving data, serve what you have and fix staleness in a background job.

Every one of these is the same underlying move: draw a hard line between the part of your system that deals with somebody else's reality and the part that deals with yours, then make the crossing explicit. The source will change. The question is only whether the change lands in one file or in forty.

If you are building something on a source you do not control and want a second pair of eyes on where the boundaries should go, that is the kind of work I do.