Stop Creating Duplicates on Write: Idempotent Upserts in HubSpot

Published June 14, 2026

TL;DR: Deduplication usually gets framed as a cleanup job, merging records that already collided. The cheaper fix is upstream: stop the integration from creating the duplicate in the first place. A write that is not idempotent makes a new record every time it runs without a stable match key, so a retried webhook, a re-run backfill, or a network timeout that actually succeeded all leave a second copy behind. The fix is an idempotent upsert: give each record a unique external key, write through the endpoint that matches on that key, and treat create-or-update as one operation. Done that way, running the same job twice is a no-op, not a doubling.

A duplicate is usually a write that ran twice

The standard story about duplicates is that two teams typed the same customer in two ways. That happens, but the higher-volume source is mechanical: an integration that writes without a stable identity. Three everyday cases produce it. A webhook delivery is retried because the first response was slow, so the handler runs twice. A nightly backfill is re-run after a partial failure and re-inserts the rows it already inserted. A request times out on the client side but actually succeeded on HubSpot’s side, so the job retries and creates a second record for a write that already landed. None of these is a data-entry mistake. Each is a write that was not safe to repeat.

The reason this matters is that the network guarantees you have are weak. Any distributed write can be delivered more than once, so the only durable defense is to make the write idempotent: repeating it produces the same end state as doing it once. For CRM records, that means matching on a key that is stable across runs rather than minting a new record each time.

The unique key is the whole mechanism

Idempotency needs something to match on, and the record ID HubSpot assigns on creation does not exist until after the first write, so it cannot be the key for that write. The durable pattern is to carry your own identifier from the source system into a HubSpot property that is marked unique, then match on that property instead of on email or on nothing.

HubSpot supports unique-value properties for exactly this: a custom property flagged so that no two records of the object can share the value, which makes it a safe match key for an upsert (HubSpot Knowledge Base, create and edit properties). Populate it with the source system’s primary key, the value that already uniquely identifies the row where the data comes from. Email is tempting but wrong for this job: people change emails and share inboxes, so it is neither stable nor guaranteed unique in practice. A source-system ID is both.

Upsert, do not create

With a unique key in place, the write becomes an upsert: create the record if the key is new, update it if the key already exists, in one operation. HubSpot’s batch endpoints support addressing a record by a unique property value through the idProperty parameter rather than by the HubSpot object ID, which is what lets a job update-or-create against your own key without a separate lookup (HubSpot CRM objects and batch API). The job stops asking “does this record exist yet” and simply asserts the desired state for that key. Run it once, the record is correct. Run it again, the record is still correct, and nothing new is created.

This is the structural difference between a create call and an upsert. A bare create says “make a record”; repeated, it makes many. An upsert says “make this key look like this”; repeated, it converges. The second is the only one that is safe to retry, and retries are not optional, they are how reliable systems handle the network they actually run on.

Make retries safe on purpose

Idempotent writes and a retry policy are two halves of the same design. Because the upsert is safe to repeat, the job can retry aggressively without fear: on a 429 it reads the Retry-After header and waits exactly that long, and on a 5xx or a client timeout it backs off and tries the same upsert again (HubSpot API usage and rate limits). The timeout case is the subtle one: a request that timed out on your side may have committed on HubSpot’s side, so a blind retry of a plain create would duplicate. With an upsert on a unique key, that same retry is harmless, because the second write matches the record the first one created. The retry policy is what makes the job finish; the idempotent key is what makes the retry safe.

How to know your writes are idempotent

A write path that is not idempotent looks fine until volume or a retry exposes it. Three checks surface it before it corrupts the table:

Run it twice on a sandbox. Execute the same batch against a test portal twice and confirm the record count does not change on the second run. A count that doubles is a create masquerading as an upsert.
Confirm the unique constraint exists. Verify the match property is actually flagged unique on the object, not just conventionally unique, so HubSpot enforces it rather than trusting the job to.
Inspect the retry handler. Confirm that the timeout and 5xx paths re-issue an upsert keyed on the external ID, not a fresh create. The bug usually hides in the error branch, the code path that only runs when something already went wrong.

Where this sits in the foundation

Write-time idempotency and after-the-fact deduplication are the same problem at two different moments. Idempotent upserts stop new duplicates from entering; a deduplication and survivorship pass resolves the ones already there. Do the cleanup without fixing the write path and the duplicates simply grow back by the next run, which is why integration design is part of a data foundation audit and not a separate concern. The cheapest duplicate is the one the write never created.

The takeaway

If a HubSpot integration can create the same record twice, assume it eventually will, because the network guarantees that retries happen. Give each record a unique external key from the source system, write through an upsert that matches on that key with idProperty, and pair it with a retry policy that leans on the upsert being safe to repeat. Then prove it by running the same job twice and watching the count hold steady. Done that way, deduplication stops being a recurring cleanup and becomes a property the write path simply has.

If your duplicate count keeps climbing no matter how often you merge, the write path is the cause this method addresses. Start with a Data Foundation Audit, where integration idempotency is one of the layers we check before the duplicates are treated as a data-entry problem.

Sources

HubSpot Knowledge Base, Create and edit properties (unique-value properties): https://knowledge.hubspot.com/properties/create-and-edit-properties
HubSpot Developers, CRM objects and batch API (upsert by idProperty): https://developers.hubspot.com/docs/api/crm/contacts
HubSpot Developers, API usage details and rate limits (429 / Retry-After): https://developers.hubspot.com/docs/api/usage-details

← Back to all field notes