Why Your Scheduled API Sync Keeps Missing Data

Your scheduled sync ran fine last night. Green checkmark, no errors, log says success. And you're still missing records.

That's not a bug in your scheduler. It's a gap in what "success" actually means. HTTP 200 tells you the server accepted your request. It says nothing about whether the data you got back is complete, valid, or safe to trust. On projects I've built with multiple sales channels and seller platforms, I've debugged syncs that looked perfectly healthy in the logs while quietly dropping 5 to 10 percent of records every month. The fix was almost never the scheduler. It was an unvalidated API response, a pagination reset under load, or a rate limit nobody documented.

The 200 OK Trap: Your Sync Completes, But Your Data Doesn't

HTTP 200 is a transport-layer signal. It means the request reached the server and the server responded without a transport error. It does not mean the payload is complete. Nothing in the HTTP spec obligates the server to guarantee your JSON is whole.

Most sync jobs log success on two conditions: the status code is in the 2xx range, and the body parses without throwing an error. They never ask the harder question: is every record I expected actually in this payload? If you're pulling 10,000 records and the response gets truncated at 9,500, your parser doesn't care. It sees valid JSON, it finishes the loop, it logs "sync complete." You don't find out anything is wrong until someone downstream notices numbers don't add up.

This is the pattern behind most of the "silent" data loss I've seen. The failure isn't in the network layer, it's in the validation layer, where nobody checks the response against what was expected.

Pagination Breaks Under Load: Why Cursors Skip Pages

If your sync walks through pages using offset and limit, or a cursor token, it's exposed to a failure mode that rarely shows up in testing: pagination resets or skips under concurrent load. A sync that starts while the API is under heavy write traffic can get a cursor that jumps ahead, or an offset that resets to a stale point, with no error response at all.

Picture a scheduled sync that fires while the source system is still processing a batch of recent writes. The cursor moves in a way that skips a chunk of records. Your next scheduled run starts from where the cursor says it left off, which means those skipped records are just gone from your sync's perspective. Smaller or internal APIs usually don't document it at all. You find out only when your reconciliation report comes up short.

If you're building or maintaining an integration against a third-party API, it's worth reading up on common mistakes that cause the most support headaches, pagination handling being one of the recurring ones.

Race Conditions Between Sync Runs: When Overlapping Jobs Eat Data

Here's a question worth asking about your own cron setup: how long does your sync actually take to run, and how often is it scheduled? If the job takes longer than the interval between runs, you will eventually get two instances running at once. That's not a hypothetical, it's a matter of time.

When two sync runs overlap, they can collide on the same records, skip batches entirely, or write duplicate inserts. When two sync runs overlap without proper locking mechanisms, they can collide on the same records, skip batches, or create duplicate inserts. Proper handling means a distributed lock, a status flag on the sync record itself, or an idempotency key that rejects a duplicate write instead of silently creating one.

The database ended up with stale or missing stock counts, not because any single sync failed, but because they stepped on each other. If you're running WooCommerce with more than one sales channel, this is worth reading alongside my piece on why plugins alone won't stop overselling.

Silent Rate Limits and Soft Throttling: The Hidden Data Loss

Not every API tells you when it's throttling you. Plenty of production APIs have soft rate limits that quietly drop requests or reduce the result set instead of returning a clean 429. Your sync finishes, your logs show no errors, and you got 80 percent of the records because the other 20 percent were silently dropped by a rate limiter you didn't know existed.

You won't catch this in your application logs. You'll catch it in a reconciliation report, usually days after the fact, when someone asks why a batch of orders or customers never showed up. The fix starts with visibility: log response sizes on every call, check response headers for rate-limit counters if the API exposes them, and compare your fetched counts against a source-of-truth count on the API side whenever you can get one. I've written more on this pattern in why integrations fail silent and how to fix it, which covers the retry side of this problem in more depth.

Timestamp-Based Sync Windows: Clock Skew and Backfilled Data

A lot of syncs fetch "everything created or modified since the last run" using a stored timestamp. This works until your server clock drifts from the source system's clock, even by a small amount, at which point you have a gap that never gets fetched. Timestamp-based syncs are vulnerable to missed windows if the source system clock drifts from the sync server's clock.

There's a second, sneakier version of this problem: backfilled records. Payment systems and order management platforms often let records get backdated after the fact, a refund applied against an earlier order, a correction to a transaction date. If your sync window has already passed by the time that record's timestamp lands, your timestamp-based sync will never see it. It's not late, it's invisible.

A more resilient pattern is to stop trusting the timestamp window as your only filter. Fetch a wider slice, say the last several hours, on every run, and deduplicate using a unique record ID rather than relying on the timestamp to do the filtering for you. It costs a bit more in API calls. It costs a lot less in missing data.

How to Audit Your Sync Right Now

You don't need to rebuild your whole integration to find out if this is happening to you. Start with a full reconciliation between your source API and your destination database. Count the records on both sides. Spot-check a sample. Log every response size and status code your sync receives, not just whether it succeeded.

Then add validation as a real step in the pipeline, not an afterthought. After every sync run, check that required fields are present in every record, that the count you received matches what you expected, and that nothing arrived truncated. If webhooks are part of your setup, make sure you understand how they interact with your scheduled sync in the first place. I've covered that groundwork in how your tools actually talk to each other.

The mindset shift that matters most: track sync success and data completeness as two separate metrics. A sync that reports success but silently drops a small percentage of records every month is worse than one that fails loudly and retries, because nobody goes looking for a problem that never throws an error.

If you're staring at a dashboard that's been quietly wrong for months and you're not sure where to start looking, that's exactly the kind of audit I do for clients through API and webhook integration work. Sometimes the fix is a few lines of validation. Sometimes it's a redesign of how the sync fetches data at all.

How confident are you, right now, that your last sync actually got everything it was supposed to?

FAQ

How do I know if my sync is actually losing data?

Log every API response size and record count, then run a full reconciliation against the source API monthly. Compare row counts and spot-check records. If counts don't match, compare timestamps and IDs to find gaps. Most teams discover data loss only when auditing after a customer complaint.

Should I use webhooks instead of scheduled syncs?

Not instead of, alongside. Webhooks are event-driven and catch near-real-time changes, but they can fail and require retry logic with idempotency keys. Scheduled syncs are slower but act as a reconciliation catch-all. Use both: webhooks for speed, syncs for safety.

What's the simplest way to prevent pagination from skipping pages?

Fetch a timestamp or cursor on every page and store it. On the next sync, start from that marker, not from the beginning. For non-cursor APIs, fetch the last N hours every time and deduplicate by unique ID, rather than trusting offset alone.

Do I need an idempotency key for scheduled syncs?

Yes, if syncs can overlap or retry. Store a hash of the batch or use a sync window timestamp as a unique key. When a sync reruns or overlaps, the database rejects duplicate key inserts, preventing duplicate records.