Building a Health Data Pipeline: The Hard Parts Nobody Warns You About

You scope it as a two-day integration: call the HealthKit API, read the step count, store it. Then you ship, and the bug reports start. One user’s step total is double what their phone shows. Another’s sleep is logged on the wrong day. A third connected a Garmin and now has three overlapping heart-rate streams. None of that was in the API docs, because none of it is an API problem.

Connecting to a health data source is easy. Making that data trustworthy, consistent, and usable across every device your users own is the actual work, and it’s a data-engineering problem wearing an integration costume. The API call is maybe 5% of the job. This is a field guide to the other 95%: the stages of a real health data pipeline, the trap waiting at each one, and why “just pull the data” is where the problems start, not end.

The pipeline, in one picture

A production health data pipeline has six stages, and only the first one looks like the integration you scoped. Everything downstream is reconciliation and interpretation:

Read down the “trap” column and the pattern is clear: the pipeline isn’t hard to connect, it’s hard to trust. Let’s walk the stages that surprise teams.

Getting data in is the easy part

Ingestion is well documented per platform, but two things break the assumptions you brought from a normal API: partial permissions and per-source data models.

On iOS you read from HealthKit [1]; on Android from Health Connect [2]; from a Garmin, Oura, or Fitbit you go through a cloud API with OAuth. Each has its own model, and we’ve covered the platform split in HealthKit vs Health Connect. The part that trips teams up isn’t the connection, it’s what you get after it.

Health permissions are granular and partial. A user can grant steps but not heart rate, sleep but not workouts, and revoke any of it later without telling you. So you cannot assume a field exists just because you asked for it, and “no data” is ambiguous: it might mean the user didn’t move, didn’t grant access, or doesn’t own a device that produces it. Every stage downstream has to treat presence of data as a variable, not a given. That single fact, that your input is always partial, shapes the entire rest of the pipeline.

It arrives late, out of order, and in bulk

You do not control when health data shows up, and the timing is genuinely adversarial to naive pipelines.

Three realities collide here:

• Late and out of order. A watch that was offline syncs hours later; a workout from this morning lands this afternoon, after you already computed today’s totals. Your pipeline has to be able to revise the past, not just append to the present.
• Backfill in bulk, and only once. When a user first connects, you don’t get today’s data, you get months of history at once. That flood has to reconcile with everything that arrives afterward, which means your ingestion has to be idempotent: re-processing the same sample must never double-count it. It also has a ceiling you don’t set. Every provider caps how far back it will serve, from a few days to a couple of years depending on the source and the data type, and anything older than that window is not retrievable later. The backfill you capture on day one is usually the only history of that user you will ever have, which makes a bug in your first ingestion run permanently expensive.
• Push where you can, poll where you must. iOS offers background delivery so the OS can wake your app on new samples [1]; Health Connect exposes a changes API you check on your own schedule [2]. Cloud wearable APIs vary: Garmin, Fitbit, and Oura all offer push notifications or webhooks, others leave you polling within rate limits, and even webhooks need a polling fallback for the deliveries that never arrive. Freshness is a per-source negotiation, not a guarantee.

The design consequence is that a health pipeline is not an append-only log. It’s a system that continuously corrects a moving picture of the past, which is a much harder thing to build than the “fetch and store” you scoped. In practice that starts with making writes idempotent, so a re-sync or backfill replaying the same samples can never double-count:

-- Re-syncs and backfills re-send rows you already have.
-- Append-only INSERT double-counts them; keyed upsert doesn't.
INSERT INTO samples (user_id, source_id, type, start_at, end_at, value)
VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT (user_id, source_id, type, start_at)
DO UPDATE SET end_at = EXCLUDED.end_at, value = EXCLUDED.value;

The key includes the source, not just the user and time, because the same minute can legitimately hold samples from two devices. Which sample wins is the dedup stage’s job, not the write path’s.

Every source speaks a different dialect

The same metric means different things depending on which device produced it, so without normalization, any score or comparison you build is unreliable.

Providers differ in units (kilocalories vs kilojoules, meters vs steps vs strides), schemas (what one calls a “workout” another splits into segments), and sampling rates (a continuous heart-rate stream vs periodic spot checks). A “sleep” record from a phone estimating from motion is not the same measurement as a “sleep” record from a ring reading heart-rate variability, even though both arrive under the same name.

Normalization is the layer that makes those comparable, and it’s genuinely involved: we go deep on it in how to normalize wearable data across providers. The reason it matters beyond tidiness: the moment you compute a score, a trend, or a cross-user comparison, you’re implicitly asserting the inputs mean the same thing. If they don’t, the output is confidently wrong.

Closely related, and often conflated, is deduplication. A single morning walk can appear three times at once: steps from the phone, a workout from the watch, and an activity from a third-party app, with overlapping time ranges and different values. Sum them naively and you double or triple count. Correct dedup means understanding each sample’s source and priority and reconciling overlaps per data type, without dropping anything real, which is its own deep problem we cover in deduplicating health data across wearables.

You can’t unit-test the truth

There is no ground truth for most of what this pipeline decides, so “is it correct?” is a question your test suite cannot answer.

When a phone and a watch disagree about last night’s sleep, no oracle tells you which one was right. When dedup collapses three overlapping records into a single walk, nothing confirms the merged total matches what the person actually did. You can test that your code does what you specified. You cannot test that your specification matches the world, and those two things drift apart quietly, because a wrong answer looks exactly like a right one in a database.

What works instead is triangulation. Keep a small internal cohort with real ground truth, a research-grade device or a manual log, and check derived values against it. Then watch distributions rather than rows: if median sleep duration across your population moves twenty minutes after a release, something changed, whether or not a single test failed. At scale, monitoring your own aggregates is the verification layer, and it is worth building before you need it rather than after a score goes visibly wrong.

Health data lives in local time

Health is experienced in local time, stored in absolute time, and the gap between the two is a bug factory.

“Last night’s sleep” and “today’s steps” are local-time concepts. But samples are timestamped in absolute time, often UTC, sometimes with an offset, sometimes without. Aggregate a “day” wrong and you split one night’s sleep across two dates, or land a workout on the wrong day for a user who flew across timezones. The bug is usually one innocent-looking line:

// Wrong: buckets by the server's clock. An 11 pm walk in Auckland
// lands on tomorrow's date, and travel scrambles history.
const day = sample.timestamp.toISOString().slice(0, 10);

// Right: bucket in the timezone the sample was recorded in,
// which means storing that timezone with the sample.
const day = formatInTimeZone(sample.timestamp, sample.tzId, "yyyy-MM-dd");

Daylight-saving transitions add days that are 23 or 25 hours long, which quietly break any window that assumes 24. None of this is exotic; it hits any user who travels or any product that reports daily totals, which is all of them. Handling it correctly means carrying timezone context with every sample and defining “a day” deliberately rather than assuming the server’s clock.

You will never have complete data

Partial data is the normal case, not the exception, and a pipeline that assumes completeness produces confident nonsense on real users.

Between partial permissions, users who own no wearable, devices worn inconsistently, and sync gaps, most users most of the time have an incomplete picture. Two implications follow. First, cold-start: a brand-new user, or one with only a phone, still needs a usable experience before rich data exists, which is a product and modeling problem, not just an engineering one. Second, honest gaps: your pipeline needs to distinguish “zero” from “unknown,” because treating a missing night of sleep as zero sleep will wreck any average or score built on top. Designing for sparsity from the start is far cheaper than retrofitting it after your metrics start lying.

You can’t tell a broken pipeline from a lazy user

A silent pipeline and a sedentary Sunday look identical in the data, so observability for a health pipeline means monitoring per-source freshness, not error rates.

Normal services alert on failures. A health pipeline’s worst failures don’t throw errors: a revoked permission, a watch left in a drawer, a webhook subscription that silently expired, a provider that changed a rate limit. In every case the symptom is the same, data just stops, and “no data” is exactly what a resting user also produces. If you only monitor exceptions, you’ll discover these weeks later, from a user complaining their score is stuck.

What works is treating freshness as the health check: track “expected time since last sample” per user, per source, per data type, against that source’s normal cadence, and alert on the deviation, not the absence. A phone that hasn’t reported steps in 36 hours is a signal; a lab source quiet for two months is business as usual. Getting this right has a product payoff too, because “this user’s data stopped flowing” is also your earliest churn indicator, days or weeks before the uninstall.

The ground shifts under you

Every source you integrate is a dependency another company can deprecate, and they do, regularly, which is why a pipeline is a maintenance commitment and not a project.

This isn’t hypothetical. The Fitbit Web API, the workhorse of a decade of integrations, shuts down in September 2026, with existing OAuth tokens not transferring to its Google-run replacement, meaning every integration must migrate and every user must re-consent [4]. Google already deprecated Google Fit in favor of Health Connect. Apple removed iPhone-based sleep tracking in iOS 18, taking a data source away from every product that depended on it overnight. And beyond the headline deprecations there’s the steady drip: new data types each platform cycle, schema revisions, OAuth scope changes, rate-limit adjustments.

The engineering consequence is that source integrations need to be built as replaceable modules behind your own internal schema, so a migration is contained rather than load-bearing. The planning consequence is blunter: whoever owns the pipeline owns a permanent line item of migration work that arrives on other companies’ schedules, not yours.

Raw samples aren’t insight, and it’s all PHI

The reason to build the pipeline is to turn raw samples into something a product can act on, and every stage of it operates under a health-data compliance bar.

A list of heart-rate values is not a readiness signal; a week of steps is not a fitness trend. Turning samples into scores, biomarkers, or features requires personal baselines (what’s normal for this user), validation (does the derived signal mean what you claim), and a defensible method, which is a different discipline from moving data around. This derivation is where the product value actually lives, and it’s why “we store the raw data” is the start of the work, not the end.

It’s also the stage that breaks the shape of everything above it. Every other problem in this article is an engineering problem, and engineering problems shrink when you add engineers. This one doesn’t. A validated score needs a labelled dataset, which needs a study, which needs an ethics protocol and recruited participants and a full seasonal cycle of collection before a model can be trained at all. No amount of headcount shortens that, and it is the reason teams that scope a pipeline and then decide to add scoring find the second half took longer than the first. We put our own numbers behind it in what building it yourself actually costs.

Serving it back to your product is its own stage, and it’s shaped by everything above. Health data is high-volume time series (a single user can produce hundreds of thousands of samples a year), your queries are almost always “by local day for this user,” and because late data revises the past, derived values need recomputation or versioning rather than write-once storage. Add retention decisions (keep raw forever, or aggregates only?) and you have a real storage design problem, not a table you add on Friday.

Wrapping all of it: this is among the most sensitive data a person owns, so consent, encryption, retention, and PHI handling are not a final step but a property of every stage [3]. If you serve regulated markets, the bar rises again, and we keep a practical HIPAA-compliant health data APIs checklist for exactly that.

One compliance requirement deserves its own callout because it cuts against the grain of everything above: deletion has to propagate. Under GDPR-style erasure rights, and under your own consent promises, “delete my data” means the raw samples, the normalized copies, the deduplicated views, the derived scores computed from them, the caches, and eventually the backups. A pipeline built as layers of derived data is, by construction, a machine for spreading one person’s information across many stores, and a permission revoked mid-stream is a small deletion event you’ll process weekly, not a rare edge case. If erasure isn’t designed in at each stage, it becomes an archaeology project at exactly the moment a regulator or a user is watching.

So, build or buy?

Here’s the honest read after walking the whole pipeline: none of these problems is unsolvable, and all of them are permanent. Platforms change their APIs, providers add and deprecate data types, new devices ship with new quirks, and each one is ongoing maintenance, not a one-time build. The question is not whether you can build this. A capable team can. The question is whether health data plumbing is the thing your engineers should spend their time on.

The honest test is simple: is the pipeline your product, or an input to it? If normalizing health data across every device is your differentiator, build it and own it. If health data is a feature that makes your actual product smarter, most of the effort above is undifferentiated heavy lifting, and it competes for the same engineering hours as the thing your users actually pay for.

We costed our own build rather than leave that as a judgement call: the roles, the person-years and the fifteen-month critical path are in what building it yourself actually costs, and the longer argument sits alongside it.

Either way, the takeaway is the same, and it’s the thing the two-day-integration estimate missed: a health data pipeline is not an integration you finish. It’s a system you maintain, one that continuously reconciles partial, late, mismatched, duplicated data from sources you don’t control into something trustworthy enough to build on. Scope it as that from the start, and the bug reports at the top of this article never happen.