Syncing#

A sync reads source records and commits a consistent new state to your lake. You can query the previous snapshot while a run is in progress; readers move to the new snapshot only after its commit succeeds.

The lifecycle#

1. Supernova discovers the tables and columns visible to the source credential.

2. The run reads new or changed source records, resuming from its saved position where the source supports one.

3. Rows merge by source key. A later value for the same key replaces the earlier current value.

4. Delete detection marks rows with _deleted = true in the same commit as the merge.

5. The committed snapshot becomes visible to new queries.

A new source syncs when you connect it. The default recurring interval is 4 hours. The source's Sync history tab shows each run and its outcome.

Incremental and complete reads#

An incremental source exposes a cursor such as an update timestamp, event id, or page token. Supernova stores that position only after the corresponding data is durable. If a run stops, its replacement resumes from the last durable position.

A complete read lists the source again and reconciles it with the lake. This costs more source requests, but it is the only truthful way to detect deletes for some APIs. Salesforce performs complete reads until its per-lake reconciliation gate is armed; it then prefers SystemModstamp, falls back to LastModifiedDate, and finally CreatedDate for objects that expose neither preferred field.

Fixed and discovered schemas#

Fixed-schema sources such as Stripe define their tables in Supernova. A code change adds a new source column to that contract.

Discovered-schema sources such as PostgreSQL, Airtable, and Salesforce ask the source for metadata. Their Tables tab is the exact reference for the connected account. A newly discovered column appears in the schema-change feed before it can surprise a dependent model.

Deletes#

Supernova preserves deleted source rows and sets _deleted instead of erasing history. Current-state queries should say so explicitly:

select id, email, _synced_at
from titan.hubspot.contacts
where not _deleted

Delete detection varies by source. Some APIs send delete events. Others require a complete key sweep. The connector page names that edge when it exists.

Consistency#

One table commit is atomic: a query sees either the snapshot before the run or the snapshot after it, never half of each. Related tables may finish at different moments, so use their _synced_at values when a report requires a common freshness boundary.