Warehouse ingestion
Updated August 2026
Postgres continuous sync
Postgres ingestion starts with a complete table copy, then continuously applies inserts, updates, and deletes from the transaction log. AWS DMS runs the replication; Embrasure keeps logical and physical table names independent from that transport.
Full load and continuous CDC
This is the default when a Postgres source passes preflight. A shared AWS DMS pool performs full-load-and-cdc: it copies existing rows first, then stays attached to PostgreSQL logical replication for later changes.
Stable primary keys let the destination merge updates and represent source deletes as durable soft-delete rows. The destination table name remains a normal warehouse name; it does not expose whether DMS or a batch reader transported the data.
Preflight connects with the supplied source user and blocks provisioning until the direct endpoint, logical replication setting, role membership, replication capacity, table visibility, key selection, and supported column types are all safe.
Required setup
Direct endpoint
Use the database's direct PostgreSQL endpoint. PgBouncer, transaction poolers, and RDS Proxy do not expose the replication protocol that continuous sync requires.
Logical replication
Set wal_level=logical. On Amazon RDS for PostgreSQL, set rds.logical_replication=1 in the parameter group and reboot the instance before preflight.
Source permissions
Grant CONNECT on the database, USAGE on each selected schema, SELECT on each selected table, and either PostgreSQL REPLICATION or the Amazon RDS rds_replication role.
Primary key
Every synced table needs a stable primary key so Embrasure can update the warehouse copy of each row.
Replication capacity
Keep at least one logical replication slot and one WAL sender available. Preflight checks max_replication_slots and max_wal_senders before creating AWS resources.
Network access
Allow the Embrasure connector egress IP to reach the direct endpoint on the PostgreSQL port, and use SSL. Private-only databases need an agreed private network path.
show wal_level;
show max_replication_slots;
show max_wal_senders;Cursor and xmin batch fallback
Cursor and PostgreSQL xmin ingestion remain available when continuous replication cannot be enabled. They are explicit batch modes, not the Postgres default. Cursor mode needs a reliably updated column; xmin avoids that column but may scan the source table on every run. Neither mode captures hard deletes.
For large cursor-mode tables, add a covering index on the cursor and primary key columns. Preflight blocks unsafe large-table scans.
create index concurrently if not exists orders_updated_at_id_idx
on public.orders (updated_at, id);Column types
Embrasure reads structured Postgres catalog metadata before deciding how to copy a column. If a type cannot be represented safely, the column is skipped with a visible reason instead of being coerced into the wrong shape.
- Common Postgres values stay queryable as native analytical types.
- Source-specific values preserve their original meaning, even when that means text encoding.
- Unsupported values are visible and explainable instead of causing unclear warehouse failures.
- Primary key and cursor columns are stricter than normal columns because they control sync correctness.
| Native scalars | Integers, floating point numbers, booleans, strings, UUIDs, dates, times, and timestamps are copied into matching analytical types. |
|---|---|
| Decimals | Numeric and decimal values preserve precision and scale when the destination supports them. Values outside destination limits are not silently narrowed. |
| JSON | json and jsonb columns are copied as JSON text. This keeps the source value intact without guessing a nested warehouse schema. |
| Arrays | Arrays of supported scalar values are copied as JSON array text. Arrays of unsupported or composite values are skipped. |
| Binary | bytea values are copied as base64 text so query engines read them consistently. |
| Enums and domains | Enum, domain, range, network, geometric, and custom scalar-like values are copied as text unless a safer native mapping is explicitly supported. |
| Unsupported | Composite types, arrays of composite types, pseudo types, trigger types, handler types, and event types are skipped rather than copied incorrectly. |
Primary keys and incremental fields
If a selected primary key or batch cursor column is missing or unsupported, Embrasure pauses or disables that table until the configuration is corrected. For auto-discovered non-key columns, Embrasure can remove stale or unsupported columns from the selection and continue syncing the rest of the table.
Likely next additions
The next compatibility additions are focused on common production Postgres extensions and application patterns:
- citext as string for emails and usernames.
- hstore as JSON text for key-value columns.
- interval as text with source metadata.
- PostGIS geometry and geography as WKT or GeoJSON text with SRID metadata.
- Richer enum and domain metadata so agents can understand allowed values.
Limitations
- The initial full load can take time on large tables. Continuous changes are buffered and applied after each table's initial copy catches up.
- New tables are not silently added to a running DMS replication configuration. Add them through a new or rebuilt ingestion connection so the full-load boundary stays explicit.
- Additive columns can evolve through the destination schema. Dropped, renamed, incompatible, or key columns still require review.
- Cursor and xmin modes remain available as explicit batch fallbacks, but they do not capture hard deletes.
What gets stored
Embrasure stores the synced warehouse tables plus operational metadata such as selected tables, checkpoints, row counts, sync status, errors, and preflight results.