Skip to content
Embrasure

Embrasure FlowBeta · Open source · Apache-2.0

Postgres to Iceberg
in under a second.

Flow copies your Postgres tables into Apache Iceberg, then streams every insert, update and delete as it commits.

p95 0.48 to 0.93 s in 4 of 7 load tests

View on GitHubBook a demo

Watch Flow work.

Four sessions on the demo stack in Flow’s repo.

A row updated in psql shows up in Trino’s next read of the Iceberg table.

Two terminal panes. In psql, an order is updated, then a transaction inserts three rows, deletes one and changes a key. After each commit a Trino query of the Iceberg table returns the new rows.

  1. Stream

    A row updated in psql shows up in Trino’s next read of the Iceberg table.

    psql
    UPDATE 1
    trino
    {"id":1,"status":"shipped"}
    psql
    COMMIT
    trino
    {"id":20,"status":"paid"}

  2. Connect

    check verifies the source, init copies the tables, run streams from there.

    check
    source check: 0 failed, 0 warnings
    init
    "event":"bootstrap_completed","tables":3
    status
    "state": "running", "ready": true

  3. Recover

    After a catalog outage and a SIGKILL, Postgres and Iceberg both count 1,003 rows.

    status
    "pending_transactions": 1
    status
    "pending_transactions": 0
    compose
    Container embrasure-flow-demo-flow-1 Killed
    trino
    "1003"

  4. Compact

    Flow rewrites small files into larger ones while commits continue.

    metrics
    flow_table_l0_files{table_id="16388"} 8
    metrics
    flow_compactions_total{table_id="16388",kind="data"} 2
    metrics
    flow_table_l0_files{table_id="16388"} 3
    metrics
    flow_catalog_commits_total{table_id="16388"} 15

How Flow works

One Rust process reads the WAL, journals each transaction, writes Parquet and commits to Iceberg.

Common questions.

What does Flow need from Postgres?

PostgreSQL 14 to 18 with wal_level = logical, a REPLICATION login and a publication. embrasure-flow check --source tells you what’s missing. Getting started

Where do the tables live?

In your bucket, as Iceberg v2 or v3 tables behind an Iceberg REST catalog. Readers need only the catalog and the files.

What happens when something fails?

By default, Flow acknowledges Postgres only after the Iceberg commit, so after a crash it replays from its journal. If the catalog or object store goes away, it retries and catches up. The fault suite

Does compaction stop ingestion?

No. It runs in the same process while changes keep publishing, and it reconciles rewrites made by Trino or Spark.

Is it ready for production?

Not yet. Flow is in beta. It doesn’t support partitioned source tables or Iceberg partition specs, and it has no high availability.

How does Flow fit with Embrasure?

Flow is how the Embrasure Warehouse ingests Postgres and Supabase. You can also run it yourself, under Apache-2.0.

Stream your
first table.

The demo runs Postgres, MinIO, an Iceberg REST catalog, Trino and Flow in Docker Compose.

View on GitHubBook a demo
git clone https://github.com/EmbrasureAI/flow.git
cd flow
docker compose -f demo/compose.yaml up --build -d