Scalable incremental data aggregation on Postgres and Citus

  • The hook, for me, was a few paragraphs in:

    > We tested this approach for a CDN use case and found that a 4-node Citus database cluster can simultaneously:

    > - Ingest and aggregate over a million rows per second

    > - Keep the rollup table up-to-date within ~10s

    > - Answer analytical queries in under 10ms.

  • The example sounds pretty similar to Cloudflare's old pipeline:

    https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...

  • Isnt there a race condition ? you get the sequence before acquiring a lock.