<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Terraform on Chronicles of Data Enginenering</title>
    <link>https://blog.veskovujovic.me/tags/terraform/</link>
    <description>Recent content in Terraform on Chronicles of Data Enginenering</description>
    <image>
      <title>Chronicles of Data Enginenering</title>
      <url>https://blog.veskovujovic.me/general/floppy.jpg</url>
      <link>https://blog.veskovujovic.me/general/floppy.jpg</link>
    </image>
    <generator>Hugo -- 0.148.0</generator>
    <language>en-us</language>
    <lastBuildDate>Sat, 14 Nov 2026 22:14:14 +0100</lastBuildDate>
    <atom:link href="https://blog.veskovujovic.me/tags/terraform/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>CDC on AWS with PostgreSQL, Debezium, Kafka and ClickHouse</title>
      <link>https://blog.veskovujovic.me/posts/cdc-on-aws-postgresql-to-clickhouse/</link>
      <pubDate>Sat, 14 Nov 2026 22:14:14 +0100</pubDate>
      <guid>https://blog.veskovujovic.me/posts/cdc-on-aws-postgresql-to-clickhouse/</guid>
      <description>Build a CDC pipeline on AWS with Terraform: RDS PostgreSQL, Debezium on Kafka Connect, Kafka on Amazon MSK and ClickHouse. Plus how a replication slot work.</description>
      <content:encoded><![CDATA[<p><img alt="cdc-postgres-to-clickhouse-cover" loading="lazy" src="https://blog.veskovujovic.me/posts/cdc-postgres-to-clickhouse/cover.png"></p>
<h2 id="intro">Intro</h2>
<p>Your analytics team wants fresh numbers, and your PostgreSQL database is not built for them. You can run heavy queries on production and slow down the app. Or you can copy the data once a night and work with yesterday&rsquo;s numbers.</p>
<p>Or maybe you want to <code>squeeze</code> the data in real time (incrementaly) from production system and in parallel build new one. Let&rsquo;s say if client reached all the limits of <code>vertical scaling</code>.</p>
<p>Change Data Capture (CDC) solves both challenges. PostgreSQL already writes every insert, update and delete to its write-ahead log (WAL).</p>
<p>CDC reads that log and streams each change in another database or service.</p>
<p>In this post we build a CDC pipeline fully on <strong>AWS with Terraform: RDS for PostgreSQL, Debezium running on MSK Connect, Kafka on Amazon MSK, and ClickHouse</strong> at the end.</p>
<p>You will learn:</p>
<ul>
<li>What Debezium and MSK Connect do, in plain words</li>
<li>How replication slots work, and how a stopped connector can fill up your production database&rsquo;s disk</li>
<li>What to change to make the pipeline highly available</li>
<li>Why so many companies put ClickHouse at the end of this kind of pipeline</li>
</ul>
<h2 id="the-pipeline-in-one-picture">The Pipeline in One Picture</h2>
<p><img alt="CDC pipeline on AWS: RDS PostgreSQL streams changes through Debezium on MSK Connect into Kafka on Amazon MSK, and ClickHouse on EC2 consumes them into a ReplacingMergeTree table" loading="lazy" src="https://blog.veskovujovic.me/posts/cdc-postgres-to-clickhouse/aws_cdc_pipeline.png"></p>
<ul>
<li><strong>RDS for PostgreSQL</strong> holds a <code>transactions</code> table with dummy card payments. The only setting it needs for CDC is <code>rds.logical_replication = 1</code>.</li>
<li><strong>Debezium</strong> reads the WAL through a replication slot and turns every row change into a JSON event.</li>
<li><strong>Amazon MSK</strong> stores those events in the topic <code>cdc.public.transactions</code>.</li>
<li><strong>ClickHouse</strong> reads the topic and keeps the latest version of every row, ready for analytical queries.</li>
</ul>
<h2 id="debezium-and-msk-connect-in-two-minutes">Debezium and MSK Connect in Two Minutes</h2>
<p><strong>Debezium</strong> is an open source CDC tool. For PostgreSQL it uses logical decoding with the built-in <code>pgoutput</code> plugin, so you install nothing on the database. On the first start it takes a snapshot of the table. After that it streams every change. Each event says what happened (<code>c</code> for create, <code>u</code> for update, <code>d</code> for delete, <code>r</code> for snapshot read) and what the row looks like.</p>
<p>Debezium runs inside <strong>Kafka Connect</strong>, the framework for moving data in and out of Kafka. Running Kafka Connect yourself means managing workers, upgrades and scaling.</p>
<p><strong>MSK Connect</strong> is the managed version on AWS. You give it a plugin and a config, and AWS runs it.</p>
<p>Three things to know before you start:</p>
<ol>
<li><strong>The plugin must be a ZIP or a JAR.</strong> Debezium ships as a <code>.tar.gz</code>, so you download it and zip it again. The demo repo has a script for this.</li>
<li><strong>Pick Kafka Connect 3.7.x.</strong> It runs on Java 17, and Debezium 3.x needs Java 17. On the older 2.7.1 option, a new Debezium connector simply isn&rsquo;t found.</li>
<li><strong>The connector always has one task.</strong> One replication slot has one reader. You scale MSK Connect up (more capacity per worker), not out.</li>
</ol>
<p>The important part of the connector config:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-0-1"><a class="lnlinks" href="#hl-0-1"> 1</a>
</span><span class="lnt" id="hl-0-2"><a class="lnlinks" href="#hl-0-2"> 2</a>
</span><span class="lnt" id="hl-0-3"><a class="lnlinks" href="#hl-0-3"> 3</a>
</span><span class="lnt" id="hl-0-4"><a class="lnlinks" href="#hl-0-4"> 4</a>
</span><span class="lnt" id="hl-0-5"><a class="lnlinks" href="#hl-0-5"> 5</a>
</span><span class="lnt" id="hl-0-6"><a class="lnlinks" href="#hl-0-6"> 6</a>
</span><span class="lnt" id="hl-0-7"><a class="lnlinks" href="#hl-0-7"> 7</a>
</span><span class="lnt" id="hl-0-8"><a class="lnlinks" href="#hl-0-8"> 8</a>
</span><span class="lnt" id="hl-0-9"><a class="lnlinks" href="#hl-0-9"> 9</a>
</span><span class="lnt" id="hl-0-10"><a class="lnlinks" href="#hl-0-10">10</a>
</span><span class="lnt" id="hl-0-11"><a class="lnlinks" href="#hl-0-11">11</a>
</span><span class="lnt" id="hl-0-12"><a class="lnlinks" href="#hl-0-12">12</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-properties" data-lang="properties"><span class="line"><span class="cl"><span class="na">connector.class</span><span class="o">=</span><span class="s">io.debezium.connector.postgresql.PostgresConnector</span>
</span></span><span class="line"><span class="cl"><span class="na">plugin.name</span><span class="o">=</span><span class="s">pgoutput</span>
</span></span><span class="line"><span class="cl"><span class="na">slot.name</span><span class="o">=</span><span class="s">debezium_slot</span>
</span></span><span class="line"><span class="cl"><span class="na">publication.name</span><span class="o">=</span><span class="s">dbz_publication</span>
</span></span><span class="line"><span class="cl"><span class="na">table.include.list</span><span class="o">=</span><span class="s">public.transactions</span>
</span></span><span class="line"><span class="cl"><span class="na">decimal.handling.mode</span><span class="o">=</span><span class="s">string</span>
</span></span><span class="line"><span class="cl"><span class="na">heartbeat.interval.ms</span><span class="o">=</span><span class="s">10000</span>
</span></span><span class="line"><span class="cl"><span class="na">heartbeat.action.query</span><span class="o">=</span><span class="s">UPDATE public.debezium_heartbeat SET ts = now() WHERE id = 1</span>
</span></span><span class="line"><span class="cl"><span class="na">transforms</span><span class="o">=</span><span class="s">unwrap</span>
</span></span><span class="line"><span class="cl"><span class="na">transforms.unwrap.type</span><span class="o">=</span><span class="s">io.debezium.transforms.ExtractNewRecordState</span>
</span></span><span class="line"><span class="cl"><span class="na">transforms.unwrap.delete.tombstone.handling.mode</span><span class="o">=</span><span class="s">rewrite</span>
</span></span><span class="line"><span class="cl"><span class="na">transforms.unwrap.add.fields</span><span class="o">=</span><span class="s">op,lsn,source.ts_ms</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p><code>ExtractNewRecordState</code> flattens Debezium&rsquo;s before/after envelope into a plain row. A delete becomes a row with <code>__deleted = true</code>, which makes life easy for ClickHouse.</p>
<p>You can find all the code for setup here: <a href="https://github.com/vesko-vujovic/cdc-postgres-msk-clickhouse">https://github.com/vesko-vujovic/cdc-postgres-msk-clickhouse</a></p>
<h2 id="replication-slots-how-cdc-can-take-down-production">Replication Slots: How CDC Can Take Down Production</h2>
<h3 id="what-a-replication-slot-is">What a replication slot is</h3>
<p>PostgreSQL writes every change to the <strong>WAL (write ahead log)</strong> before it touches the table.</p>
<p>The WAL is a long list of 16 MB files, and every position in it has an address called an <strong>LSN</strong> (log sequence number).</p>
<p>Once PostgreSQL has safely written changes to the tables, it recycles (deletes) the old WAL files to save disk.</p>
<p>A <strong>replication slot</strong> is an object inside the database that remembers how far one reader has read.</p>
<p><code>The slot is a sticky note on a page that says &quot;someone still needs everything after this page.&quot; The cleaner can't throw away anything after the note until the reader moves it forward.</code></p>
<p>A slot tracks two positions:</p>
<ul>
<li><code>confirmed_flush_lsn</code>: the last change the reader has confirmed it has stored safely.</li>
<li><code>restart_lsn</code>: the oldest WAL the database must keep, so the reader can continue from there after a restart.</li>
</ul>
<p>A <strong>logical</strong> slot, the kind Debezium uses, does one more thing. It decodes the raw WAL into row changes (insert this row, update that one) through an output plugin, here <code>pgoutput</code>. The <strong>publication</strong> decides which tables go into that stream.</p>
<p>For our pipeline it works like this:</p>
<ol>
<li>On its first start, Debezium creates <code>debezium_slot</code> and takes a snapshot of the table.</li>
<li>It reads changes from the slot and sends them to Kafka.</li>
<li>Each time Kafka Connect commits its offsets, Debezium tells PostgreSQL &ldquo;I have everything up to this LSN.&rdquo; The slot moves forward, and PostgreSQL can recycle the older WAL.</li>
</ol>
<p>Two rules matter for the rest of this section. A slot has <strong>one reader</strong> at a time. And it <strong>lives in the database</strong>, not in the connector, so it survives restarts, crashes and even deleting the connector. You can see every slot in <code>pg_replication_slots</code>.</p>
<h3 id="how-it-breaks">How it breaks</h3>
<p>That is what makes CDC reliable. If Debezium restarts, it continues from the slot and loses nothing. It is also the danger:</p>
<ul>
<li>The connector stops, crashes, or someone deletes it.</li>
<li>The slot stays. It is not attached to the connector, it lives in the database.</li>
<li><strong>The application keeps writing. PostgreSQL keeps every WAL file the slot hasn&rsquo;t confirmed.</strong></li>
<li>The disk fills up. On RDS, the instance goes into <code>storage-full</code> and stops accepting writes.</li>
</ul>
<p>Nothing is wrong with the database or the data. A forgotten <code>replication_slot</code> is enough to stop production.</p>
<p>There is an RDS twist too. RDS writes to its own internal tables in the background. If your captured tables are quiet, Debezium gets no events and never confirms a newer position, so WAL piles up anyway. That is why the config has a <strong>heartbeat</strong>: every 10 seconds Debezium updates a small table, gets an event back and moves the slot forward.</p>
<h3 id="the-initial-snapshot-can-hurt-too">The initial snapshot can hurt too</h3>
<p>The first load is the other risky moment. With <code>snapshot.mode=initial</code>, Debezium creates the slot first and then reads the whole table in one long transaction. On a table with hundreds of millions of rows, that can run for hours, and three things happen at once:</p>
<ul>
<li><strong>The slot holds WAL for the whole snapshot.</strong> It was created before the snapshot started, and it can&rsquo;t move forward until the snapshot is done. A busy database keeps writing WAL the entire time.</li>
<li><strong>The long transaction blocks vacuum.</strong> PostgreSQL can&rsquo;t clean up dead rows that the open transaction might still see, so tables and indexes bloat and queries get slower.</li>
<li><strong>The full table scan competes with the application</strong> for CPU, disk I/O and memory, right on the production primary.</li>
</ul>
<p>So a snapshot that takes too long fills the disk the same way a stopped connector does, and it also slows down everything else on the way.</p>
<p>What helps:</p>
<ul>
<li><strong>Use incremental snapshots for big tables.</strong> Start with <code>snapshot.mode=no_data</code>, so Debezium only streams new changes, then trigger an incremental snapshot through a signal table. Debezium reads the table in small chunks between streaming events, with no long transaction, and the slot keeps moving.</li>
<li><strong>Run the first load when traffic is low</strong>, and watch slot lag and <code>FreeStorageSpace</code> while it runs.</li>
<li><strong>Make sure the disk has room</strong> for all the WAL the database writes during the snapshot window.</li>
</ul>
<p>How to protect yourself:</p>
<ul>
<li>
<p><strong>Alarm on <code>OldestReplicationSlotLag</code></strong> and <code>FreeStorageSpace</code> in CloudWatch. The demo creates both.</p>
</li>
<li>
<p><strong>Check slots from SQL</strong> when something looks wrong:</p>
</li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-1-1"><a class="lnlinks" href="#hl-1-1">1</a>
</span><span class="lnt" id="hl-1-2"><a class="lnlinks" href="#hl-1-2">2</a>
</span><span class="lnt" id="hl-1-3"><a class="lnlinks" href="#hl-1-3">3</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">slot_name</span><span class="p">,</span><span class="w"> </span><span class="n">active</span><span class="p">,</span><span class="w"> </span><span class="n">wal_status</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">pg_wal_lsn_diff</span><span class="p">(</span><span class="n">pg_current_wal_lsn</span><span class="p">(),</span><span class="w"> </span><span class="n">restart_lsn</span><span class="p">))</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">retained_wal</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_replication_slots</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><ul>
<li><strong>Consider <code>max_slot_wal_keep_size</code>.</strong> It caps how much WAL one slot can hold. The trade-off: when the cap is hit, PostgreSQL invalidates the slot and Debezium has to snapshot the whole table again. A full disk is worse than a re-snapshot, so for most teams the cap is worth it.</li>
<li><strong>Don&rsquo;t count on storage autoscaling.</strong> It hides the problem until the bill shows up.</li>
</ul>
<h2 id="run-the-demo">Run the Demo</h2>
<p>The whole setup is in the repo: <a href="https://github.com/vesko-vujovic/cdc-postgres-msk-clickhouse">github.com/vesko-vujovic/cdc-postgres-msk-clickhouse</a>. You need Terraform, the AWS CLI and an AWS account.</p>
<p><strong>1. Build it</strong></p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-2-1"><a class="lnlinks" href="#hl-2-1">1</a>
</span><span class="lnt" id="hl-2-2"><a class="lnlinks" href="#hl-2-2">2</a>
</span><span class="lnt" id="hl-2-3"><a class="lnlinks" href="#hl-2-3">3</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">make infra       <span class="c1"># VPC, RDS, MSK, ClickHouse box (~30 min, MSK is the slow part)</span>
</span></span><span class="line"><span class="cl">make init        <span class="c1"># table, debezium user, publication, ClickHouse tables, 10k rows</span>
</span></span><span class="line"><span class="cl">make connector   <span class="c1"># start Debezium: snapshot first, then streaming</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>The connector starts in a separate step because the table, user and publication must exist before Debezium connects.</p>
<p><code>The RDS instance</code>
<img alt="The cdc-demo RDS for PostgreSQL instance available in the AWS console after make infra: db.t4g.micro in eu-west-1a" loading="lazy" src="https://blog.veskovujovic.me/posts/cdc-postgres-to-clickhouse/rds.png"></p>
<p><code>The MSK Kafka instance</code>
<img alt="The cdc-demo Amazon MSK cluster active in the AWS console after make infra: provisioned, Kafka 3.8.x, 2 brokers" loading="lazy" src="https://blog.veskovujovic.me/posts/cdc-postgres-to-clickhouse/kafka_msk.png"></p>
<p><strong>2. Watch changes flow</strong></p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-3-1"><a class="lnlinks" href="#hl-3-1">1</a>
</span><span class="lnt" id="hl-3-2"><a class="lnlinks" href="#hl-3-2">2</a>
</span><span class="lnt" id="hl-3-3"><a class="lnlinks" href="#hl-3-3">3</a>
</span><span class="lnt" id="hl-3-4"><a class="lnlinks" href="#hl-3-4">4</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">make compare          <span class="c1"># same row count and total in PostgreSQL and ClickHouse</span>
</span></span><span class="line"><span class="cl">make churn <span class="nv">N</span><span class="o">=</span><span class="m">2000</span>     <span class="c1"># settle, refund and delete transactions</span>
</span></span><span class="line"><span class="cl">make compare          <span class="c1"># matches again a few seconds later</span>
</span></span><span class="line"><span class="cl">make latency          <span class="c1"># p50 / p99 lag from commit to ClickHouse</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>3. Break it</strong></p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-4-1"><a class="lnlinks" href="#hl-4-1">1</a>
</span><span class="lnt" id="hl-4-2"><a class="lnlinks" href="#hl-4-2">2</a>
</span><span class="lnt" id="hl-4-3"><a class="lnlinks" href="#hl-4-3">3</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">make stop-connector   <span class="c1"># deletes the connector, the slot stays behind</span>
</span></span><span class="line"><span class="cl">make flood <span class="nv">N</span><span class="o">=</span><span class="m">1000000</span>  <span class="c1"># 1M inserts plus an update of every row</span>
</span></span><span class="line"><span class="cl">make slot             <span class="c1"># active = f, retained_wal keeps growing</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p><strong>4. Fix it</strong></p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-5-1"><a class="lnlinks" href="#hl-5-1">1</a>
</span><span class="lnt" id="hl-5-2"><a class="lnlinks" href="#hl-5-2">2</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">make start-connector  <span class="c1"># resumes from the stored offset, no new snapshot</span>
</span></span><span class="line"><span class="cl">make slot             <span class="c1"># retained_wal goes back down as Debezium catches up</span>
</span></span></code></pre></td></tr></table>
</div>
</div><p>The recovery works without a new snapshot because the worker config sets a fixed <code>offset.storage.topic</code>. By default, every new MSK Connect connector gets its own offset topic, so a recreated connector forgets where it was and starts over.</p>
<p>When you are done, run <code>make destroy</code>. <strong>MSK, MSK Connect, RDS and the NAT gateway</strong> all bill by the hour.</p>
<h2 id="making-it-highly-available">Making It Highly Available</h2>
<p>The demo runs on two AZs to keep costs down. For a real workload with a lot of data:</p>
<ul>
<li><strong>Kafka:</strong> 3 brokers in 3 AZs, replication factor 3 and <code>min.insync.replicas = 2</code>. You can lose one AZ without losing data or stopping writes. In the demo this is one variable: <code>az_count = 3</code>.</li>
<li><strong>MSK Connect:</strong> you can&rsquo;t add tasks, so give the worker more capacity (MCUs) when throughput grows. If the worker dies, MSK Connect restarts it and Debezium continues from its offset. Keep the fixed offset topic so this also holds when you recreate the connector.</li>
<li><strong>PostgreSQL:</strong> the slot exists only on the primary. Without extra setup, a failover means a new slot and a new snapshot. PostgreSQL 17 on RDS can sync logical slots to a read replica, so the connector can continue after promotion. The slot has to be created with failover enabled, so check that your Debezium version supports it.</li>
<li><strong>Many tables:</strong> split them across a few connectors instead of one huge one. Each connector has its own slot, so a problem in one doesn&rsquo;t block the others. Every slot is one more thing to monitor, though.</li>
<li><strong>ClickHouse:</strong> use replicated tables on a cluster, or ClickHouse Cloud, instead of a single node.</li>
</ul>
<h2 id="why-clickhouse-at-the-end">Why ClickHouse at the End</h2>
<p>PostgreSQL stores data row by row, which is right for an app that reads and writes single orders.</p>
<p>Analytics does the opposite. &ldquo;Total refunds per merchant last month&rdquo; reads two columns across millions of rows.</p>
<p><code>ClickHouse stores each column separately and compresses it, so that query reads a small part of the data and comes back in milliseconds to seconds.</code></p>
<p>That is why companies use it as the destination for CDC data. It handles <code>real-time dashboards, user-facing analytics and reporting</code> without touching the production database. It&rsquo;s also open source, so you can run it yourself, as in this demo, or use the managed cloud version.</p>
<p>Two ClickHouse features make the CDC part simple:</p>
<ol>
<li><strong>The Kafka table engine</strong> reads the topic directly, and a materialized view inserts every batch into the real table. You don&rsquo;t need a second connector.</li>
<li><strong><code>ReplacingMergeTree(_version, _is_deleted)</code></strong> keeps only the newest version of each row. The PostgreSQL log position (LSN) is the version, because it only goes up. A delete sets <code>_is_deleted = 1</code>, and the row disappears.</li>
</ol>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-6-1"><a class="lnlinks" href="#hl-6-1"> 1</a>
</span><span class="lnt" id="hl-6-2"><a class="lnlinks" href="#hl-6-2"> 2</a>
</span><span class="lnt" id="hl-6-3"><a class="lnlinks" href="#hl-6-3"> 3</a>
</span><span class="lnt" id="hl-6-4"><a class="lnlinks" href="#hl-6-4"> 4</a>
</span><span class="lnt" id="hl-6-5"><a class="lnlinks" href="#hl-6-5"> 5</a>
</span><span class="lnt" id="hl-6-6"><a class="lnlinks" href="#hl-6-6"> 6</a>
</span><span class="lnt" id="hl-6-7"><a class="lnlinks" href="#hl-6-7"> 7</a>
</span><span class="lnt" id="hl-6-8"><a class="lnlinks" href="#hl-6-8"> 8</a>
</span><span class="lnt" id="hl-6-9"><a class="lnlinks" href="#hl-6-9"> 9</a>
</span><span class="lnt" id="hl-6-10"><a class="lnlinks" href="#hl-6-10">10</a>
</span><span class="lnt" id="hl-6-11"><a class="lnlinks" href="#hl-6-11">11</a>
</span><span class="lnt" id="hl-6-12"><a class="lnlinks" href="#hl-6-12">12</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">cdc</span><span class="p">.</span><span class="n">transactions</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">id</span><span class="w">          </span><span class="n">UInt64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">merchant</span><span class="w">    </span><span class="n">LowCardinality</span><span class="p">(</span><span class="n">String</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">amount</span><span class="w">      </span><span class="nb">Decimal</span><span class="p">(</span><span class="mi">12</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">status</span><span class="w">      </span><span class="n">LowCardinality</span><span class="p">(</span><span class="n">String</span><span class="p">),</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">_version</span><span class="w">    </span><span class="n">UInt64</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="n">_is_deleted</span><span class="w"> </span><span class="n">UInt8</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="c1">-- other columns left out
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">ENGINE</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ReplacingMergeTree</span><span class="p">(</span><span class="n">_version</span><span class="p">,</span><span class="w"> </span><span class="n">_is_deleted</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><p>If you wonder what is <code>ReplacingMergeTree</code> it  is a specialized table engine in ClickHouse that extends the standard MergeTree engine to automatically remove duplicate rows and handle updates</p>
<p>One thing to remember: ClickHouse removes old versions in the background, not on insert. Add <code>FINAL</code> to queries that must see exactly one row per id:</p>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt" id="hl-7-1"><a class="lnlinks" href="#hl-7-1">1</a>
</span><span class="lnt" id="hl-7-2"><a class="lnlinks" href="#hl-7-2">2</a>
</span><span class="lnt" id="hl-7-3"><a class="lnlinks" href="#hl-7-3">3</a>
</span><span class="lnt" id="hl-7-4"><a class="lnlinks" href="#hl-7-4">4</a>
</span><span class="lnt" id="hl-7-5"><a class="lnlinks" href="#hl-7-5">5</a>
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">merchant</span><span class="p">,</span><span class="w"> </span><span class="k">sum</span><span class="p">(</span><span class="n">amount</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">total</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">cdc</span><span class="p">.</span><span class="n">transactions</span><span class="w"> </span><span class="k">FINAL</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">WHERE</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;settled&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">merchant</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">total</span><span class="w"> </span><span class="k">DESC</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></td></tr></table>
</div>
</div><h2 id="conclusion">Conclusion</h2>
<ul>
<li><strong>CDC reads the database log</strong>, so your application doesn&rsquo;t change and production gets no extra queries. Debezium on MSK Connect plus Kafka on Amazon MSK gives you that as a managed setup.</li>
<li><strong>The replication slot is the part that can hurt you.</strong> A stopped connector makes PostgreSQL keep WAL until the disk is full. Alarm on slot lag, use a heartbeat, and think about <code>max_slot_wal_keep_size</code>.</li>
<li><strong>ClickHouse is a good place to end up.</strong> The Kafka engine and <code>ReplacingMergeTree</code> turn a stream of changes into a table you can query, without another connector.
Clone the repo, run <code>make up</code>, then stop the connector and watch the slot grow. Seeing it happen in a demo is a lot cheaper than seeing it happen in production.</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
