Merge ~cjwatson/launchpad:db-kafka-offset into launchpad:db-devel

[Colin Watson]

Add KafkaOffset table

In order to consume events from Kafka, we'll need a way to keep track of our position in any given event stream. This is tracked by topic name and partition (with the number of partitions depending on the number of parallel consumers of any given topic). Updating a table of (topic, partition, offset) in the same transaction as we handle whatever else we need to do for that event gives us exactly-once event delivery, and seems to integrate quite nicely into the rest of Launchpad.

[Ines Almeida (ines-almeida)]

LGTM

review: Approve

Unmerged commits

ee8a49c... by Colin Watson on 2023-03-13

Add KafkaOffset table

In order to consume events from Kafka, we'll need a way to keep track of
our position in any given event stream. This is tracked by topic name
and partition (with the number of partitions depending on the number of
parallel consumers of any given topic). Updating a table of (topic,
partition, offset) in the same transaction as we handle whatever else we
need to do for that event gives us exactly-once event delivery, and
seems to integrate quite nicely into the rest of Launchpad.

	docs:0 (build)
	lint:0 (build)
	mypy:0 (build)

1 → 3 of 3 results

First • Previous • Next • Last

diff --git a/database/schema/patch-2211-18-0.sql b/database/schema/patch-2211-18-0.sql
new file mode 100644
index 0000000..c0c5e5d
--- /dev/null
+++ b/database/schema/patch-2211-18-0.sql
@@ -0,0 +1,16 @@
+-- Copyright 2023 Canonical Ltd.  This software is licensed under the
+-- GNU Affero General Public License version 3 (see the file LICENSE).
+
+SET client_min_messages=ERROR;
+
+CREATE TABLE KafkaOffset (
+    id serial PRIMARY KEY,
+    topic text NOT NULL,
+    partition integer NOT NULL,
+    stored_offset integer NOT NULL
+);
+
+CREATE UNIQUE INDEX kafkaoffset__topic__partition__key
+    ON KafkaOffset (topic, partition);
+
+INSERT INTO LaunchpadDatabaseRevision VALUES (2211, 18, 0);