Merge ~athos-ribeiro/ubuntu-docker-images/+git/kafka:add-entrypoint into ~ubuntu-docker-images/ubuntu-docker-images/+git/kafka:3.1-22.04

[Athos Ribeiro]

This MP changes the kafka image to use an entrypoint to start kafka by default and allow specifying a zookeeper server through env vars.

Based on change requests, it also splits the image in two different images: kafka and zookeeper.

[Sergio Durigan Junior (sergiodj)]

Thanks for the MP, Athos.

LGTM modulo a small nit in the entrypoint script. Let's see what Valentin thinks :-).

review: Approve

[Athos Ribeiro (athos-ribeiro)]

Thanks, Sergio!

I changed the entrypoint to address your comment

[Valentin Viennot (valentinviennot)]

Thanks Athos! I added a small tweak for providing default values for the environment variables.

However, thinking through this, I think that having Zookeeper and Kafka be the same image leads to this quite weird and very complex configuration.

For a first, this might be fine. But if the build infra permits to use targets, I have made a counter proposal : https://git.launchpad.net/~valentinviennot/ubuntu-docker-images/+git/kafka/commit/?h=split-images using targets to build an image for Kafka and one for Zookeeper. Let me know and I will create new repos if this is supported.

Otherwise, we shall move forward with this implementation.

review: Approve

[Athos Ribeiro (athos-ribeiro)]

Thanks, Valentin!

AFAICT, LP does not support building to targets.

One option to create a new zookeeper image would be to maintain a new Dockerfile with all the contents of the kafka one plus the additional stage you proposed (or even letting them differ a bit on entrypoints, exposed ports, etc. At first I considered layering the zookeeper image on top of the kafka one, but that would imply a new build chain order which our tooling is not yet ready to support.

With this, we would maintain a Dockerfile.kafka and a Dockerfile.zookeeper in this repository (LP does support specifying which Dockerfile to use). Would that be any better than the current state?

[Valentin Viennot (valentinviennot)]

> to create a new zookeeper image would be to maintain a new Dockerfile with all the contents of the kafka

Good point. Not ideal, but it might be a solution indeed. From a UX perspective at least, it would be a great improvement. (and it opens the door for building more specific images in the future, only including the necessary binaries.) I’m leaving the implementation details up to you. In [1], I forked a step before, resulting in a common unnamed base and two different images for Zookeeper and Kafka.

[1] https://git.launchpad.net/~valentinviennot/ubuntu-docker-images/+git/kafka/tree/Dockerfile?h=split-images

> On 20 Apr 2022, at 15:55, Athos Ribeiro <email address hidden> wrote:
>
> Thanks, Valentin!
>
> AFAICT, LP does not support building to targets.
>
> One option to create a new zookeeper image would be to maintain a new Dockerfile with all the contents of the kafka one plus the additional stage you proposed (or even letting them differ a bit on entrypoints, exposed ports, etc. At first I considered layering the zookeeper image on top of the kafka one, but that would imply a new build chain order which our tooling is not yet ready to support.
>
> With this, we would maintain a Dockerfile.kafka and a Dockerfile.zookeeper in this repository (LP does support specifying which Dockerfile to use). Would that be any better than the current state?
> --
> https://code.launchpad.net/~athos-ribeiro/ubuntu-docker-images/+git/kafka/+merge/419704
> You are reviewing the proposed merge of ~athos-ribeiro/ubuntu-docker-images/+git/kafka:add-entrypoint into ~canonical-server/ubuntu-docker-images/+git/kafka:3.1-22.04.
>

[Athos Ribeiro (athos-ribeiro)]

Hi Valentin, Sergio,

I split the Dockerfile in two by using Jinja2 for templating from a common Dockerfile here.

This way, we will not need to keep maintaining two different Dockerfiles for these images.

I also fixed all the examples to adapt to the new format.

Once approved, I will proceed to fix the integration tests for these images.

[Valentin Viennot (valentinviennot)]

LGTM! Good Jinja trick :)

review: Approve

[Sergio Durigan Junior (sergiodj)]

LGTM, thanks Athos.

review: Approve

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..d526d56
--- /dev/null
+++ b/README.md
@@ -0,0 +1,41 @@
+# Ubuntu Kafka image set
+
+This repository contains the source code for the Ubuntu kafka image set,
+composed by the following images:
+
+- `ubuntu/kafka`; and
+- `ubuntu/zookeeper`.
+
+Since the images are composed by the same artifacts, they could share the same
+Dockerfile in a multistage build. In this case, builds could be set to target
+specific stages to generate the different images. Since this is not yet
+supported in Launchpad, we split the images in two different Dockerfiles:
+
+- `oci/Dockerfile.kafka`; and
+- `oci/Dockerfile.zookeeper`.
+
+The images generated from these Dockerfiles share most of their layers.
+Therefore, they are generated by a common
+[Jinjna2](https://pypi.org/project/Jinja2/) template, located at
+
+- `template/Dockerfile.j2`
+
+To perform changes in the images, e.g., when preparing images for a new Ubuntu
+release, make the changes in the template described above and run the renderer
+script (`render-dockerfiles.py`) to generate the new versions of the
+Dockerfiles.
+
+To run the renderer, you will need Python and Jinja2 installed. From an Ubuntu
+system:
+
+```
+apt install python3-jinja2
+```
+
+Then, you can generate the Dockerfiles and test the new image builds with
+
+```
+python render-dockerfiles.py
+docker build -t ubuntu/kafka:edge -f oci/Dockerfile.kafka oci
+docker build -t ubuntu/zookeeper:edge -f oci/Dockerfile.zookeeper oci
+```
diff --git a/examples/README.md b/examples/README.md
index 0ea1490..9e7a10d 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -21,8 +21,9 @@ directory, you could use this service with:
 ```
 docker run  --rm -it \
   --network=examples_default \
+  --entrypoint=SOME_KAFKA_BINARY \
   ubuntu/kafka:edge \
-  SOME_KAFKA_BINARY_AND_OPTIONS
+  SOME_KAFKA_BINARY_OPTIONS
 ```
 
 and use `kafka:9092` as your kafka server parameter.
@@ -50,6 +51,6 @@ $ microk8s kubectl apply -f kafka-deployment.yml
 You can now try your Kafka setup within microk8s. For instance, you could
 use some of the Kafka shipped binaries with
 
-`microk8s kubectl run -it --rm  kafka-client --restart=Never --image ubuntu/kafka:edge -- SOME_KAFKA_BINARY_AND_OPTIONS`
+`microk8s kubectl run -it --rm  kafka-client --restart=Never --image ubuntu/kafka:edge --command -- SOME_KAFKA_BINARY_AND_OPTIONS`
 
 and use `kafka-service:9092` as your kafka server parameter.
diff --git a/examples/docker-compose.yml b/examples/docker-compose.yml
index 2ff529b..f33caab 100644
--- a/examples/docker-compose.yml
+++ b/examples/docker-compose.yml
@@ -3,13 +3,13 @@ version: '2'
 services:
   kafka:
     image: ubuntu/kafka:edge
-    command: kafka-server-start.sh /etc/kafka/server.properties --override zookeeper.connect=zookeeper:2181
+    environment:
+      - ZOOKEEPER_HOST=zookeeper
     ports:
       - '9092:9092'
     depends_on:
       - zookeeper
   zookeeper:
-    image: ubuntu/kafka:edge
-    command: zookeeper-server-start.sh /etc/kafka/zookeeper.properties
+    image: ubuntu/zookeeper:edge
     ports:
       - '2181:2181'
diff --git a/examples/kafka-deployment.yml b/examples/kafka-deployment.yml
index 44dc71b..20943b4 100644
--- a/examples/kafka-deployment.yml
+++ b/examples/kafka-deployment.yml
@@ -28,7 +28,10 @@ spec:
       containers:
       - name: kafka
         image: ubuntu/kafka:edge
-        args: ["kafka-server-start.sh", "/etc/kafka/server.properties", "--override", "zookeeper.connect=zookeeper-service:2181", "--override", "advertised.listeners=PLAINTEXT://kafka-service:9092"]
+        env:
+          - name: ZOOKEEPER_HOST
+            value: zookeeper-service
+        args: ["/etc/kafka/server.properties", "--override", "advertised.listeners=PLAINTEXT://kafka-service:9092"]
         ports:
         - containerPort: 9092
           name: kafka
@@ -62,8 +65,7 @@ spec:
     spec:
       containers:
       - name: zookeeper
-        image: ubuntu/kafka:edge
-        args: ["zookeeper-server-start.sh", "/etc/kafka/zookeeper.properties"]
+        image: ubuntu/zookeeper:edge
         ports:
         - containerPort: 2181
           name: zookeeper
diff --git a/oci/Dockerfile.kafka b/oci/Dockerfile.kafka
new file mode 100644
index 0000000..cc15f9b
--- /dev/null
+++ b/oci/Dockerfile.kafka
@@ -0,0 +1,47 @@
+FROM ubuntu:jammy AS snap-installer
+
+RUN set -eux; \
+	apt-get update && \
+	DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y && \
+	DEBIAN_FRONTEND=noninteractive apt-get install -y \
+		jq curl ca-certificates squashfs-tools && \
+	# taken from https://snapcraft.io/docs/build-on-docker
+	# Alternatively, we can install snapd, and issue `snap download kafka`
+	curl -L -H 'Snap-CDN: none' $(curl -H 'X-Ubuntu-Series: 16'  -H "X-Ubuntu-Architecture: $(dpkg --print-architecture)" 'https://api.snapcraft.io/api/v1/snaps/details/kafka?channel=rock/edge' | jq '.download_url' -r) --output kafka.snap && \
+	mkdir -p /snap && unsquashfs -d /snap/kafka kafka.snap
+
+FROM ubuntu:jammy
+
+LABEL maintainer="Ubuntu Server team <[e-mail]>"
+
+ENV TZ=UTC PATH=/opt/kafka/bin:$PATH
+
+COPY --from=snap-installer /snap/kafka/opt/kafka /opt/kafka
+
+# Copy the manifest files from the snap
+COPY --from=snap-installer /snap/kafka/snap/snapcraft.yaml /usr/share/rocks/
+COPY --from=snap-installer /snap/kafka/snap/manifest.yaml /usr/share/rocks/
+
+RUN set -eux; \
+	ln -s /opt/kafka/config /etc/kafka; \
+	apt-get update; \
+	DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y; \
+	DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+		openjdk-11-jre-headless ca-certificates tzdata \
+		; \
+	DEBIAN_FRONTEND=noninteractive apt-get remove --purge --auto-remove -y; \
+	rm -rf /var/lib/apt/lists/*; \
+# create deb manifest
+	(echo "# os-release" && cat /etc/os-release && echo "# dpkg-query" && dpkg-query -f '${db:Status-Abbrev},${binary:Package},${Version},${source:Package},${Source:Version}\n' -W) > /usr/share/rocks/dpkg.query
+
+EXPOSE 9092
+
+COPY ./kafka-entrypoint.sh /usr/local/bin/entrypoint.sh
+
+# Default values to relate the Zookeeper and Kafka instances
+# Note that 172.17.0.1 is the default IP for the Docker host (host.docker.internal)
+# For more configuration changes, change the configuration file (e.g., /etc/kafka/server.properties)
+ENV ZOOKEEPER_PORT=2181 ZOOKEEPER_HOST=172.17.0.1
+
+ENTRYPOINT ["entrypoint.sh"]
+CMD ["/etc/kafka/server.properties"]
diff --git a/Dockerfile b/oci/Dockerfile.zookeeper
similarity index 94%
rename from Dockerfile
rename to oci/Dockerfile.zookeeper
index f26b84b..6a07666 100644
--- a/Dockerfile
+++ b/oci/Dockerfile.zookeeper
@@ -34,4 +34,7 @@ RUN set -eux; \
 # create deb manifest
 	(echo "# os-release" && cat /etc/os-release && echo "# dpkg-query" && dpkg-query -f '${db:Status-Abbrev},${binary:Package},${Version},${source:Package},${Source:Version}\n' -W) > /usr/share/rocks/dpkg.query
 
-EXPOSE 2181 9092
+EXPOSE 2181
+
+ENTRYPOINT ["/opt/kafka/bin/zookeeper-server-start.sh"]
+CMD ["/etc/kafka/zookeeper.properties"]
diff --git a/oci/kafka-entrypoint.sh b/oci/kafka-entrypoint.sh
new file mode 100755
index 0000000..7d9e96e
--- /dev/null
+++ b/oci/kafka-entrypoint.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+ERR_MSG="E: If either ZOOKEEPER_HOST or ZOOKEEPER_PORT is defined, both must be defined"
+if [ -n "${ZOOKEEPER_HOST}" ] || [ -n "${ZOOKEEPER_PORT}" ]; then
+  if [ -z "${ZOOKEEPER_HOST}" ]; then
+    echo "E: ZOOKEEPER_HOST is not defined"
+    echo "${ERR_MSG}"
+    exit 1
+  elif [ -z "${ZOOKEEPER_PORT}" ]; then
+    echo "E: ZOOKEEPER_PORT is not defined"
+    echo "${ERR_MSG}"
+    exit 1
+  fi
+  exec /opt/kafka/bin/kafka-server-start.sh "${@}" --override zookeeper.connect="${ZOOKEEPER_HOST}":"${ZOOKEEPER_PORT}"
+else
+  exec /opt/kafka/bin/kafka-server-start.sh "${@}"
+fi
diff --git a/render-dockerfiles.py b/render-dockerfiles.py
new file mode 100644
index 0000000..c31d519
--- /dev/null
+++ b/render-dockerfiles.py
@@ -0,0 +1,17 @@
+from jinja2 import Environment, FileSystemLoader
+
+KAFKA_DF_PATH = 'oci/Dockerfile.kafka'
+ZOOKEEPER_DF_PATH = 'oci/Dockerfile.zookeeper'
+TEMPLATE_DIR = 'template'
+TEMPLATE = 'Dockerfile.j2'
+
+env = Environment(loader=FileSystemLoader(TEMPLATE_DIR))
+template = env.get_template(TEMPLATE)
+kafka_df = template.render(image='kafka')
+zookeeper_df = template.render(image='zookeeper')
+
+with open(KAFKA_DF_PATH, 'w') as f:
+    f.write(kafka_df)
+
+with open(ZOOKEEPER_DF_PATH, 'w') as f:
+    f.write(zookeeper_df)
diff --git a/template/Dockerfile.j2 b/template/Dockerfile.j2
new file mode 100644
index 0000000..2581520
--- /dev/null
+++ b/template/Dockerfile.j2
@@ -0,0 +1,53 @@
+FROM ubuntu:jammy AS snap-installer
+
+RUN set -eux; \
+	apt-get update && \
+	DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y && \
+	DEBIAN_FRONTEND=noninteractive apt-get install -y \
+		jq curl ca-certificates squashfs-tools && \
+	# taken from https://snapcraft.io/docs/build-on-docker
+	# Alternatively, we can install snapd, and issue `snap download kafka`
+	curl -L -H 'Snap-CDN: none' $(curl -H 'X-Ubuntu-Series: 16'  -H "X-Ubuntu-Architecture: $(dpkg --print-architecture)" 'https://api.snapcraft.io/api/v1/snaps/details/kafka?channel=rock/edge' | jq '.download_url' -r) --output kafka.snap && \
+	mkdir -p /snap && unsquashfs -d /snap/kafka kafka.snap
+
+FROM ubuntu:jammy
+
+LABEL maintainer="Ubuntu Server team <[e-mail]>"
+
+ENV TZ=UTC PATH=/opt/kafka/bin:$PATH
+
+COPY --from=snap-installer /snap/kafka/opt/kafka /opt/kafka
+
+# Copy the manifest files from the snap
+COPY --from=snap-installer /snap/kafka/snap/snapcraft.yaml /usr/share/rocks/
+COPY --from=snap-installer /snap/kafka/snap/manifest.yaml /usr/share/rocks/
+
+RUN set -eux; \
+	ln -s /opt/kafka/config /etc/kafka; \
+	apt-get update; \
+	DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y; \
+	DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+		openjdk-11-jre-headless ca-certificates tzdata \
+		; \
+	DEBIAN_FRONTEND=noninteractive apt-get remove --purge --auto-remove -y; \
+	rm -rf /var/lib/apt/lists/*; \
+# create deb manifest
+	(echo "# os-release" && cat /etc/os-release && echo "# dpkg-query" && dpkg-query -f '${db:Status-Abbrev},${binary:Package},${Version},${source:Package},${Source:Version}\n' -W) > /usr/share/rocks/dpkg.query
+{% if image == 'kafka' %}
+EXPOSE 9092
+
+COPY ./kafka-entrypoint.sh /usr/local/bin/entrypoint.sh
+
+# Default values to relate the Zookeeper and Kafka instances
+# Note that 172.17.0.1 is the default IP for the Docker host (host.docker.internal)
+# For more configuration changes, change the configuration file (e.g., /etc/kafka/server.properties)
+ENV ZOOKEEPER_PORT=2181 ZOOKEEPER_HOST=172.17.0.1
+
+ENTRYPOINT ["entrypoint.sh"]
+CMD ["/etc/kafka/server.properties"]
+{% elif image == 'zookeeper' %}
+EXPOSE 2181
+
+ENTRYPOINT ["/opt/kafka/bin/zookeeper-server-start.sh"]
+CMD ["/etc/kafka/zookeeper.properties"]
+{% endif %}