Merge ~jk0ne/charm-k8s-discourse/+git/discourse-k8s-image-builder:master into charm-k8s-discourse:master

[Jay Kuri]

This adds various log handling tweaks along with other cleanup. This also adds the migration sync workaround during pod startup.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Tom Haddon (mthaddon)]

Some comments inline. Overall looks good, mostly just comment formatting, and a few questions about variables.

review: Needs Fixing

[Tom Haddon (mthaddon)]

I'm not quite sure if you're ready for review on this or still working on it - I think you're ready for review (it's best to add a comment and select "Resubmit" to let reviewers know you're done with changes). Two very small comments inline, but otherwise this is mergeable.

[Jay Kuri (jk0ne)]

Made the CONTAINER_APP_NAME change requested in previous comments.

review: Needs Resubmitting

[Tom Haddon (mthaddon)]

LGTM

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision acff79e6ece1be0da13262bed4302aae6d4fe5cb

diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 0000000..7baf5cb
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,54 @@
+FROM ubuntu:focal
+
+# ARGS become environment variables, but can be overridden using the
+# --build-arg var=foo option to docker build. This allows you to have a
+# default build image, but customize certain options such as app version or
+# userids, etc.
+ARG CONTAINER_APP_NAME
+ARG CONTAINER_APP_VERSION
+ARG CONTAINER_APP_USERNAME
+ARG CONTAINER_APP_UID
+ARG CONTAINER_APP_GROUP
+ARG CONTAINER_APP_GID
+
+# Copy any args we got into the environment.
+ENV CONTAINER_APP_NAME ${CONTAINER_APP_NAME:-discourse}
+ENV CONTAINER_APP_VERSION ${CONTAINER_APP_VERSION:-v2.4.3}
+ENV CONTAINER_APP_USERNAME ${CONTAINER_APP_USERNAME:-discourse}
+ENV CONTAINER_APP_UID ${CONTAINER_APP_UID:-200}
+ENV CONTAINER_APP_GROUP ${CONTAINER_APP_GROUP:-discourse}
+ENV CONTAINER_APP_GID ${CONTAINER_APP_GID:-200}
+
+# CONTAINER_APP_ROOT is where files related to this application go. This
+# environment variable is available in the build scripts. This should usually be
+# a subdirectory of /srv.
+ENV CONTAINER_APP_ROOT=/srv/discourse
+
+# Application specific environment variables go here.
+ENV GEM_HOME ${CONTAINER_APP_ROOT}/.gem
+
+# We don't want packages prompting us during install.
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Copy our build and runtime scripts into the image.
+COPY --chown=${CONTAINER_APP_UID}:${CONTAINER_APP_GID} build_scripts /srv/build_scripts
+
+# The setup_base_layer script performs steps that are very unlikely to change
+# for this image.
+RUN /srv/build_scripts/setup_base_layer
+
+# The get_app_dependencies script gets any application-specific packages needed
+# for this build.
+RUN /srv/build_scripts/get_app_dependencies
+
+# Run build process.
+RUN /srv/build_scripts/build_app
+
+# Run post-build cleanup.
+RUN /srv/build_scripts/cleanup_post_build
+
+# Copy run-time scripts into the container.
+COPY --chown=${CONTAINER_APP_UID}:${CONTAINER_APP_GID} scripts /srv/scripts
+
+# Set our entrypoint.
+ENTRYPOINT /srv/scripts/pod_start
diff --git a/build_scripts/build_app b/build_scripts/build_app
new file mode 100755
index 0000000..c313240
--- /dev/null
+++ b/build_scripts/build_app
@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+# build_app
+#
+# This builds the application. This should include retrieving the app if it is
+# built from source, as well as any build, install and post-install setup.
+
+echo "Building ${CONTAINER_APP_NAME} ${CONTAINER_APP_VERSION} in ${CONTAINER_APP_ROOT}..."
+
+mkdir -p ${GEM_HOME}
+chown ${CONTAINER_APP_USERNAME}:${CONTAINER_APP_GROUP} ${GEM_HOME}
+
+cd ${CONTAINER_APP_ROOT}
+git clone --branch ${CONTAINER_APP_VERSION} https://github.com/discourse/discourse.git ${CONTAINER_APP_VERSION}
+chown -R ${CONTAINER_APP_USERNAME}:${CONTAINER_APP_GROUP} ${CONTAINER_APP_VERSION}
+ln -s ${CONTAINER_APP_ROOT}/${CONTAINER_APP_VERSION} app
+cd ${CONTAINER_APP_ROOT}/app
+
+# This must be done as the discourse user in order to avoid permission 
+# problems later.
+su -s /bin/bash -c 'bin/bundle install' ${CONTAINER_APP_USERNAME} 
+
+# If intermediate files are generated by the build process or other files are
+# generated that are not needed at runtime, remove them here to save image size.
+# If we don't do this they will become part of the image even if removed later.
+find ${CONTAINER_APP_ROOT} -name tmp -type d -exec rm -rf {} +
diff --git a/build_scripts/cleanup_post_build b/build_scripts/cleanup_post_build
new file mode 100755
index 0000000..2c2fe6d
--- /dev/null
+++ b/build_scripts/cleanup_post_build
@@ -0,0 +1,12 @@
+#!/bin/bash
+#
+# cleanup_post_build
+#
+# This does any post-build steps that need to be run and removes anything that
+# should not exist in the image. Note that removals here do not save space, they
+# simply remove the file references from the running image.
+
+apt-get autoremove
+apt-get clean
+
+rm -rf /srv/build_scripts
diff --git a/build_scripts/get_app_dependencies b/build_scripts/get_app_dependencies
new file mode 100755
index 0000000..65b267a
--- /dev/null
+++ b/build_scripts/get_app_dependencies
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# get_app_dependencies
+#
+# Installs application-specific dependencies for this image.
+ 
+apt-get install -y brotli \
+                   gifsicle \
+                   git \
+                   imagemagick \
+                   jhead \
+                   jpegoptim \
+                   libjpeg-turbo-progs \
+                   libpq-dev \
+                   libxml2-dev \
+                   libxslt1-dev \
+                   libz-dev \
+                   node-uglify \
+                   optipng \
+                   pngquant \
+                   postgresql-client \
+                   postgresql-client-common \
+                   redis-tools \
+                   ruby2.7 \
+                   ruby2.7-dev \
+                   zlib1g-dev
+
diff --git a/build_scripts/setup_base_layer b/build_scripts/setup_base_layer
new file mode 100755
index 0000000..9d3584e
--- /dev/null
+++ b/build_scripts/setup_base_layer
@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# setup_base_layer 
+#
+# This sets up the base image with common tools and packages that are needed for
+# most images.
+#
+# Update our packages.
+apt-get update
+
+# Set our timezone to UTC.
+ln -s /usr/share/zoneinfo/UTC /etc/localtime
+
+# Get our base packages.
+apt-get install -y gettext-base tzdata ubuntu-dev-tools git
+
+# Create our application group.
+/usr/sbin/addgroup --gid ${CONTAINER_APP_GID} ${CONTAINER_APP_GROUP}
+
+# Create our application user - this also creates our app root. 
+/usr/sbin/adduser --uid ${CONTAINER_APP_UID} --home ${CONTAINER_APP_ROOT} --gid ${CONTAINER_APP_GID} --system ${CONTAINER_APP_USERNAME}
diff --git a/scripts/app_launch b/scripts/app_launch
new file mode 100755
index 0000000..87f7166
--- /dev/null
+++ b/scripts/app_launch
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+cd ${CONTAINER_APP_ROOT}/app
+
+export RAILS_ENV=${DISCOURSE_RAILS_ENVIRONMENT:-production}
+export UNICORN_BIND_ALL=0.0.0.0
+
+if [ "${DISCOURSE_ENABLE_LOCAL_SIDEKIQ}" == "true" ]; then
+    export UNICORN_SIDEKIQS=1
+fi
+
+# Putting the tail's in the background gets the various logs outputting to
+# stdout. The sed allows us to distinguish where a given log line comes form.
+touch ${CONTAINER_APP_ROOT}/app/log/production.log
+touch ${CONTAINER_APP_ROOT}/app/log/unicorn-stderr.log
+tail -f ${CONTAINER_APP_ROOT}/app/log/production.log |sed 's/^/discourse: /' &
+tail -f ${CONTAINER_APP_ROOT}/app/log/unicorn-stderr.log |sed 's/^/unicorn-error: /' 1>&2 &
+
+su -s /bin/bash -c "bin/bundle exec unicorn -c ${CONTAINER_APP_ROOT}/app/config/unicorn.conf.rb" ${CONTAINER_APP_USERNAME} 2>&1
diff --git a/scripts/assets/discourse.conf.tmpl b/scripts/assets/discourse.conf.tmpl
new file mode 100644
index 0000000..4cc13b9
--- /dev/null
+++ b/scripts/assets/discourse.conf.tmpl
@@ -0,0 +1,21 @@
+hostname = $DISCOURSE_HOSTNAME 
+developer_emails = $DISCOURSE_DEVELOPER_EMAILS
+serve_static_assets = $DISCOURSE_SERVE_STATIC_ASSETS 
+db_host = $DISCOURSE_POSTGRES_HOST 
+db_username = $DISCOURSE_POSTGRES_USERNAME
+db_password = $DISCOURSE_POSTGRES_PASSWORD
+db_name = $DISCOURSE_POSTGRES_DB_NAME
+smtp_domain = $DISCOURSE_SMTP_DOMAIN
+smtp_address = $DISCOURSE_SMTP_ADDRESS
+smtp_port = $DISCOURSE_SMTP_PORT
+smtp_openssl_verify_mode = $DISCOURSE_SMTP_OPENSSL_VERIFY_MODE
+smtp_user_name = $DISCOURSE_SMTP_USER_NAME
+smtp_password = $DISCOURSE_SMTP_PASSWORD
+smtp_authentication = $DISCOURSE_SMTP_AUTHENTICATION
+enable_cors = $DISCOURSE_ENABLE_CORS
+saml_target_url = $DISCOURSE_SAML_TARGET_URL
+saml_cert_fingerprint = $DISCOURSE_CERT_FINGERPRINT
+saml_full_screen_login = $DISCOURSE_FULL_SCREEN_LOGIN
+refresh_maxmind_db_during_precompile_days = 0
+redis_host = $DISCOURSE_REDIS_HOST
+redis_port = $DISCOURSE_REDIS_PORT
diff --git a/scripts/pod_exit b/scripts/pod_exit
new file mode 100755
index 0000000..cff0efc
--- /dev/null
+++ b/scripts/pod_exit
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# pod_exit
+#
+# Run when a pod is terminating, assuming it's terminating normally. This gives
+# us a chance to do any api-calls, notifications or last-minute cleanup prior to
+# pod exit.
+ 
+SETUP_EXITCODE=$1
+APP_EXITCODE=$2
+
+echo "Setup exited with exit code ${SETUP_EXITCODE}"
+echo "App exited with exit code ${APP_EXITCODE}"
+
+exit ${APP_EXITCODE} 
diff --git a/scripts/pod_setup b/scripts/pod_setup
new file mode 100755
index 0000000..83e57ff
--- /dev/null
+++ b/scripts/pod_setup
@@ -0,0 +1,47 @@
+#!/bin/bash
+
+cd ${CONTAINER_APP_ROOT}/app
+
+DISCOURSE_CONF=${CONTAINER_APP_ROOT}/app/config/discourse.conf
+
+export RAILS_ENV=${DISCOURSE_RAILS_ENVIRONMENT:-production}
+
+# Generate our discourse.conf: Load config file and remove any lines we don't
+# have config values for.
+/usr/bin/envsubst < /srv/scripts/assets/discourse.conf.tmpl | grep -vE '= *$' > $DISCOURSE_CONF
+
+### MIGRATION LOCK WORKAROUND STARTS HERE
+#
+# Try to get an exclusive lock via redis prior to migration so that we don't
+# have multiple units running migration at once. This is a workaround because we
+# don't have an official Juju way to do this coordination at this point. When we
+# do, this will go away.
+MIGRATE_LOCK=$(redis-cli -h $DISCOURSE_REDIS_HOST --raw setnx discourse_migrate_lock $HOSTNAME)
+if [ ${MIGRATE_LOCK} -eq 1 ]; then
+    # If we are here, we got the migrate lock, so we can do the migrate.
+    echo "Starting migration at " $(date)
+    su -s /bin/bash -c "bin/bundle exec rake db:migrate RAILS_ENV=$RAILS_ENV" ${CONTAINER_APP_USERNAME}
+    redis-cli -h $DISCOURSE_REDIS_HOST --raw del discourse_migrate_lock >/dev/null
+else
+    RETRIES=5
+    while [ "${RETRIES}" -gt 0 ]; do
+        MIGRATE_HOST=$(redis-cli -h $DISCOURSE_REDIS_HOST --raw get discourse_migrate_lock)
+        if [ -z "${MIGRATE_HOST}" ]; then
+            break;
+        else
+            echo "Migrate lock found, Migrate running on ${MIGRATE_HOST}, waiting 90s."
+            sleep 90
+        fi
+        RETRIES=$(($RETRIES - 1))
+    done
+
+    if [ $RETRIES -le 0 ]; then
+        echo "Retried too many times. Can't proceed while migrate is unfinished. Exiting."
+        exit 1
+    fi
+fi
+### MIGRATION LOCK WORKAROUND ENDS HERE
+
+# It is safe to build assets in the background to improve startup time.
+export SKIP_NODE_UGLIFY="true"
+su -s /bin/bash -c "bin/bundle exec rake assets:precompile RAILS_ENV=$RAILS_ENV" ${CONTAINER_APP_USERNAME} 2>&1 |sed 's/^/asset-build: /' &
diff --git a/scripts/pod_start b/scripts/pod_start
new file mode 100755
index 0000000..fd3d95f
--- /dev/null
+++ b/scripts/pod_start
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# pod_start
+#
+# This script is the primary entrypoint for the pod. It should start up the
+# primary service and output log data to stdout. It should also exit when the
+# application exits in order for the pod to terminate properly.
+
+# Steps:
+#  1) Run any pre-launch setup.
+#  2) Launch application - should not exit until application exits.
+#  3) Do any pod-exit cleanup.
+
+SETUP_EXIT="notrun"
+APP_EXIT="notrun"
+
+# Run pod_setup script first.
+echo "Pod setup starting..."
+/srv/scripts/pod_setup
+SETUP_EXIT=$?
+
+if [ $SETUP_EXIT -eq 0 ]; then
+    echo "${CONTAINER_APP_NAME} setup completed successfully."
+else
+    echo "${CONTAINER_APP_NAME} setup failed with exit code: " $SETUP_EXIT
+fi
+
+
+# If we exited pod_setup ok, we launch the app.
+if [ $SETUP_EXIT -eq 0 ]; then
+    echo "Launching ${CONTAINER_APP_NAME}"
+    /srv/scripts/app_launch
+    APP_EXIT=$?
+    echo "${CONTAINER_APP_NAME} exited with code ${APP_EXIT}"
+fi
+
+# Run pod_exit with either the pod_setup exit code, or the app_launch exit code
+# as the first argument.
+/srv/scripts/pod_exit $SETUP_EXIT $APP_EXIT