Merge ~jk0ne/charm-k8s-discourse/+git/discourse-k8s-image-builder:master into charm-k8s-discourse:master

Proposed by Jay Kuri
Status: Merged
Merged at revision: acff79e6ece1be0da13262bed4302aae6d4fe5cb
Proposed branch: ~jk0ne/charm-k8s-discourse/+git/discourse-k8s-image-builder:master
Merge into: charm-k8s-discourse:master
Diff against target: 341 lines (+281/-0)
10 files modified
Dockerfile (+54/-0)
build_scripts/build_app (+26/-0)
build_scripts/cleanup_post_build (+12/-0)
build_scripts/get_app_dependencies (+27/-0)
build_scripts/setup_base_layer (+21/-0)
scripts/app_launch (+19/-0)
scripts/assets/discourse.conf.tmpl (+21/-0)
scripts/pod_exit (+15/-0)
scripts/pod_setup (+47/-0)
scripts/pod_start (+39/-0)
Reviewer Review Type Date Requested Status
Tom Haddon Approve
Jay Kuri (community) Needs Resubmitting
Canonical IS Reviewers Pending
Review via email: mp+387079@code.launchpad.net

Commit message

This adds various log handling tweaks along with other cleanup. This also adds the migration sync workaround during pod startup.

To post a comment you must log in.
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
Tom Haddon (mthaddon) wrote :

Some comments inline. Overall looks good, mostly just comment formatting, and a few questions about variables.

review: Needs Fixing
Revision history for this message
Jay Kuri (jk0ne) :
Revision history for this message
Tom Haddon (mthaddon) wrote :

I'm not quite sure if you're ready for review on this or still working on it - I think you're ready for review (it's best to add a comment and select "Resubmit" to let reviewers know you're done with changes). Two very small comments inline, but otherwise this is mergeable.

Revision history for this message
Jay Kuri (jk0ne) wrote :

Made the CONTAINER_APP_NAME change requested in previous comments.

review: Needs Resubmitting
Revision history for this message
Tom Haddon (mthaddon) wrote :

LGTM

review: Approve
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Change successfully merged at revision acff79e6ece1be0da13262bed4302aae6d4fe5cb

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/Dockerfile b/Dockerfile
0new file mode 1006440new file mode 100644
index 0000000..7baf5cb
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,54 @@
1FROM ubuntu:focal
2
3# ARGS become environment variables, but can be overridden using the
4# --build-arg var=foo option to docker build. This allows you to have a
5# default build image, but customize certain options such as app version or
6# userids, etc.
7ARG CONTAINER_APP_NAME
8ARG CONTAINER_APP_VERSION
9ARG CONTAINER_APP_USERNAME
10ARG CONTAINER_APP_UID
11ARG CONTAINER_APP_GROUP
12ARG CONTAINER_APP_GID
13
14# Copy any args we got into the environment.
15ENV CONTAINER_APP_NAME ${CONTAINER_APP_NAME:-discourse}
16ENV CONTAINER_APP_VERSION ${CONTAINER_APP_VERSION:-v2.4.3}
17ENV CONTAINER_APP_USERNAME ${CONTAINER_APP_USERNAME:-discourse}
18ENV CONTAINER_APP_UID ${CONTAINER_APP_UID:-200}
19ENV CONTAINER_APP_GROUP ${CONTAINER_APP_GROUP:-discourse}
20ENV CONTAINER_APP_GID ${CONTAINER_APP_GID:-200}
21
22# CONTAINER_APP_ROOT is where files related to this application go. This
23# environment variable is available in the build scripts. This should usually be
24# a subdirectory of /srv.
25ENV CONTAINER_APP_ROOT=/srv/discourse
26
27# Application specific environment variables go here.
28ENV GEM_HOME ${CONTAINER_APP_ROOT}/.gem
29
30# We don't want packages prompting us during install.
31ENV DEBIAN_FRONTEND=noninteractive
32
33# Copy our build and runtime scripts into the image.
34COPY --chown=${CONTAINER_APP_UID}:${CONTAINER_APP_GID} build_scripts /srv/build_scripts
35
36# The setup_base_layer script performs steps that are very unlikely to change
37# for this image.
38RUN /srv/build_scripts/setup_base_layer
39
40# The get_app_dependencies script gets any application-specific packages needed
41# for this build.
42RUN /srv/build_scripts/get_app_dependencies
43
44# Run build process.
45RUN /srv/build_scripts/build_app
46
47# Run post-build cleanup.
48RUN /srv/build_scripts/cleanup_post_build
49
50# Copy run-time scripts into the container.
51COPY --chown=${CONTAINER_APP_UID}:${CONTAINER_APP_GID} scripts /srv/scripts
52
53# Set our entrypoint.
54ENTRYPOINT /srv/scripts/pod_start
diff --git a/build_scripts/build_app b/build_scripts/build_app
0new file mode 10075555new file mode 100755
index 0000000..c313240
--- /dev/null
+++ b/build_scripts/build_app
@@ -0,0 +1,26 @@
1#!/bin/bash
2#
3# build_app
4#
5# This builds the application. This should include retrieving the app if it is
6# built from source, as well as any build, install and post-install setup.
7
8echo "Building ${CONTAINER_APP_NAME} ${CONTAINER_APP_VERSION} in ${CONTAINER_APP_ROOT}..."
9
10mkdir -p ${GEM_HOME}
11chown ${CONTAINER_APP_USERNAME}:${CONTAINER_APP_GROUP} ${GEM_HOME}
12
13cd ${CONTAINER_APP_ROOT}
14git clone --branch ${CONTAINER_APP_VERSION} https://github.com/discourse/discourse.git ${CONTAINER_APP_VERSION}
15chown -R ${CONTAINER_APP_USERNAME}:${CONTAINER_APP_GROUP} ${CONTAINER_APP_VERSION}
16ln -s ${CONTAINER_APP_ROOT}/${CONTAINER_APP_VERSION} app
17cd ${CONTAINER_APP_ROOT}/app
18
19# This must be done as the discourse user in order to avoid permission
20# problems later.
21su -s /bin/bash -c 'bin/bundle install' ${CONTAINER_APP_USERNAME}
22
23# If intermediate files are generated by the build process or other files are
24# generated that are not needed at runtime, remove them here to save image size.
25# If we don't do this they will become part of the image even if removed later.
26find ${CONTAINER_APP_ROOT} -name tmp -type d -exec rm -rf {} +
diff --git a/build_scripts/cleanup_post_build b/build_scripts/cleanup_post_build
0new file mode 10075527new file mode 100755
index 0000000..2c2fe6d
--- /dev/null
+++ b/build_scripts/cleanup_post_build
@@ -0,0 +1,12 @@
1#!/bin/bash
2#
3# cleanup_post_build
4#
5# This does any post-build steps that need to be run and removes anything that
6# should not exist in the image. Note that removals here do not save space, they
7# simply remove the file references from the running image.
8
9apt-get autoremove
10apt-get clean
11
12rm -rf /srv/build_scripts
diff --git a/build_scripts/get_app_dependencies b/build_scripts/get_app_dependencies
0new file mode 10075513new file mode 100755
index 0000000..65b267a
--- /dev/null
+++ b/build_scripts/get_app_dependencies
@@ -0,0 +1,27 @@
1#!/bin/bash
2#
3# get_app_dependencies
4#
5# Installs application-specific dependencies for this image.
6
7apt-get install -y brotli \
8 gifsicle \
9 git \
10 imagemagick \
11 jhead \
12 jpegoptim \
13 libjpeg-turbo-progs \
14 libpq-dev \
15 libxml2-dev \
16 libxslt1-dev \
17 libz-dev \
18 node-uglify \
19 optipng \
20 pngquant \
21 postgresql-client \
22 postgresql-client-common \
23 redis-tools \
24 ruby2.7 \
25 ruby2.7-dev \
26 zlib1g-dev
27
diff --git a/build_scripts/setup_base_layer b/build_scripts/setup_base_layer
0new file mode 10075528new file mode 100755
index 0000000..9d3584e
--- /dev/null
+++ b/build_scripts/setup_base_layer
@@ -0,0 +1,21 @@
1#!/bin/bash
2#
3# setup_base_layer
4#
5# This sets up the base image with common tools and packages that are needed for
6# most images.
7#
8# Update our packages.
9apt-get update
10
11# Set our timezone to UTC.
12ln -s /usr/share/zoneinfo/UTC /etc/localtime
13
14# Get our base packages.
15apt-get install -y gettext-base tzdata ubuntu-dev-tools git
16
17# Create our application group.
18/usr/sbin/addgroup --gid ${CONTAINER_APP_GID} ${CONTAINER_APP_GROUP}
19
20# Create our application user - this also creates our app root.
21/usr/sbin/adduser --uid ${CONTAINER_APP_UID} --home ${CONTAINER_APP_ROOT} --gid ${CONTAINER_APP_GID} --system ${CONTAINER_APP_USERNAME}
diff --git a/scripts/app_launch b/scripts/app_launch
0new file mode 10075522new file mode 100755
index 0000000..87f7166
--- /dev/null
+++ b/scripts/app_launch
@@ -0,0 +1,19 @@
1#!/bin/bash
2
3cd ${CONTAINER_APP_ROOT}/app
4
5export RAILS_ENV=${DISCOURSE_RAILS_ENVIRONMENT:-production}
6export UNICORN_BIND_ALL=0.0.0.0
7
8if [ "${DISCOURSE_ENABLE_LOCAL_SIDEKIQ}" == "true" ]; then
9 export UNICORN_SIDEKIQS=1
10fi
11
12# Putting the tail's in the background gets the various logs outputting to
13# stdout. The sed allows us to distinguish where a given log line comes form.
14touch ${CONTAINER_APP_ROOT}/app/log/production.log
15touch ${CONTAINER_APP_ROOT}/app/log/unicorn-stderr.log
16tail -f ${CONTAINER_APP_ROOT}/app/log/production.log |sed 's/^/discourse: /' &
17tail -f ${CONTAINER_APP_ROOT}/app/log/unicorn-stderr.log |sed 's/^/unicorn-error: /' 1>&2 &
18
19su -s /bin/bash -c "bin/bundle exec unicorn -c ${CONTAINER_APP_ROOT}/app/config/unicorn.conf.rb" ${CONTAINER_APP_USERNAME} 2>&1
diff --git a/scripts/assets/discourse.conf.tmpl b/scripts/assets/discourse.conf.tmpl
0new file mode 10064420new file mode 100644
index 0000000..4cc13b9
--- /dev/null
+++ b/scripts/assets/discourse.conf.tmpl
@@ -0,0 +1,21 @@
1hostname = $DISCOURSE_HOSTNAME
2developer_emails = $DISCOURSE_DEVELOPER_EMAILS
3serve_static_assets = $DISCOURSE_SERVE_STATIC_ASSETS
4db_host = $DISCOURSE_POSTGRES_HOST
5db_username = $DISCOURSE_POSTGRES_USERNAME
6db_password = $DISCOURSE_POSTGRES_PASSWORD
7db_name = $DISCOURSE_POSTGRES_DB_NAME
8smtp_domain = $DISCOURSE_SMTP_DOMAIN
9smtp_address = $DISCOURSE_SMTP_ADDRESS
10smtp_port = $DISCOURSE_SMTP_PORT
11smtp_openssl_verify_mode = $DISCOURSE_SMTP_OPENSSL_VERIFY_MODE
12smtp_user_name = $DISCOURSE_SMTP_USER_NAME
13smtp_password = $DISCOURSE_SMTP_PASSWORD
14smtp_authentication = $DISCOURSE_SMTP_AUTHENTICATION
15enable_cors = $DISCOURSE_ENABLE_CORS
16saml_target_url = $DISCOURSE_SAML_TARGET_URL
17saml_cert_fingerprint = $DISCOURSE_CERT_FINGERPRINT
18saml_full_screen_login = $DISCOURSE_FULL_SCREEN_LOGIN
19refresh_maxmind_db_during_precompile_days = 0
20redis_host = $DISCOURSE_REDIS_HOST
21redis_port = $DISCOURSE_REDIS_PORT
diff --git a/scripts/pod_exit b/scripts/pod_exit
0new file mode 10075522new file mode 100755
index 0000000..cff0efc
--- /dev/null
+++ b/scripts/pod_exit
@@ -0,0 +1,15 @@
1#!/bin/bash
2
3# pod_exit
4#
5# Run when a pod is terminating, assuming it's terminating normally. This gives
6# us a chance to do any api-calls, notifications or last-minute cleanup prior to
7# pod exit.
8
9SETUP_EXITCODE=$1
10APP_EXITCODE=$2
11
12echo "Setup exited with exit code ${SETUP_EXITCODE}"
13echo "App exited with exit code ${APP_EXITCODE}"
14
15exit ${APP_EXITCODE}
diff --git a/scripts/pod_setup b/scripts/pod_setup
0new file mode 10075516new file mode 100755
index 0000000..83e57ff
--- /dev/null
+++ b/scripts/pod_setup
@@ -0,0 +1,47 @@
1#!/bin/bash
2
3cd ${CONTAINER_APP_ROOT}/app
4
5DISCOURSE_CONF=${CONTAINER_APP_ROOT}/app/config/discourse.conf
6
7export RAILS_ENV=${DISCOURSE_RAILS_ENVIRONMENT:-production}
8
9# Generate our discourse.conf: Load config file and remove any lines we don't
10# have config values for.
11/usr/bin/envsubst < /srv/scripts/assets/discourse.conf.tmpl | grep -vE '= *$' > $DISCOURSE_CONF
12
13### MIGRATION LOCK WORKAROUND STARTS HERE
14#
15# Try to get an exclusive lock via redis prior to migration so that we don't
16# have multiple units running migration at once. This is a workaround because we
17# don't have an official Juju way to do this coordination at this point. When we
18# do, this will go away.
19MIGRATE_LOCK=$(redis-cli -h $DISCOURSE_REDIS_HOST --raw setnx discourse_migrate_lock $HOSTNAME)
20if [ ${MIGRATE_LOCK} -eq 1 ]; then
21 # If we are here, we got the migrate lock, so we can do the migrate.
22 echo "Starting migration at " $(date)
23 su -s /bin/bash -c "bin/bundle exec rake db:migrate RAILS_ENV=$RAILS_ENV" ${CONTAINER_APP_USERNAME}
24 redis-cli -h $DISCOURSE_REDIS_HOST --raw del discourse_migrate_lock >/dev/null
25else
26 RETRIES=5
27 while [ "${RETRIES}" -gt 0 ]; do
28 MIGRATE_HOST=$(redis-cli -h $DISCOURSE_REDIS_HOST --raw get discourse_migrate_lock)
29 if [ -z "${MIGRATE_HOST}" ]; then
30 break;
31 else
32 echo "Migrate lock found, Migrate running on ${MIGRATE_HOST}, waiting 90s."
33 sleep 90
34 fi
35 RETRIES=$(($RETRIES - 1))
36 done
37
38 if [ $RETRIES -le 0 ]; then
39 echo "Retried too many times. Can't proceed while migrate is unfinished. Exiting."
40 exit 1
41 fi
42fi
43### MIGRATION LOCK WORKAROUND ENDS HERE
44
45# It is safe to build assets in the background to improve startup time.
46export SKIP_NODE_UGLIFY="true"
47su -s /bin/bash -c "bin/bundle exec rake assets:precompile RAILS_ENV=$RAILS_ENV" ${CONTAINER_APP_USERNAME} 2>&1 |sed 's/^/asset-build: /' &
diff --git a/scripts/pod_start b/scripts/pod_start
0new file mode 10075548new file mode 100755
index 0000000..fd3d95f
--- /dev/null
+++ b/scripts/pod_start
@@ -0,0 +1,39 @@
1#!/bin/bash
2
3# pod_start
4#
5# This script is the primary entrypoint for the pod. It should start up the
6# primary service and output log data to stdout. It should also exit when the
7# application exits in order for the pod to terminate properly.
8
9# Steps:
10# 1) Run any pre-launch setup.
11# 2) Launch application - should not exit until application exits.
12# 3) Do any pod-exit cleanup.
13
14SETUP_EXIT="notrun"
15APP_EXIT="notrun"
16
17# Run pod_setup script first.
18echo "Pod setup starting..."
19/srv/scripts/pod_setup
20SETUP_EXIT=$?
21
22if [ $SETUP_EXIT -eq 0 ]; then
23 echo "${CONTAINER_APP_NAME} setup completed successfully."
24else
25 echo "${CONTAINER_APP_NAME} setup failed with exit code: " $SETUP_EXIT
26fi
27
28
29# If we exited pod_setup ok, we launch the app.
30if [ $SETUP_EXIT -eq 0 ]; then
31 echo "Launching ${CONTAINER_APP_NAME}"
32 /srv/scripts/app_launch
33 APP_EXIT=$?
34 echo "${CONTAINER_APP_NAME} exited with code ${APP_EXIT}"
35fi
36
37# Run pod_exit with either the pod_setup exit code, or the app_launch exit code
38# as the first argument.
39/srv/scripts/pod_exit $SETUP_EXIT $APP_EXIT

Subscribers

People subscribed via source and target branches