Merge lp:~widelands-dev/widelands-website/wlwebsite_docker into lp:widelands-website

[SirVer]

Experimental branch, not sure if this should be merged.

I set up a docker workflow to run the website against a real mysql database on any development system (windows, mac, linux) easily. The goals here are

1) get rid of installing dependencies (like the python modules) on dev machines.
2) make sure all devs have always the exactly same set of python modules and python version,
3) dev workflows can use external tools (like wl_map_info) also on their dev machine, i.e. less differences between production and dev,
4) quicker setup for new devs.

How does this work? The flow is based around a tool named Docker (https://www.docker.com/). Docker runs containers and links them together. You can think of a container as a zero-cost virtual Linux machine (when running on Linux).

A container is described by a Dockerfile - which is a Build rule for assembling these virtual machines. And the core of this patch is a Dockerfile for the website. The idea is now to run a container with a mysql database[1], then build the website docker container and run it against this mysql. This means you run the full setup of the website on your dev machine.

Read the patch starting from the Makefile, then into the Dockerfile.

The workflow to get this up and running is this:

1) Install docker and do the docker tutorial[2] to get a basic understanding of docker.
2) bring up in one terminal the mysql database (make run_mysql).
3) build an image from the dockerfile: make docker
4) initialize the database and create a superuser: make init
5) run the website and try to upload a map: make serve

[1] https://hub.docker.com/_/mysql/
[2] https://docs.docker.com/engine/getstarted/

[lp:~widelands-dev/widelands-website/wlwebsite_docker]

453. By SirVer on 2017-02-11

More documentation.

[kaputtnik (franku)]

Sounds interesting.

I will test it in the next days.

[kaputtnik (franku)]

I got a failure on make init:

django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
make: *** [Makefile:32: init] Fehler 1

I am unsure about the benefits:

> 1) get rid of installing dependencies (like the python modules)
> on dev machines.
Instead one has to install other things (docker itself, docker-files).

> 2) make sure all devs have always the exactly same set of python
> modules and python version,
Good point. On the other side: If one is using a more up to date version we are able to find a trap early, before deploying.

> 3) dev workflows can use external tools (like wl_map_info) also
> on their dev machine, i.e. less differences between production and dev,
Also a good point. On the other side a website dev should have current trunk from widelands somewhere on his disk, to develop the encyclopedia or the documentation. So it should be possible to give also access to wl_map_info.

> 4) quicker setup for new devs.
I don't know. The whole setup relies on docker which basics have to be understood. From my point of view having our README is much clearer and cleaner than reading and executing the docker example. Using docker will hide some of the installation process to a new dev which could lead into other questions.

Also running docker means each docker-file has some access to the root file system. The 'make init' command produce *.pyc files owned by the root user here.

Regarding mysql, and database in general:
The main Problem for a new dev is that he has no data. All the forum and wiki is empty. For testing purposes a dev has to spend some time to get such data.
I had worked on a text file regarding setting up and maintain a mysql database containing some hints on using utf8, collation, backup and so on. Maybe this is the better way to go?

All this installations takes round about 2 GB of disk space. Can't say why, but i don't like that...

I think this proposal is meant to get more developers for the webpage, but i don't believe it would help here because the process is as 'complicated' as using the README. Maybe i am wrong...

Unmerged revisions

453. By SirVer on 2017-02-11

More documentation.

452. By SirVer on 2017-02-11

Folded database running into Makefile to not require docker compose.

451. By SirVer on 2017-02-11

First working iteration of a dockerized dev environment.

=== added file 'Dockerfile'
--- Dockerfile	1970-01-01 00:00:00 +0000
+++ Dockerfile	2017-02-11 21:31:29 +0000
@@ -0,0 +1,49 @@
+# This image will be based on the ubuntu:zesty image: https://hub.docker.com/_/ubuntu/
+FROM ubuntu:zesty
+ENV PYTHONUNBUFFERED 1
+
+# Some magic to make sure that the uid and gid of created files matches your
+# own. Ignore.
+ARG UID
+ARG GID
+RUN useradd user -u $UID -g $GID -m -s /bin/bash 
+
+# Install add-apt-repository, so we can add the Widelands repository.
+RUN apt-get update && apt-get install -y \
+   software-properties-common
+
+RUN add-apt-repository ppa:widelands-dev/widelands-daily
+
+# Install all software we require.
+RUN apt-get update && apt-get install -y \
+   git \
+   libjpeg-dev \
+   libmysqlclient-dev \
+   libpng-dev \
+   python \
+   python-pip \
+   widelands \
+   xvfb \
+   && rm -rf /var/lib/apt/lists/*
+
+RUN mkdir /src/ && mkdir /code && chown -R user /src /code
+
+# Copy the 'pip_requirements.txt' file into /code. This has also the benefit
+# that whenever this file changes, i.e. our requirements changes, docker will
+# rerun this step and build a new image.
+ADD pip_requirements.txt /code
+RUN pip install -r /code/pip_requirements.txt
+
+# Make sure that when we run this container, we can find the binaries installed by Widelands which end up
+# in /usr/games.
+ENV PATH /usr/games:$PATH
+
+ADD run_docker.sh /code
+
+EXPOSE 8000
+# TODO(sirver): this should be enabled, but leads to wrong writes
+# USER user
+WORKDIR /code
+
+# This is the entrypoint if you run this image without any further commands.
+CMD ["sh", "run_docker.sh"]
 
=== added file 'Makefile'
--- Makefile	1970-01-01 00:00:00 +0000
+++ Makefile	2017-02-11 21:31:29 +0000
@@ -0,0 +1,51 @@
+# Arguments to run the website container, we use this a couple of times below.
+# --link db:db means that the container called 'db' is reachable as 'db' inside this container, i.e. 
+#  ping db will work.
+#  -v means "mount the current directory as /code in the container, i.e. all
+#  changes the container writes there are visible in the current directory and
+#  vice versa.
+#  Note the --rm (remove). Each time we run this command, we create a new
+#  container from the image 'wlwebsite' and delete it immediately after it
+#  exits again.
+DOCKER_RUN:=docker run -it --rm \
+	--link db:db \
+	-p 8000:8000 \
+	-v $(shell pwd):/code \
+	wlwebsite
+
+# Bring up a mysql server. The database will be completely empty the first time
+# this runs, but if you press ctrl-c, docker will not delete this container
+# (you could say it hibernates). starting it again gives you access to all your
+# data again.
+run_mysql:
+	docker run -it \
+		--name db \
+		-e MYSQL_DATABASE=wl_django \
+		-e MYSQL_ROOT_PASSWORD=bluba \
+		-e MYSQL_USER=widelands \
+		-e MYSQL_PASSWORD=widelands \
+		-p 3306:3306 \
+		mysql
+
+# Creates a super user and runs ./manage.py migrate
+init:
+	$(DOCKER_RUN) ./manage.py migrate
+	$(DOCKER_RUN) ./manage.py createsuperuser
+
+# Builds the 'Dockerfile' into an image called 'wlwebsite'. This image can then
+# be run which results in a container.
+docker:
+	docker build . -t wlwebsite \
+		--build-arg UID=$(shell id -u) \
+		--build-arg GID=$(shell id -g)
+
+# Runs ./run_docker.sh inside the container. See Dockerfile.
+serve:
+	$(DOCKER_RUN)
+
+# This will execute the CMD in the Dockerfile which runs the django debug server. Use like this:
+# CMD="<cmd>" make cmd. For example
+# CMD="find / | grep foo" make cmd 
+# lists all files that contain foo in their filename inside the container.
+cmd:
+	$(DOCKER_RUN) ${CMD}
 
=== modified file 'pip_requirements.txt'
--- pip_requirements.txt	2017-01-21 12:39:23 +0000
+++ pip_requirements.txt	2017-02-11 21:31:29 +0000
@@ -24,4 +24,3 @@
 untokenize==0.1.1
 -e git://github.com/kerin/django-sphinx.git#egg=django-sphinx
 bleach==1.4.3
-
 
=== added file 'run_docker.sh'
--- run_docker.sh	1970-01-01 00:00:00 +0000
+++ run_docker.sh	2017-02-11 21:31:29 +0000
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+Xvfb :99 -screen 0 1024x768x16 &> xvfb.log &
+export DISPLAY=:99.0
+
+python manage.py runserver 0.0.0.0:8000
 
=== modified file 'settings.py'
--- settings.py	2017-01-21 12:39:23 +0000
+++ settings.py	2017-02-11 21:31:29 +0000
@@ -12,19 +12,6 @@
 
 MANAGERS = ADMINS
 
-DATABASES = {
-    'default': {
-        'ENGINE': 'django.db.backends.sqlite3',
-        'NAME': 'dev.db',
-        'USER': '',      # Not used with sqlite3.
-        'PASSWORD': '',  # Not used with sqlite3.
-        # Set to empty string for localhost. Not used with sqlite3.
-        'HOST': '',
-        # Set to empty string for default. Not used with sqlite3.
-        'PORT': '',
-    }
-}
-
 # Local time zone for this installation. Choices can be found here:
 # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
 # although not all choices may be available on all operating systems.
@@ -48,7 +35,7 @@
 
 # Absolute path to the directory that holds media.
 # Example: "/home/media/media.lawrence.com/"
-MEDIA_ROOT = ''
+MEDIA_ROOT = '/code/media_root'
 
 # URL that handles the media served from MEDIA_ROOT. Make sure to use a
 # trailing slash if there is a path component (optional in other cases).
@@ -245,7 +232,7 @@
 ###############################
 # Sphinx (Search prog) Config #
 ###############################
-USE_SPHINX = True
+USE_SPHINX = False
 SPHINX_API_VERSION = 0x116
 
 ############