Merge lp:~lazypower/charms/trusty/docker/trunk into lp:charms/trusty/docker

[Charles Butler]

Updates to v0.1.6 of the docker charm, which adds AUFS backend storage support.

The existing charm deploys device mapper, leaving this enabled by default is not desireable as Docker Inc has depreciated support for this storage engine. Its possible to upgrade existing clusters, however the upgrade path takes some manual intervention of exporting and re-importing the containers.

This has potential side-effects for data-containers, and existing services which will prevent them from starting. Full export and recovery instructions are included in the README along with validation of AUFS storage driver from a devicemapper upgrade have been included in the test suite.

[Matt Bruzek (mbruzek)]

Hey Chuck,

Thanks for this update to the docker charm. As we discussed over IRC this branch does not merge cleanly with the trusty/docker that is in the store.

Please fix the branch and resubmit. Thank you.

review: Needs Fixing

[Charles Butler (lazypower)]

I've cleaned up the tree, refactored the tests and I'm getting greens in charmbox. I think we're ready for another round of reviews on this branch.

[Matt Bruzek (mbruzek)]

Nice work here Chuck. I was able to deploy and pull an image. Passes bundletester PASS: 4 Total: 4 (711.994322 sec). I know this aufs change fixes storage problems in the charm.

+1 LGTM

review: Approve

[lp:~lazypower/charms/trusty/docker/trunk]

7. By Matt Bruzek on 2015-09-15

[lazypower] Updates to v0.1.6 of the docker charm, which adds AUFS backend storage support.

=== modified file 'Makefile'
--- Makefile	2015-07-10 20:52:16 +0000
+++ Makefile	2015-09-14 21:25:51 +0000
@@ -2,11 +2,14 @@
 
 build: tox lint test
 
+virtualenv:
+	virtualenv .venv
+
 tox:
 /usr/bin/tox:
 	sudo apt-get install -y  python-tox python-dev python-virtualenv
 
-lint: /usr/bin/tox 
+lint: /usr/bin/tox
 	tox -e lint
 
 
 
=== modified file 'README.md'
--- README.md	2015-07-17 13:22:34 +0000
+++ README.md	2015-09-14 21:25:51 +0000
@@ -1,4 +1,4 @@
-# Docker ![](https://d3oypxn00j2a10.cloudfront.net/0.12.10/img/nav/docker-logo-loggedout.png)
+# Docker Charm
 
 [![Build Status](http://drone.dasroot.net/api/badge/github.com/chuckbutler/docker-charm/status.svg?branch=master)](http://drone.dasroot.net/github.com/chuckbutler/docker-charm)
 
@@ -20,20 +20,20 @@
 
 # Using the Docker Charm
 
-Step by step instructions on using the docker charm:
+Step by step instructions on using the Docker charm:
 
-    juju deploy cs:trusty/docker
+    juju deploy cs:~lazypower/trusty/docker
 
 ## Scale out Usage
 
-Scaling out the docker service is as simple as adding additional docker units
+Scaling out the Docker service is as simple as adding additional Docker units
 to expand your cluster. However, you will need an SDN solution to provide cross
 host networking. See the Known Limitations and issues about this.
 
 # Configuration
 
 - latest : By default the charm assumes installation from the ubuntu
-repositories. If you wish to deploy the latest upstream docker runtime enable
+repositories. If you wish to deploy the latest upstream Docker runtime enable
 this option.
 
 - version : String representation of the version you wish to deploy. This helps
@@ -47,13 +47,22 @@
     leverage a yaml file to spin up and manage multiple containers that
     comprise a single application stack.
 
+- aufs : **new as of v0.1.6** Defaults the backend storage driver to AUFS. The
+    older option of device mapper was horribly broken in most setups, and has
+    been completely depreciated by the Docker foundation. Disable to keep the
+    DeviceMapper backend. - Not recommended. **note** this will break existing
+    containers if you upgrade existing setups. Ensure you account for this when
+    upgrading your Docker clusters that are previously deployed with this charm.
+
+    > See blurb under Known Issues for migration instructions.
+
 ## Known Limitations and Issues
 
 
 #### AWS t1.micro
 
 Performance will suffer on an AWS t1.micro unit - as it has such a limited amount of ram. Between
-the juju unit-agent, and the docker daemon + workloads - you will only be able to run the smallest
+the juju unit-agent, and the Docker daemon + workloads - you will only be able to run the smallest
 of deployments on them. Thus it is not recommended.
 
 #### Local Provider Blockers
@@ -71,7 +80,7 @@
 
 #### Host Only Networking
 
- By default, docker deploys a host-only bridge adapter. Containers are able to communicate with one
+ By default, Docker deploys a host-only bridge adapter. Containers are able to communicate with one
  another if you forward host ports to the containers using the `-p` option. More on this in the
 [Deploying Containers]({{site.url}}/user/deploying-containers.html) docpage.
 
@@ -81,12 +90,21 @@
 
 #### Offline Environments
 
-There is no support for installation of the docker service in
+There is no support for installation of the Docker service in
 [offline environments](https://jujucharms.com/docs/howto-offline-charms).
 There is however [a bug](https://github.com/chuckbutler/docker-charm/issues/13) to track the
 progress of this feature.
 
-
+#### AUFS Upgrade Stopped my containers from working
+
+If you have older containers deployed and running, you will need to pause them
+and export. Once the tarballs of the containers have been exported - upgrade
+your cluster and reimport following the CLI instructions below as a guide
+
+    docker export <<container id>> > mycontainer-latest.tgz
+    # upgrade
+    docker import -i mycontainer-latest - mycontainer:latest
+    docker run <<options>> mycontainer:latest
 
 
 # Contact Information
 
=== modified file 'config.yaml'
--- config.yaml	2015-07-10 20:52:16 +0000
+++ config.yaml	2015-09-14 21:25:51 +0000
@@ -7,7 +7,7 @@
       installs docker.io from the Ubuntu package archive."
   version:
     type: string
-    default: 1.7.0
+    default: 1.8.1-0~trusty
     description: |
       When latest = true, specify the version to install from the PPA.
       latest = false will always assume use the current stable in distro.
@@ -16,4 +16,12 @@
     default: true
     description: |
         Install docker compose, the formation launch utility
-
+  aufs:
+    type: boolean
+    default: true
+    description: |
+        Change the backend storage driver from devicemapper to AUFS.
+        The device mapper storage engine is known to be problematic, and AUFS
+        is not enabled by default in Ubuntu. This will install AUFS support and
+        swap the backend. **NOTE** It is not recommended to reconfigure this post
+        deployment. You may incur dataloss.
 
=== modified file 'hooks/python-pkgs.txt'
--- hooks/python-pkgs.txt	2015-06-02 17:33:17 +0000
+++ hooks/python-pkgs.txt	2015-09-14 21:25:51 +0000
@@ -1,5 +1,5 @@
 -e git+https://github.com/whitmo/ansible-charm.git#egg=AnsibleCharm
 charmhelpers
 ansible
-path.py
+path.py <= 7.7.1
 docker-py
 
=== removed file 'hooks/setup.py'
--- hooks/setup.py	2015-07-17 13:22:34 +0000
+++ hooks/setup.py	1970-01-01 00:00:00 +0000
@@ -1,22 +0,0 @@
-import subprocess
-import sys
-import os
-
-
-def pre_install():
-    """
-    Do any setup required before the install hook.
-    """
-    try:
-        import charmhelpers  # noqa
-        import ansiblecharm  # noqa
-        from path import path  # noqa
-    except ImportError:
-        subprocess.check_call(['hooks/setup.sh'])
-        subprocess.check_call("pip install -r hooks/python-pkgs.txt",
-                              shell=True)
-
-        from path import path
-
-        pth = str(path(os.environ['CHARM_DIR']) / 'src/ansiblecharm')
-        sys.path.append(pth)
 
=== removed file 'hooks/setup.sh'
--- hooks/setup.sh	2015-06-10 20:39:25 +0000
+++ hooks/setup.sh	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-#!/bin/bash
-set -e
-
-apt-get install -y \
-    python-dev \
-    python-pip \
-    git
-
-easy_install -U pip
 
=== modified file 'metadata.yaml'
--- metadata.yaml	2015-07-10 20:52:16 +0000
+++ metadata.yaml	2015-09-14 21:25:51 +0000
@@ -10,8 +10,8 @@
 provides:
   docker-containers:
     interface: containers
-#  events:
-#    interface: docker-socket
+  events:
+    interface: docker-socket
 requires:
   network:
     interface: overlay-network
 
=== added file 'modules/__init__.py'
=== modified file 'playbooks/config-changed.yaml'
--- playbooks/config-changed.yaml	2015-06-10 20:39:25 +0000
+++ playbooks/config-changed.yaml	2015-09-14 21:25:51 +0000
@@ -13,21 +13,61 @@
     service_label: "service={{ service_name }}"
     unit_label: "unit={{ local_unit }}"
 
+- name: grab release info
+  shell: uname -r
+  register: rel
+  when: aufs == true
+
+- set_fact:
+    thisrel: "{{ rel.stdout }}"
+  when: aufs == true
+
+- name: Install AUFS backend supporting packages
+  apt: name={{item}} state=present
+  with_items:
+      - "linux-image-extra-{{thisrel}}"
+      - aufs-tools
+  when: aufs == true
+
+- name: stop docker to remove device mapper
+  service: "name={{docker_version_name}} state=stopped"
+  when: aufs == true and latest == true
+  notify:
+    - restart docker
+
+- name: Remove Devicemapper backend if present
+  file: path=/var/lib/docker/devicemapper state=absent
+  when: aufs == true and latest == true
+
+- name: Register AUFS option
+  docker_opts: action=add
+               key="storage-driver" val="aufs"
+               yaml="{{ opts_yaml }}"
+  when: aufs == true and latest == true
+  notify:
+    - calculate docker opts
+    - render docker defaults
+    - restart docker
+
 - name: set service name as a label
   docker_opts: action=add
                key=label val="{{ service_label }}"
                yaml="{{ opts_yaml }}"
+  when: latest == true
   notify:
     - calculate docker opts
     - render docker defaults
+    - restart docker
 
 - name: set unit as a label
   docker_opts: action=add
                key=label val="{{ unit_label }}"
                yaml="{{ opts_yaml }}"
+  when: latest == true
   notify:
     - calculate docker opts
     - render docker defaults
+    - restart docker
 
 - name: Install docker compose
   include: install-compose.yaml
 
=== modified file 'playbooks/latest-docker.yaml'
--- playbooks/latest-docker.yaml	2015-06-03 17:16:46 +0000
+++ playbooks/latest-docker.yaml	2015-09-14 21:25:51 +0000
@@ -4,15 +4,20 @@
   when: universe_installed
 
 - name: Add upstream apt key
-  apt_key: id=36A1D7869245C8950F966E92D8576A8BA88D21E9 keyserver=keyserver.ubuntu.com
+  apt_key: id=58118E89F3A912897C070ADBF76221572C52609D keyserver=keyserver.ubuntu.com
 
 - name: Add Upstream apt repository
-  apt_repository: repo='deb https://get.docker.com/ubuntu docker main' state=present
+  apt_repository: repo='deb https://apt.dockerproject.org/repo ubuntu-trusty main' state=present
 
 - debug: msg="Installing latest"
 
 - name: Install required packages.
-  apt: name=lxc-docker-{{ version }} state=present update_cache=yes
+  apt: name=docker-engine={{version}} state=present update_cache=yes
+  when: version != ""
+
+- name: Install docker-engine metapackage
+  apt: name=docker-engine state=present update_cache=yes
+  when: version == ""
 
 - set_fact:
-    docker_version_name: docker
 \ No newline at end of file
+    docker_version_name: docker
 
=== modified file 'playbooks/network-relation-changed.yaml'
--- playbooks/network-relation-changed.yaml	2015-06-03 17:16:46 +0000
+++ playbooks/network-relation-changed.yaml	2015-09-14 21:25:51 +0000
@@ -32,6 +32,7 @@
   notify:
     - calculate docker opts
     - render docker defaults
+    - restart docker
 
 - name: set subnet daemon option
   docker_opts: action=set key=mtu val="{{ flannel_mtu }}" yaml="{{opts_yaml}}"
@@ -39,6 +40,7 @@
   notify:
     - calculate docker opts
     - render docker defaults
+    - restart docker
 
 - name: Halt docker service
   service: name={{docker_version_name}} state=stopped
 
=== modified file 'playbooks/site.yaml'
--- playbooks/site.yaml	2015-06-03 17:16:46 +0000
+++ playbooks/site.yaml	2015-09-14 21:25:51 +0000
@@ -3,12 +3,12 @@
     opts_yaml: /etc/ansible/docker-opts.yaml
     docker_version_name: false
   handlers:
-    - name: restart docker
-      service: "name={{docker_version_name}} state=restarted"
     - name: calculate docker opts
       docker_opts: "action=read yaml={{opts_yaml}}"
     - name: render docker defaults
       template: "src=templates/docker-defaults dest=/etc/default/{{docker_version_name}}"
+    - name: restart docker
+      service: "name={{docker_version_name}} state=restarted"
   tasks:
     - name: Check Installation Status
       include: installation-status.yaml
 
=== modified file 'tests/10-deploy-test'
--- tests/10-deploy-test	2015-06-02 17:33:17 +0000
+++ tests/10-deploy-test	2015-09-14 21:25:51 +0000
@@ -15,7 +15,7 @@
         """ This method is run once at class creation. """
         self.deployment = amulet.Deployment(series='trusty')
 
-        self.deployment.add('docker')
+        self.deployment.add('docker', 'local:trusty/docker')
 
         try:
             self.deployment.setup(timeout=seconds)
@@ -66,20 +66,33 @@
     def test_latest_config_option(self):
         """ Set config option to latest and verify docker is installed """
         self.deployment.configure('docker', {'latest': True,
-                                             'version': '1.5.0'})
+                                             'version': '1.8.1-0~trusty'})
         self.deployment.sentry.wait()
-        command = 'dpkg -l lxc-docker-1.5.0'
+        import time
+        print("sleeping 15 seconds because reasons")
+        time.sleep(15)
+        command = 'dpkg -l docker-engine'
         output, code = self.docker_unit.run(command)
-        print(output)
         if output.find('ii') == -1:
             message = 'Could not upgrade docker to latest!'
             amulet.raise_status(amulet.FAIL, msg=message)
         command = 'dpkg -l docker.io'
         output, code = self.docker_unit.run(command)
-        print(output)
         if output.find('ii') != -1:
             message = "Leftover docker.io package found"
             amulet.raise_status(amulet.Fail, msg=message)
 
+    def test_latest_config_option_aufs_enabled(self):
+        '''
+        This test has a byproduct of actually checking to make sure
+        we are cleaning up the broken devicemapper leftovers. Which
+        is totally cool
+        '''
+        # Random failures that require a sleep... so remove the sleep
+        # and recycle the service a second time.
+        command = "docker info"
+        output, code = self.docker_unit.run(command)
+        assert "aufs" in output
+
 if __name__ == '__main__':
     unittest.main()
 
=== modified file 'tests/tests.yaml'
--- tests/tests.yaml	2015-07-10 20:52:16 +0000
+++ tests/tests.yaml	2015-09-14 21:25:51 +0000
@@ -1,4 +1,3 @@
 virtualenv: false
 makefile:
   - lint
-  - unit_test