Merge lp:~psivaa/uci-engine/django-in-mthood into lp:uci-engine/mthood

[Para Siva]

Fixes the python-django installation related issues in mthood private cloud. This completes the changes needed for the deployment in the private cloud.

Fixes the python-django installation related issues in mthood private cloud. This completes the changes needed for the deployment in the private cloud.

[Andy Doan (doanac)]

On 05/11/2014 07:07 AM, Parameswaran Sivatharman wrote:

> === modified file 'charms/precise/python-django/hooks/hooks.py'
> @@ -347,7 +348,6 @@
> remote_host = run("relation-get private-address")
> return remote_host
>
> -
> def get_unit_host():

check out the deletion of that line. I suspect that might cause a pep8
warning. functions need 2 blank lines between them.

> @@ -365,6 +365,15 @@
> with open(destination, 'w') as inject_file:
> inject_file.write(str(template))
>
> +def import_ci_ppa_keys():

ditto on the 2 blank lines

> + cmd = "apt-key adv --keyserver keyserver.ubuntu.com " \
> + "--recv-keys F7F9102D6A8DFC40"
> + try:
> + subprocess.check_call(cmd.split(' '))
> + except:
> + juju_log(MSG_ERROR, "Error importing repo key ")

That's a bit awkward to write a string and split. how about do this:

  def import_ci_ppa_keys():
     cmd = ['apt-key', 'adv', '--keyserver', 'keyserver.ubuntu.com',
            '--recv-keys', 'F7F9102D6A8DFC40']
     try:
         subprocess.check_call(cmd)
     except:
         juju_log(MSG_ERROR, "Error importing repo key ")

Also - what is "F7F9102D6A8DFC40"? Seems like this should probably come
from config.yaml?

You also might name the function "import_ppa_keys" so this isn't totally
tied to ci-airlines (I know it is totally tied to ci-airlines, but lets
not make the problem worse).

> @@ -510,7 +534,8 @@
> else:
> run('git clone %s %s' % (repos_url, vcs_clone_dir))
> elif vcs == 'bzr' or vcs == 'bazaar' or vcs == 'branch':
> - run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
> + run('http_proxy=%s https_proxy=%s bzr branch %s %s'
> + % (proxy_url, proxy_url, repos_url, vcs_clone_dir))

not sure - is this safe when no http_proxy was defined?

[lp:~psivaa/uci-engine/django-in-mthood]

414. By Para Siva on 2014-05-13

pep8 fixes on changes and getting gpg fingerprint from config

415. By Para Siva on 2014-05-15

Execute proxy commands only if proxy is defined

416. By Para Siva on 2014-05-15

pep8 fixes on changes and getting gpg fingerprint from config-pep8fix2

[Para Siva (psivaa)]

Thanks doanac for the comments. They all should be fixed now. Could you please verify?

One note on pep8 warnings is that hooks.py in this charm throws quite a lot of warnings about lines that haven't been changed as part of this MP. I haven't attempted to fix all of them but only the lines that changed due to time limitation.

[Andy Doan (doanac)]

tried out inline comments for this.

[lp:~psivaa/uci-engine/django-in-mthood]

417. By Para Siva on 2014-05-15

Taking the mthood specific vars from env

[Para Siva (psivaa)]

r417 should fix the inline comments for r416

[lp:~psivaa/uci-engine/django-in-mthood]

418. By Para Siva on 2014-06-06

proxy-url and ppa-gpg-key renaming to be consistent with restish and rabbitmq charms

[Para Siva (psivaa)]

Curious if I could get a nod for this to go in so that we don't lose this work. This part alone is functionally in working order, although there are other tweaks needed in some other components for mthood to be fully functional.
This is in a state where it can be picked up later if mthood needs development.

[Vincent Ladeuil (vila)]

Let's park that now indeed.

=== modified file 'charms/precise/python-django/config.yaml'
--- charms/precise/python-django/config.yaml	2014-03-10 22:25:00 +0000
+++ charms/precise/python-django/config.yaml	2014-06-06 10:05:35 +0000
@@ -22,6 +22,14 @@
         type: string
         default: ""
         description: The vcs url to checkout.
+    proxy-url:
+        type: string
+        default: ""
+        description: The http proxy url for private clouds.
+    ppa-gpg-key:
+        type: string
+        default: ""
+        description: GPG key fingerprint for the PPA
     repos_username:
         type: string
         default: ""
 
=== modified file 'charms/precise/python-django/hooks/hooks.py'
--- charms/precise/python-django/hooks/hooks.py	2014-03-10 22:25:00 +0000
+++ charms/precise/python-django/hooks/hooks.py	2014-06-06 10:05:35 +0000
@@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# vim: et ai ts=4 sw=4:
 
 import base64
 import json
@@ -10,12 +9,13 @@
 import subprocess
 import sys
 import time
+import textwrap
 from pwd import getpwnam
 from grp import getgrnam
 from random import choice
 
 CHARM_PACKAGES = ["python-pip", "python-jinja2", "mercurial", "git-core",
-                  "subversion", "bzr", "gettext"]
+                  "subversion", "bzr", "gettext", "tsocks"]
 
 INJECTED_WARNING = """
 #------------------------------------------------------------------------------
@@ -352,6 +352,7 @@
     this_host = run("unit-get private-address")
     return this_host.strip()
 
+
 def process_template(template_name, template_vars, destination):
     # --- exported service configuration file
     from jinja2 import Environment, FileSystemLoader
@@ -365,6 +366,16 @@
     with open(destination, 'w') as inject_file:
         inject_file.write(str(template))
 
+
+def import_ppa_keys():
+    cmd = ['apt-key', 'adv', '--keyserver', 'keyserver.ubuntu.com',
+           '--recv-keys', ppa_gpg_key]
+    try:
+        subprocess.check_call(cmd)
+    except:
+        juju_log(MSG_ERROR, "Error importing repo key")
+
+
 def configure_and_install(rel):
 
     def _import_key(id):
@@ -377,12 +388,16 @@
 
     if rel == 'distro':
         return apt_get_install("python-django")
+
     elif rel[:4] == "ppa:":
+        if ppa_gpg_key:
+            import_ppa_keys()
         src = rel
         subprocess.check_call(["add-apt-repository", "-y", src])
         apt_get_update()
 
         return apt_get_install("python-django")
+
     elif rel[:3] == "deb":
         l = len(rel.split('|'))
         if l ==  2:
@@ -465,6 +480,17 @@
     subprocess.check_call(cmd, shell=True)
 
 
+def setup_apt_proxy():
+    apt_conf= textwrap.dedent("""\
+        Acquire::http::proxy "%s";
+        Acquire::https::proxy "%s";
+        Acquire::HTTP::PROXY "%s";
+        Acquire::HTTPS::PROXY "%s";
+        """ %  (proxy_url, proxy_url, proxy_url, proxy_url))
+    with open('/etc/apt/apt.conf.d/95proxies', 'w') as f:
+        f.write(apt_conf)
+
+
 ###############################################################################
 # Hook functions
 ###############################################################################
@@ -475,6 +501,8 @@
             time.sleep(10)
         else:
             break
+    if proxy_url:
+        setup_apt_proxy()
 
     configure_and_install(django_version)
 
@@ -510,7 +538,11 @@
         else:
             run('git clone %s %s' % (repos_url, vcs_clone_dir))
     elif vcs == 'bzr' or vcs == 'bazaar' or vcs == 'branch':
-        run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
+        if proxy_url:
+            run('http_proxy=%s https_proxy=%s  bzr branch %s %s'
+                % (proxy_url, proxy_url, repos_url, vcs_clone_dir))
+        else:
+            run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
     elif vcs == 'svn' or vcs == 'subversion':
         run('svn co %s %s' % (repos_url, vcs_clone_dir))
     elif vcs == 'tarball':
@@ -810,6 +842,8 @@
 repos_username = config_data['repos_username']
 repos_password = config_data['repos_password']
 repos_branch = config_data['repos_branch']
+proxy_url = config_data['proxy-url']
+ppa_gpg_key = config_data['ppa-gpg-key']
 
 project_template_extension = config_data['project_template_extension']
 project_template_url = config_data['project_template_url']
 
=== modified file 'juju-deployer/deploy.py'
--- juju-deployer/deploy.py	2014-05-07 01:25:30 +0000
+++ juju-deployer/deploy.py	2014-06-06 10:05:35 +0000
@@ -56,7 +56,7 @@
           value))
 
 
-def check_environment():
+def check_environment(args):
     '''Cheetah requires environment variables for populating our templates'''
     cheetah_vars = {
         'OS_USERNAME': None,
@@ -77,6 +77,12 @@
         'CI_MASTER_PPA': 'ppa:ci-engineering-airline/ci-archive',
     }
     missing_required = []
+    if args.mthood_deployment:
+        os.environ['CI_PPA'] = 'ppa:canonical-ci-engineering/mthood-phase-0'
+        if not os.environ.get('CI_PRIVATE_HTTP_PROXY'):
+            missing_required.append('CI_PRIVATE_HTTP_PROXY')
+        if not os.environ.get('CI_PPA_GPG_KEY'):
+            missing_required.append('CI_PPA_GPG_KEY')
     for key in cheetah_vars:
         if key not in os.environ or len(os.environ[key]) == 0:
             default = cheetah_vars[key]
@@ -352,13 +358,16 @@
     parser.add_argument('--private-launchpad', action='store_true',
                         help='''Sets the environment to use proper Launchpad
                         instance.''', default=False)
+    parser.add_argument('--mthood-deployment', action='store_true',
+                        help='''Sets the environment to use mthood specific
+                        variables.''', default=False)
     return parser.parse_args(args)
 
 
 def main():
     args = _get_args()
 
-    check_environment()
+    check_environment(args)
 
     # You can configure the system to use private PPAs only or by providing
     # the option --private-ppas-only or by setting the environment variable
 
=== modified file 'juju-deployer/ppa-assigner.yaml.tmpl'
--- juju-deployer/ppa-assigner.yaml.tmpl	2014-03-10 22:25:00 +0000
+++ juju-deployer/ppa-assigner.yaml.tmpl	2014-06-06 10:05:35 +0000
@@ -9,6 +9,8 @@
                 repos_url: ${CI_BRANCH}
                 additional_distro_packages: python-oauth, python-launchpadlib, python-yaml, python-tastypie, python-django-south
                 django_version: ${CI_PPA}
+                proxy-url: ${CI_PRIVATE_HTTP_PROXY}
+                ppa-gpg-key: ${CI_PPA_GPG_KEY}
                 application_path: ppa-assigner/
                 django_settings: ppa_assigner.settings
                 django_south: True
 
=== modified file 'juju-deployer/ticket-system.yaml.tmpl'
--- juju-deployer/ticket-system.yaml.tmpl	2014-02-25 14:14:43 +0000
+++ juju-deployer/ticket-system.yaml.tmpl	2014-06-06 10:05:35 +0000
@@ -18,6 +18,8 @@
                 repos_url: ${CI_BRANCH}
                 additional_distro_packages: python-lazr.enum, python-yaml, python-tastypie, python-django-south
                 django_version: ${CI_PPA}
+                proxy-url: ${CI_PRIVATE_HTTP_PROXY}
+                ppa-gpg-key: ${CI_PPA_GPG_KEY}
                 unit-config: include-base64://configs/unit_config.yaml
                 application_path: ticket_system/
                 django_settings: ticket_system.settings