Merge lp:~psivaa/uci-engine/django-in-mthood into lp:uci-engine/mthood

Proposed by Para Siva
Status: Merged
Merged at revision: 417
Proposed branch: lp:~psivaa/uci-engine/django-in-mthood
Merge into: lp:uci-engine/mthood
Prerequisite: lp:~psivaa/uci-engine/lander-jenkins-plugin
Diff against target: 204 lines (+60/-5)
5 files modified
charms/precise/python-django/config.yaml (+8/-0)
charms/precise/python-django/hooks/hooks.py (+37/-3)
juju-deployer/deploy.py (+11/-2)
juju-deployer/ppa-assigner.yaml.tmpl (+2/-0)
juju-deployer/ticket-system.yaml.tmpl (+2/-0)
To merge this branch: bzr merge lp:~psivaa/uci-engine/django-in-mthood
Reviewer Review Type Date Requested Status
Canonical CI Engineering Pending
Review via email: mp+219118@code.launchpad.net

Commit message

Fixes the python-django installation related issues in mthood private cloud. This completes the changes needed for the deployment in the private cloud.

Description of the change

Fixes the python-django installation related issues in mthood private cloud. This completes the changes needed for the deployment in the private cloud.

To post a comment you must log in.
Revision history for this message
Andy Doan (doanac) wrote :

On 05/11/2014 07:07 AM, Parameswaran Sivatharman wrote:

> === modified file 'charms/precise/python-django/hooks/hooks.py'
> @@ -347,7 +348,6 @@
> remote_host = run("relation-get private-address")
> return remote_host
>
> -
> def get_unit_host():

check out the deletion of that line. I suspect that might cause a pep8
warning. functions need 2 blank lines between them.

> @@ -365,6 +365,15 @@
> with open(destination, 'w') as inject_file:
> inject_file.write(str(template))
>
> +def import_ci_ppa_keys():

ditto on the 2 blank lines

> + cmd = "apt-key adv --keyserver keyserver.ubuntu.com " \
> + "--recv-keys F7F9102D6A8DFC40"
> + try:
> + subprocess.check_call(cmd.split(' '))
> + except:
> + juju_log(MSG_ERROR, "Error importing repo key ")

That's a bit awkward to write a string and split. how about do this:

  def import_ci_ppa_keys():
     cmd = ['apt-key', 'adv', '--keyserver', 'keyserver.ubuntu.com',
            '--recv-keys', 'F7F9102D6A8DFC40']
     try:
         subprocess.check_call(cmd)
     except:
         juju_log(MSG_ERROR, "Error importing repo key ")

Also - what is "F7F9102D6A8DFC40"? Seems like this should probably come
from config.yaml?

You also might name the function "import_ppa_keys" so this isn't totally
tied to ci-airlines (I know it is totally tied to ci-airlines, but lets
not make the problem worse).

> @@ -510,7 +534,8 @@
> else:
> run('git clone %s %s' % (repos_url, vcs_clone_dir))
> elif vcs == 'bzr' or vcs == 'bazaar' or vcs == 'branch':
> - run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
> + run('http_proxy=%s https_proxy=%s bzr branch %s %s'
> + % (proxy_url, proxy_url, repos_url, vcs_clone_dir))

not sure - is this safe when no http_proxy was defined?

414. By Para Siva

pep8 fixes on changes and getting gpg fingerprint from config

415. By Para Siva

Execute proxy commands only if proxy is defined

416. By Para Siva

pep8 fixes on changes and getting gpg fingerprint from config-pep8fix2

Revision history for this message
Para Siva (psivaa) wrote :

Thanks doanac for the comments. They all should be fixed now. Could you please verify?

One note on pep8 warnings is that hooks.py in this charm throws quite a lot of warnings about lines that haven't been changed as part of this MP. I haven't attempted to fix all of them but only the lines that changed due to time limitation.

Revision history for this message
Andy Doan (doanac) wrote :

tried out inline comments for this.

417. By Para Siva

Taking the mthood specific vars from env

Revision history for this message
Para Siva (psivaa) wrote :

r417 should fix the inline comments for r416

418. By Para Siva

proxy-url and ppa-gpg-key renaming to be consistent with restish and rabbitmq charms

Revision history for this message
Para Siva (psivaa) wrote :

Curious if I could get a nod for this to go in so that we don't lose this work. This part alone is functionally in working order, although there are other tweaks needed in some other components for mthood to be fully functional.
This is in a state where it can be picked up later if mthood needs development.

Revision history for this message
Vincent Ladeuil (vila) wrote :

Let's park that now indeed.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'charms/precise/python-django/config.yaml'
2--- charms/precise/python-django/config.yaml 2014-03-10 22:25:00 +0000
3+++ charms/precise/python-django/config.yaml 2014-06-06 10:05:35 +0000
4@@ -22,6 +22,14 @@
5 type: string
6 default: ""
7 description: The vcs url to checkout.
8+ proxy-url:
9+ type: string
10+ default: ""
11+ description: The http proxy url for private clouds.
12+ ppa-gpg-key:
13+ type: string
14+ default: ""
15+ description: GPG key fingerprint for the PPA
16 repos_username:
17 type: string
18 default: ""
19
20=== modified file 'charms/precise/python-django/hooks/hooks.py'
21--- charms/precise/python-django/hooks/hooks.py 2014-03-10 22:25:00 +0000
22+++ charms/precise/python-django/hooks/hooks.py 2014-06-06 10:05:35 +0000
23@@ -1,5 +1,4 @@
24 #!/usr/bin/env python
25-# vim: et ai ts=4 sw=4:
26
27 import base64
28 import json
29@@ -10,12 +9,13 @@
30 import subprocess
31 import sys
32 import time
33+import textwrap
34 from pwd import getpwnam
35 from grp import getgrnam
36 from random import choice
37
38 CHARM_PACKAGES = ["python-pip", "python-jinja2", "mercurial", "git-core",
39- "subversion", "bzr", "gettext"]
40+ "subversion", "bzr", "gettext", "tsocks"]
41
42 INJECTED_WARNING = """
43 #------------------------------------------------------------------------------
44@@ -352,6 +352,7 @@
45 this_host = run("unit-get private-address")
46 return this_host.strip()
47
48+
49 def process_template(template_name, template_vars, destination):
50 # --- exported service configuration file
51 from jinja2 import Environment, FileSystemLoader
52@@ -365,6 +366,16 @@
53 with open(destination, 'w') as inject_file:
54 inject_file.write(str(template))
55
56+
57+def import_ppa_keys():
58+ cmd = ['apt-key', 'adv', '--keyserver', 'keyserver.ubuntu.com',
59+ '--recv-keys', ppa_gpg_key]
60+ try:
61+ subprocess.check_call(cmd)
62+ except:
63+ juju_log(MSG_ERROR, "Error importing repo key")
64+
65+
66 def configure_and_install(rel):
67
68 def _import_key(id):
69@@ -377,12 +388,16 @@
70
71 if rel == 'distro':
72 return apt_get_install("python-django")
73+
74 elif rel[:4] == "ppa:":
75+ if ppa_gpg_key:
76+ import_ppa_keys()
77 src = rel
78 subprocess.check_call(["add-apt-repository", "-y", src])
79 apt_get_update()
80
81 return apt_get_install("python-django")
82+
83 elif rel[:3] == "deb":
84 l = len(rel.split('|'))
85 if l == 2:
86@@ -465,6 +480,17 @@
87 subprocess.check_call(cmd, shell=True)
88
89
90+def setup_apt_proxy():
91+ apt_conf= textwrap.dedent("""\
92+ Acquire::http::proxy "%s";
93+ Acquire::https::proxy "%s";
94+ Acquire::HTTP::PROXY "%s";
95+ Acquire::HTTPS::PROXY "%s";
96+ """ % (proxy_url, proxy_url, proxy_url, proxy_url))
97+ with open('/etc/apt/apt.conf.d/95proxies', 'w') as f:
98+ f.write(apt_conf)
99+
100+
101 ###############################################################################
102 # Hook functions
103 ###############################################################################
104@@ -475,6 +501,8 @@
105 time.sleep(10)
106 else:
107 break
108+ if proxy_url:
109+ setup_apt_proxy()
110
111 configure_and_install(django_version)
112
113@@ -510,7 +538,11 @@
114 else:
115 run('git clone %s %s' % (repos_url, vcs_clone_dir))
116 elif vcs == 'bzr' or vcs == 'bazaar' or vcs == 'branch':
117- run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
118+ if proxy_url:
119+ run('http_proxy=%s https_proxy=%s bzr branch %s %s'
120+ % (proxy_url, proxy_url, repos_url, vcs_clone_dir))
121+ else:
122+ run('bzr branch %s %s' % (repos_url, vcs_clone_dir))
123 elif vcs == 'svn' or vcs == 'subversion':
124 run('svn co %s %s' % (repos_url, vcs_clone_dir))
125 elif vcs == 'tarball':
126@@ -810,6 +842,8 @@
127 repos_username = config_data['repos_username']
128 repos_password = config_data['repos_password']
129 repos_branch = config_data['repos_branch']
130+proxy_url = config_data['proxy-url']
131+ppa_gpg_key = config_data['ppa-gpg-key']
132
133 project_template_extension = config_data['project_template_extension']
134 project_template_url = config_data['project_template_url']
135
136=== modified file 'juju-deployer/deploy.py'
137--- juju-deployer/deploy.py 2014-05-07 01:25:30 +0000
138+++ juju-deployer/deploy.py 2014-06-06 10:05:35 +0000
139@@ -56,7 +56,7 @@
140 value))
141
142
143-def check_environment():
144+def check_environment(args):
145 '''Cheetah requires environment variables for populating our templates'''
146 cheetah_vars = {
147 'OS_USERNAME': None,
148@@ -77,6 +77,12 @@
149 'CI_MASTER_PPA': 'ppa:ci-engineering-airline/ci-archive',
150 }
151 missing_required = []
152+ if args.mthood_deployment:
153+ os.environ['CI_PPA'] = 'ppa:canonical-ci-engineering/mthood-phase-0'
154+ if not os.environ.get('CI_PRIVATE_HTTP_PROXY'):
155+ missing_required.append('CI_PRIVATE_HTTP_PROXY')
156+ if not os.environ.get('CI_PPA_GPG_KEY'):
157+ missing_required.append('CI_PPA_GPG_KEY')
158 for key in cheetah_vars:
159 if key not in os.environ or len(os.environ[key]) == 0:
160 default = cheetah_vars[key]
161@@ -352,13 +358,16 @@
162 parser.add_argument('--private-launchpad', action='store_true',
163 help='''Sets the environment to use proper Launchpad
164 instance.''', default=False)
165+ parser.add_argument('--mthood-deployment', action='store_true',
166+ help='''Sets the environment to use mthood specific
167+ variables.''', default=False)
168 return parser.parse_args(args)
169
170
171 def main():
172 args = _get_args()
173
174- check_environment()
175+ check_environment(args)
176
177 # You can configure the system to use private PPAs only or by providing
178 # the option --private-ppas-only or by setting the environment variable
179
180=== modified file 'juju-deployer/ppa-assigner.yaml.tmpl'
181--- juju-deployer/ppa-assigner.yaml.tmpl 2014-03-10 22:25:00 +0000
182+++ juju-deployer/ppa-assigner.yaml.tmpl 2014-06-06 10:05:35 +0000
183@@ -9,6 +9,8 @@
184 repos_url: ${CI_BRANCH}
185 additional_distro_packages: python-oauth, python-launchpadlib, python-yaml, python-tastypie, python-django-south
186 django_version: ${CI_PPA}
187+ proxy-url: ${CI_PRIVATE_HTTP_PROXY}
188+ ppa-gpg-key: ${CI_PPA_GPG_KEY}
189 application_path: ppa-assigner/
190 django_settings: ppa_assigner.settings
191 django_south: True
192
193=== modified file 'juju-deployer/ticket-system.yaml.tmpl'
194--- juju-deployer/ticket-system.yaml.tmpl 2014-02-25 14:14:43 +0000
195+++ juju-deployer/ticket-system.yaml.tmpl 2014-06-06 10:05:35 +0000
196@@ -18,6 +18,8 @@
197 repos_url: ${CI_BRANCH}
198 additional_distro_packages: python-lazr.enum, python-yaml, python-tastypie, python-django-south
199 django_version: ${CI_PPA}
200+ proxy-url: ${CI_PRIVATE_HTTP_PROXY}
201+ ppa-gpg-key: ${CI_PPA_GPG_KEY}
202 unit-config: include-base64://configs/unit_config.yaml
203 application_path: ticket_system/
204 django_settings: ticket_system.settings

Subscribers

People subscribed via source and target branches