Ubuntu CI Engine

Merge lp:~ursinha/uci-engine/private-lp-mthood into lp:uci-engine/mthood

  1. private-lp-mthood
  2. Merge into mthood
Proposed by Ursula Junque
Status: Merged
Approved by: Ursula Junque
Approved revision: 442
Merged at revision: 412
Proposed branch: lp:~ursinha/uci-engine/private-lp-mthood
Merge into: lp:uci-engine/mthood
Diff against target: 587 lines (+228/-117)
14 files modified
branch-source-builder/bsbuilder/upload_package.py (+19/-0)
branch-source-builder/cupstream2distro/launchpadmanager.py (+21/-29)
branch-source-builder/cupstream2distro/packagemanager.py (+1/-1)
branch-source-builder/cupstream2distro/settings.py (+1/-5)
branch-source-builder/watch_ppa.py (+5/-5)
ci-utils/ci_utils/launchpad.py (+30/-2)
create_private_lp_creds.py (+57/-0)
image-builder/imagebuilder/cloud_image.py (+2/-4)
image-builder/imagebuilder/tests/test_modify_cloud_image.py (+1/-1)
juju-deployer/configs/unit_config.yaml.tmpl (+6/-2)
juju-deployer/deploy.py (+17/-1)
ppa-assigner/ppa_assigner/launchpad.py (+62/-61)
ppa-assigner/ppa_assigner/models.py (+3/-3)
ppa-assigner/ppa_assigner/settings.py (+3/-3)
To merge this branch: bzr merge lp:~ursinha/uci-engine/private-lp-mthood
Reviewer Review Type Date Requested Status
Francis Ginther Approve
Andy Doan (community) Approve
Review via email: mp+218538@code.launchpad.net

Commit message

This branch contains all changes required for mthood to access a private lp instance.

Description of the change

This branch contains all changes required for mthood to access a private lp instance.

The original lp:~ursinha/uci-engine/private-lp branch was created from lp:uci-engine, not lp:uci-engine/mthood, contanining a bit more changes than only private lp related ones. In this branch I applied only relevant changes on top of the mthood branch.

To post a comment you must log in.
Revision history for this message
Andy Doan (doanac) wrote :

> === modified file 'branch-source-builder/cupstream2distro/launchpadmanager.py'

> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))

I don't think we need sys.path.append stuff in our code anymore.

> === modified file 'branch-source-builder/cupstream2distro/packagemanager.py'

> - cmd = ["dput", ppa,
> + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> "{}_{}_source.changes".format(source, version_for_source_file)]

I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
like we should either make a constant for that, or (and I don't know the
code structure/flow, have pass this value down to the function via
upload_package.py

> === modified file 'branch-source-builder/cupstream2distro/settings.py'

> +PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)

This is beyond the scope of this MP, but a tip: We now have a
unit_config module: ci_utils.unit_config that was written to handle this
stuff for all our services.

> === modified file 'branch-source-builder/watch_ppa.py'

def get_launchpad_log(url):
> '''Retrieves the build log from launchpad (thanks to celso).'''
> if not url:
> return None
> - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> - lp = launchpadmanager.get_launchpad()
> + api_url = launchpad.get_lp_api_url(api_version="devel")
> + lp = launchpad.lp_login()
> return lp._browser.get(api_url)

looks like the "url" parameter to this function is now ignored. Should
we remove it and refactor the caller of it?

> === added file 'create_private_lp_creds.py'

> +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))

again - don't think you need to alter sys.path anymore.

> +web_root ="https://mthoodapi.lacinonac.com/"

I don't understand the details well enough, but should this value come
from our unit_config rather than being hard-coded? Actually it looks
like deploy.py might be setting this in the unit_config?

Revision history for this message
Francis Ginther (fginther) wrote :

Approve, let's get this in!

review: Approve
Revision history for this message
Francis Ginther (fginther) wrote :

Oops, I see failure in my tarmac.sh run. Let me dig into it first.

Revision history for this message
Francis Ginther (fginther) wrote :

I found the failure:
======================================================================
FAIL: test_pep8_conformance (ci_utils.tests.test_style.TestPep8)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.YUCsZ3kgPK/local/lib/python2.7/site-packages/ucitests-0.1.2-py2.7.egg/ucitests/styles.py", line 84, in test_pep8_conformance
    '\n'.join(self.report._msgs))
AssertionError: /tmp/engine/private-lp-mthood/ci-utils/ci_utils/launchpad.py:49:80: E501 line too long (81 > 79 characters)

Can you fix this pep8 error before we merge this please?

review: Needs Information
lp:~ursinha/uci-engine/private-lp-mthood updated
440. By Ursula Junque

Fixed pep8 error on launchpad.py

Revision history for this message
Ursula Junque (ursinha) wrote :

Thanks for reviewing, Andy.

> > === modified file 'branch-source-
> builder/cupstream2distro/launchpadmanager.py'
>
> > +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
>
> I don't think we need sys.path.append stuff in our code anymore.

This branch is on top of mthood one, that is a tad far from lp:uci-engine/trunk. Are these things you mention available since when? I think this should be changed in case we're merging this with trunk, otherwise I'm not sure it's going to work? (same apply for other comments regarding things we have now and didn't have when this was created).

>
> > === modified file 'branch-source-builder/cupstream2distro/packagemanager.py'
>
> > - cmd = ["dput", ppa,
> > + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> > "{}_{}_source.changes".format(source, version_for_source_file)]
>
> I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
> like we should either make a constant for that, or (and I don't know the
> code structure/flow, have pass this value down to the function via
> upload_package.py

I'll leave this for Francis to comment, he implemented that part and knows the constraints better than I do.

>
> > === modified file 'branch-source-builder/cupstream2distro/settings.py'
>
> > +PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)
>
> This is beyond the scope of this MP, but a tip: We now have a
> unit_config module: ci_utils.unit_config that was written to handle this
> stuff for all our services.
>
> > === modified file 'branch-source-builder/watch_ppa.py'
>
> def get_launchpad_log(url):
> > '''Retrieves the build log from launchpad (thanks to celso).'''
> > if not url:
> > return None
> > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > - lp = launchpadmanager.get_launchpad()
> > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > + lp = launchpad.lp_login()
> > return lp._browser.get(api_url)
>
> looks like the "url" parameter to this function is now ignored. Should
> we remove it and refactor the caller of it?
>
> > === added file 'create_private_lp_creds.py'
>
> > +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
>
> again - don't think you need to alter sys.path anymore.
>
> > +web_root ="https://mthoodapi.lacinonac.com/"
>
> I don't understand the details well enough, but should this value come
> from our unit_config rather than being hard-coded? Actually it looks
> like deploy.py might be setting this in the unit_config?

This script is only a helper to create the credentials to the private lp instance, I think in trunk the original create_lp_creds.py doesn't even exist anymore. I agree this could be done using values on unit_config, but as this is something that should go away soon I didn't bother much.

Revision history for this message
Andy Doan (doanac) wrote :

On 05/07/2014 12:12 PM, Ursula Junque wrote:
> Thanks for reviewing, Andy.
>
>>> === modified file 'branch-source-
>> builder/cupstream2distro/launchpadmanager.py'
>>
>>> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
>>
>> I don't think we need sys.path.append stuff in our code anymore.
>
> This branch is on top of mthood one, that is a tad far from lp:uci-engine/trunk. Are these things you mention available since when? I think this should be changed in case we're merging this with trunk, otherwise I'm not sure it's going to work? (same apply for other comments regarding things we have now and didn't have when this was created).

yes. it was added here:

<http://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/mthood/revision/320>

>>> +web_root ="https://mthoodapi.lacinonac.com/"
>>
>> I don't understand the details well enough, but should this value come
>> from our unit_config rather than being hard-coded? Actually it looks
>> like deploy.py might be setting this in the unit_config?
>
> This script is only a helper to create the credentials to the private lp instance, I think in trunk the original create_lp_creds.py doesn't even exist anymore. I agree this could be done using values on unit_config, but as this is something that should go away soon I didn't bother much.

ah - okay. makes sense then.

Revision history for this message
Ursula Junque (ursinha) wrote :

Oops, I missed this.

>
> > === modified file 'branch-source-builder/watch_ppa.py'
>
> def get_launchpad_log(url):
> > '''Retrieves the build log from launchpad (thanks to celso).'''
> > if not url:
> > return None
> > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > - lp = launchpadmanager.get_launchpad()
> > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > + lp = launchpad.lp_login()
> > return lp._browser.get(api_url)
>
> looks like the "url" parameter to this function is now ignored. Should
> we remove it and refactor the caller of it?

You are not only right, I think you found a bug. :) I'm fixing it right now.

>
> > === added file 'create_private_lp_creds.py'
>
> > +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
>
> again - don't think you need to alter sys.path anymore.
>
> > +web_root ="https://mthoodapi.lacinonac.com/"
>
> I don't understand the details well enough, but should this value come
> from our unit_config rather than being hard-coded? Actually it looks
> like deploy.py might be setting this in the unit_config?

lp:~ursinha/uci-engine/private-lp-mthood updated
441. By Ursula Junque

Fixing bug spotted by andy during review (thanks andy), build log correct url is used now

442. By Ursula Junque

Removing sys.path changes as per andy's review, it's not required anymore

Revision history for this message
Ursula Junque (ursinha) wrote :

> I found the failure:
> ======================================================================
> FAIL: test_pep8_conformance (ci_utils.tests.test_style.TestPep8)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/tmp/tmp.YUCsZ3kgPK/local/lib/python2.7/site-
> packages/ucitests-0.1.2-py2.7.egg/ucitests/styles.py", line 84, in
> test_pep8_conformance
> '\n'.join(self.report._msgs))
> AssertionError: /tmp/engine/private-lp-mthood/ci-
> utils/ci_utils/launchpad.py:49:80: E501 line too long (81 > 79 characters)
>
> Can you fix this pep8 error before we merge this please?

Fixed in 440.

Revision history for this message
Ursula Junque (ursinha) wrote :

> On 05/07/2014 12:12 PM, Ursula Junque wrote:
> > Thanks for reviewing, Andy.
> >
> >>> === modified file 'branch-source-
> >> builder/cupstream2distro/launchpadmanager.py'
> >>
> >>> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
> >>
> >> I don't think we need sys.path.append stuff in our code anymore.
> >
> > This branch is on top of mthood one, that is a tad far from lp:uci-
> engine/trunk. Are these things you mention available since when? I think this
> should be changed in case we're merging this with trunk, otherwise I'm not
> sure it's going to work? (same apply for other comments regarding things we
> have now and didn't have when this was created).
>
> yes. it was added here:
>
> <http://bazaar.launchpad.net/~canonical-ci-engineering/uci-
> engine/mthood/revision/320>

Fixed in 442.

Revision history for this message
Ursula Junque (ursinha) wrote :

> > > === modified file 'branch-source-builder/watch_ppa.py'
> >
> > def get_launchpad_log(url):
> > > '''Retrieves the build log from launchpad (thanks to celso).'''
> > > if not url:
> > > return None
> > > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > > - lp = launchpadmanager.get_launchpad()
> > > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > > + lp = launchpad.lp_login()
> > > return lp._browser.get(api_url)
> >
> > looks like the "url" parameter to this function is now ignored. Should
> > we remove it and refactor the caller of it?
>
> You are not only right, I think you found a bug. :) I'm fixing it right now.

Fixed in 441.

Revision history for this message
Francis Ginther (fginther) wrote :

> > > === modified file 'branch-source-
> builder/cupstream2distro/packagemanager.py'
> >
> > > - cmd = ["dput", ppa,
> > > + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> > > "{}_{}_source.changes".format(source,
> version_for_source_file)]
> >
> > I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
> > like we should either make a constant for that, or (and I don't know the
> > code structure/flow, have pass this value down to the function via
> > upload_package.py
>
> I'll leave this for Francis to comment, he implemented that part and knows the
> constraints better than I do.

A constant is more appropriate and will make it easier if we need to relocate the file. And when thinking about the proper location, /srv/$service_name/etc/dput.cf is probably a better fit.

I'll work on a fix for this.

Revision history for this message
Andy Doan (doanac) wrote :

LGTM. i'll defer to fginther on the rest.

review: Approve
Revision history for this message
Francis Ginther (fginther) wrote :

I'd like to work on the dput.cf location and use of a constant change in a follow on MP. Approving this as all of the other comments have been addressed.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'branch-source-builder/bsbuilder/upload_package.py'
--- branch-source-builder/bsbuilder/upload_package.py 2014-03-14 10:37:28 +0000
+++ branch-source-builder/bsbuilder/upload_package.py 2014-05-07 18:17:44 +0000
@@ -23,9 +23,11 @@
23import shutil23import shutil
24import sys24import sys
25import tempfile25import tempfile
26import textwrap
26import urllib227import urllib2
2728
28from cupstream2distro import packagemanager29from cupstream2distro import packagemanager
30from ci_utils import unit_config
2931
3032
31def parse_arguments():33def parse_arguments():
@@ -53,6 +55,22 @@
53 return directory55 return directory
5456
5557
58def _create_dput_config():
59 dput_cf = '/tmp/dput.cf'
60 if not os.path.exists(dput_cf):
61 with open(dput_cf, 'w') as f:
62 os.fchmod(f.fileno(), 0700)
63 template = textwrap.dedent('''\
64 [ppa]
65 login = anonymous
66 fqdn = {}
67 method = ftp
68 allow_unsigned_uploads = 0
69 incoming = ~%(ppa)s/ubuntu
70 ''')
71 f.write(template.format(unit_config.get('launchpad_upload_url')))
72
73
56def get_url_contents(url):74def get_url_contents(url):
57 try:75 try:
58 req = urllib2.Request(url)76 req = urllib2.Request(url)
@@ -121,6 +139,7 @@
121139
122def create_upload_list(subtickets):140def create_upload_list(subtickets):
123 '''Parses the subtickets into the upload list.'''141 '''Parses the subtickets into the upload list.'''
142 _create_dput_config()
124 upload_list = get_source_files(subtickets)143 upload_list = get_source_files(subtickets)
125 parse_source_files(upload_list)144 parse_source_files(upload_list)
126 return upload_list145 return upload_list
127146
=== modified file 'branch-source-builder/cupstream2distro/launchpadmanager.py'
--- branch-source-builder/cupstream2distro/launchpadmanager.py 2014-02-19 16:34:46 +0000
+++ branch-source-builder/cupstream2distro/launchpadmanager.py 2014-05-07 18:17:44 +0000
@@ -23,35 +23,24 @@
23import lazr23import lazr
24import logging24import logging
25import os25import os
26launchpad = None26import sys
2727
28from .settings import ARCHS_TO_EVENTUALLY_IGNORE, ARCHS_TO_UNCONDITIONALLY_IGNORE, VIRTUALIZED_PPA_ARCH, CRED_FILE_PATH, COMMON_LAUNCHPAD_CACHE_DIR28from ci_utils import launchpad
29
30from .settings import (ARCHS_TO_EVENTUALLY_IGNORE,
31 ARCHS_TO_UNCONDITIONALLY_IGNORE, VIRTUALIZED_PPA_ARCH,
32 CRED_FILE_PATH, COMMON_LAUNCHPAD_CACHE_DIR,
33 PRIVATE_LAUNCHPAD)
34my_launchpad = None
2935
3036
31def get_launchpad(use_staging=False, use_cred_file=os.path.expanduser(CRED_FILE_PATH)):37def get_launchpad(use_staging=False, use_cred_file=os.path.expanduser(CRED_FILE_PATH)):
32 '''Get THE Launchpad'''38 '''Get THE Launchpad'''
33 global launchpad39 global my_launchpad
34 if not launchpad:40 if not my_launchpad:
35 if use_staging:41 my_launchpad = launchpad.lp_login()
36 server = 'staging'42
37 else:43 return my_launchpad
38 server = 'production'
39
40 launchpadlib_dir = COMMON_LAUNCHPAD_CACHE_DIR
41 if not os.path.exists(launchpadlib_dir):
42 os.makdedirs(launchpadlib_dir)
43
44 if use_cred_file:
45 launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
46 version='devel', # devel because copyPackage is only available there
47 credentials_file=use_cred_file,
48 launchpadlib_dir=launchpadlib_dir)
49 else:
50 launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
51 version='devel', # devel because copyPackage is only available there
52 launchpadlib_dir=launchpadlib_dir)
53
54 return launchpad
5544
5645
57def get_ubuntu():46def get_ubuntu():
@@ -110,14 +99,17 @@
110def get_available_and_all_archs(series, ppa=None):99def get_available_and_all_archs(series, ppa=None):
111 '''Return a set of available archs, and the all arch'''100 '''Return a set of available archs, and the all arch'''
112 available_arch = set()101 available_arch = set()
113 if ppa and ppa.require_virtualized:102 if PRIVATE_LAUNCHPAD:
103 # Hack to workaround having only archindep arch enabled for PPAs in
104 # the private lp instance.
105 archindep = series.nominatedarchindep.architecture_tag
106 available_arch = set([archindep])
107 arch_all_arch = archindep
108 elif ppa and ppa.require_virtualized:
114 available_arch = set(VIRTUALIZED_PPA_ARCH)109 available_arch = set(VIRTUALIZED_PPA_ARCH)
115 arch_all_arch = VIRTUALIZED_PPA_ARCH[0]110 arch_all_arch = VIRTUALIZED_PPA_ARCH[0]
116 else:111 else:
117 for arch in series.architectures:112 for arch in series.architectures:
118 # HACK: filters armel as it's still seen as available on raring: https://launchpad.net/bugs/1077257
119 if arch.architecture_tag == "armel":
120 continue
121 available_arch.add(arch.architecture_tag)113 available_arch.add(arch.architecture_tag)
122 if arch.is_nominated_arch_indep:114 if arch.is_nominated_arch_indep:
123 arch_all_arch = arch.architecture_tag115 arch_all_arch = arch.architecture_tag
124116
=== modified file 'branch-source-builder/cupstream2distro/packagemanager.py'
--- branch-source-builder/cupstream2distro/packagemanager.py 2014-02-19 16:34:46 +0000
+++ branch-source-builder/cupstream2distro/packagemanager.py 2014-05-07 18:17:44 +0000
@@ -471,7 +471,7 @@
471 here = os.getcwd()471 here = os.getcwd()
472 os.chdir(source_directory)472 os.chdir(source_directory)
473 version_for_source_file = version.split(':')[-1]473 version_for_source_file = version.split(':')[-1]
474 cmd = ["dput", ppa,474 cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
475 "{}_{}_source.changes".format(source, version_for_source_file)]475 "{}_{}_source.changes".format(source, version_for_source_file)]
476 if subprocess.call(cmd) != 0:476 if subprocess.call(cmd) != 0:
477 if here:477 if here:
478478
=== modified file 'branch-source-builder/cupstream2distro/settings.py'
--- branch-source-builder/cupstream2distro/settings.py 2014-03-14 10:37:28 +0000
+++ branch-source-builder/cupstream2distro/settings.py 2014-05-07 18:17:44 +0000
@@ -50,14 +50,9 @@
50 print('Unable to use unit_config(%s), defaulting values' % path)50 print('Unable to use unit_config(%s), defaulting values' % path)
51 return config51 return config
52_cfg = _unit_config()52_cfg = _unit_config()
53LAUNCHPAD_PPA_OWNER = _cfg.get('launchpad_ppa_owner', None)
54LAUNCHPAD_USER = _cfg.get('launchpad_user', None)
55LAUNCHPAD_API_BASE = _cfg.get(
56 'launchpad_api_base', 'https://api.launchpad.net/1.0')
57OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)53OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)
58OAUTH_TOKEN = _cfg.get('oauth_token', None)54OAUTH_TOKEN = _cfg.get('oauth_token', None)
59OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)55OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)
60OAUTH_REALM = _cfg.get('oauth_realm', 'https://api.launchpad.net/')
6156
62if not os.path.exists(CRED_FILE_PATH):57if not os.path.exists(CRED_FILE_PATH):
63 with open(CRED_FILE_PATH, 'w') as f:58 with open(CRED_FILE_PATH, 'w') as f:
@@ -73,6 +68,7 @@
73BOT_KEY = "B879A3E9"68BOT_KEY = "B879A3E9"
7469
75# selected arch for building arch:all packages70# selected arch for building arch:all packages
71PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)
76VIRTUALIZED_PPA_ARCH = ["i386", "amd64"]72VIRTUALIZED_PPA_ARCH = ["i386", "amd64"]
77# an arch we will ignore for publication if latest published version in dest doesn't build it73# an arch we will ignore for publication if latest published version in dest doesn't build it
78ARCHS_TO_EVENTUALLY_IGNORE = set(['powerpc', 'arm64', 'ppc64el'])74ARCHS_TO_EVENTUALLY_IGNORE = set(['powerpc', 'arm64', 'ppc64el'])
7975
=== modified file 'branch-source-builder/watch_ppa.py'
--- branch-source-builder/watch_ppa.py 2014-03-14 10:37:28 +0000
+++ branch-source-builder/watch_ppa.py 2014-05-07 18:17:44 +0000
@@ -33,8 +33,7 @@
33from cupstream2distro.settings import (33from cupstream2distro.settings import (
34 TIME_BETWEEN_PPA_CHECKS, TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH)34 TIME_BETWEEN_PPA_CHECKS, TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH)
3535
36sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))36from ci_utils import dump_stack, launchpad
37from ci_utils import dump_stack
38from ci_utils.ticket_states import (37from ci_utils.ticket_states import (
39 SubTicketWorkflowStep,38 SubTicketWorkflowStep,
40 SubTicketWorkflowStepStatus39 SubTicketWorkflowStepStatus
@@ -99,9 +98,10 @@
99 '''Retrieves the build log from launchpad (thanks to celso).'''98 '''Retrieves the build log from launchpad (thanks to celso).'''
100 if not url:99 if not url:
101 return None100 return None
102 api_url = url.replace('/launchpad.net/', '/api.launchpad.net/devel/')101 api_url = launchpad.get_lp_api_url(api_version="devel")
103 lp = launchpadmanager.get_launchpad()102 url = "{}/{}".format(api_url, "/".join(build.build_log_url.split("/")[3:]))
104 return lp._browser.get(api_url)103 lp = launchpad.lp_login()
104 return lp._browser.get(url)
105105
106106
107def get_build_logs(datastore, ppa, package):107def get_build_logs(datastore, ppa, package):
108108
=== modified file 'ci-utils/ci_utils/launchpad.py'
--- ci-utils/ci_utils/launchpad.py 2014-03-14 03:43:36 +0000
+++ ci-utils/ci_utils/launchpad.py 2014-05-07 18:17:44 +0000
@@ -44,9 +44,37 @@
44def lp_login():44def lp_login():
45 lib = '/tmp/launchpadlib'45 lib = '/tmp/launchpadlib'
46 store = _get_credential_store()46 store = _get_credential_store()
47 return Launchpad.login_with('ci-engine', 'production',47 return Launchpad.login_with('ci-engine', service_root=get_lp_root_url(),
48 credential_store=store, launchpadlib_dir=lib)48 credential_store=store, launchpadlib_dir=lib,
49 version=unit_config.get(
50 'launchpad_api_version'))
4951
5052
51def get_lp_user_name():53def get_lp_user_name():
52 return unit_config.get('launchpad_user')54 return unit_config.get('launchpad_user')
55
56
57def get_lp_ppa_owner():
58 return unit_config.get('launchpad_ppa_owner')
59
60
61def get_lp_root_url():
62 return unit_config.get('launchpad_api_root_url')
63
64
65def get_lp_api_url(api_version=None):
66 if not api_version:
67 api_version = unit_config.get('launchpad_api_version')
68 return "{}/{}/".format(get_lp_root_url(), api_version)
69
70
71def get_private_ppa_sources_list_url(lp, ppa):
72 if lp:
73 user = lp.people[get_lp_user_name()]
74 return user.getArchiveSubscriptionURL(archive=ppa)
75 return None
76
77
78def get_ppa_sources_list_url(team, repo_name):
79 api_url = get_lp_root_url().replace("https", "http").replace("api", "ppa")
80 return "{}/{}/{}/ubuntu".format(api_url, team, repo_name)
5381
=== added file 'create_private_lp_creds.py'
--- create_private_lp_creds.py 1970-01-01 00:00:00 +0000
+++ create_private_lp_creds.py 2014-05-07 18:17:44 +0000
@@ -0,0 +1,57 @@
1#!/usr/bin/env python
2
3import os
4import subprocess
5import sys
6
7import launchpadlib.uris
8from launchpadlib.launchpad import Launchpad
9from lazr.restfulclient import errors
10sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
11from ci_utils import dump_stack
12
13web_root = "https://mthoodapi.lacinonac.com/"
14
15old = launchpadlib.uris.web_root_for_service_root
16
17
18def web_root_for_service_root(service_root):
19 if service_root == 'https://mthoodapi.lacinonac.com/':
20 return 'https://mthood.lacinonac.com/'
21 return old(service_root)
22
23launchpadlib.uris.web_root_for_service_root = web_root_for_service_root
24
25creds_path = os.path.abspath('lp_creds.txt')
26
27
28def create():
29 try:
30 lp = Launchpad.login_with('ci-airline', consumer_name='ci-airline',
31 service_root=web_root, version='devel')
32 except errors.HTTPError as e:
33 if e.response['status'] == '403':
34 sys.stderr.write('Ok, not creating credentials then\n')
35 exit(1)
36 raise
37
38 creds = lp.credentials
39 creds.save_to_path(creds_path)
40 sys.stderr.write('Credentials created in {}\n'.format(creds_path))
41
42 print('export CI_OAUTH_CONSUMER_KEY="{}"'.format(creds.consumer.key))
43 print('export CI_OAUTH_TOKEN={}'.format(creds.access_token.key))
44 print('export CI_OAUTH_TOKEN_SECRET={}'.format(creds.access_token.secret))
45 print('export CI_LAUNCHPAD_USER={}'.format(lp.me.name))
46 # This should be a team that owns the ppa build pool
47 print('export CI_LAUNCHPAD_PPA_OWNER={}'.format(lp.me.name))
48 return 0
49
50
51def main(args):
52 create()
53
54
55if __name__ == '__main__':
56 dump_stack.install_stack_dump_signal()
57 sys.exit(main(sys.argv[1:]))
058
=== modified file 'image-builder/imagebuilder/cloud_image.py'
--- image-builder/imagebuilder/cloud_image.py 2014-03-17 16:08:17 +0000
+++ image-builder/imagebuilder/cloud_image.py 2014-05-07 18:17:44 +0000
@@ -65,11 +65,9 @@
65 ppa = lp.people[team].getPPAByName(name=repo)65 ppa = lp.people[team].getPPAByName(name=repo)
66 keys[ppa.name] = ppa.signing_key_fingerprint66 keys[ppa.name] = ppa.signing_key_fingerprint
67 if ppa.private:67 if ppa.private:
68 #lp.me won't work, you have to get yourself by person68 url = launchpad.get_private_ppa_sources_list_url(lp, ppa)
69 user = lp.people[launchpad.get_lp_user_name()]
70 url = user.getArchiveSubscriptionURL(archive=ppa)
71 else:69 else:
72 url = 'http://ppa.launchpad.net/{}/{}/ubuntu'.format(team, repo)70 url = launchpad.get_ppa_sources_list_url(team, repo)
73 aptlines.append('deb {} {} main'.format(url, series))71 aptlines.append('deb {} {} main'.format(url, series))
74 aptlines.append('deb-src {} {} main'.format(url, series))72 aptlines.append('deb-src {} {} main'.format(url, series))
75 return aptlines, keys73 return aptlines, keys
7674
=== modified file 'image-builder/imagebuilder/tests/test_modify_cloud_image.py'
--- image-builder/imagebuilder/tests/test_modify_cloud_image.py 2014-03-15 16:46:11 +0000
+++ image-builder/imagebuilder/tests/test_modify_cloud_image.py 2014-05-07 18:17:44 +0000
@@ -28,6 +28,7 @@
28 @mock.patch('imagebuilder.cloud_image.launchpad')28 @mock.patch('imagebuilder.cloud_image.launchpad')
29 def test_parse_ppas(self, launchpad):29 def test_parse_ppas(self, launchpad):
30 launchpad.get_lp_user_name.return_value = 'foo'30 launchpad.get_lp_user_name.return_value = 'foo'
31 launchpad.get_private_ppa_sources_list_url.return_value = 'foo_url'
3132
32 lp = mock.MagicMock()33 lp = mock.MagicMock()
33 ppa = mock.Mock()34 ppa = mock.Mock()
@@ -35,7 +36,6 @@
35 ppa.signing_key_fingerprint = 'foofp'36 ppa.signing_key_fingerprint = 'foofp'
36 ppa.private = True37 ppa.private = True
37 lp.people['foo'].getPPAByName.return_value = ppa38 lp.people['foo'].getPPAByName.return_value = ppa
38 lp.people['foo'].getArchiveSubscriptionURL.return_value = 'foo_url'
3939
40 series = 'precise'40 series = 'precise'
41 deblines, keys = cloud_image._parse_ppas(lp, ['ppa:/foo/bar'], series)41 deblines, keys = cloud_image._parse_ppas(lp, ['ppa:/foo/bar'], series)
4242
=== modified file 'juju-deployer/configs/unit_config.yaml.tmpl'
--- juju-deployer/configs/unit_config.yaml.tmpl 2014-03-15 23:15:38 +0000
+++ juju-deployer/configs/unit_config.yaml.tmpl 2014-05-07 18:17:44 +0000
@@ -8,13 +8,17 @@
8# for launchpad apis8# for launchpad apis
9# and see: https://help.launchpad.net/API/SigningRequests9# and see: https://help.launchpad.net/API/SigningRequests
10# and see ../../create_lp_creds.py10# and see ../../create_lp_creds.py
11launchpad_api_base: https://api.launchpad.net/1.011launchpad_api_root_url: $CI_LAUNCHPAD_API_ROOT_URL
12launchpad_api_version: $CI_LAUNCHPAD_API_VERSION
12launchpad_user: $CI_LAUNCHPAD_USER13launchpad_user: $CI_LAUNCHPAD_USER
13launchpad_ppa_owner: $CI_LAUNCHPAD_PPA_OWNER14launchpad_ppa_owner: $CI_LAUNCHPAD_PPA_OWNER
14oauth_consumer_key: $CI_OAUTH_CONSUMER_KEY15launchpad_upload_url: $CI_LAUNCHPAD_UPLOAD_URL
16oauth_consumer_key: "$CI_OAUTH_CONSUMER_KEY"
15oauth_token: $CI_OAUTH_TOKEN17oauth_token: $CI_OAUTH_TOKEN
16oauth_token_secret: $CI_OAUTH_TOKEN_SECRET18oauth_token_secret: $CI_OAUTH_TOKEN_SECRET
1719
20private_launchpad: $CI_PRIVATE_LAUNCHPAD
21
18ppa_pattern: "ci-pool-\\d+"22ppa_pattern: "ci-pool-\\d+"
1923
20private_ppas_only: $CI_PRIVATE_PPAS_ONLY24private_ppas_only: $CI_PRIVATE_PPAS_ONLY
2125
=== modified file 'juju-deployer/deploy.py'
--- juju-deployer/deploy.py 2014-03-14 23:53:04 +0000
+++ juju-deployer/deploy.py 2014-05-07 18:17:44 +0000
@@ -65,7 +65,11 @@
65 'CI_PAYLOAD_URL': 'n/a',65 'CI_PAYLOAD_URL': 'n/a',
66 'CI_LAUNCHPAD_USER': None,66 'CI_LAUNCHPAD_USER': None,
67 'CI_LAUNCHPAD_PPA_OWNER': None,67 'CI_LAUNCHPAD_PPA_OWNER': None,
68 'CI_LAUNCHPAD_API_ROOT_URL': 'https://api.launchpad.net/',
69 'CI_LAUNCHPAD_API_VERSION': 'devel',
70 'CI_LAUNCHPAD_UPLOAD_URL': 'ppa.launchpad.net',
68 'CI_PRIVATE_PPAS_ONLY': '0',71 'CI_PRIVATE_PPAS_ONLY': '0',
72 'CI_PRIVATE_LAUNCHPAD': 'False',
69 'CI_OAUTH_CONSUMER_KEY': None,73 'CI_OAUTH_CONSUMER_KEY': None,
70 'CI_OAUTH_TOKEN': None,74 'CI_OAUTH_TOKEN': None,
71 'CI_OAUTH_TOKEN_SECRET': None,75 'CI_OAUTH_TOKEN_SECRET': None,
@@ -345,7 +349,9 @@
345 when building packages.''')349 when building packages.''')
346 parser.add_argument('specific', metavar='<config.yaml>', nargs='*',350 parser.add_argument('specific', metavar='<config.yaml>', nargs='*',
347 help='Use a specific list of juju-deployer configs.')351 help='Use a specific list of juju-deployer configs.')
348352 parser.add_argument('--private-launchpad', action='store_true',
353 help='''Sets the environment to use proper Launchpad
354 instance.''', default=False)
349 return parser.parse_args(args)355 return parser.parse_args(args)
350356
351357
@@ -361,6 +367,16 @@
361 # variable value.367 # variable value.
362 check_and_set_ppa_privacy(args.private_ppas_only)368 check_and_set_ppa_privacy(args.private_ppas_only)
363369
370 if args.private_launchpad:
371 os.environ['CI_LAUNCHPAD_API_ROOT_URL'] = (
372 'https://mthoodapi.lacinonac.com/')
373 os.environ['CI_LAUNCHPAD_UPLOAD_URL'] = (
374 'mthood.lacinonac.com')
375 os.environ['CI_PRIVATE_LAUNCHPAD'] = 'True'
376 print('Private Launchpad instance: ENABLED')
377 else:
378 print('Private Launchpad instance: DISABLED')
379
364 check_bootstrapped()380 check_bootstrapped()
365 check_dependencies()381 check_dependencies()
366382
367383
=== modified file 'ppa-assigner/ppa_assigner/launchpad.py'
--- ppa-assigner/ppa_assigner/launchpad.py 2014-03-14 10:37:28 +0000
+++ ppa-assigner/ppa_assigner/launchpad.py 2014-05-07 18:17:44 +0000
@@ -22,65 +22,6 @@
22from oauth import oauth22from oauth import oauth
2323
2424
25def lp_collection(url):
26 '''iterate through each item returned in an lp_collection'''
27 resp = _lp_get_authenticated(url)
28 resp = resp.read()
29 data = json.loads(resp)
30
31 for d in data['entries']:
32 yield d
33
34 if 'next_collection_link' in data:
35 for d in lp_collection(data['next_collection_link']):
36 yield d
37
38
39def _oauth_headers():
40 '''oauth header required for a launchpad api post
41
42 https://help.launchpad.net/API/SigningRequests
43 '''
44 values = [
45 'OAuth realm="%s"' % settings.OAUTH_REALM,
46 'oauth_version="1.0"',
47 'oauth_consumer_key="%s"' % settings.OAUTH_CONSUMER_KEY,
48 'oauth_token="%s"' % settings.OAUTH_TOKEN,
49 'oauth_signature_method="PLAINTEXT"',
50 'oauth_signature="%%26%s"' % settings.OAUTH_TOKEN_SECRET,
51 'oauth_timestamp="%d"' % int(oauth.time.time()),
52 'oauth_nonce="%s"' % oauth.generate_nonce(),
53 ]
54 return {
55 'Authorization': ', '.join(values),
56 'Accept': 'application/json',
57 }
58
59
60def _lp_post(url, data):
61 headers = _oauth_headers()
62 headers['Content-Type'] = 'application/x-www-form-urlencoded'
63 req = urllib2.Request(url, urllib.urlencode(data), headers)
64 return urllib2.urlopen(req)
65
66
67def _lp_get_authenticated(url):
68 '''An authenticated GET to obtain also private information.'''
69 req = urllib2.Request(url, headers=_oauth_headers())
70 return urllib2.urlopen(req)
71
72
73def get_publishing_history(ppa):
74 '''A generator for the source and binary publish history of a ppa.'''
75 url = settings.LAUNCHPAD_API_BASE + '/~%s/+archive/%s/?ws.op='
76 url = url % (settings.LAUNCHPAD_PPA_OWNER, ppa.split('/')[-1])
77
78 for x in lp_collection(url + 'getPublishedSources'):
79 yield x
80 for x in lp_collection(url + 'getPublishedBinaries'):
81 yield x
82
83
84def request_ppa_clean(ppa):25def request_ppa_clean(ppa):
85 '''call requestDeletion operation on ppa's source and binary pub history'''26 '''call requestDeletion operation on ppa's source and binary pub history'''
86 for x in get_publishing_history(ppa):27 for x in get_publishing_history(ppa):
@@ -116,8 +57,9 @@
11657
117def _sync_source(ppa, source_name, version, pocket, archive):58def _sync_source(ppa, source_name, version, pocket, archive):
118 '''Used for populating a PPA so we can run integration tests.'''59 '''Used for populating a PPA so we can run integration tests.'''
119 url = 'https://api.launchpad.net/1.0/~%s/+archive/%s'60 url = '{}/~{}/+archive/{}'.format(settings.LAUNCHPAD_API_URL,
120 url = url % (settings.LAUNCHPAD_PPA_OWNER, ppa.split('/')[-1])61 settings.LAUNCHPAD_PPA_OWNER,
62 ppa.split('/')[-1])
121 data = {63 data = {
122 'ws.op': 'syncSource',64 'ws.op': 'syncSource',
123 'from_archive': archive,65 'from_archive': archive,
@@ -129,3 +71,62 @@
129 c = resp.getcode()71 c = resp.getcode()
130 if c != 200:72 if c != 200:
131 raise urllib2.HTTPError(url, c, resp.read(), resp.header.headers, None)73 raise urllib2.HTTPError(url, c, resp.read(), resp.header.headers, None)
74
75
76def get_publishing_history(ppa):
77 '''A generator for the source and binary publish history of a ppa.'''
78 url = '{}/~{}/+archive/{}/?ws.op='.format(settings.LAUNCHPAD_API_URL,
79 settings.LAUNCHPAD_PPA_OWNER,
80 ppa.split('/')[-1])
81 for x in _lp_collection(url + 'getPublishedSources'):
82 yield x
83 for x in _lp_collection(url + 'getPublishedBinaries'):
84 yield x
85
86
87def _lp_collection(url):
88 '''iterate through each item returned in an lp_collection'''
89 resp = _lp_get_authenticated(url)
90 resp = resp.read()
91 data = json.loads(resp)
92
93 for d in data['entries']:
94 yield d
95
96 if 'next_collection_link' in data:
97 for d in _lp_collection(data['next_collection_link']):
98 yield d
99
100
101def _oauth_headers():
102 '''oauth header required for a launchpad api post
103
104 https://help.launchpad.net/API/SigningRequests
105 '''
106 values = [
107 'OAuth realm="%s"' % settings.OAUTH_REALM,
108 'oauth_version="1.0"',
109 'oauth_consumer_key="%s"' % settings.OAUTH_CONSUMER_KEY,
110 'oauth_token="%s"' % settings.OAUTH_TOKEN,
111 'oauth_signature_method="PLAINTEXT"',
112 'oauth_signature="%%26%s"' % settings.OAUTH_TOKEN_SECRET,
113 'oauth_timestamp="%d"' % int(oauth.time.time()),
114 'oauth_nonce="%s"' % oauth.generate_nonce(),
115 ]
116 return {
117 'Authorization': ', '.join(values),
118 'Accept': 'application/json',
119 }
120
121
122def _lp_post(url, data):
123 headers = _oauth_headers()
124 headers['Content-Type'] = 'application/x-www-form-urlencoded'
125 req = urllib2.Request(url, urllib.urlencode(data), headers)
126 return urllib2.urlopen(req)
127
128
129def _lp_get_authenticated(url):
130 '''An authenticated GET to obtain also private information.'''
131 req = urllib2.Request(url, headers=_oauth_headers())
132 return urllib2.urlopen(req)
132133
=== modified file 'ppa-assigner/ppa_assigner/models.py'
--- ppa-assigner/ppa_assigner/models.py 2014-03-14 23:53:04 +0000
+++ ppa-assigner/ppa_assigner/models.py 2014-05-07 18:17:44 +0000
@@ -83,13 +83,13 @@
83 @staticmethod83 @staticmethod
84 def populate_from_launchpad():84 def populate_from_launchpad():
85 '''Get list of PPAs from launchpad and ensure they exist here.'''85 '''Get list of PPAs from launchpad and ensure they exist here.'''
86 url = settings.LAUNCHPAD_API_BASE + '/~%s/ppas'86 url = '{}/~{}/ppas'.format(settings.LAUNCHPAD_API_URL,
87 url = url % settings.LAUNCHPAD_PPA_OWNER87 settings.LAUNCHPAD_PPA_OWNER)
88 logging.info('populating ppa model from: %s', url)88 logging.info('populating ppa model from: %s', url)
89 try:89 try:
90 ppas = {}90 ppas = {}
91 pat = re.compile(settings.PPA_PATTERN)91 pat = re.compile(settings.PPA_PATTERN)
92 for ppa in launchpad.lp_collection(url):92 for ppa in launchpad._lp_collection(url):
93 if pat.match(ppa['name']):93 if pat.match(ppa['name']):
94 ppa_name = 'ppa:{}/{}'.format(settings.LAUNCHPAD_PPA_OWNER,94 ppa_name = 'ppa:{}/{}'.format(settings.LAUNCHPAD_PPA_OWNER,
95 ppa['name'])95 ppa['name'])
9696
=== modified file 'ppa-assigner/ppa_assigner/settings.py'
--- ppa-assigner/ppa_assigner/settings.py 2014-03-14 23:53:04 +0000
+++ ppa-assigner/ppa_assigner/settings.py 2014-05-07 18:17:44 +0000
@@ -44,12 +44,12 @@
4444
45LAUNCHPAD_USER = _cfg.get('launchpad_user', None)45LAUNCHPAD_USER = _cfg.get('launchpad_user', None)
46LAUNCHPAD_PPA_OWNER = _cfg.get('launchpad_ppa_owner', None)46LAUNCHPAD_PPA_OWNER = _cfg.get('launchpad_ppa_owner', None)
47LAUNCHPAD_API_BASE = _cfg.get(47LAUNCHPAD_API_URL = "{}/{}".format(_cfg.get('launchpad_api_root_url', None),
48 'launchpad_api_base', 'https://api.launchpad.net/1.0')48 _cfg.get('launchpad_api_version'))
49OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)49OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)
50OAUTH_TOKEN = _cfg.get('oauth_token', None)50OAUTH_TOKEN = _cfg.get('oauth_token', None)
51OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)51OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)
52OAUTH_REALM = _cfg.get('oauth_realm', 'https://api.launchpad.net/')52OAUTH_REALM = _cfg.get('launchpad_api_root_url')
53PPA_PATTERN = _cfg.get('ppa_pattern', r'ci-pool-\d+')53PPA_PATTERN = _cfg.get('ppa_pattern', r'ci-pool-\d+')
54PRIVATE_PPAS_ONLY = _cfg.get('private_ppas_only', None)54PRIVATE_PPAS_ONLY = _cfg.get('private_ppas_only', None)
5555

Subscribers

People subscribed via source and target branches