Merge lp:~ursinha/uci-engine/private-lp-mthood into lp:uci-engine/mthood

[Ursula Junque]

This branch contains all changes required for mthood to access a private lp instance.

This branch contains all changes required for mthood to access a private lp instance.

The original lp:~ursinha/uci-engine/private-lp branch was created from lp:uci-engine, not lp:uci-engine/mthood, contanining a bit more changes than only private lp related ones. In this branch I applied only relevant changes on top of the mthood branch.

[Andy Doan (doanac)]

> === modified file 'branch-source-builder/cupstream2distro/launchpadmanager.py'

> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))

I don't think we need sys.path.append stuff in our code anymore.

> === modified file 'branch-source-builder/cupstream2distro/packagemanager.py'

> - cmd = ["dput", ppa,
> + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> "{}_{}_source.changes".format(source, version_for_source_file)]

I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
like we should either make a constant for that, or (and I don't know the
code structure/flow, have pass this value down to the function via
upload_package.py

> === modified file 'branch-source-builder/cupstream2distro/settings.py'

> +PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)

This is beyond the scope of this MP, but a tip: We now have a
unit_config module: ci_utils.unit_config that was written to handle this
stuff for all our services.

> === modified file 'branch-source-builder/watch_ppa.py'

def get_launchpad_log(url):
> '''Retrieves the build log from launchpad (thanks to celso).'''
> if not url:
> return None
> - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> - lp = launchpadmanager.get_launchpad()
> + api_url = launchpad.get_lp_api_url(api_version="devel")
> + lp = launchpad.lp_login()
> return lp._browser.get(api_url)

looks like the "url" parameter to this function is now ignored. Should
we remove it and refactor the caller of it?

> === added file 'create_private_lp_creds.py'

> +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))

again - don't think you need to alter sys.path anymore.

> +web_root ="https://mthoodapi.lacinonac.com/"

I don't understand the details well enough, but should this value come
from our unit_config rather than being hard-coded? Actually it looks
like deploy.py might be setting this in the unit_config?

[Francis Ginther (fginther)]

Approve, let's get this in!

review: Approve

[Francis Ginther (fginther)]

Oops, I see failure in my tarmac.sh run. Let me dig into it first.

[Francis Ginther (fginther)]

I found the failure:
======================================================================
FAIL: test_pep8_conformance (ci_utils.tests.test_style.TestPep8)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmp.YUCsZ3kgPK/local/lib/python2.7/site-packages/ucitests-0.1.2-py2.7.egg/ucitests/styles.py", line 84, in test_pep8_conformance
    '\n'.join(self.report._msgs))
AssertionError: /tmp/engine/private-lp-mthood/ci-utils/ci_utils/launchpad.py:49:80: E501 line too long (81 > 79 characters)

Can you fix this pep8 error before we merge this please?

review: Needs Information

[lp:~ursinha/uci-engine/private-lp-mthood]

440. By Ursula Junque on 2014-05-07

Fixed pep8 error on launchpad.py

[Ursula Junque (ursinha)]

Thanks for reviewing, Andy.

> > === modified file 'branch-source-
> builder/cupstream2distro/launchpadmanager.py'
>
> > +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
>
> I don't think we need sys.path.append stuff in our code anymore.

This branch is on top of mthood one, that is a tad far from lp:uci-engine/trunk. Are these things you mention available since when? I think this should be changed in case we're merging this with trunk, otherwise I'm not sure it's going to work? (same apply for other comments regarding things we have now and didn't have when this was created).

>
> > === modified file 'branch-source-builder/cupstream2distro/packagemanager.py'
>
> > - cmd = ["dput", ppa,
> > + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> > "{}_{}_source.changes".format(source, version_for_source_file)]
>
> I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
> like we should either make a constant for that, or (and I don't know the
> code structure/flow, have pass this value down to the function via
> upload_package.py

I'll leave this for Francis to comment, he implemented that part and knows the constraints better than I do.

>
> > === modified file 'branch-source-builder/cupstream2distro/settings.py'
>
> > +PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)
>
> This is beyond the scope of this MP, but a tip: We now have a
> unit_config module: ci_utils.unit_config that was written to handle this
> stuff for all our services.
>
> > === modified file 'branch-source-builder/watch_ppa.py'
>
> def get_launchpad_log(url):
> > '''Retrieves the build log from launchpad (thanks to celso).'''
> > if not url:
> > return None
> > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > - lp = launchpadmanager.get_launchpad()
> > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > + lp = launchpad.lp_login()
> > return lp._browser.get(api_url)
>
> looks like the "url" parameter to this function is now ignored. Should
> we remove it and refactor the caller of it?
>
> > === added file 'create_private_lp_creds.py'
>
> > +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
>
> again - don't think you need to alter sys.path anymore.
>
> > +web_root ="https://mthoodapi.lacinonac.com/"
>
> I don't understand the details well enough, but should this value come
> from our unit_config rather than being hard-coded? Actually it looks
> like deploy.py might be setting this in the unit_config?

This script is only a helper to create the credentials to the private lp instance, I think in trunk the original create_lp_creds.py doesn't even exist anymore. I agree this could be done using values on unit_config, but as this is something that should go away soon I didn't bother much.

[Andy Doan (doanac)]

On 05/07/2014 12:12 PM, Ursula Junque wrote:
> Thanks for reviewing, Andy.
>
>>> === modified file 'branch-source-
>> builder/cupstream2distro/launchpadmanager.py'
>>
>>> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
>>
>> I don't think we need sys.path.append stuff in our code anymore.
>
> This branch is on top of mthood one, that is a tad far from lp:uci-engine/trunk. Are these things you mention available since when? I think this should be changed in case we're merging this with trunk, otherwise I'm not sure it's going to work? (same apply for other comments regarding things we have now and didn't have when this was created).

yes. it was added here:

<http://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/mthood/revision/320>

>>> +web_root ="https://mthoodapi.lacinonac.com/"
>>
>> I don't understand the details well enough, but should this value come
>> from our unit_config rather than being hard-coded? Actually it looks
>> like deploy.py might be setting this in the unit_config?
>
> This script is only a helper to create the credentials to the private lp instance, I think in trunk the original create_lp_creds.py doesn't even exist anymore. I agree this could be done using values on unit_config, but as this is something that should go away soon I didn't bother much.

ah - okay. makes sense then.

[Ursula Junque (ursinha)]

Oops, I missed this.

>
> > === modified file 'branch-source-builder/watch_ppa.py'
>
> def get_launchpad_log(url):
> > '''Retrieves the build log from launchpad (thanks to celso).'''
> > if not url:
> > return None
> > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > - lp = launchpadmanager.get_launchpad()
> > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > + lp = launchpad.lp_login()
> > return lp._browser.get(api_url)
>
> looks like the "url" parameter to this function is now ignored. Should
> we remove it and refactor the caller of it?

You are not only right, I think you found a bug. :) I'm fixing it right now.

>
> > === added file 'create_private_lp_creds.py'
>
> > +sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
>
> again - don't think you need to alter sys.path anymore.
>
> > +web_root ="https://mthoodapi.lacinonac.com/"
>
> I don't understand the details well enough, but should this value come
> from our unit_config rather than being hard-coded? Actually it looks
> like deploy.py might be setting this in the unit_config?

[lp:~ursinha/uci-engine/private-lp-mthood]

441. By Ursula Junque on 2014-05-07

Fixing bug spotted by andy during review (thanks andy), build log correct url is used now

442. By Ursula Junque on 2014-05-07

Removing sys.path changes as per andy's review, it's not required anymore

[Ursula Junque (ursinha)]

> I found the failure:
> ======================================================================
> FAIL: test_pep8_conformance (ci_utils.tests.test_style.TestPep8)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/tmp/tmp.YUCsZ3kgPK/local/lib/python2.7/site-
> packages/ucitests-0.1.2-py2.7.egg/ucitests/styles.py", line 84, in
> test_pep8_conformance
> '\n'.join(self.report._msgs))
> AssertionError: /tmp/engine/private-lp-mthood/ci-
> utils/ci_utils/launchpad.py:49:80: E501 line too long (81 > 79 characters)
>
> Can you fix this pep8 error before we merge this please?

Fixed in 440.

[Ursula Junque (ursinha)]

> On 05/07/2014 12:12 PM, Ursula Junque wrote:
> > Thanks for reviewing, Andy.
> >
> >>> === modified file 'branch-source-
> >> builder/cupstream2distro/launchpadmanager.py'
> >>
> >>> +sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
> >>
> >> I don't think we need sys.path.append stuff in our code anymore.
> >
> > This branch is on top of mthood one, that is a tad far from lp:uci-
> engine/trunk. Are these things you mention available since when? I think this
> should be changed in case we're merging this with trunk, otherwise I'm not
> sure it's going to work? (same apply for other comments regarding things we
> have now and didn't have when this was created).
>
> yes. it was added here:
>
> <http://bazaar.launchpad.net/~canonical-ci-engineering/uci-
> engine/mthood/revision/320>

Fixed in 442.

[Ursula Junque (ursinha)]

> > > === modified file 'branch-source-builder/watch_ppa.py'
> >
> > def get_launchpad_log(url):
> > > '''Retrieves the build log from launchpad (thanks to celso).'''
> > > if not url:
> > > return None
> > > - api_url = url.replace('/launchpad.net/','/api.launchpad.net/devel/')
> > > - lp = launchpadmanager.get_launchpad()
> > > + api_url = launchpad.get_lp_api_url(api_version="devel")
> > > + lp = launchpad.lp_login()
> > > return lp._browser.get(api_url)
> >
> > looks like the "url" parameter to this function is now ignored. Should
> > we remove it and refactor the caller of it?
>
> You are not only right, I think you found a bug. :) I'm fixing it right now.

Fixed in 441.

[Francis Ginther (fginther)]

> > > === modified file 'branch-source-
> builder/cupstream2distro/packagemanager.py'
> >
> > > - cmd = ["dput", ppa,
> > > + cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
> > > "{}_{}_source.changes".format(source,
> version_for_source_file)]
> >
> > I'm being a little over-protective, but as per: "/tmp/dput.cf" - Seems
> > like we should either make a constant for that, or (and I don't know the
> > code structure/flow, have pass this value down to the function via
> > upload_package.py
>
> I'll leave this for Francis to comment, he implemented that part and knows the
> constraints better than I do.

A constant is more appropriate and will make it easier if we need to relocate the file. And when thinking about the proper location, /srv/$service_name/etc/dput.cf is probably a better fit.

I'll work on a fix for this.

[Andy Doan (doanac)]

LGTM. i'll defer to fginther on the rest.

review: Approve

[Francis Ginther (fginther)]

I'd like to work on the dput.cf location and use of a constant change in a follow on MP. Approving this as all of the other comments have been addressed.

review: Approve

=== modified file 'branch-source-builder/bsbuilder/upload_package.py'
--- branch-source-builder/bsbuilder/upload_package.py	2014-03-14 10:37:28 +0000
+++ branch-source-builder/bsbuilder/upload_package.py	2014-05-07 18:17:44 +0000
@@ -23,9 +23,11 @@
 import shutil
 import sys
 import tempfile
+import textwrap
 import urllib2
 
 from cupstream2distro import packagemanager
+from ci_utils import unit_config
 
 
 def parse_arguments():
@@ -53,6 +55,22 @@
     return directory
 
 
+def _create_dput_config():
+    dput_cf = '/tmp/dput.cf'
+    if not os.path.exists(dput_cf):
+        with open(dput_cf, 'w') as f:
+            os.fchmod(f.fileno(), 0700)
+            template = textwrap.dedent('''\
+            [ppa]
+            login                  = anonymous
+            fqdn                   = {}
+            method                 = ftp
+            allow_unsigned_uploads = 0
+            incoming               = ~%(ppa)s/ubuntu
+            ''')
+            f.write(template.format(unit_config.get('launchpad_upload_url')))
+
+
 def get_url_contents(url):
     try:
         req = urllib2.Request(url)
@@ -121,6 +139,7 @@
 
 def create_upload_list(subtickets):
     '''Parses the subtickets into the upload list.'''
+    _create_dput_config()
     upload_list = get_source_files(subtickets)
     parse_source_files(upload_list)
     return upload_list

=== modified file 'branch-source-builder/cupstream2distro/launchpadmanager.py'
--- branch-source-builder/cupstream2distro/launchpadmanager.py	2014-02-19 16:34:46 +0000
+++ branch-source-builder/cupstream2distro/launchpadmanager.py	2014-05-07 18:17:44 +0000
@@ -23,35 +23,24 @@
 import lazr
 import logging
 import os
-launchpad = None
-
-from .settings import ARCHS_TO_EVENTUALLY_IGNORE, ARCHS_TO_UNCONDITIONALLY_IGNORE, VIRTUALIZED_PPA_ARCH, CRED_FILE_PATH, COMMON_LAUNCHPAD_CACHE_DIR
+import sys
+
+from ci_utils import launchpad
+
+from .settings import (ARCHS_TO_EVENTUALLY_IGNORE,
+                       ARCHS_TO_UNCONDITIONALLY_IGNORE, VIRTUALIZED_PPA_ARCH,
+                       CRED_FILE_PATH, COMMON_LAUNCHPAD_CACHE_DIR,
+                       PRIVATE_LAUNCHPAD)
+my_launchpad = None
 
 
 def get_launchpad(use_staging=False, use_cred_file=os.path.expanduser(CRED_FILE_PATH)):
     '''Get THE Launchpad'''
-    global launchpad
-    if not launchpad:
-        if use_staging:
-            server = 'staging'
-        else:
-            server = 'production'
-
-        launchpadlib_dir = COMMON_LAUNCHPAD_CACHE_DIR
-        if not os.path.exists(launchpadlib_dir):
-            os.makdedirs(launchpadlib_dir)
-
-        if use_cred_file:
-            launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
-                                             version='devel',  # devel because copyPackage is only available there
-                                             credentials_file=use_cred_file,
-                                             launchpadlib_dir=launchpadlib_dir)
-        else:
-            launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
-                                             version='devel',  # devel because copyPackage is only available there
-                                             launchpadlib_dir=launchpadlib_dir)
-
-    return launchpad
+    global my_launchpad
+    if not my_launchpad:
+        my_launchpad = launchpad.lp_login()
+
+    return my_launchpad
 
 
 def get_ubuntu():
@@ -110,14 +99,17 @@
 def get_available_and_all_archs(series, ppa=None):
     '''Return a set of available archs, and the all arch'''
     available_arch = set()
-    if ppa and ppa.require_virtualized:
+    if PRIVATE_LAUNCHPAD:
+        # Hack to workaround having only archindep arch enabled for PPAs in
+        # the private lp instance.
+        archindep = series.nominatedarchindep.architecture_tag
+        available_arch = set([archindep])
+        arch_all_arch = archindep
+    elif ppa and ppa.require_virtualized:
         available_arch = set(VIRTUALIZED_PPA_ARCH)
         arch_all_arch = VIRTUALIZED_PPA_ARCH[0]
     else:
         for arch in series.architectures:
-            # HACK: filters armel as it's still seen as available on raring: https://launchpad.net/bugs/1077257
-            if arch.architecture_tag == "armel":
-                continue
             available_arch.add(arch.architecture_tag)
             if arch.is_nominated_arch_indep:
                 arch_all_arch = arch.architecture_tag

=== modified file 'branch-source-builder/cupstream2distro/packagemanager.py'
--- branch-source-builder/cupstream2distro/packagemanager.py	2014-02-19 16:34:46 +0000
+++ branch-source-builder/cupstream2distro/packagemanager.py	2014-05-07 18:17:44 +0000
@@ -471,7 +471,7 @@
         here = os.getcwd()
         os.chdir(source_directory)
     version_for_source_file = version.split(':')[-1]
-    cmd = ["dput", ppa,
+    cmd = ["dput", "-u", "-c", "/tmp/dput.cf", ppa,
            "{}_{}_source.changes".format(source, version_for_source_file)]
     if subprocess.call(cmd) != 0:
         if here:

=== modified file 'branch-source-builder/cupstream2distro/settings.py'
--- branch-source-builder/cupstream2distro/settings.py	2014-03-14 10:37:28 +0000
+++ branch-source-builder/cupstream2distro/settings.py	2014-05-07 18:17:44 +0000
@@ -50,14 +50,9 @@
         print('Unable to use unit_config(%s), defaulting values' % path)
     return config
 _cfg = _unit_config()
-LAUNCHPAD_PPA_OWNER = _cfg.get('launchpad_ppa_owner', None)
-LAUNCHPAD_USER = _cfg.get('launchpad_user', None)
-LAUNCHPAD_API_BASE = _cfg.get(
-    'launchpad_api_base', 'https://api.launchpad.net/1.0')
 OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)
 OAUTH_TOKEN = _cfg.get('oauth_token', None)
 OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)
-OAUTH_REALM = _cfg.get('oauth_realm', 'https://api.launchpad.net/')
 
 if not os.path.exists(CRED_FILE_PATH):
     with open(CRED_FILE_PATH, 'w') as f:
@@ -73,6 +68,7 @@
 BOT_KEY = "B879A3E9"
 
 # selected arch for building arch:all packages
+PRIVATE_LAUNCHPAD = _cfg.get('private_launchpad', None)
 VIRTUALIZED_PPA_ARCH = ["i386", "amd64"]
 # an arch we will ignore for publication if latest published version in dest doesn't build it
 ARCHS_TO_EVENTUALLY_IGNORE = set(['powerpc', 'arm64', 'ppc64el'])

=== modified file 'branch-source-builder/watch_ppa.py'
--- branch-source-builder/watch_ppa.py	2014-03-14 10:37:28 +0000
+++ branch-source-builder/watch_ppa.py	2014-05-07 18:17:44 +0000
@@ -33,8 +33,7 @@
 from cupstream2distro.settings import (
     TIME_BETWEEN_PPA_CHECKS, TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH)
 
-sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
-from ci_utils import dump_stack
+from ci_utils import dump_stack, launchpad
 from ci_utils.ticket_states import (
     SubTicketWorkflowStep,
     SubTicketWorkflowStepStatus
@@ -99,9 +98,10 @@
     '''Retrieves the build log from launchpad (thanks to celso).'''
     if not url:
         return None
-    api_url = url.replace('/launchpad.net/', '/api.launchpad.net/devel/')
-    lp = launchpadmanager.get_launchpad()
-    return lp._browser.get(api_url)
+    api_url = launchpad.get_lp_api_url(api_version="devel")
+    url = "{}/{}".format(api_url, "/".join(build.build_log_url.split("/")[3:]))
+    lp = launchpad.lp_login()
+    return lp._browser.get(url)
 
 
 def get_build_logs(datastore, ppa, package):

=== modified file 'ci-utils/ci_utils/launchpad.py'
--- ci-utils/ci_utils/launchpad.py	2014-03-14 03:43:36 +0000
+++ ci-utils/ci_utils/launchpad.py	2014-05-07 18:17:44 +0000
@@ -44,9 +44,37 @@
 def lp_login():
     lib = '/tmp/launchpadlib'
     store = _get_credential_store()
-    return Launchpad.login_with('ci-engine', 'production',
-                                credential_store=store, launchpadlib_dir=lib)
+    return Launchpad.login_with('ci-engine', service_root=get_lp_root_url(),
+                                credential_store=store, launchpadlib_dir=lib,
+                                version=unit_config.get(
+                                    'launchpad_api_version'))
 
 
 def get_lp_user_name():
     return unit_config.get('launchpad_user')
+
+
+def get_lp_ppa_owner():
+    return unit_config.get('launchpad_ppa_owner')
+
+
+def get_lp_root_url():
+    return unit_config.get('launchpad_api_root_url')
+
+
+def get_lp_api_url(api_version=None):
+    if not api_version:
+        api_version = unit_config.get('launchpad_api_version')
+    return "{}/{}/".format(get_lp_root_url(), api_version)
+
+
+def get_private_ppa_sources_list_url(lp, ppa):
+    if lp:
+        user = lp.people[get_lp_user_name()]
+        return user.getArchiveSubscriptionURL(archive=ppa)
+    return None
+
+
+def get_ppa_sources_list_url(team, repo_name):
+    api_url = get_lp_root_url().replace("https", "http").replace("api", "ppa")
+    return "{}/{}/{}/ubuntu".format(api_url, team, repo_name)

=== added file 'create_private_lp_creds.py'
--- create_private_lp_creds.py	1970-01-01 00:00:00 +0000
+++ create_private_lp_creds.py	2014-05-07 18:17:44 +0000
@@ -0,0 +1,57 @@
+#!/usr/bin/env python
+
+import os
+import subprocess
+import sys
+
+import launchpadlib.uris
+from launchpadlib.launchpad import Launchpad
+from lazr.restfulclient import errors
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), './ci-utils'))
+from ci_utils import dump_stack
+
+web_root = "https://mthoodapi.lacinonac.com/"
+
+old = launchpadlib.uris.web_root_for_service_root
+
+
+def web_root_for_service_root(service_root):
+    if service_root == 'https://mthoodapi.lacinonac.com/':
+        return 'https://mthood.lacinonac.com/'
+    return old(service_root)
+
+launchpadlib.uris.web_root_for_service_root = web_root_for_service_root
+
+creds_path = os.path.abspath('lp_creds.txt')
+
+
+def create():
+    try:
+        lp = Launchpad.login_with('ci-airline', consumer_name='ci-airline',
+                                  service_root=web_root, version='devel')
+    except errors.HTTPError as e:
+        if e.response['status'] == '403':
+            sys.stderr.write('Ok, not creating credentials then\n')
+            exit(1)
+        raise
+
+    creds = lp.credentials
+    creds.save_to_path(creds_path)
+    sys.stderr.write('Credentials created in {}\n'.format(creds_path))
+
+    print('export CI_OAUTH_CONSUMER_KEY="{}"'.format(creds.consumer.key))
+    print('export CI_OAUTH_TOKEN={}'.format(creds.access_token.key))
+    print('export CI_OAUTH_TOKEN_SECRET={}'.format(creds.access_token.secret))
+    print('export CI_LAUNCHPAD_USER={}'.format(lp.me.name))
+    # This should be a team that owns the ppa build pool
+    print('export CI_LAUNCHPAD_PPA_OWNER={}'.format(lp.me.name))
+    return 0
+
+
+def main(args):
+    create()
+
+
+if __name__ == '__main__':
+    dump_stack.install_stack_dump_signal()
+    sys.exit(main(sys.argv[1:]))

=== modified file 'image-builder/imagebuilder/cloud_image.py'
--- image-builder/imagebuilder/cloud_image.py	2014-03-17 16:08:17 +0000
+++ image-builder/imagebuilder/cloud_image.py	2014-05-07 18:17:44 +0000
@@ -65,11 +65,9 @@
         ppa = lp.people[team].getPPAByName(name=repo)
         keys[ppa.name] = ppa.signing_key_fingerprint
         if ppa.private:
-            #lp.me won't work, you have to get yourself by person
-            user = lp.people[launchpad.get_lp_user_name()]
-            url = user.getArchiveSubscriptionURL(archive=ppa)
+            url = launchpad.get_private_ppa_sources_list_url(lp, ppa)
         else:
-            url = 'http://ppa.launchpad.net/{}/{}/ubuntu'.format(team, repo)
+            url = launchpad.get_ppa_sources_list_url(team, repo)
         aptlines.append('deb {} {} main'.format(url, series))
         aptlines.append('deb-src {} {} main'.format(url, series))
     return aptlines, keys

=== modified file 'image-builder/imagebuilder/tests/test_modify_cloud_image.py'
--- image-builder/imagebuilder/tests/test_modify_cloud_image.py	2014-03-15 16:46:11 +0000
+++ image-builder/imagebuilder/tests/test_modify_cloud_image.py	2014-05-07 18:17:44 +0000
@@ -28,6 +28,7 @@
     @mock.patch('imagebuilder.cloud_image.launchpad')
     def test_parse_ppas(self, launchpad):
         launchpad.get_lp_user_name.return_value = 'foo'
+        launchpad.get_private_ppa_sources_list_url.return_value = 'foo_url'
 
         lp = mock.MagicMock()
         ppa = mock.Mock()
@@ -35,7 +36,6 @@
         ppa.signing_key_fingerprint = 'foofp'
         ppa.private = True
         lp.people['foo'].getPPAByName.return_value = ppa
-        lp.people['foo'].getArchiveSubscriptionURL.return_value = 'foo_url'
 
         series = 'precise'
         deblines, keys = cloud_image._parse_ppas(lp, ['ppa:/foo/bar'], series)

=== modified file 'juju-deployer/configs/unit_config.yaml.tmpl'
--- juju-deployer/configs/unit_config.yaml.tmpl	2014-03-15 23:15:38 +0000
+++ juju-deployer/configs/unit_config.yaml.tmpl	2014-05-07 18:17:44 +0000
@@ -8,13 +8,17 @@
 # for launchpad apis
 # and see: https://help.launchpad.net/API/SigningRequests
 # and see ../../create_lp_creds.py
-launchpad_api_base: https://api.launchpad.net/1.0
+launchpad_api_root_url: $CI_LAUNCHPAD_API_ROOT_URL
+launchpad_api_version: $CI_LAUNCHPAD_API_VERSION
 launchpad_user: $CI_LAUNCHPAD_USER
 launchpad_ppa_owner: $CI_LAUNCHPAD_PPA_OWNER
-oauth_consumer_key: $CI_OAUTH_CONSUMER_KEY
+launchpad_upload_url: $CI_LAUNCHPAD_UPLOAD_URL
+oauth_consumer_key: "$CI_OAUTH_CONSUMER_KEY"
 oauth_token: $CI_OAUTH_TOKEN
 oauth_token_secret: $CI_OAUTH_TOKEN_SECRET
 
+private_launchpad: $CI_PRIVATE_LAUNCHPAD
+
 ppa_pattern: "ci-pool-\\d+"
 
 private_ppas_only: $CI_PRIVATE_PPAS_ONLY

=== modified file 'juju-deployer/deploy.py'
--- juju-deployer/deploy.py	2014-03-14 23:53:04 +0000
+++ juju-deployer/deploy.py	2014-05-07 18:17:44 +0000
@@ -65,7 +65,11 @@
         'CI_PAYLOAD_URL': 'n/a',
         'CI_LAUNCHPAD_USER': None,
         'CI_LAUNCHPAD_PPA_OWNER': None,
+        'CI_LAUNCHPAD_API_ROOT_URL': 'https://api.launchpad.net/',
+        'CI_LAUNCHPAD_API_VERSION': 'devel',
+        'CI_LAUNCHPAD_UPLOAD_URL': 'ppa.launchpad.net',
         'CI_PRIVATE_PPAS_ONLY': '0',
+        'CI_PRIVATE_LAUNCHPAD': 'False',
         'CI_OAUTH_CONSUMER_KEY': None,
         'CI_OAUTH_TOKEN': None,
         'CI_OAUTH_TOKEN_SECRET': None,
@@ -345,7 +349,9 @@
                         when building packages.''')
     parser.add_argument('specific', metavar='<config.yaml>', nargs='*',
                         help='Use a specific list of juju-deployer configs.')
-
+    parser.add_argument('--private-launchpad', action='store_true',
+                        help='''Sets the environment to use proper Launchpad
+                        instance.''', default=False)
     return parser.parse_args(args)
 
 
@@ -361,6 +367,16 @@
     # variable value.
     check_and_set_ppa_privacy(args.private_ppas_only)
 
+    if args.private_launchpad:
+        os.environ['CI_LAUNCHPAD_API_ROOT_URL'] = (
+            'https://mthoodapi.lacinonac.com/')
+        os.environ['CI_LAUNCHPAD_UPLOAD_URL'] = (
+            'mthood.lacinonac.com')
+        os.environ['CI_PRIVATE_LAUNCHPAD'] = 'True'
+        print('Private Launchpad instance: ENABLED')
+    else:
+        print('Private Launchpad instance: DISABLED')
+
     check_bootstrapped()
     check_dependencies()
 

=== modified file 'ppa-assigner/ppa_assigner/launchpad.py'
--- ppa-assigner/ppa_assigner/launchpad.py	2014-03-14 10:37:28 +0000
+++ ppa-assigner/ppa_assigner/launchpad.py	2014-05-07 18:17:44 +0000
@@ -22,65 +22,6 @@
 from oauth import oauth
 
 
-def lp_collection(url):
-    '''iterate through each item returned in an lp_collection'''
-    resp = _lp_get_authenticated(url)
-    resp = resp.read()
-    data = json.loads(resp)
-
-    for d in data['entries']:
-        yield d
-
-    if 'next_collection_link' in data:
-        for d in lp_collection(data['next_collection_link']):
-            yield d
-
-
-def _oauth_headers():
-    '''oauth header required for a launchpad api post
-
-    https://help.launchpad.net/API/SigningRequests
-    '''
-    values = [
-        'OAuth realm="%s"' % settings.OAUTH_REALM,
-        'oauth_version="1.0"',
-        'oauth_consumer_key="%s"' % settings.OAUTH_CONSUMER_KEY,
-        'oauth_token="%s"' % settings.OAUTH_TOKEN,
-        'oauth_signature_method="PLAINTEXT"',
-        'oauth_signature="%%26%s"' % settings.OAUTH_TOKEN_SECRET,
-        'oauth_timestamp="%d"' % int(oauth.time.time()),
-        'oauth_nonce="%s"' % oauth.generate_nonce(),
-    ]
-    return {
-        'Authorization': ', '.join(values),
-        'Accept': 'application/json',
-    }
-
-
-def _lp_post(url, data):
-    headers = _oauth_headers()
-    headers['Content-Type'] = 'application/x-www-form-urlencoded'
-    req = urllib2.Request(url, urllib.urlencode(data), headers)
-    return urllib2.urlopen(req)
-
-
-def _lp_get_authenticated(url):
-    '''An authenticated GET to obtain also private information.'''
-    req = urllib2.Request(url, headers=_oauth_headers())
-    return urllib2.urlopen(req)
-
-
-def get_publishing_history(ppa):
-    '''A generator for the source and binary publish history of a ppa.'''
-    url = settings.LAUNCHPAD_API_BASE + '/~%s/+archive/%s/?ws.op='
-    url = url % (settings.LAUNCHPAD_PPA_OWNER, ppa.split('/')[-1])
-
-    for x in lp_collection(url + 'getPublishedSources'):
-        yield x
-    for x in lp_collection(url + 'getPublishedBinaries'):
-        yield x
-
-
 def request_ppa_clean(ppa):
     '''call requestDeletion operation on ppa's source and binary pub history'''
     for x in get_publishing_history(ppa):
@@ -116,8 +57,9 @@
 
 def _sync_source(ppa, source_name, version, pocket, archive):
     '''Used for populating a PPA so we can run integration tests.'''
-    url = 'https://api.launchpad.net/1.0/~%s/+archive/%s'
-    url = url % (settings.LAUNCHPAD_PPA_OWNER, ppa.split('/')[-1])
+    url = '{}/~{}/+archive/{}'.format(settings.LAUNCHPAD_API_URL,
+                                      settings.LAUNCHPAD_PPA_OWNER,
+                                      ppa.split('/')[-1])
     data = {
         'ws.op': 'syncSource',
         'from_archive': archive,
@@ -129,3 +71,62 @@
     c = resp.getcode()
     if c != 200:
         raise urllib2.HTTPError(url, c, resp.read(), resp.header.headers, None)
+
+
+def get_publishing_history(ppa):
+    '''A generator for the source and binary publish history of a ppa.'''
+    url = '{}/~{}/+archive/{}/?ws.op='.format(settings.LAUNCHPAD_API_URL,
+                                              settings.LAUNCHPAD_PPA_OWNER,
+                                              ppa.split('/')[-1])
+    for x in _lp_collection(url + 'getPublishedSources'):
+        yield x
+    for x in _lp_collection(url + 'getPublishedBinaries'):
+        yield x
+
+
+def _lp_collection(url):
+    '''iterate through each item returned in an lp_collection'''
+    resp = _lp_get_authenticated(url)
+    resp = resp.read()
+    data = json.loads(resp)
+
+    for d in data['entries']:
+        yield d
+
+    if 'next_collection_link' in data:
+        for d in _lp_collection(data['next_collection_link']):
+            yield d
+
+
+def _oauth_headers():
+    '''oauth header required for a launchpad api post
+
+    https://help.launchpad.net/API/SigningRequests
+    '''
+    values = [
+        'OAuth realm="%s"' % settings.OAUTH_REALM,
+        'oauth_version="1.0"',
+        'oauth_consumer_key="%s"' % settings.OAUTH_CONSUMER_KEY,
+        'oauth_token="%s"' % settings.OAUTH_TOKEN,
+        'oauth_signature_method="PLAINTEXT"',
+        'oauth_signature="%%26%s"' % settings.OAUTH_TOKEN_SECRET,
+        'oauth_timestamp="%d"' % int(oauth.time.time()),
+        'oauth_nonce="%s"' % oauth.generate_nonce(),
+    ]
+    return {
+        'Authorization': ', '.join(values),
+        'Accept': 'application/json',
+    }
+
+
+def _lp_post(url, data):
+    headers = _oauth_headers()
+    headers['Content-Type'] = 'application/x-www-form-urlencoded'
+    req = urllib2.Request(url, urllib.urlencode(data), headers)
+    return urllib2.urlopen(req)
+
+
+def _lp_get_authenticated(url):
+    '''An authenticated GET to obtain also private information.'''
+    req = urllib2.Request(url, headers=_oauth_headers())
+    return urllib2.urlopen(req)

=== modified file 'ppa-assigner/ppa_assigner/models.py'
--- ppa-assigner/ppa_assigner/models.py	2014-03-14 23:53:04 +0000
+++ ppa-assigner/ppa_assigner/models.py	2014-05-07 18:17:44 +0000
@@ -83,13 +83,13 @@
     @staticmethod
     def populate_from_launchpad():
         '''Get list of PPAs from launchpad and ensure they exist here.'''
-        url = settings.LAUNCHPAD_API_BASE + '/~%s/ppas'
-        url = url % settings.LAUNCHPAD_PPA_OWNER
+        url = '{}/~{}/ppas'.format(settings.LAUNCHPAD_API_URL,
+                                   settings.LAUNCHPAD_PPA_OWNER)
         logging.info('populating ppa model from: %s', url)
         try:
             ppas = {}
             pat = re.compile(settings.PPA_PATTERN)
-            for ppa in launchpad.lp_collection(url):
+            for ppa in launchpad._lp_collection(url):
                 if pat.match(ppa['name']):
                     ppa_name = 'ppa:{}/{}'.format(settings.LAUNCHPAD_PPA_OWNER,
                                                   ppa['name'])

=== modified file 'ppa-assigner/ppa_assigner/settings.py'
--- ppa-assigner/ppa_assigner/settings.py	2014-03-14 23:53:04 +0000
+++ ppa-assigner/ppa_assigner/settings.py	2014-05-07 18:17:44 +0000
@@ -44,12 +44,12 @@
 
 LAUNCHPAD_USER = _cfg.get('launchpad_user', None)
 LAUNCHPAD_PPA_OWNER = _cfg.get('launchpad_ppa_owner', None)
-LAUNCHPAD_API_BASE = _cfg.get(
-    'launchpad_api_base', 'https://api.launchpad.net/1.0')
+LAUNCHPAD_API_URL = "{}/{}".format(_cfg.get('launchpad_api_root_url', None),
+                                   _cfg.get('launchpad_api_version'))
 OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)
 OAUTH_TOKEN = _cfg.get('oauth_token', None)
 OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)
-OAUTH_REALM = _cfg.get('oauth_realm', 'https://api.launchpad.net/')
+OAUTH_REALM = _cfg.get('launchpad_api_root_url')
 PPA_PATTERN = _cfg.get('ppa_pattern', r'ci-pool-\d+')
 PRIVATE_PPAS_ONLY = _cfg.get('private_ppas_only', None)