Merge lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script into lp:snappy-ecosystem-tests

[Omer Akram]

script: build snapd in lxd container to enable staging servers

script: build snapd in lxd container to enable staging servers

[Omer Akram (om26er)]

Tested with Ubuntu 16.04.1 and 16.10 hosts, worked with both.

[I Ahmad (iahmad)]

Could you please add some instructions about how to use it, as well as if it is standalone utility then add these instructions to the README file.

[Omer Akram (om26er)]

Updated the branch with some cleanups and also added instructions on how to setup the environment.

Please make sure to have all updates applied to Ubuntu 16.04 host when running.

[Omer Akram (om26er)]

I initially wrote this script in bash but later ported it to python. So if anyone interested in the shell version look here: https://pastebin.canonical.com/178467/

[I Ahmad (iahmad)]

Go ahead with merge, we will know if any issues or improvements needed once test cases are added.

review: Approve

[platform-qa-bot (platform-qa-bot)]

review: Approve (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-autoland/1/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-land-mp/2655/console

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:8
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/10/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1567/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/10/rebuild

review: Needs Fixing (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Minor comments inline plus if we really want to move from bash to python then maybe we should take advantage of the available python libraries and replace the subprocess calls with python calls

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

9. By Omer Akram on 2017-02-09

reduce duplicated call

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:9
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/11/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1573/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/11/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

Thanks for the review, I have replied to your comments

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

10. By Omer Akram on 2017-02-09

merge with trunk

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:10
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/12/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1574/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/12/rebuild

review: Needs Fixing (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Replies inline. Thanks

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

11. By Omer Akram on 2017-02-12

Replace lxd shell calls with pylxd

12. By Omer Akram on 2017-02-12

Merge with trunk

13. By Omer Akram on 2017-02-12

remove logging config, to be set by the test runner

14. By Omer Akram on 2017-02-12

better worded errors

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:12
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/20/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1592/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/20/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:14
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/21/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1593/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/21/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

Added replies and pushed changes.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:14
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/22/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1594/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/22/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

15. By Omer Akram on 2017-02-12

add more docstrings, make pylint happier

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:15
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/23/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1595/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/23/rebuild

review: Needs Fixing (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Hi Omer, I still find this confusing. Starting with a Container class that has a container attribute. I also don't understand why we redefine methods like start, stop, delete, etc instead of calling the ones provided by the library. I've re-written this just as an example of what I mean: https://pastebin.ubuntu.com/23988552/

review: Needs Information

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

16. By Omer Akram on 2017-02-14

Incorporate changes suggested by sbalda, more improvements

[Omer Akram (om26er)]

I updated the approach based on some of the suggestions in your paste. I still kept a class that inherits from pylxd.models.Container as we want to extend a bit of functionality of that class.

I have also addressed some of your concerns as well.

Note: there is an outstanding issue with inheritance of Container class, I have sent a query to pylxd upstream developers for that.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:16
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/29/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1607/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/29/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:16
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/30/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1608/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/30/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

17. By Omer Akram on 2017-02-14

address pylint errors

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:17
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/32/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1610/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/32/rebuild

review: Needs Fixing (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Looks better to me, however, there's an issue to be fixed.

By the way, I think the inheritance implemented in this mp is causing the issue in pylxd: https://github.com/lxc/pylxd/pull/220
If you tried the same sequence of events but without recreating the object, it is working just fine:
https://pastebin.ubuntu.com/23997027/

Last but not least, if you want to redefine the Container methods, like exec or delete, you'll have to do it in pylxd because they'll always use their class to create the objects. Another option is to move those methods to the builder and use the Container class provided by pylxd

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

18. By Omer Akram on 2017-02-14

Container inheritance is broken, move to delegation

[Omer Akram (om26er)]

Thanks for the review, I have pushed the updated code. I have moved away from inheritance as that's broken upstream. The approach in the pull request that I created was rejected by upstream.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:18
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/37/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1622/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/37/rebuild

review: Needs Fixing (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Minor comments inline. I'm abstaining from now since I still think that the way this is modeled is not right. Maybe someone else could chime in here

review: Abstain

[Omer Akram (om26er)]

Replied inline. I believe there is nothing wrong with the current approach, we are just trying to extend the pylxd.models.Container class with delegation as inheritance is not supported upstream.

[Santiago Baldassin (sbaldassin)]

Replies inline

review: Abstain

[Santiago Baldassin (sbaldassin)]

By the way, the current approach forces you to keep two instances of the same container created by pylxd

[Heber Parrucci (heber013)]

Comments inline. Thanks!

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

19. By Omer Akram on 2017-02-16

Remove custom exception

20. By Omer Akram on 2017-02-16

pylint happy

21. By Omer Akram on 2017-02-16

merge with trunk

22. By Omer Akram on 2017-02-16

make pylint finally super happy

[Omer Akram (om26er)]

Thanks heber, replied inline and also pylint should be happy now.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:22
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/50/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1636/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/50/rebuild

review: Needs Fixing (continuous-integration)

[Heber Parrucci (heber013)]

Reply inline

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

23. By Omer Akram on 2017-02-17

Some sanity checks to Container code, improve project structure

24. By Omer Akram on 2017-02-17

Update readme

25. By Omer Akram on 2017-02-17

Remove delegate updation code

[Omer Akram (om26er)]

I have made some structural changes to the code and replied inline.

[Omer Akram (om26er)]

Santiago, added replies.

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

26. By Omer Akram on 2017-02-17

Add message on missing commandline arguments for snapd builder script

27. By Omer Akram on 2017-02-17

ignore pylint errors as its not able to find members of the class as they are dynamic

[Santiago Baldassin (sbaldassin)]

Minor comment inline

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

28. By Omer Akram on 2017-02-18

Replace sys.argv with argparse

29. By Omer Akram on 2017-02-18

pylint happy

[Omer Akram (om26er)]

This branch is 10 days old and I would now expect that we land it instead of more nitpicks.

[Santiago Baldassin (sbaldassin)]

Minor comment inline. By the way, please do not expect a branch to be merged based on the time it has been for out there for review, which is not a valid reason to merge a change.
Based on the previous comments/discussions, I'm abstaining

review: Abstain

[Heber Parrucci (heber013)]

Comments inline.
One thing, I was getting an error when the sh creates the virtual env:

http://paste.ubuntu.com/24033643/

It is because "pylxd" depends on "cryptography" and it has a ssl extention. I solved the problem by installing OpenSSL development package manually:
$ sudo apt-get install libssl-dev

How do we solve this issue automatically?

review: Needs Fixing

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:29
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/59/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1646/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/59/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

Heber, replied inline. Not sure how we can fix that error. Probably instead of virtualenv we could resort to a container, then installing that dep would be plain and simple and won't pollute anything else.

[Omer Akram (om26er)]

Santiago, replied in line.

If we really want a code to work we have to land it :) Playing around in circles does not help. This branch was initially ported from raw lxd commands to pylxd with no concrete benefits and now after all the reasons were provided on why the Container class looks the way it is because inheritance is not supported by upstream and current approach is very natural for classes that cannot be inherited, it feels kind of like "my way or the highway" ;)

[Heber Parrucci (heber013)]

Comments inline

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

30. By Omer Akram on 2017-02-20

remove pylint suppress thats not needed

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:30
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/62/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1649/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/62/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

Heber, replied inline.

[Santiago Baldassin (sbaldassin)]

> Santiago, replied in line.
>
> If we really want a code to work we have to land it :) Playing around in

With all do respect, I strongly disagree. We do NOT have to land code that we think is not ready to be landed just because we want the code to work.

> circles does not help. This branch was initially ported from raw lxd commands
> to pylxd with no concrete benefits and now after all the reasons were provided

Using pylxd does have concrete benefits on the long term, that's why we decided to use it. If you think that the previous approach is the right one, I'd be happy to revisit the approach once again

> on why the Container class looks the way it is because inheritance is not
> supported by upstream and current approach is very natural for classes that
> cannot be inherited, it feels kind of like "my way or the highway" ;)

I'm sorry Homer but Heber and myself clearly expressed that we'd preferred a different approach and yet you intend to get this merged

[Santiago Baldassin (sbaldassin)]

Reply inline

review: Abstain

[Omer Akram (om26er)]

minor query inline.

[Santiago Baldassin (sbaldassin)]

reply inline

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:30
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/63/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1650/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/63/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

31. By Omer Akram on 2017-02-20

merge with trunk

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:31
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/68/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1655/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/68/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:31
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/70/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1657/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/70/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:31
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/71/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1658/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/71/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:31
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/72/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1659/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/72/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

32. By Santiago Baldassin on 2017-02-21

Ignoring chrome_installer to make sure it is not causing side effects

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:32
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/74/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1661/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/74/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/snapd-staging-build-script]

33. By Santiago Baldassin on 2017-02-21

Adding TODOs and updating pylint.cfg

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:33
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/75/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1662/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/75/rebuild

review: Approve (continuous-integration)

[Heber Parrucci (heber013)]

Code LGTM and TODOs make sense.

review: Approve

=== modified file 'README.rst'
--- README.rst	2017-02-16 14:24:21 +0000
+++ README.rst	2017-02-21 13:46:31 +0000
@@ -98,3 +98,19 @@
 
 If you are behind a proxy you can specify --proxy option
 
+
+How to setup snapd staging environment
+======================================
+
+Requirements
+1. Ubuntu 16.04.1+ with all updates applied
+2. LXD >= 2.4.1
+
+To enable staging for snapd, it needs to be re-built with special flags, the script in
+tests/utils/build_snapd_staging.py have all the bits in place to achieve that. It can be
+run as a standalone script or could be incorporated into snapd tests. Just run the script
+to setup the environment.
+
+$ bzr branch lp:snappy-ecosystem-tests
+$ cd snappy-ecosystem-tests/
+$ python3 -m snappy_ecosystem_tests.snapd.staging_builder my_new_container_name 16.10

=== modified file 'pylint.cfg'
--- pylint.cfg	2017-02-15 19:35:54 +0000
+++ pylint.cfg	2017-02-21 13:46:31 +0000
@@ -304,7 +304,7 @@
 # List of class names for which member attributes should not be checked (useful
 # for classes with dynamically set attributes). This supports the use of
 # qualified names.
-ignored-classes=optparse.Values,thread._local,_thread._local
+ignored-classes=optparse.Values,thread._local,_thread._local,ContainerManager
 
 # List of members which are set dynamically and missed by pylint inference
 # system, and so shouldn't trigger E1101 when accessed. Python regular

=== modified file 'requirements.txt'
--- requirements.txt	2017-02-15 14:22:53 +0000
+++ requirements.txt	2017-02-21 13:46:31 +0000
@@ -14,4 +14,5 @@
 pexpect
 pymacaroons==0.9.2
 requests-toolbelt==0.6.0
-chromedriver_installer
+#chromedriver_installer
+pylxd

=== added file 'snappy_ecosystem_tests/helpers/snapd/staging_builder.py'
--- snappy_ecosystem_tests/helpers/snapd/staging_builder.py	1970-01-01 00:00:00 +0000
+++ snappy_ecosystem_tests/helpers/snapd/staging_builder.py	2017-02-21 13:46:31 +0000
@@ -0,0 +1,150 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Snappy Ecosystem Tests
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""Module to enable staging environment for snapd."""
+
+import logging
+import sys
+from time import sleep
+
+from snappy_ecosystem_tests.utils.lxd import Container, launch_container
+
+CONTAINER_HOME = '/root'
+DIRECTORY_CLONE = '{}/src/github.com/snapcore/snapd'.format(CONTAINER_HOME)
+ENV_STORE_API_ROOT = 'UBUNTU_STORE_API_ROOT_URL'
+ENV_STORE_SEARCH_ROOT = 'UBUNTU_STORE_SEARCH_ROOT_URL'
+ENV_STORE_UPLOAD_ROOT = 'UBUNTU_STORE_UPLOAD_ROOT_URL'
+ENV_SSO_API_ROOT = 'UBUNTU_SSO_API_ROOT_URL'
+PACKAGES_SNAPD = ['snapd', 'snap-confine', 'ubuntu-core-launcher']
+COMMAND_REMOVE_SNAPD = 'apt purge -y {}'.format(' '.join(PACKAGES_SNAPD))
+COMMAND_APT_BUILD_DEP = 'apt build-dep -y ./'
+COMMAND_APT_INSTALL = 'apt install -y {}'
+COMMAND_APT_INSTALL_STAGING_DEBS = COMMAND_APT_INSTALL.format(
+    ' '.join(['../{}*.deb'.format(pkg) for pkg in PACKAGES_SNAPD]))
+COMMAND_APT_UPDATE = 'apt update'
+COMMAND_APT_UPGRADE = 'apt dist-upgrade -y'
+COMMAND_BUILD_SNAPD = 'DEB_BUILD_OPTIONS="nocheck testkeys" ' \
+                      'dpkg-buildpackage -tc -b'
+COMMAND_EXPORT_STAGING_STORE_VAR = 'echo SNAPPY_USE_STAGING_STORE=1 >> ' \
+                                   '/etc/environment'
+COMMAND_GET_GOVENDOR = 'go get -v github.com/kardianos/govendor'
+COMMAND_GIT_CLONE = 'git clone {} {}'
+COMMAND_GOVENDOR_SYNC = 'govendor sync'
+REPOSITORY_GIT = 'https://github.com/snapcore/snapd'
+URL_API_ROOT = 'https://myapps.developer.staging.ubuntu.com/dev/api/'
+URL_SEARCH_ROOT = 'https://search.apps.staging.ubuntu.com/'
+URL_UPLOAD_ROOT = 'https://upload.apps.staging.ubuntu.com/'
+URL_SSO_API_ROOT = 'https://login.staging.ubuntu.com/api/v2/'
+
+CONTAINER_SETUP = [
+    COMMAND_REMOVE_SNAPD,
+    COMMAND_APT_UPDATE,
+    # Hold open-iscsi which is causing apt upgrade errors while running
+    # inside container.
+    'apt-mark hold open-iscsi',
+    COMMAND_APT_UPGRADE,
+    # Needed for snaps to work inside a lxd container
+    #  ref: https://stgraber.org/2016/12/07/running-snaps-in-lxd-containers/
+    COMMAND_APT_INSTALL.format('squashfuse'),
+    COMMAND_GIT_CLONE.format(REPOSITORY_GIT, DIRECTORY_CLONE),
+    {'command': COMMAND_APT_BUILD_DEP, 'cwd': DIRECTORY_CLONE},
+    {'command': COMMAND_GET_GOVENDOR, 'cwd': DIRECTORY_CLONE},
+    {'command': COMMAND_GOVENDOR_SYNC, 'cwd': DIRECTORY_CLONE},
+    {'command': COMMAND_BUILD_SNAPD, 'cwd': DIRECTORY_CLONE},
+    COMMAND_EXPORT_STAGING_STORE_VAR,
+    {'command': COMMAND_APT_INSTALL_STAGING_DEBS, 'cwd': DIRECTORY_CLONE}
+]
+
+CONTAINER_ENV_VARS = {
+    'GOPATH': CONTAINER_HOME,
+    'SNAPPY_USE_STAGING_STORE': '1',
+    ENV_STORE_API_ROOT: URL_API_ROOT,
+    ENV_STORE_SEARCH_ROOT: URL_SEARCH_ROOT,
+    ENV_STORE_UPLOAD_ROOT: URL_UPLOAD_ROOT,
+    ENV_SSO_API_ROOT: URL_SSO_API_ROOT,
+}
+
+LOGGER = logging.getLogger(__name__)
+
+
+class StagingBuilder:
+    """Class to build and activate snapd staging environment."""
+    def __init__(self, container):
+        if not isinstance(container, Container):
+            raise RuntimeError(
+                '`container` must be an instance of '
+                'snappy_ecosystem_tests.utils.lxd.Container')
+        self.container = container
+
+    def _setup_staging(self):
+        """Export environment variables required to enable snapd/snapcraft
+        staging."""
+        LOGGER.info('Exporting staging environment variables')
+        updated_path = '{}/bin:{}'.format(
+            CONTAINER_HOME, self.container.execute('echo $PATH'))
+        CONTAINER_ENV_VARS['PATH'] = updated_path
+        for key, value in CONTAINER_ENV_VARS.items():
+            self.container.set_environment_variable(key, value, save=False)
+        self.container.save(wait=True)
+        LOGGER.info('Exported staging environment variables')
+
+    def _wait_for_internet(self, attempts_count=10, attempt_interval=1):
+        """Wait for internet connectivity in the container."""
+        for _ in range(attempts_count):
+            if self.container.is_internet_working():
+                break
+            else:
+                LOGGER.info('Sleeping for %s second(s)', attempt_interval)
+                sleep(attempt_interval)
+        else:
+            LOGGER.error(
+                'Failed to get internet access in container %s after %s '
+                'seconds', self.container.name,
+                attempts_count * attempt_interval)
+            sys.exit(1)
+
+    def _setup_and_build(self):
+        """Setup the container, pulls and builds snapd with staging enabled.
+        """
+        for command in CONTAINER_SETUP:
+            if isinstance(command, dict):
+                self.container.execute(**command)
+            else:
+                self.container.execute(command)
+
+    def build(self):
+        """Setup lxd container environment for snapd staging enabled builds."""
+        self._setup_staging()
+        self._wait_for_internet()
+        self._setup_and_build()
+
+
+if __name__ == '__main__':
+    # TODO remove local import
+    import argparse
+    logging.basicConfig(level=logging.INFO)
+    PARSER = argparse.ArgumentParser(
+        description='Builder for snapd staging environment.')
+    PARSER.add_argument('container_name', help='name for the new container.')
+    PARSER.add_argument(
+        'ubuntu_release', help='Ubuntu release version for the container.')
+    ARGS = PARSER.parse_args()
+    StagingBuilder(launch_container(name=ARGS.container_name,
+                                    release=ARGS.ubuntu_release)).build()

=== added file 'snappy_ecosystem_tests/utils/lxd.py'
--- snappy_ecosystem_tests/utils/lxd.py	1970-01-01 00:00:00 +0000
+++ snappy_ecosystem_tests/utils/lxd.py	2017-02-21 13:46:31 +0000
@@ -0,0 +1,125 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Snappy Ecosystem Tests
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""Module to manipulate lxd containers."""
+
+import logging
+import shlex
+
+import pylxd
+from pylxd import models
+
+DEFAULT_CONTAINER_CONFIG = {
+    'name': 'snapd',
+    'source': {
+        'type': 'image',
+        'protocol': 'simplestreams',
+        'server': 'https://cloud-images.ubuntu.com/daily',
+        'alias': '16.04'
+    }
+}
+DOMAIN_PING = 'ubuntu.com'
+LOGGER = logging.getLogger(__name__)
+
+
+def launch_container(name, release, delete_existing=True):
+    """Launch a lxd container of the requested name and release and return
+    its object."""
+    lxd_client = pylxd.Client()
+    if delete_existing and lxd_client.containers.exists(name):
+        Container(lxd_client.containers.get(name)).delete()
+    container_config = DEFAULT_CONTAINER_CONFIG.copy()
+    container_config['source']['alias'] = release
+    container_config['name'] = name
+    container = Container(
+        lxd_client.containers.create(container_config, wait=True))
+    container.start(wait=True)
+    return container
+
+
+class Container:
+    """A Class representing a lxd container.
+
+    Extends and overrides some functionalities of pylxd.models.Container
+    instance through delegation since that class does not support inheritance.
+    """
+    def __init__(self, container_instance):
+        # TODO: move to inheritance once upstream supports it.
+        if not isinstance(container_instance, models.Container):
+            raise RuntimeError(
+                '`container_instance` must be an instance of '
+                'pylxd.models.Container'
+            )
+        #TODO: Remove this as the StageBuilder class already has an
+        # instance of the container
+        self._delegate = container_instance
+
+    def __getattr__(self, name):
+        """Set the container instance as delegate."""
+        return getattr(self._delegate, name)
+
+    @staticmethod
+    def _assemble_container_commands(command, cwd=''):
+        """Format and split the requested command to be container friendly."""
+        exec_command_base = 'sh -c \'{}\''
+        if cwd:
+            command = 'cd {}; {}'.format(cwd, command)
+        LOGGER.info('Run command: %s', command)
+        return shlex.split(exec_command_base.format(command))
+
+    #TODO move the following methods to another place which does not require
+    # a container class
+    def execute(self, command, environment=None, cwd=''):
+        """Execute command in the container."""
+        response = self._delegate.execute(
+            self._assemble_container_commands(command, cwd=cwd),
+            environment=environment or {}
+        )
+        if response.exit_code == 0:
+            LOGGER.info(response.stdout)
+            return response.stdout.strip()
+        raise ValueError(response.stderr)
+
+    def set_environment_variable(self, key, value, save=True):
+        """Exports environment variable to be available to
+        Container.execute()."""
+        self._delegate.config.update({'environment.{}'.format(key): value})
+        if save:
+            self._delegate.save(wait=True)
+
+    def is_running(self):
+        """Returns bool representing whether the container is running."""
+        return self._delegate.status == 'Running'
+
+    def delete(self, stop=True, wait=True):
+        """Stops and deletes the container."""
+        if stop and self.is_running():
+            self._delegate.stop(wait=True)
+        LOGGER.info('Deleting container %s', self.name)
+        self._delegate.delete(wait=wait)
+        LOGGER.info('Deleted container %s', self.name)
+
+    def is_internet_working(self, ping_domain=DOMAIN_PING):
+        """Return bool representing if the container have internet access."""
+        try:
+            self.execute('ping -c 1 {}'.format(ping_domain))
+            return True
+        except ValueError:
+            return False