Merge lp:~noise/tanuki-agent/tutorial-two-point-oh into lp:tanuki-agent

[Bret Barker]

First pass of new tutorial for product-based auth, GM testing.

First pass of new tutorial for product-based auth, GM testing. Lots of TODO placeholders waiting until more code lands to use proper API endpoints and payloads.

[Roberto Alsina (ralsina)]

LGTM. Made a change in my branch to make the agent print the list of accessible packages on registration like this tutorial says.

review: Approve

=== modified file 'docs/Makefile'
--- docs/Makefile	2015-11-04 15:57:51 +0000
+++ docs/Makefile	2015-11-23 20:04:49 +0000
@@ -1,5 +1,8 @@
-docs: tutorial.md notes.md api-reference.md agent-reference.md
-	../env/bin/python3 build.py tutorial.md -o tutorial.html
-	../env/bin/python3 build.py notes.md -o index.html
-	../env/bin/python3 build.py api-reference.md -o api-reference.html
-	../env/bin/python3 build.py agent-reference.md -o agent-reference.html
+%.html: %.md 
+	../env/bin/python build.py -o $@ $<
+
+all: $(addsuffix .html, $(basename $(wildcard *.md)))
+	mv notes.html index.html
+
+clean:
+	rm -f *.html

=== added file 'docs/tutorial2.md'
--- docs/tutorial2.md	1970-01-01 00:00:00 +0000
+++ docs/tutorial2.md	2015-11-23 20:04:49 +0000
@@ -0,0 +1,467 @@
+Title:   Getting Started with Ubuntu Product Integration
+Version: 20151119
+
+Ubuntu Product Integration is a system for automated integration
+testing of products built upon Ubuntu Core. Once configured, the system will
+monitor for updates to components of your product and run automated
+tests against target devices to determine if those updates are
+compatible, and allow you to control the flow of those updates to your
+customers.
+
+This tutorial will guide you step-by-step to create a snap for an application,
+define a product that includes that application, create a laboratory
+environment where integration tests for that product can run, and set up the
+system so that it automatically executes tests in that laboratory when new
+versions of the snap become available.
+
+The document is organized as follows:
+
+-   **[What you will need:](#what-you-will-need)** A description of everything needed to complete the tutorial.
+-   **[Lab setup:](#lab-setup)** Create and configure your own laboratory.
+-   **[Gold Master Candidate Testing:](#gold-master-candidate-testing)** Testing of Product images composed from the stable channel, to support releasing Gold Master Images.
+-   **[Product update testing:](#product-update-testing)** Testing of software updates prior to releasing them to customers.
+-   **[First-boot update testing:](#first-boot-update-testing)** First-boot update scenario tesitng.
+
+<div class="panel panel-info">
+  <div class="panel-heading">
+    <h3 class="panel-title">Conventions</h3>
+  </div>
+  <div class="panel-body">
+    From time to time you will need to execute commands in a terminal. This
+    will be indicated using <code>this particular font and background color</code>,
+    normally on a separate line. This font will also be used to specify
+    filenames or any other cases where spelling matters.
+  </div>
+</div>
+
+
+## What you will need
+
+In order to complete this tutorial, you will need the following resources:
+
+-   A host computer running [Ubuntu 14.04 (64-bit)](http://www.ubuntu.com/download/desktop), with an SD card reader and at least 6 GB free drive space.
+-   A [Raspberry Pi 2 Model B](https://www.raspberrypi.org/products/raspberry-pi-2-model-b/).
+-   A 4GB (minimum) Micro SDHC card.
+-   The host computer and Raspberry Pi connected on the same local ethernet network.
+-   Access to the Internet from the host computer.
+-   An optional serial USB cable, for debugging the RPI boot sequence.
+-   Ubuntu One OAuth credentials. This will be covered below
+    during [Lab setup](#lab-setup).
+
+## Lab setup
+
+In order to run tests on your product, the system needs a host computer to serve as a
+bridge between your device and itself. This computer can be as simple as your
+desktop, or part of a full device laboratory.
+
+For the purposes of this tutorial, your desktop computer will serve as the
+host and you will use a Raspberry Pi as the product device. The host computer
+will run a small application called the Agent, which will communicate with
+Canonical's servers and manage integration tests on the Raspberry Pi.
+
+### Setting up the host computer
+
+On your host computer, first install the latest Snappy tools:
+
+    sudo add-apt-repository ppa:snappy-dev/tools
+    sudo apt-get update
+    sudo apt-get dist-upgrade
+    sudo apt-get install snappy-tools
+
+See [Snappy Getting Started Guide](https://developer.ubuntu.com/en/snappy/start/) for further details.
+
+Also install the software dependencies of the Agent:
+
+    sudo apt-get install python-requests pv
+
+Next you will install and configure the Agent. The most recent release of the Agent can always be found at [this location](https://spi.canonical.com/assets/agent.tar.gz).
+
+Download and install the Agent from your host computer as follows:
+
+    wget https://spi.canonical.com/assets/agent.tar.gz
+    tar -zxf agent.tar.gz
+    cd agent-binaries
+
+### Configuring the Agent
+
+Once the Agent is installed, you will need to create its configuration:
+
+TODO: make interactive-login generate default config as well to condense these steps
+
+    ./agent.py --default-config > config.ini
+
+The Agent authenticates using an Ubuntu One account . To use it, first create an
+[Ubuntu One account](https://login.ubuntu.com), then instruct the Agent to put
+your credentials in a configuration file:
+
+    ./agent.py --interactive-login config.ini
+
+<div class="panel panel-danger">
+  <div class="panel-heading">
+    <h3 class="panel-title">Security warning</h3>
+  </div>
+  <div class="panel-body">
+    The config file now has data (<code>consumer_secret</code>,
+    <code>token_secret</code>, and <code>token_key</code>) that are equivalent
+    to your password. You are responsible for storing this securely. The device
+    on which the Agent runs should be protected from unauthorized access.
+  </div>
+</div>
+
+
+Note: It is best practice to create a separate Ubuntu One account to use for your Agents, but we'll keep things simple for the purposes of this tutorial.
+
+### Testing Agent Authorization
+
+Before we proceed, let's ensure that your Agent is properly configured.
+
+    ./agent.py --register config.ini
+
+You should see a message similiar to the following:
+
+    [agent-queue] Registered at https://spi.canonical.com/...TODO
+    [] ... available products: [ ]
+
+Please double check your credentials and config.ini if you instead see an error similiar to the following:
+
+    {
+      "id": "unauthorized",
+      "message": "unauthorized"
+    }
+
+Once successfully registered, you can do a test run of the Agent:
+
+    ./agent.py -v --once config.ini
+
+If successful, this will produce output similar to the following:
+
+    [agent-queue] Registered at https://spi.canonical.com/... TODO
+    [agent-queue] No tests at https://spi.canonical.com/... TODO
+
+
+### Agent Provisioning Kit setup
+
+When the Agent receives tests from the server, it will try to run the commands to provision, set up, and run the tests.
+
+Together these scripts are called a "Provisioning Kit." A sample kit has been provided for this tutorial and as a starting point for your own projects.
+
+Edit ```config.ini``` and replace the following three fields:
+
+    provcommand = PATH_TO_AGENT/rpi2-sample-provkit/provision
+    setupcommand = PATH_TO_AGENT/rpi2-sample-provkit/setup
+    testcommand = PATH_TO_AGENT/rpi2-sample-provkit/runtest
+
+<div class="panel panel-warning">
+  <div class="panel-heading">
+    <h3 class="panel-title">Path sensitivity</h3>
+  </div>
+  <div class="panel-body">
+     The paths in <code>config.ini</code> must be absolute. Be sure to replace
+     the correct values.
+  </div>
+</div>
+
+
+### Raspberry Pi Setup
+
+Next, you will do a tgest provisioning to ensure your Raspberry Pi is setup properly.
+
+Download an already prepared sample image:
+
+    wget https://spi.canonical.com/assets/spi-test_pi2.img.xz
+
+Verify that the MD5 hash for the downloaded file matches the value shown below:
+
+    md5sum spi-test_pi2.img.xz
+    be7cf02999e576aa5eb76e7e85713812  spi-test_pi2.img.xz
+
+Because remotely flashing the Raspberry Pi is a feature still in active
+development, you will need to manually flash an SD card with the image.
+
+On the host machine, insert your SD card and determine which device it is
+(using ```df -h``` or other tools), then copy the image to your flash device:
+
+    xzcat spi-test_pi2.img | pv -s 4g | sudo dd of=/dev/sdX bs=32M
+    sync
+
+<div class="panel panel-danger">
+  <div class="panel-heading">
+    <h3 class="panel-title">Careful device selection</h3>
+  </div>
+  <div class="panel-body">
+     Be careful select the correct output device ('of' flag). The dd command
+     will offer no prompt, nor any protection from accidentally selecting your
+     hard drive.
+  </div>
+</div>
+
+This should take less than five minutes on most modern hardware. For additional instructions see [Getting started with a Raspberry Pi 2](https://developer.ubuntu.com/en/snappy/start/raspberry-pi-2/)
+
+When the flashing is complete, insert the SD card in your Raspberry Pi 2, power
+it on, and wait for it to boot. This typically takes less than a minute to
+complete.
+
+You can check whether it's online just by running:
+
+    ping webdm.local
+
+If the device is not accessible, you can troubleshoot by using a serial cable
+to [obtain a console on the Raspberry Pi](http://elinux.org/RPi_Serial_Connection).
+Note that the pins are the same for both versions of the Raspberry Pi.
+
+Alternatively, it may be that the image is not properly written. If the device
+fails to boot, put the SD card back in the host machine and write the image a
+second time. Verify the write was succesful by confirming that the following
+commands both return the same hash:
+
+    sudo dd if=/dev/sdX | head -c `stat -c "%s" spi-test_pi2.img` | pv -s `stat -c "%s" spi-test_pi2.img` | sha1sum
+    sha1sum spi-test_pi2.img
+
+
+## Creating a Product
+
+Now it is time to create your first product based on Ubuntu Core.
+
+A special type of snap called a "Gadget Snap" (sometimes refererred to as an "OEM Snap" in the current docs) is used to define the components of a device image, i.e. the software stack that powers your device. This includes specifying the base Ubuntu OS, the specific kernel to use, pre-installed applications, configuring hardware access, etc. See [Snappy OEM Documentation](https://developer.ubuntu.com/en/snappy/guides/oem/) for more details.
+
+Product Integration data is associated to a snap that you own, which may be either an Application or Gadget snap. 
+
+For the purposes of this tutorial, you will use the reference ```rpi2.canonical``` gadget snap, and define your product based on an Application snap that you will own. For a real product you would create and manage your own Gadget snap as well.
+
+
+### Publishing an Application Snap
+
+We will start by having you publish a copy of our test application snap under your own MyApps account.
+
+First, download the test snap:
+    wget https://spi.canonical.com/assets/test-snap_1_all.snap
+
+Then you need to publish the snap under your MyApps account, as follows:
+
+- visit https://myapps.developer.ubuntu.com/dev/click-apps/upload/
+- drag and drop or otherwise select the example snap for upload
+- select "rolling-core" as the release
+- select "Developer Tools" as the department
+- select "GNU GPL v3" as the license
+- select "stable" as the channel
+- click "Submit to the Ubuntu store"
+
+If all went well, you should now see a page for the package, named like "test-snap.<username>". It will also state "Package status is Ready to Publish".
+
+Note: when you have a separate Ubuntu One user for your Agents, this is where you would share access to the Product, by sharing this "primary snap" with the Agent user.
+
+Note: Currently only private packages can be shared; public package sharing is under development.
+
+Now click the "Publish" button on the test-snap.<username> MyApps page. You will then see details showing that sequence 1 is published in all channels.
+
+### Defining the Product
+
+So far we have published the application snap, and can build an image with it pre-installed on top of the reference Rasberry Pi Gadget snap. Now we want to codify that arrangement by defining a Product in the Ubuntu Product Integration system.
+
+For the purposes of Product Integration testing, we are only concerned about the list of snaps that compose the product and some additional metadata decribed here:
+
+-   ```name```: A short name to identify this Product, does not need to be unique.
+-   ```release```: the Ubuntu Core release this products aims for (currently "15.04-core" or "rolling-core" as known by the Store)
+-   ```base_image_reference```: optional (TODO: we should make it optional as it shouldn't be needed in all-snaps land anymore)
+-   ```snaps```: the list of snap names that that comprise the product
+
+TODO: instruct them to export SSO_USERNAME or have the agent login step do it.
+
+TODO: sidebar about the api_example script, oauth, etc.
+
+In the agent directory on your computer (where you already authenticated during
+the [Lab setup](#lab-setup) above), issue the following HTTP request using the api_example.py helper script included with the Agent distribution:
+
+    ./api_example.py -X POST config.ini https://spi.canonical.com/products --data '{
+        "name": "test-product.$SSO_USERNAME",
+        "primary_snap_name": "test-snap.$SSO_USERNAME",
+        "release": "rolling-core",
+        "snaps": [
+            "ubuntu-core.canonical",
+            "pi2.canonical",
+            "webdm.canonical",
+            "test-snap.$SSO_USERNAME"
+        ]
+    }'
+
+output will contain the ID newly created product definition. You will need to export this as an environment variable for use in later API calls as follows:
+
+    export PRODUCT_ID=<id from output above>
+
+NOTE/TODO: we are not yet in all-snaps land so there is no kernel snap listed here.
+
+
+## Testing
+
+### Defining a Test
+
+Next you will define your test scenario via a Test Specification, with the following fields:
+
+-   ```name```: a unique reference to identify tests and their results
+-   ```product_id```: the unique id for the Product to test
+-   ```channels```: a list of channel pairs. 'base' is the target/live channel generally, and 'update' is where the system will look for changes to trigger update test runs (covered in a later chapter).
+-   ```test_payload```: information for the test to be run; in this case we put where the tests can be downloaded, and the command to trigger their execution
+
+Execute the following command to create this Test Specification:
+
+    ./api_example.py -X POST config.ini https://spi.canonical.com/products/$PRODUCT_ID/tests/specs --data '{
+        "name": "Tutorial Example Test Specification",
+        "product_id": "$PRODUCT_ID",
+        "channels": [ ['base': 'stable', 'update': 'edge'] ],
+        "test_payload": {
+            "tarball_url": "https://spi.canonical.com/assets/test_tarball2.tar.gz",
+            "test_cmd": "./test"
+        }
+    }'
+
+At this point the system is configured for your product and tests, and will supervise the Store for new versions of the snaps listed in your Product.
+
+TODO: note about not being in all-snaps land yet.
+
+
+### First test run
+
+TODO: system should trigger an initial GM test when the TestSpec is created.
+
+This test will be what is called a Gold Master Candidate test. It is triggered by a change to any of the component snaps in the monitored base channel defined in the Test Specification (stable in this case). The server will generate a Test Opportunity that instructs the provisioning kit script to compose a new GM candidate image from the Product's snaps, using the current sequences published in the base channel, provision that image, and then run tests against it. Later chapters of the tutorial will discuss additional testing scenarios.
+
+When we first defined our Test Specification, the system generated a new Test Opportunity to test the initial Gold Master Candidate image, since that combination of snaps had not yet been tested.
+
+Further GM tests will be automatically generated each time a component snap is updated in the base_channel.
+
+You can now run the agent to consume and run your first test. Run the following command:
+
+    ./agent.py -v --once config.ini
+
+As remote flashing of the Raspberry Pi is still in active development, the
+Raspberry Pi will need to be set up manually. After receiving the test to run,
+the Agent will output instructions explaining how to write the composed image
+to the device. Please follow those instructions and let the Agent know when
+you are ready by pressing the Enter key when prompted.
+
+After executing the test, the Agent will automatically retrieve the
+results and send them back to Canonical's servers. You can query these results
+in the next section, [Checking test progress and results](#checking-test-progress-and-results).
+
+
+### Checking test progress and results
+
+At any time, you can check the progress of tests generated in the
+system:
+
+TODO: we need a way to get our test opp id now that we are not manually triggering
+
+    ./api_example.py config.ini \
+    https://spi.canonical.com/products/$PRODUCT_ID/tests/events?test_opportunity_id=<test_opportunity_id>
+
+This produces a large JSON document containing all the events that relate to this test, for example:
+
+    {"event_logs": [
+        {
+        "updated_at": "2015-10-28T18:35:38.197000+00:00",
+        "events": [
+            {
+            "timestamp": "2015-10-28T18:34:56.614000+00:00",
+            "event_type": "QUEUED_FOR_AGENTS"
+            },
+            {
+            "platform": "spi-tutorial-raspi2",
+            "timestamp": "2015-10-28T18:34:58.988000+00:00",
+            "agent_group": "default",
+            "agent_id": "special",
+            "event_type": "PICKED_BY_AGENT"
+            },
+
+The ```event_type``` field will describe what happened, there is a timestamp, and extra information related
+to the event.
+
+It will also contain all the information relevant to the test, such as:
+
+      "queued_at": "2015-10-28T18:34:56.614000+00:00",
+      "test_spec_id": "2991e1d1-ae89-4afe-9801-140207526f21",
+      "spec_name": "Tutorial Example Test Specification",
+      "image_unique_id": "5e6ff512-87b6-427e-917d-615c4d1c0aec",
+
+As well as the test itself in the ```test_opportunity``` field.
+
+You can also check the results for an specific image (you can get the image
+id from the response of the query when you told the system above that a
+new snap revision was available):
+
+    ./api_example.py config.ini \
+    https://spi.canonical.com/products/$PRODUCT_ID/tests/events?image_unique_id=<unique_image_id>
+
+Or you may directly poll for specific results using filters, as below:
+
+    ./api_example.py config.ini https://spi.canonical.com/products/$PRODUCT_ID/results... TODO
+
+The result query will return a document describing one or more test results.
+Each will contain, among other things, the pass or failure status of the test
+run, as well as all related information such as the manifest, test spec, and so
+on:
+
+    {
+    "test_results": [
+        {
+        "created_at": "2015-10-28T18:34:56.614000+00:00",
+        "result_id": "7348dac1-9191-454c-93f0-cedb463aa477",
+        "test_status": "PASSED",
+        "manifest": {
+        ...
+      "test_spec": {
+        "platform": "spi-tutorial-raspi2",
+        "image_name": "tutorial-raspi2-image",
+        "test_payload": {
+          "test_cmd": "./test",
+          "tarball_url": "https://spi.canonical.com/assets/test_tarball.tar.gz"
+        },
+        "name": "Tutorial Example Test Specification",
+        "id": "2991e1d1-ae89-4afe-9801-140207526f21",
+        "created_at": "2015-10-28T16:42:42.989056",
+        "active": true
+      },
+    ...
+
+### Artifacts and logs
+
+Logs containing the output and errors of the provisioning kit's scripts are always stored in ```logs/```.
+Saving those logs and any artifacts the provisioning kit creates is the responsibility of your test command.
+
+The example provisioning kit will create a tarball with the test opportunity, the test result
+and all the provisioning kit logs, using this code:
+
+    ARTIFACTS = [
+        'spi_test_result.json',
+        'spi_test_opportunity.json',
+        'logs/provisioning.log',
+        'logs/setup.log',
+        'logs/testing.log'
+    ]
+
+    [...]
+
+    artifact_dir = os.getenv('ARTIFACT_DIR', '/tmp')
+    save_cmd = 'tar czvf {} {}'.format(
+        os.path.join(artifact_dir, test_opp['id'] + '.tar.gz'),
+        ' '.join(ARTIFACTS),
+    )
+    log.info('Saving artifact with %s', save_cmd)
+    subprocess.check_call(save_cmd, shell=True)
+
+Here, the tarball is stored in /tmp by default, but you can override it in the ```[environment]``` section of your ```config.ini```:
+
+    [environment]
+    ARTIFACT_DIR='/home/user/artifacts'
+
+#### Generated GM Candidate images
+
+The images that are generated by the provisioning kit may be stored for later use. For example if tests pass and you would like to release the image you will want to retrieve the binary from the Agent host.
+
+The sample provisioning kit stores the images in a ./images subdirector of the agent installation. You are responsible for handling transport of the images to a storage system of your choice.
+
+Note: the sample kit does not remove old images and they can quickly eat a lot of disk space.
+
+## Update Testing
+
+Future revisions of this document will cover update testing scenarios allowing for verifying updates before new versions are made available to customers
+