Merge lp:~frankban/juju-quickstart/joyent into lp:juju-quickstart

[Francesco Banconi]

Add support for the Joyent provider.

Tests: `make check`.

QA:
QA is not straightforward for this branch, due to the
fact this needs to be tested on a not yet released
juju-core. Sorry about that.
1) Check out the 1.18 juju-core branch (lp:juju-core/1.18),
   and compile it.
2) Edit the quickstart/settings.py file included in this branch:
   set `JUJU_CMD` to point to the juju 1.18 path.
3) Sign up for a Joyent account from http://www.joyent.com/.
4) Run `.venv/bin/python juju-quickstart -i` and create a new
   joyent environment: DO NOT bootstrap it from quickstart, just
   create it and exit (^X).
5) Use juju 1.18 to bootstrap the joyent account, e.g.
   `juju bootstrap -e joyent --upload-tools`. This is required
   because quickstart does not support uploading tools.
6) After a minute, and while the environment is bootstrapping,
   open another terminal and re-run quickstart, select the
   joyent account you created and hit "use"
   (or just invoke quickstart with `-e joyent`).
7) Ensure everything proceeds as expected, and wait
   until the quickstart opens the GUI.
Done, destroy the environment, thank you!

https://codereview.appspot.com/83880044/

[Francesco Banconi (frankban)]

Reviewers: mp+214022_code.launchpad.net,

Message:
Please take a look.

Description:
Add support for the Joyent provider.

Tests: `make check`.

QA:
QA is not straightforward for this branch, due to the
fact this needs to be tested on a not yet released
juju-core. Sorry about that.
1) Check out the 1.18 juju-core branch (lp:juju-core/1.18),
    and compile it.
2) Edit the quickstart/settings.py file included in this branch:
    set `JUJU_CMD` to point to the juju 1.18 path.
3) Sign up for a Joyent account from http://www.joyent.com/.
4) Run `.venv/bin/python juju-quickstart -i` and create a new
    joyent environment: DO NOT bootstrap it from quickstart, just
    create it and exit (^X).
5) Use juju 1.18 to bootstrap the joyent account, e.g.
    `juju bootstrap -e joyent --upload-tools`. This is required
    because quickstart does not support uploading tools.
6) After a minute, and while the environment is bootstrapping,
    open another terminal and re-run quickstart, select the
    joyent account you created and hit "use"
    (or just invoke quickstart with `-e joyent`).
7) Ensure everything proceeds as expected, and wait
    until the quickstart opens the GUI.
Done, destroy the environment, thank you!

https://code.launchpad.net/~frankban/juju-quickstart/joyent/+merge/214022

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/83880044/

Affected files (+117, -4 lines):
   A [revision details]
   M quickstart/models/envs.py
   M quickstart/models/fields.py
   M quickstart/tests/models/test_envs.py
   M quickstart/tests/models/test_fields.py

[Brad Crittenden (bac)]

Code LGTM. On to QA.

https://codereview.appspot.com/83880044/diff/1/quickstart/tests/models/test_fields.py
File quickstart/tests/models/test_fields.py (right):

https://codereview.appspot.com/83880044/diff/1/quickstart/tests/models/test_fields.py#newcode468
quickstart/tests/models/test_fields.py:468:
field.validate(unicode(__file__))
Nice use of __file__. Nicer then creating a temp file and destroying
it.

https://codereview.appspot.com/83880044/

[Brad Crittenden (bac)]

When doing QA and bootstrapping the joyent environment I get

http://paste.ubuntu.com/7199671/

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py
File quickstart/models/envs.py (right):

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py#newcode621
quickstart/models/envs.py:621: 'manta-user', required=True,
I'm running quickstart for the first time to set up joyent. I don't
know what the manta username and ssh keys are. Are they the same as my
SDC ones? I only have one Joyent username and ssh key.

Kind of confusing.

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py#newcode634
quickstart/models/envs.py:634: 'private-key-path', label='provate key
path', required=False,
typo: provate

https://codereview.appspot.com/83880044/

[lp:~frankban/juju-quickstart/joyent]

67. By Francesco Banconi on 2014-04-03

Fix typo.

[Brad Crittenden (bac)]

For the private key file path, the default is listed as ~/.ssh/id_rsa

If you try to enter '~/.ssh/id_rsa_joyent', quickstart does not properly
expand the path and gives an error. Using
'/home/bac/.ssh/id_rsa_joyent' was accepted.

https://codereview.appspot.com/83880044/

[Brad Crittenden (bac)]

QA-OK after lots of struggles, mainly the issue with
passphrase-protected ssh key, which is not quickstart's problem.

Though if juju doesn't screen for it perhaps we should?

https://codereview.appspot.com/83880044/

[Francesco Banconi (frankban)]

On 2014/04/03 20:06:11, bac wrote:
> For the private key file path, the default is listed as ~/.ssh/id_rsa

> If you try to enter '~/.ssh/id_rsa_joyent', quickstart does not
properly expand
> the path and gives an error. Using '/home/bac/.ssh/id_rsa_joyent' was
accepted.

Good catch. Fixed the FilePathField validation to use expanduser.

https://codereview.appspot.com/83880044/

[Francesco Banconi (frankban)]

On 2014/04/03 20:22:37, bac wrote:
> QA-OK after lots of struggles, mainly the issue with
passphrase-protected ssh
> key, which is not quickstart's problem.

> Though if juju doesn't screen for it perhaps we should?

Thanks a lot Brad, and sorry again for the difficult QA.
As you mentioned, this is something that must be handled by juju-core,
so, given this will land later, I'd be inclined to wait for the core
fix.

https://codereview.appspot.com/83880044/

[lp:~frankban/juju-quickstart/joyent]

68. By Francesco Banconi on 2014-04-04

Changes as per review.

[Francesco Banconi (frankban)]

Please take a look.

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py
File quickstart/models/envs.py (right):

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py#newcode621
quickstart/models/envs.py:621: 'manta-user', required=True,
On 2014/04/03 17:23:50, bac wrote:
> I'm running quickstart for the first time to set up joyent. I don't
know what
> the manta username and ssh keys are. Are they the same as my SDC
ones? I only
> have one Joyent username and ssh key.

> Kind of confusing.

Good point Brad, updated the fields description.

https://codereview.appspot.com/83880044/diff/1/quickstart/models/envs.py#newcode634
quickstart/models/envs.py:634: 'private-key-path', label='provate key
path', required=False,
On 2014/04/03 17:23:50, bac wrote:
> typo: provate

Done.

https://codereview.appspot.com/83880044/

[Brad Crittenden (bac)]

Changes all LGTM. Thanks.

https://codereview.appspot.com/83880044/

[lp:~frankban/juju-quickstart/joyent]

69. By Francesco Banconi on 2014-04-10

Merged trunk.

[Francesco Banconi (frankban)]

*** Submitted:

Add support for the Joyent provider.

Tests: `make check`.

QA:
QA is not straightforward for this branch, due to the
fact this needs to be tested on a not yet released
juju-core. Sorry about that.
1) Check out the 1.18 juju-core branch (lp:juju-core/1.18),
    and compile it.
2) Edit the quickstart/settings.py file included in this branch:
    set `JUJU_CMD` to point to the juju 1.18 path.
3) Sign up for a Joyent account from http://www.joyent.com/.
4) Run `.venv/bin/python juju-quickstart -i` and create a new
    joyent environment: DO NOT bootstrap it from quickstart, just
    create it and exit (^X).
5) Use juju 1.18 to bootstrap the joyent account, e.g.
    `juju bootstrap -e joyent --upload-tools`. This is required
    because quickstart does not support uploading tools.
6) After a minute, and while the environment is bootstrapping,
    open another terminal and re-run quickstart, select the
    joyent account you created and hit "use"
    (or just invoke quickstart with `-e joyent`).
7) Ensure everything proceeds as expected, and wait
    until the quickstart opens the GUI.
Done, destroy the environment, thank you!

R=bac
CC=
https://codereview.appspot.com/83880044

https://codereview.appspot.com/83880044/

=== modified file 'quickstart/models/envs.py'
--- quickstart/models/envs.py	2014-04-04 14:03:16 +0000
+++ quickstart/models/envs.py	2014-04-10 09:23:37 +0000
@@ -418,6 +418,13 @@
     azure_locations = (
         'East US', 'West US', 'West Europe', 'North Europe',
         'Southeast Asia', 'East Asia')
+    # The first URL is used as the default one.
+    joyent_sdc_urls = (
+        'https://us-west-1.api.joyentcloud.com',
+        'https://us-east-1.api.joyentcloud.com',
+        'https://us-sw-1.api.joyentcloud.com',
+        'https://us-ams-1.api.joyentcloud.com',
+    )
     # Define the env_type_db dictionary: this is done inside this function in
     # order to avoid instantiating fields at import time.
     # This is an ordered dict so that views can expose options to create new
@@ -566,7 +573,7 @@
                 label='management subscription ID',
                 help='this information can be retrieved from '
                      'https://manage.windowsazure.com (Settings)'),
-            fields.StringField(
+            fields.FilePathField(
                 'management-certificate-path', required=True,
                 label='management certificate path',
                 help='the path to the pem file associated to the certificate '
@@ -584,6 +591,58 @@
             is_default_field,
         ),
     }
+    env_type_db['joyent'] = {
+        'label': 'Joyent',
+        'description': (
+            'The joyent provider enables you to run Juju on the Joyent cloud. '
+            'This process requires you to have a Joyent account. If you have '
+            'not signed up for one yet, it can obtained at '
+            'http://www.joyent.com/.'
+        ),
+        'fields': (
+            provider_field,
+            name_field,
+            admin_secret_field,
+            default_series_field,
+            fields.StringField(
+                'sdc-user', required=True,
+                label='SDC user name',
+                help='The Smart Data Center account name. '
+                     'This is usually your Joyent user name.'),
+            fields.StringField(
+                'sdc-key-id', required=True,
+                label='SDC key id',
+                help='the SSH public key fingerprint used to connect to the '
+                     'Smart Data Center'),
+            fields.ChoiceField(
+                'sdc-url', choices=joyent_sdc_urls, label='SDC URL',
+                required=False, default=joyent_sdc_urls[0],
+                help='the region URL to use for SDC'),
+            fields.StringField(
+                'manta-user', required=True,
+                label='manta user name',
+                help='The Manta Storage Service account name. '
+                     'This is usually your Joyent user name.'),
+            fields.StringField(
+                'manta-key-id', required=True,
+                label='manta key id',
+                help='The SSH public key fingerprint used to connect to the '
+                     'Manta Storage Service. This is usually the same SSH '
+                     'fingerprint provided as SDC key id.'),
+            fields.StringField(
+                'manta-url', label='manta URL', required=False,
+                default='https://us-east.manta.joyent.com',
+                help='the region URL to use for manta'),
+            fields.FilePathField(
+                'private-key-path', label='private key path', required=False,
+                default='~/.ssh/id_rsa',
+                help='the path to the SSH private key'),
+            fields.StringField(
+                'algorithm', label='SSH algorithm', required=False,
+                default='rsa-sha256'),
+            is_default_field,
+        ),
+    }
     env_type_db['local'] = {
         'label': 'local (LXC)',
         'description': (

=== modified file 'quickstart/models/fields.py'
--- quickstart/models/fields.py	2014-01-14 14:42:20 +0000
+++ quickstart/models/fields.py	2014-04-10 09:23:37 +0000
@@ -30,6 +30,7 @@
 
 from __future__ import unicode_literals
 
+import os
 import uuid
 
 
@@ -293,6 +294,24 @@
             )
 
 
+class FilePathField(StringField):
+    """A string field storing a path to an existing file."""
+
+    def validate(self, value):
+        """Check the field is set if required.
+
+        If the field is set, also check that the file exists.
+        """
+        # The parent field ensures the value is set if required.
+        super(FilePathField, self).validate(value)
+        value = self.normalize(value)
+        # If the value is not set, just return without validating the path.
+        if value is None:
+            return
+        if not os.path.isfile(os.path.expanduser(value)):
+            raise ValueError('file not found in the specified path')
+
+
 class SuggestionsStringField(StringField):
     """A string field storing possible value suggestions."""
 

=== modified file 'quickstart/tests/models/test_envs.py'
--- quickstart/tests/models/test_envs.py	2014-01-28 20:35:42 +0000
+++ quickstart/tests/models/test_envs.py	2014-04-10 09:23:37 +0000
@@ -780,6 +780,21 @@
         self.assert_fields(expected, env_metadata)
         self.assert_required_fields(expected_required, env_metadata)
 
+    def test_joyent_environment(self):
+        # The joyent environment metadata includes the expected fields.
+        self.assertIn('joyent', self.env_type_db)
+        env_metadata = self.env_type_db['joyent']
+        expected = [
+            'type', 'name', 'admin-secret', 'default-series',
+            'sdc-user', 'sdc-key-id', 'sdc-url',
+            'manta-user', 'manta-key-id', 'manta-url',
+            'private-key-path', 'algorithm', 'is-default']
+        expected_required = [
+            'type', 'name', 'sdc-user', 'sdc-key-id', 'manta-user',
+            'manta-key-id', 'is-default']
+        self.assert_fields(expected, env_metadata)
+        self.assert_required_fields(expected_required, env_metadata)
+
 
 class TestGetSupportedEnvTypes(unittest.TestCase):
 
@@ -790,6 +805,7 @@
             ('ec2', 'Amazon EC2'),
             ('openstack', 'OpenStack (or HP Public Cloud)'),
             ('azure', 'Windows Azure'),
+            ('joyent', 'Joyent'),
             ('local', 'local (LXC)'),
         ]
         obtained_env_types = envs.get_supported_env_types(env_type_db)

=== modified file 'quickstart/tests/models/test_fields.py'
--- quickstart/tests/models/test_fields.py	2014-01-14 15:46:08 +0000
+++ quickstart/tests/models/test_fields.py	2014-04-10 09:23:37 +0000
@@ -19,8 +19,11 @@
 from __future__ import unicode_literals
 
 from contextlib import contextmanager
+import os
 import unittest
 
+import mock
+
 from quickstart.models import fields
 from quickstart.tests import helpers
 
@@ -457,6 +460,37 @@
             field.validate(None)
 
 
+class TestFilePathField(TestStringField):
+
+    field_class = fields.FilePathField
+
+    def test_validation_success(self):
+        # The validation succeeds if the value is set and the path exists.
+        field = self.field_class('path')
+        with self.assert_not_raises(ValueError):
+            field.validate(unicode(__file__))
+
+    def test_validation_expanduser(self):
+        # The ~ construct is properly expanded.
+        field = self.field_class('path')
+        basename, filename = os.path.split(unicode(__file__))
+        with self.assert_not_raises(ValueError):
+            with mock.patch('os.environ', {'HOME': basename}):
+                field.validate('~/{}'.format(filename))
+
+    def test_validation_error_file_not_found(self):
+        # A ValueError is raised if the path does not exist.
+        field = self.field_class('path')
+        with self.assert_value_error('file not found in the specified path'):
+            field.validate('__no-such-file__')
+
+    def test_validation_error_directory(self):
+        # A ValueError is raised if the path is a directory.
+        field = self.field_class('path')
+        with self.assert_value_error('file not found in the specified path'):
+            field.validate('/tmp')
+
+
 class TestSuggestionsStringField(TestStringField):
 
     field_class = fields.SuggestionsStringField