Merge lp:~frankban/juju-quickstart/manual-provider into lp:juju-quickstart

[Francesco Banconi]

Add support for the manual provider.

Also reorder the provider fields so
that required ones come first.
The hope for this is to improve the
user experience while creating
environments.

Tests: `make check`.

QA: use quickstart as usual to set up
environments.
To test the manual provider, I've done
the following:

- I used a separate vmware virtual machine:
  let's assume its address is 192.168.1.5.
- Install the SSH server in the vm:
        `sudo apt-get install ssh`.
- Copy your ssh credential over:
        `scp -rp ~/.ssh 192.168.1.5:~`.
- Allow sudo power without password for your user on the vm, by
  editing the /etc/sudoers file and adding the following:
        $USER ALL=(ALL) NOPASSWD: ALL
  where $USER is your user name.
- Run `.venv/bin/python juju-quickstart -i`, select the option to
  create a new "Manual Provisioning" environment, select a name for it and
  use the vm address as the "bootstrap host", save the environment
  and then run it!
- wait for the environment to be ready, then run quickstart again
  with the same environment to check it recognizes it's
  already bootstrapped.
Done, thank you!

https://codereview.appspot.com/172410043/

[Francesco Banconi (frankban)]

Reviewers: mp+241595_code.launchpad.net,

Message:
Please take a look.

Description:
Add support for the manual provider.

Also reorder the provider fields so
that required ones come first.
The hope for this is to improve the
user experience while creating
environments.

Tests: `make check`.

QA: use quickstart as usual to set up
environments.
To test the manual provider, I've done
the following:

- I used a separate vmware virtual machine:
   let's assume its address is 192.168.1.5.
- Install the SSH server in the vm:
         `sudo apt-get install ssh`.
- Copy your ssh credential over:
         `scp -rp ~/.ssh 192.168.1.5:~`.
- Allow sudo power without password for your user on the vm, by
   editing the /etc/sudoers file and adding the following:
         $USER ALL=(ALL) NOPASSWD: ALL
   where $USER is your user name.
- Run `.venv/bin/python juju-quickstart -i`, select the option to
   create a new "Manual Provisioning" environment, select a name for it
and
   use the vm address as the "bootstrap host", save the environment
   and then run it!
- wait for the environment to be ready, then run quickstart again
   with the same environment to check it recognizes it's
   already bootstrapped.
Done, thank you!

https://code.launchpad.net/~frankban/juju-quickstart/manual-provider/+merge/241595

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/172410043/

Affected files (+145, -76 lines):
   A [revision details]
   M quickstart/models/envs.py
   M quickstart/tests/cli/test_views.py
   M quickstart/tests/models/test_envs.py

[Richard Harding (rharding)]

LGTM and will QA. Ideally it would be cool if our QA instructions were
using guimaas to just bring up notes and then hook them into juju.

https://codereview.appspot.com/172410043/diff/1/quickstart/models/envs.py
File quickstart/models/envs.py (right):

https://codereview.appspot.com/172410043/diff/1/quickstart/models/envs.py#newcode720
quickstart/models/envs.py:720: help='The host name of the machine where
the bootstrap '
name/ip address?

https://codereview.appspot.com/172410043/diff/1/quickstart/models/envs.py#newcode732
quickstart/models/envs.py:732: fields.IntField(
ugh, I think this is going away soon. I'll ping wallyworld and see if
it's in 1.21. If that's true it might be worth just waiting a few weeks
for 1.21 and skipping this.

https://codereview.appspot.com/172410043/

[Richard Harding (rharding)]

looks like the removal of provider storage won't be in 1.21 so ignore
that hope of not needing it.

https://codereview.appspot.com/172410043/

[Richard Harding (rharding)]

QA:

I used a digital ocean droplet

- created droplet with my ssh key
- had to ssh root@xxx to login and set an initial root password
- added the 'ocean' envirnoment using juju-quickstart -i
- failed at first as it said no tools for my environment so I retired
with --upload-tools
- that ran and worked, gui loaded up in a new window for me
- reran quickstart and it worked and loaded the gui window again for me.
Really pretty quick <3

So my one QA question is if we should always upload tools? I guess I'm
on 1.21-beta1 so maybe it's that vs the manual provider. Never mind.

https://codereview.appspot.com/172410043/

[Madison Scott-Clary (makyo)]

LGTM, no QA. THanks for this!

https://codereview.appspot.com/172410043/

[Francesco Banconi (frankban)]

Thanks for the reviews!

https://codereview.appspot.com/172410043/diff/1/quickstart/models/envs.py
File quickstart/models/envs.py (right):

https://codereview.appspot.com/172410043/diff/1/quickstart/models/envs.py#newcode720
quickstart/models/envs.py:720: help='The host name of the machine where
the bootstrap '
On 2014/11/12 22:08:24, rharding wrote:
> name/ip address?

Done.

https://codereview.appspot.com/172410043/

[lp:~frankban/juju-quickstart/manual-provider]

115. By Francesco Banconi on 2014-11-13

Fix typo.

[Francesco Banconi (frankban)]

*** Submitted:

Add support for the manual provider.

Also reorder the provider fields so
that required ones come first.
The hope for this is to improve the
user experience while creating
environments.

Tests: `make check`.

QA: use quickstart as usual to set up
environments.
To test the manual provider, I've done
the following:

- I used a separate vmware virtual machine:
   let's assume its address is 192.168.1.5.
- Install the SSH server in the vm:
         `sudo apt-get install ssh`.
- Copy your ssh credential over:
         `scp -rp ~/.ssh 192.168.1.5:~`.
- Allow sudo power without password for your user on the vm, by
   editing the /etc/sudoers file and adding the following:
         $USER ALL=(ALL) NOPASSWD: ALL
   where $USER is your user name.
- Run `.venv/bin/python juju-quickstart -i`, select the option to
   create a new "Manual Provisioning" environment, select a name for it
and
   use the vm address as the "bootstrap host", save the environment
   and then run it!
- wait for the environment to be ready, then run quickstart again
   with the same environment to check it recognizes it's
   already bootstrapped.
Done, thank you!

R=rharding, matthew.scott
CC=
https://codereview.appspot.com/172410043

https://codereview.appspot.com/172410043/

=== modified file 'quickstart/models/envs.py'
--- quickstart/models/envs.py	2014-11-11 11:10:33 +0000
+++ quickstart/models/envs.py	2014-11-13 08:56:33 +0000
@@ -456,9 +456,8 @@
         ),
         'fields': (
             provider_field,
+            # Required fields.
             name_field,
-            admin_secret_field,
-            default_series_field,
             fields.PasswordField(
                 'access-key', label='access key', required=True,
                 help='The access key to use to authenticate to AWS. '
@@ -475,12 +474,15 @@
                 'secret-key', label='secret key', required=True,
                 help='The secret key to use to authenticate to AWS. '
                      'See the access key help above.'),
-            fields.AutoGeneratedStringField(
-                'control-bucket', label='control bucket', required=False,
-                help='the globally unique S3 bucket name'),
+            # Optional fields.
             fields.ChoiceField(
                 'region', choices=ec2_regions, default='us-east-1',
                 label='region', required=False, help='the ec2 region to use'),
+            admin_secret_field,
+            default_series_field,
+            fields.AutoGeneratedStringField(
+                'control-bucket', label='control bucket', required=False,
+                help='the globally unique S3 bucket name'),
             is_default_field,
         ),
     }
@@ -496,20 +498,8 @@
         ),
         'fields': (
             provider_field,
+            # Required fields.
             name_field,
-            admin_secret_field,
-            default_series_field,
-            fields.BoolField(
-                'use-floating-ip', label='use floating IP', allow_mixed=False,
-                required=True,
-                help='Specifies whether the use of a floating IP address is '
-                     'required to give the nodes a public IP address. '
-                     'Some installations assign public IP addresses by '
-                     'default without requiring a floating IP address. '
-                     'For HP Public Cloud, floating IP must be activated.'),
-            fields.AutoGeneratedStringField(
-                'control-bucket', label='control bucket', required=False,
-                help='the globally unique swift bucket name'),
             fields.SuggestionsStringField(
                 'auth-url', label='authentication URL', required=True,
                 help='The Keystone URL to use in the authentication process. '
@@ -521,11 +511,20 @@
                 help='The OpenStack tenant name. For HP Public Cloud, this is '
                      'listed as the project name on the '
                      'https://horizon.hpcloud.com/landing/ page.'),
+            fields.BoolField(
+                'use-floating-ip', label='use floating IP', allow_mixed=False,
+                required=True,
+                help='Specifies whether the use of a floating IP address is '
+                     'required to give the nodes a public IP address. '
+                     'Some installations assign public IP addresses by '
+                     'default without requiring a floating IP address. '
+                     'For HP Public Cloud, floating IP must be activated.'),
             fields.SuggestionsStringField(
                 'region', label='region', required=True,
                 help='The OpenStack region to use. '
                      'For HP Public Cloud, use one of the following:',
                 suggestions=hp_regions),
+            # Optional fields.
             fields.ChoiceField(
                 'auth-mode', label='authentication mode', required=False,
                 default='userpass', choices=('userpass', 'keypair'),
@@ -549,6 +548,11 @@
             fields.PasswordField(
                 'secret-key', label='secret key', required=False,
                 help='the secret key to use for the keypair authentication'),
+            fields.AutoGeneratedStringField(
+                'control-bucket', label='control bucket', required=False,
+                help='the globally unique swift bucket name'),
+            admin_secret_field,
+            default_series_field,
             is_default_field,
         ),
     }
@@ -564,12 +568,8 @@
         ),
         'fields': (
             provider_field,
+            # Required fields.
             name_field,
-            admin_secret_field,
-            default_series_field,
-            fields.ChoiceField(
-                'location', choices=azure_locations, label='location',
-                required=True, help='the region to use'),
             fields.StringField(
                 'management-subscription-id', required=True,
                 label='management subscription ID',
@@ -590,6 +590,12 @@
                      'https://manage.windowsazure.com (Storage). '
                      'You must create the storage account in the same '
                      'region/location specified by the location key value.'),
+            fields.ChoiceField(
+                'location', choices=azure_locations, label='location',
+                required=True, help='the region to use'),
+            # Optional fields.
+            admin_secret_field,
+            default_series_field,
             is_default_field,
         ),
     }
@@ -606,8 +612,7 @@
         'fields': (
             provider_field,
             name_field,
-            admin_secret_field,
-            default_series_field,
+            # SDC fields.
             fields.StringField(
                 'sdc-user', required=True,
                 label='SDC user name',
@@ -622,6 +627,7 @@
                 'sdc-url', choices=joyent_sdc_urls, label='SDC URL',
                 required=False, default=joyent_sdc_urls[0],
                 help='the region URL to use for SDC'),
+            # Manta fields.
             fields.StringField(
                 'manta-user', required=True,
                 label='manta user name',
@@ -637,6 +643,7 @@
                 'manta-url', label='manta URL', required=False,
                 default='https://us-east.manta.joyent.com',
                 help='the region URL to use for manta'),
+            # Optional fields.
             fields.FilePathField(
                 'private-key-path', label='private key path', required=False,
                 default='~/.ssh/id_rsa',
@@ -644,6 +651,8 @@
             fields.StringField(
                 'algorithm', label='SSH algorithm', required=False,
                 default='rsa-sha256'),
+            admin_secret_field,
+            default_series_field,
             is_default_field,
         ),
     }
@@ -662,6 +671,7 @@
         ),
         'fields': (
             provider_field,
+            # Required fields.
             name_field,
             fields.StringField(
                 'maas-server', label='server address', required=True,
@@ -674,6 +684,7 @@
                      'link to the account preferences is accessible from the '
                      'drop-down menu that appears when clicking your user '
                      'name at the top-right of the page.'),
+            # Optional fields.
             fields.FilePathField(
                 'authorized-keys-path', required=False,
                 label='SSH public key file',
@@ -684,6 +695,49 @@
             is_default_field,
         ),
     }
+    env_type_db['manual'] = {
+        'label': 'Manual Provisioning',
+        'description': (
+            'Juju provides a feature called "Manual Provisioning" that '
+            'enables you to deploy Juju, and charms, to existing systems. '
+            'This is useful if you have groups of machines that you want to '
+            "use for Juju but don't want to add the complexity of a new "
+            'OpenStack or MAAS setup. It is also useful as a means of '
+            'deploying workloads to VPS providers and other cheap hosting '
+            'options. Manual provisioning enables you to run Juju on systems '
+            'that have a supported operating system installed. You will need '
+            'to ensure that you have both SSH access and sudo rights. '
+            'See https://juju.ubuntu.com/docs/config-manual.html for more '
+            'details on the manual provider configuration, including its '
+            'caveats and limitattions.'
+        ),
+        'fields': (
+            provider_field,
+            # Required fields.
+            name_field,
+            fields.StringField(
+                'bootstrap-host', label='bootstrap host', required=True,
+                help='The host name/IP address of the machine where the '
+                     'bootstrap machine agent will be started.'),
+            # Optional fields.
+            fields.StringField(
+                'bootstrap-user', label='bootstrap user', required=False,
+                help='The user name to authenticate as when connecting to the '
+                     'bootstrap machine. It defaults to the current user.'),
+            fields.StringField(
+                'storage-listen-ip', label='storage listen IP', required=False,
+                help="The IP address that the bootstrap machine's Juju "
+                     'storage server will listen on. By default, storage will '
+                     'be served on all network interfaces.'),
+            fields.IntField(
+                'storage-port', min_value=1, max_value=65535,
+                label='storage port', required=False, default=8040,
+                help="The TCP port that the bootstrap machine's Juju storage "
+                     'server will listen on.'),
+            admin_secret_field,
+            is_default_field,
+        ),
+    }
     env_type_db['local'] = {
         'label': 'local (LXC)',
         'description': (
@@ -695,25 +749,7 @@
         'fields': (
             provider_field,
             name_field,
-            admin_secret_field,
-            default_series_field,
-            fields.StringField(
-                'root-dir', label='root dir', required=False,
-                default='~/.juju/local',
-                help='the directory that is used for the storage files'),
-            fields.IntField(
-                'storage-port', min_value=1, max_value=65535,
-                label='storage port', required=False, default=8040,
-                help='override if you have multiple local providers, '
-                     'or if the default port is used by another program'),
-            fields.IntField(
-                'shared-storage-port', min_value=1, max_value=65535,
-                label='shared storage port', required=False, default=8041,
-                help='override if you have multiple local providers, '
-                     'or if the default port is used by another program'),
-            fields.StringField(
-                'network-bridge', label='network bridge', required=False,
-                default='lxcbr0', help='the LXC bridge interface to use'),
+            # LXC related fields.
             fields.BoolField(
                 'lxc-clone', label='use lxc-clone', required=False,
                 help='Use lxc-clone to create the containers used as '
@@ -725,6 +761,26 @@
                      "that there are some situations where aufs doesn't "
                      'entirely behave as intuitively as one might expect. '
                      'If not specified, aufs is not used.'),
+            fields.StringField(
+                'root-dir', label='root dir', required=False,
+                default='~/.juju/local',
+                help='the directory that is used for the storage files'),
+            fields.IntField(
+                'storage-port', min_value=1, max_value=65535,
+                label='storage port', required=False, default=8040,
+                help='override if you have multiple local providers, '
+                     'or if the default port is used by another program'),
+            fields.IntField(
+                'shared-storage-port', min_value=1, max_value=65535,
+                label='shared storage port', required=False, default=8041,
+                help='override if you have multiple local providers, '
+                     'or if the default port is used by another program'),
+            fields.StringField(
+                'network-bridge', label='network bridge', required=False,
+                default='lxcbr0', help='the LXC bridge interface to use'),
+            # Other optional fields.
+            admin_secret_field,
+            default_series_field,
             is_default_field,
         ),
     }

=== modified file 'quickstart/tests/cli/test_views.py'
--- quickstart/tests/cli/test_views.py	2014-11-10 14:11:11 +0000
+++ quickstart/tests/cli/test_views.py	2014-11-13 08:56:33 +0000
@@ -706,14 +706,14 @@
         expected_form_contents = [
             ('provider type: ', 'local'),
             ('environment name: ', 'lxc'),
-            ('admin secret: ', 'bones'),
-            ('default series: ', 'raring'),
+            ('use lxc-clone', forms.MIXED),
+            ('use lxc-clone-aufs', forms.MIXED),
             ('root dir: ', ''),
             ('storage port: ', '8888'),
             ('shared storage port: ', ''),
             ('network bridge: ', ''),
-            ('use lxc-clone', forms.MIXED),
-            ('use lxc-clone-aufs', forms.MIXED),
+            ('admin secret: ', 'bones'),
+            ('default series: ', 'raring'),
             ('default', True),
         ]
         self.assertEqual(expected_form_contents, self.get_form_contents())

=== modified file 'quickstart/tests/models/test_envs.py'
--- quickstart/tests/models/test_envs.py	2014-11-11 13:21:57 +0000
+++ quickstart/tests/models/test_envs.py	2014-11-13 08:56:33 +0000
@@ -699,10 +699,9 @@
         self.assertIn('local', self.env_type_db)
         env_metadata = self.env_type_db['local']
         expected = [
-            'type', 'name', 'admin-secret', 'default-series', 'root-dir',
+            'type', 'name', 'lxc-clone', 'lxc-clone-aufs', 'root-dir',
             'storage-port', 'shared-storage-port', 'network-bridge',
-            'lxc-clone', 'lxc-clone-aufs',
-            'is-default']
+            'admin-secret', 'default-series', 'is-default']
         expected_required = ['type', 'name', 'is-default']
         self.assert_fields(expected, env_metadata)
         self.assert_required_fields(expected_required, env_metadata)
@@ -712,8 +711,8 @@
         self.assertIn('ec2', self.env_type_db)
         env_metadata = self.env_type_db['ec2']
         expected = [
-            'type', 'name', 'admin-secret', 'default-series', 'access-key',
-            'secret-key', 'control-bucket', 'region', 'is-default']
+            'type', 'name', 'access-key', 'secret-key', 'region',
+            'admin-secret', 'default-series', 'control-bucket',  'is-default']
         expected_required = [
             'type', 'name', 'access-key', 'secret-key',
             'is-default']
@@ -725,13 +724,13 @@
         self.assertIn('openstack', self.env_type_db)
         env_metadata = self.env_type_db['openstack']
         expected = [
-            'type', 'name', 'admin-secret', 'default-series',
-            'use-floating-ip', 'control-bucket', 'auth-url', 'tenant-name',
+            'type', 'name', 'auth-url', 'tenant-name', 'use-floating-ip',
             'region', 'auth-mode', 'username', 'password', 'access-key',
-            'secret-key', 'is-default']
+            'secret-key', 'control-bucket', 'admin-secret', 'default-series',
+            'is-default']
         expected_required = [
-            'type', 'name', 'use-floating-ip',
-            'auth-url', 'tenant-name', 'region', 'is-default']
+            'type', 'name', 'auth-url', 'tenant-name', 'use-floating-ip',
+            'region', 'is-default']
         self.assert_fields(expected, env_metadata)
         self.assert_required_fields(expected_required, env_metadata)
 
@@ -740,13 +739,13 @@
         self.assertIn('azure', self.env_type_db)
         env_metadata = self.env_type_db['azure']
         expected = [
-            'type', 'name', 'admin-secret', 'default-series', 'location',
-            'management-subscription-id', 'management-certificate-path',
-            'storage-account-name', 'is-default']
+            'type', 'name', 'management-subscription-id',
+            'management-certificate-path', 'storage-account-name', 'location',
+            'admin-secret', 'default-series', 'is-default']
         expected_required = [
-            'type', 'name', 'location',
-            'management-subscription-id', 'management-certificate-path',
-            'storage-account-name', 'is-default']
+            'type', 'name', 'management-subscription-id',
+            'management-certificate-path', 'storage-account-name', 'location',
+            'is-default']
         self.assert_fields(expected, env_metadata)
         self.assert_required_fields(expected_required, env_metadata)
 
@@ -755,10 +754,9 @@
         self.assertIn('joyent', self.env_type_db)
         env_metadata = self.env_type_db['joyent']
         expected = [
-            'type', 'name', 'admin-secret', 'default-series',
-            'sdc-user', 'sdc-key-id', 'sdc-url',
-            'manta-user', 'manta-key-id', 'manta-url',
-            'private-key-path', 'algorithm', 'is-default']
+            'type', 'name', 'sdc-user', 'sdc-key-id', 'sdc-url', 'manta-user',
+            'manta-key-id', 'manta-url', 'private-key-path', 'algorithm',
+            'admin-secret', 'default-series', 'is-default']
         expected_required = [
             'type', 'name', 'sdc-user', 'sdc-key-id', 'manta-user',
             'manta-key-id', 'is-default']
@@ -778,6 +776,18 @@
         self.assert_fields(expected, env_metadata)
         self.assert_required_fields(expected_required, env_metadata)
 
+    def test_manual_environment(self):
+        # The manual environment metadata includes the expected fields.
+        self.assertIn('manual', self.env_type_db)
+        env_metadata = self.env_type_db['manual']
+        expected = [
+            'type', 'name', 'bootstrap-host', 'bootstrap-user',
+            'storage-listen-ip', 'storage-port', 'admin-secret', 'is-default']
+        expected_required = [
+            'type', 'name', 'bootstrap-host', 'is-default']
+        self.assert_fields(expected, env_metadata)
+        self.assert_required_fields(expected_required, env_metadata)
+
 
 class TestGetSupportedEnvTypes(unittest.TestCase):
 
@@ -793,6 +803,7 @@
             ('azure', 'Windows Azure'),
             ('joyent', 'Joyent'),
             ('maas', 'MAAS (bare metal)'),
+            ('manual', 'Manual Provisioning'),
             ('local', 'local (LXC)'),
         ]
         obtained_env_types = envs.get_supported_env_types(self.env_type_db)
@@ -852,14 +863,14 @@
         return [
             ('type', 'local'),
             ('name', 'lxc'),
-            ('admin-secret', 'Secret!'),
-            ('default-series', 'saucy'),
+            ('lxc-clone', False),
+            ('lxc-clone-aufs', False),
             ('root-dir', '/my/juju/local/'),
             ('storage-port', 4242),
             ('shared-storage-port', 4747),
             ('network-bridge', 'lxcbr1'),
-            ('lxc-clone', False),
-            ('lxc-clone-aufs', False),
+            ('admin-secret', 'Secret!'),
+            ('default-series', 'saucy'),
             ('is-default', True),
         ]
 
@@ -874,14 +885,14 @@
         expected = [
             ('type', 'local'),
             ('name', 'lxc'),
-            ('admin-secret', None),
-            ('default-series', None),
+            ('lxc-clone', None),
+            ('lxc-clone-aufs', None),
             ('root-dir', None),
             ('storage-port', None),
             ('shared-storage-port', None),
             ('network-bridge', None),
-            ('lxc-clone', None),
-            ('lxc-clone-aufs', None),
+            ('admin-secret', None),
+            ('default-series', None),
             ('is-default', None),
         ]
         env_data = {'type': 'local', 'name': 'lxc'}