Merge lp:~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support into lp:~chad.smith/charms/precise/block-storage-broker/trunk

[Chad Smith]

This branch does a couple things:
 1. stops the block-storage-broker from importing euca2ools.commands.euca.describevolumes and euca2ools.commands.euca.describeinstances directly to avoid being affected by internal class/module changes on next major euca release. Instead block-storage-broker sets up a couple of simple parse_ec2* functions to help parsing the output of euca-describe-instances, euca-describe-volumes and euca-describe-tags commands. The euca response types supported by the parse_ec2* functions are INSTANCE, ATTACHMENT, VOLUME and TAG.

 2. To handle the fact that euca-2.X doesn't report volume TAG responses in the euca-describe-volumes command output, _ec2_describe_volumes also calls _ec2_describe_tags to augment any volume tag data (which we use to label volumes in EC2).

 3. In using euca-describe-tags to augment existing volume data I found that tags can be orphaned once a volume is removed by an admin via euca-delete-volume, if that is the case we will ignore tags for volumes that don't exist and log a message

 4. In unit tests drop all MockEuca* classes as they are no longer needed because we mock the subprocess.check_output results for the euca-* commands we run.

By parsing the output from the euca-* commands, we shield ourselves from the internal changes in class structure definition in future euca releases. This branch adds very basic validation of euca-describe-instance output. We can add more validation in separate branches as I didn't want this MP to grow too large.

[lp:~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support]

87. By Chad Smith on 2014-03-21

merge fixes from bsb-ec2-support and resolve conflicts + lint

88. By Chad Smith on 2014-03-21

revert Makefile change

89. By Chad Smith on 2014-03-21

Ignore any orphaned TAG lines from euca-describe-tags that don't map to current existing volume-ids

[Chad Smith (chad.smith)]

Here's a deployer bundle to deploy.
rm -rf ./trusty wherever you are about to juju-deployer from to ensure you don't use cached charm checkouts

cat block-storage-trusty.yaml

common:
    services:
        storage:
            branch: lp:~chad.smith/charms/precise/storage/storage-volume-label-availability-zone
            options:
                provider: block-storage-broker
                volume_size: 9
        block-storage-broker-ec2:
            branch: lp:~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support
            options:
                provider: ec2
                key: <YOUR_EC2_ACCESS_KEY>
                endpoint: <YOUR_EC2_URL>
                secret: <YOUR_EC2_SECRET_KEY>
        postgresql:
            branch: lp:~chad.smith/charms/precise/postgresql/postgresql-using-storage-subordinate
            constraints: mem=2048
            options:
                extra-packages: python-apt postgresql-contrib postgresql-9.3-debversion
                max_connections: 500

doit:
    inherits: common
    series: trusty
    relations:
        - [postgresql, storage]
        - [storage, block-storage-broker-ec2]

juju-deployer -c block-storage-trusty.yaml doit

[Chad Smith (chad.smith)]

The bundle above works from the w/ juju 1.17.6 be wary of 1.17. < 6 as it suffers from bug 1285901

[Alberto Donato (ack)]

As discussed on IRC, I think the code could be refactored to reduce duplication, for example as in https://pastebin.canonical.com/107195/.
Tests could then mock _run_command rather than subprocess.check_output.

It should be even possible to perform the line.split("\n") in the _run_command method, returning a list of lists (so the parse_* methods don't need to perform it), I'm not sure if it fits all the use cases, though.

#1:
+ data["device"] = ""
+ data["volume_label"] = ""
+ data["instance_id"] = ""
+ data["tags"] = {"volume_name": ""}

These can be set with a single data.update()

#2:
+ volume_data.update({"instance_id": data["instance_id"]})
+ volume_data.update({"device": data["device"]})

Same as #1

#3:
+ for tag_volume_id in volume_tag_data.keys():
+ additional_tags = volume_tag_data[tag_volume_id]["tags"]
+ if tag_volume_id not in result:
+ hookenv.log(
+ "Ignoring tags for volume-id %s that doesn't exist" %
+ tag_volume_id)
             else:

The first line inside the loop can be moved inside the else clause.
Also there's an extra indentation space on the hookenv.log() call

#4:
I think you can drop the checks on the response_type in the parse_* methods, since it's already checked by call sites (related tests can be dropped then).

review: Needs Fixing

[lp:~chad.smith/charms/precise/block-storage-broker/bsb-trusty-support]

90. By Chad Smith on 2014-08-08

pull in ack's _run_command patch for simplification

[Chad Smith (chad.smith)]

Rejecting this branch in favor of dropping euca2ools and using Amazon's python SDK: python-boto as boto is more stable across Ubuntu releases than trying to parse and unify the output of euca2ools commandline utilities.

review: Disapprove

Unmerged revisions

90. By Chad Smith on 2014-08-08

pull in ack's _run_command patch for simplification

89. By Chad Smith on 2014-03-21

Ignore any orphaned TAG lines from euca-describe-tags that don't map to current existing volume-ids

88. By Chad Smith on 2014-03-21

revert Makefile change

87. By Chad Smith on 2014-03-21

merge fixes from bsb-ec2-support and resolve conflicts + lint

86. By Chad Smith on 2014-03-20

now that _ec2_describe_volumes calls euca-describe-tags, we need to mock that in unit tests. Add unit tests for parse_ec2_*_reponse functions

85. By Chad Smith on 2014-03-20

docstring updates

84. By Chad Smith on 2014-03-20

perform validation of INSTANCE reponse_type before attmpting to parse the value in the line. lint fixes in unit tests

83. By Chad Smith on 2014-03-20

parse euca-describe-instances and euca-describe-volumes output instead of tying into euca2ools.commands.euca.describeinstances directly

=== modified file 'hooks/test_util.py'
--- hooks/test_util.py	2014-03-21 17:42:00 +0000
+++ hooks/test_util.py	2014-08-08 02:34:47 +0000
@@ -1,5 +1,9 @@
 import util
-from util import StorageServiceUtil, ENVIRONMENT_MAP, generate_volume_label
+from util import (
+    StorageServiceUtil, EUCA2_INSTANCE_RESPONSE_LENGTH_V3,
+    EUCA2_INSTANCE_RESPONSE_LENGTH_V2, parse_ec2_tag_response,
+    parse_ec2_volume_response, parse_ec2_instance_response,
+    parse_ec2_attachment_response, ENVIRONMENT_MAP, generate_volume_label)
 import mocker
 import os
 import subprocess
@@ -9,7 +13,6 @@
 class TestNovaUtil(mocker.MockerTestCase):
 
     def setUp(self):
-        super(TestNovaUtil, self).setUp()
         self.maxDiff = None
         util.hookenv = TestHookenv(
             {"key": "myusername", "tenant": "myusername_project",
@@ -880,59 +883,9 @@
             message, util.hookenv._log_INFO, "Not logged- %s" % message)
 
 
-class MockEucaCommand(object):
-    def __init__(self, result):
-        self.result = result
-
-    def main(self):
-        return self.result
-
-
-class MockEucaReservation(object):
-    def __init__(self, instances):
-        self.instances = instances
-        self.id = 1
-
-
-class MockEucaInstance(object):
-    def __init__(self, instance_id=None, ip_address=None, image_id=None,
-                 instance_type=None, kernel=None, private_dns_name=None,
-                 public_dns_name=None, state=None, tags=[],
-                 availability_zone=None):
-        self.id = instance_id
-        self.ip_address = ip_address
-        self.image_id = image_id
-        self.instance_type = instance_type
-        self.kernel = kernel
-        self.private_dns_name = private_dns_name
-        self.public_dns_name = public_dns_name
-        self.state = state
-        self.tags = tags
-        self.placement = availability_zone
-
-
-class MockAttachData(object):
-    def __init__(self, device, instance_id):
-        self.device = device
-        self.instance_id = instance_id
-
-
-class MockVolume(object):
-    def __init__(self, vol_id, device, instance_id, zone, size, status,
-                 snapshot_id, tags):
-        self.id = vol_id
-        self.attach_data = MockAttachData(device, instance_id)
-        self.zone = zone
-        self.size = size
-        self.status = status
-        self.snapshot_id = snapshot_id
-        self.tags = tags
-
-
 class TestEC2Util(mocker.MockerTestCase):
 
     def setUp(self):
-        super(TestEC2Util, self).setUp()
         self.maxDiff = None
         util.hookenv = TestHookenv(
             {"key": "ec2key", "secret": "ec2password",
@@ -1005,8 +958,8 @@
         configuration options.
         """
         command = "euca-describe-instances"
-        nova_cmd = self.mocker.replace(subprocess.check_call)
-        nova_cmd(command, shell=True)
+        euca_cmd = self.mocker.replace(subprocess.check_call)
+        euca_cmd(command, shell=True)
         self.mocker.replay()
 
         self.storage.validate_credentials()
@@ -1242,39 +1195,47 @@
 
     def test_wb_ec2_describe_volumes_command_error(self):
         """
-        L{_ec2_describe_volumes} will exit in error when the euca2ools
-        C{DescribeVolumes} command fails.
+        L{_ec2_describe_volumes} will exit in error when the
+        C{euca-describe-volumes} command fails.
         """
-        euca_command = self.mocker.replace(self.storage.ec2_volume_class)
-        euca_command()
-        self.mocker.throw(SystemExit(1))
+        command = "euca-describe-volumes"
+        euca_volumes = self.mocker.replace(subprocess.check_output)
+        euca_volumes(command, shell=True)
+        self.mocker.throw(subprocess.CalledProcessError(1, command))
         self.mocker.replay()
 
         result = self.assertRaises(
             SystemExit, self.storage._ec2_describe_volumes)
         self.assertEqual(result.code, 1)
-        message = "ERROR: Couldn't contact EC2 using euca-describe-volumes"
+        message = (
+            "ERROR: Command '%s' returned non-zero exit status 1" % command)
         self.assertIn(
             message, util.hookenv._log_ERROR, "Not logged- %s" % message)
 
     def test_wb_ec2_describe_volumes_without_attached_instances(self):
         """
-        L{_ec2_describe_volumes} parses the results of euca2ools
-        C{DescribeVolumes} to create a C{dict} of volume information. When no
-        C{instance_id}s are present the volumes are not attached so no
-        C{device} or C{instance_id} information will be present.
+        L{_ec2_describe_volumes} parses the output from the
+        C{euca-describe-volumes} command and returns a C{dict} of volume
+        information. When no C{ATTACHMENT} response types are present the
+        volumes are not attached so no C{device} or C{instance_id} information
+        will be present. Any C{TAG} response types will be added to the
+        associated volume.
         """
-        volume1 = MockVolume(
-            "123-123-123", device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az1", size="10", status="available",
-            snapshot_id="some-shot", tags={})
-        volume2 = MockVolume(
-            "456-456-456", device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az2", size="8", status="available",
-            snapshot_id="some-shot", tags={"volume_name": "my volume name"})
-        euca_command = self.mocker.replace(self.storage.ec2_volume_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([volume1, volume2]))
+        volume_output = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\nVOLUME\t456-456-456\t 8\tsome-shot\t"
+            "ec2-az2\tavailable\t2014-03-19T22:00:02.580Z\n"
+            "TAG\tvolume\t456-456-456\tvolume_name\tmy volume name")
+        volume_command = "euca-describe-volumes"
+        tag_output = (
+            "TAG\t456-456-456\tvolume\tother_key\tother value")
+        tag_command = "euca-describe-tags"
+
+        euca_command = self.mocker.replace(subprocess.check_output)
+        euca_command(volume_command, shell=True)
+        self.mocker.result(volume_output)
+        euca_command(tag_command, shell=True)
+        self.mocker.result(tag_output)
         self.mocker.replay()
 
         expected = {"123-123-123": {"id": "123-123-123", "status": "available",
@@ -1290,27 +1251,30 @@
                                     "volume_label": "my volume name",
                                     "size": "8", "instance_id": "",
                                     "snapshot_id": "some-shot",
-                                    "tags": {"volume_name": "my volume name"}}}
+                                    "tags": {"volume_name": "my volume name",
+                                             "other_key": "other value"}}}
         self.assertEqual(self.storage._ec2_describe_volumes(), expected)
 
     def test_wb_ec2_describe_volumes_matches_volume_id_supplied(self):
         """
-        L{_ec2_describe_volumes} parses the results of euca2ools
-        C{DescribeVolumes} to create a C{dict} of volume information.
-        When C{volume_id} is provided return a C{dict} for the matched volume.
+        L{_ec2_describe_volumes} parses the output of the
+        C{euca-describe-volumes} command and returns a C{dict} of volume
+        information. When C{volume_id} is provided, return a C{dict} for the
+        matched volume.
         """
         volume_id = "123-123-123"
-        volume1 = MockVolume(
-            volume_id, device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az1", size="10", status="available",
-            snapshot_id="some-shot", tags={})
-        volume2 = MockVolume(
-            "456-456-456", device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az2", size="8", status="available",
-            snapshot_id="some-shot", tags={"volume_name": "my volume name"})
-        euca_command = self.mocker.replace(self.storage.ec2_volume_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([volume1, volume2]))
+
+        output = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\nVOLUME\t456-456-456\t 8\tsome-shot\t"
+            "ec2-az2\tavailable\t2014-03-19T22:00:02.580Z\n")
+        command = "euca-describe-volumes"
+
+        euca_command = self.mocker.replace(subprocess.check_output)
+        euca_command(command, shell=True)
+        self.mocker.result(output)
+        euca_command("euca-describe-tags", shell=True)
+        self.mocker.result("\n")
         self.mocker.replay()
 
         expected = {
@@ -1323,18 +1287,22 @@
 
     def test_wb_ec2_describe_volumes_unmatched_volume_id_supplied(self):
         """
-        L{_ec2_describe_volumes} parses the results of euca2ools
-        C{DescribeVolumes} to create a C{dict} of volume information.
-        When C{volume_id} is provided and unmatched, return an empty C{dict}.
+        L{_ec2_describe_volumes} parses the output of the
+        C{euca-describe-volumes} command and returns a C{dict} of volume
+        information. When C{volume_id} is provided and unmatched, return an
+        empty C{dict}.
         """
         unmatched_volume_id = "456-456-456"
-        volume1 = MockVolume(
-            "123-123-123", device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az1", size="10", status="available",
-            snapshot_id="some-shot", tags={})
-        euca_command = self.mocker.replace(self.storage.ec2_volume_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([volume1]))
+        output = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\n")
+        command = "euca-describe-volumes"
+
+        euca_command = self.mocker.replace(subprocess.check_output)
+        euca_command(command, shell=True)
+        self.mocker.result(output)
+        euca_command("euca-describe-tags", shell=True)
+        self.mocker.result("\n")
         self.mocker.replay()
 
         self.assertEqual(
@@ -1342,22 +1310,28 @@
 
     def test_wb_ec2_describe_volumes_with_attached_instances(self):
         """
-        L{_ec2_describe_volumes} parses the results of euca2ools
-        C{DescribeVolumes} to create a C{dict} of volume information. If
-        C{status} is C{in-use}, both C{device} and C{instance_id} will be
-        returned in the C{dict}.
+        L{_ec2_describe_volumes} parses the output of the
+        C{euca-describe-volumes} command and returns a C{dict} of volume
+        information. When C{status} is C{in-use}, both C{device} and
+        C{instance_id} will be returned in the C{dict}.
         """
-        volume1 = MockVolume(
-            "123-123-123", device="/dev/notshown", instance_id="notseen",
-            zone="ec2-az1", size="10", status="available",
-            snapshot_id="some-shot", tags={})
-        volume2 = MockVolume(
-            "456-456-456", device="/dev/xvdc", instance_id="i-456456",
-            zone="ec2-az2", size="8", status="in-use",
-            snapshot_id="some-shot", tags={"volume_name": "my volume name"})
-        euca_command = self.mocker.replace(self.storage.ec2_volume_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([volume1, volume2]))
+        output = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\nVOLUME\t456-456-456\t 8\tsome-shot\t"
+            "ec2-az2\tin-use\t2014-03-19T22:00:02.580Z\nATTACHMENT\t"
+            "456-456-456\ti-456456\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\nTAG\tvolume\t456-456-456\tvolume_name\t"
+            "my volume name\n")
+        command = "euca-describe-volumes"
+        tag_output = (
+            "TAG\t456-456-456\tvolume\tvolume_name\tmy volume name")
+        tag_command = "euca-describe-tags"
+
+        euca_command = self.mocker.replace(subprocess.check_output)
+        euca_command(command, shell=True)
+        self.mocker.result(output)
+        euca_command(tag_command, shell=True)
+        self.mocker.result(tag_output)
         self.mocker.replay()
 
         expected = {"123-123-123": {"id": "123-123-123", "status": "available",
@@ -1377,6 +1351,47 @@
         self.assertEqual(
             self.storage._ec2_describe_volumes(), expected)
 
+    def test_wb_ec2_describe_volumes_orphaned_tags(self):
+        """
+        L{_ec2_describe_volumes} parses the output of the
+        C{euca-describe-volumes} and augments that data with volume-related
+        tags from C{euca-describe-tags}. When C{euca-describe-tags} returns
+        information about orphaned tags for volumes that no longer exist,
+        L{_ec2_describe_volumes} will log this information and move on.
+        """
+        output = (
+            "VOLUME\t456-456-456\t 8\tsome-shot\t"
+            "ec2-az2\tin-use\t2014-03-19T22:00:02.580Z\nATTACHMENT\t"
+            "456-456-456\ti-456456\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\nTAG\tvolume\t456-456-456\tvolume_name\t"
+            "my volume name\n")
+        command = "euca-describe-volumes"
+        tag_output = (
+            "TAG\tvolume-not-here\tvolume\tvolume_name\tmy volume name\n"
+            "TAG\t456-456-456\tvolume\tvolume_name\tmy volume name")
+        tag_command = "euca-describe-tags"
+
+        euca_command = self.mocker.replace(subprocess.check_output)
+        euca_command(command, shell=True)
+        self.mocker.result(output)
+        euca_command(tag_command, shell=True)
+        self.mocker.result(tag_output)
+        self.mocker.replay()
+
+        expected = {"456-456-456": {"id": "456-456-456", "status": "in-use",
+                                    "device": "/dev/xvdc",
+                                    "availability_zone": "ec2-az2",
+                                    "volume_label": "my volume name",
+                                    "size": "8", "instance_id": "i-456456",
+                                    "snapshot_id": "some-shot",
+                                    "tags": {"volume_name": "my volume name"}}}
+        self.assertEqual(
+            self.storage._ec2_describe_volumes(), expected)
+        message = (
+            "Ignoring tags for volume-id volume-not-here that doesn't exist")
+        self.assertIn(
+            message, util.hookenv._log_INFO, "Not logged- %s" % message)
+
     def test_wb_ec2_create_volume(self):
         """
         L{_ec2_create_volume} uses the command C{euca-create-volume} to create
@@ -1392,18 +1407,17 @@
         zone = "ec2-az3"
         command = "euca-create-volume -z %s -s %s" % (zone, size)
 
-        reservation = MockEucaReservation(
-            [MockEucaInstance(
-                instance_id=instance_id, availability_zone=zone)])
-        euca_command = self.mocker.replace(self.storage.ec2_instance_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([reservation]))
         create = self.mocker.replace(subprocess.check_output)
         create(command, shell=True)
         self.mocker.result(
             "VOLUME %s 9 nova creating 2014-03-14T17:26:20\n" % volume_id)
         self.mocker.replay()
 
+        def mock_describe_instances(my_id):
+            self.assertEqual(my_id, instance_id)
+            return {"availability_zone": zone}
+        self.storage._ec2_describe_instances = mock_describe_instances
+
         def mock_describe_volumes(my_id):
             self.assertEqual(my_id, volume_id)
             return {"id": volume_id}
@@ -1436,10 +1450,10 @@
         volume_label = "postgresql/0 unit volume"
         size = 10
 
-        euca_command = self.mocker.replace(self.storage.ec2_instance_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([]))  # Empty results from euca
-        self.mocker.replay()
+        def mock_describe_instances(my_id):
+            self.assertEqual(my_id, instance_id)
+            return {}  # No results found
+        self.storage._ec2_describe_instances = mock_describe_instances
 
         result = self.assertRaises(
             SystemExit, self.storage._ec2_create_volume, size, volume_label,
@@ -1465,17 +1479,16 @@
         zone = "ec2-az3"
         command = "euca-create-volume -z %s -s %s" % (zone, size)
 
-        reservation = MockEucaReservation(
-            [MockEucaInstance(
-                instance_id=instance_id, availability_zone=zone)])
-        euca_command = self.mocker.replace(self.storage.ec2_instance_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([reservation]))
         create = self.mocker.replace(subprocess.check_output)
         create(command, shell=True)
         self.mocker.result("INSTANCE invalid-instance-type-response\n")
         self.mocker.replay()
 
+        def mock_describe_instances(my_id):
+            self.assertEqual(my_id, instance_id)
+            return {"availability_zone": zone}
+        self.storage._ec2_describe_instances = mock_describe_instances
+
         def mock_describe_volumes(my_id):
             raise Exception("_ec2_describe_volumes should not be called")
         self.storage._ec2_describe_volumes = mock_describe_volumes
@@ -1507,18 +1520,17 @@
         zone = "ec2-az3"
         command = "euca-create-volume -z %s -s %s" % (zone, size)
 
-        reservation = MockEucaReservation(
-            [MockEucaInstance(
-                instance_id=instance_id, availability_zone=zone)])
-        euca_command = self.mocker.replace(self.storage.ec2_instance_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([reservation]))
         create = self.mocker.replace(subprocess.check_output)
         create(command, shell=True)
         self.mocker.result(
             "VOLUME %s 9 nova creating 2014-03-14T17:26:20\n" % volume_id)
         self.mocker.replay()
 
+        def mock_describe_instances(my_id):
+            self.assertEqual(my_id, instance_id)
+            return {"availability_zone": zone}
+        self.storage._ec2_describe_instances = mock_describe_instances
+
         def mock_describe_volumes(my_id):
             self.assertEqual(my_id, volume_id)
             return {}  # No details found for this volume
@@ -1548,17 +1560,16 @@
         zone = "ec2-az3"
         command = "euca-create-volume -z %s -s %s" % (zone, size)
 
-        reservation = MockEucaReservation(
-            [MockEucaInstance(
-                instance_id=instance_id, availability_zone=zone)])
-        euca_command = self.mocker.replace(self.storage.ec2_instance_class)
-        euca_command()
-        self.mocker.result(MockEucaCommand([reservation]))
         create = self.mocker.replace(subprocess.check_output)
         create(command, shell=True)
         self.mocker.throw(subprocess.CalledProcessError(1, command))
         self.mocker.replay()
 
+        def mock_describe_instances(my_id):
+            self.assertEqual(my_id, instance_id)
+            return {"availability_zone": zone}
+        self.storage._ec2_describe_instances = mock_describe_instances
+
         def mock_exception(my_id):
             raise Exception("These methods should not be called")
         self.storage._ec2_describe_volumes = mock_exception
@@ -1728,3 +1739,243 @@
             (volume_id, instance_id))
         self.assertIn(
             message, util.hookenv._log_INFO, "Not logged- %s" % message)
+
+    def test_wb_ec2_describe_instances_euca_v2(self):
+        """
+        L{_ec2_describe_instances} parses the output of the command
+        C{euca-describe-instances} for euca2ools version 2.X and returns a
+        C{dict} of instance information.
+        """
+        command = "euca-describe-instances"
+        output = (
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.9\t10.138.61.2\t\t\tebs\t\t\t\t\t"
+            "paravirtual\t\t\t\t")
+
+        ec2_list = self.mocker.replace(subprocess.check_output)
+        ec2_list(command, shell=True)
+        self.mocker.result(output)
+        self.mocker.replay()
+
+        expected = {
+            "i-aaa": {
+                "ip-address": "54.81.8.9", "private-ip-address": "10.138.61.2",
+                "kernel": "aki-aaa", "instance-type": "m1.small",
+                "state": "running", "public-dns-name": "aaa.amazonaws.com",
+                "private-dns-name": "aaa.ec2.internal", "instance_id": "i-aaa",
+                "image-id": "ami-aaa", "availability_zone": "us-east-1a"}}
+
+        self.assertEqual(
+            EUCA2_INSTANCE_RESPONSE_LENGTH_V2, len(output.split("\t")))
+
+        self.assertEqual(
+            self.storage._ec2_describe_instances(), expected)
+
+    def test_wb_ec2_describe_instances_euca_invalid_output_version(self):
+        """
+        L{_ec2_describe_instances} parses the output of the command
+        C{euca-describe-instances} and if it sees an output length that is
+        unsupported, an error is logged.
+        """
+        command = "euca-describe-instances"
+        output = (
+            "INSTANCE\ti-aaa\tami-0b9c9f62\tblah.amazonaws.com\t"
+            "blah.ec2.internal\trunning\t\t0\t")
+
+        ec2_list = self.mocker.replace(subprocess.check_output)
+        ec2_list(command, shell=True)
+        self.mocker.result(output)
+        self.mocker.replay()
+
+        self.assertNotIn(
+            len(output.split("\t")),
+            [EUCA2_INSTANCE_RESPONSE_LENGTH_V3,
+             EUCA2_INSTANCE_RESPONSE_LENGTH_V3])
+
+        result = self.assertRaises(
+            SystemExit, self.storage._ec2_describe_instances)
+        self.assertEqual(result.code, 1)
+        message = "ERROR: Unsupported euca INSTANCE output format version"
+        self.assertIn(
+            message, util.hookenv._log_ERROR, "Not logged- %s" % message)
+
+    def test_wb_ec2_describe_instances_specific_instance_id(self):
+        """
+        L{_ec2_describe_instances} parses the output of the command
+        C{euca-describe-instances} for euca2ools version 3.X and returns a
+        C{dict} of instance information. When C{instance_id} is provided,
+        results are filtered to only the matching C{instance_id}.
+        """
+        command = "euca-describe-instances"
+        output = (
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.1\t10.138.61.1\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t\n"
+            "INSTANCE\ti-bbb\tami-bbb\tbbb.amazonaws.com\t"
+            "bbb.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-bbb\t\t\t"
+            "monitoring-disabled\t54.81.8.2\t10.138.61.2\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t")
+
+        ec2_list = self.mocker.replace(subprocess.check_output)
+        ec2_list(command, shell=True)
+        self.mocker.result(output)
+        self.mocker.replay()
+
+        expected = {
+            "ip-address": "54.81.8.1", "private-ip-address": "10.138.61.1",
+            "kernel": "aki-aaa", "instance-type": "m1.small",
+            "state": "running", "public-dns-name": "aaa.amazonaws.com",
+            "private-dns-name": "aaa.ec2.internal", "instance_id": "i-aaa",
+            "image-id": "ami-aaa", "availability_zone": "us-east-1a"}
+
+        self.assertEqual(
+            EUCA2_INSTANCE_RESPONSE_LENGTH_V3,
+            len(output.split("\n")[0].split("\t")))
+
+        self.assertEqual(
+            self.storage._ec2_describe_instances("i-aaa"), expected)
+
+    def test_parse_ec2_tag_response_not_tag_response(self):
+        """
+        L{parse_ec2_tag_response} returns an empty C{dict} when the
+        line does not represent a C{TAG} response type.
+        """
+        non_tag_output_lines = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\n"
+            "ATTACHMENT\t123-123-123\ti-123123\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\n"
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.1\t10.138.61.1\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t")
+        for line in non_tag_output_lines.split("\n"):
+            self.assertEqual(parse_ec2_tag_response(line), {})
+
+    def test_parse_ec2_attachment_response_not_attachment_response(self):
+        """
+        L{parse_ec2_attachment_response} returns an empty C{dict} when the
+        line does not represent a C{ATTACHMENT} response type.
+        """
+        non_attachment_output_lines = (
+            "TAG\tvolume\t456-456-456\tvolume_name\tmy volume name\n"
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\n"
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.1\t10.138.61.1\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t")
+        for line in non_attachment_output_lines.split("\n"):
+            self.assertEqual(parse_ec2_attachment_response(line), {})
+
+    def test_parse_ec2_volume_response_not_volume_response(self):
+        """
+        L{parse_ec2_volume_response} returns an empty C{dict} when the
+        line does not represent a C{VOLUME} response type.
+        """
+        non_volume_output_lines = (
+            "TAG\tvolume\t456-456-456\tvolume_name\tmy volume name\n"
+            "ATTACHMENT\t123-123-123\ti-123123\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\n"
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.1\t10.138.61.1\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t")
+        for line in non_volume_output_lines.split("\n"):
+            self.assertEqual(parse_ec2_volume_response(line), {})
+
+    def test_parse_ec2_instance_response_not_instance_response(self):
+        """
+        L{parse_ec2_instance_response} returns an empty C{dict} when the
+        line does not represent a C{INSTANCE} response type.
+        """
+        non_instance_output_lines = (
+            "TAG\tvolume\t456-456-456\tvolume_name\tmy volume name\n"
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\n"
+            "ATTACHMENT\t123-123-123\ti-123123\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\n")
+        for line in non_instance_output_lines.split("\n"):
+            self.assertEqual(parse_ec2_instance_response(line), {})
+
+    def test_parse_ec2_instance_response(self):
+        """
+        L{parse_ec2_instance_response} returns a C{dict} of the parsed
+        C{INSTANCE} response type.
+        """
+        output = (
+            "INSTANCE\ti-aaa\tami-aaa\taaa.amazonaws.com\t"
+            "aaa.ec2.internal\trunning\t\t0\t\tm1.small\t"
+            "2014-03-19T17:52:02.000Z\tus-east-1a\taki-aaa\t\t\t"
+            "monitoring-disabled\t54.81.8.1\t10.138.61.1\t\t\tebs"
+            "\t\t\t\t\tparavirtual\txen\t"
+            "5926eb12c796888691d9fa135f6e6f8ada297a9b1ecbea0c12282522400636f3"
+            "\tsg-55e2e33e,sg-29e2e342\tdefault\tfalse\t")
+
+        expected = {
+            "ip-address": "54.81.8.1", "private-ip-address": "10.138.61.1",
+            "image-id": "ami-aaa", "instance-type": "m1.small",
+            "kernel": "aki-aaa", "private-dns-name": "aaa.ec2.internal",
+            "public-dns-name": "aaa.amazonaws.com", "state": "running",
+            "availability_zone": "us-east-1a", "instance_id": "i-aaa"}
+
+        self.assertEqual(parse_ec2_instance_response(output), expected)
+
+    def test_parse_ec2_tag_response(self):
+        """
+        L{parse_ec2_tag_response} returns a C{dict} of the parsed C{TAG}
+        response type.
+        """
+        output = "TAG\tvolume\t456-456-456\tvolume_name\tmy volume name"
+
+        expected = {
+            "tag_type": "volume", "id": "456-456-456", "key": "volume_name",
+            "value": "my volume name"}
+        self.assertEqual(parse_ec2_tag_response(output), expected)
+
+    def test_parse_ec2_attachment_response(self):
+        """
+        L{parse_ec2_attachment_response} returns a C{dict} of the parsed
+        C{ATTACHMENT} response type.
+        """
+        output = (
+            "ATTACHMENT\t123-123-123\ti-123123\t/dev/xvdc\tattached\t"
+            "2014-03-19T22:00:02.000Z\n")
+
+        expected = {
+            "volume_id": "123-123-123", "instance_id": "i-123123",
+            "device": "/dev/xvdc", "attach_status": "attached"}
+        self.assertEqual(parse_ec2_attachment_response(output), expected)
+
+    def test_parse_ec2_volume_response(self):
+        """
+        L{parse_ec2_volume_response} returns a C{dict} of the parsed
+        C{VOLUME} response type.
+        """
+        output = (
+            "VOLUME\t123-123-123\t 10\tsome-shot\tec2-az1\tavailable\t"
+            "2014-03-19T22:00:02.580Z\n")
+
+        expected = {
+            "id": "123-123-123", "size": "10",
+            "snapshot_id": "some-shot", "availability_zone": "ec2-az1",
+            "status": "available"}
+        self.assertEqual(parse_ec2_volume_response(output), expected)

=== modified file 'hooks/util.py'
--- hooks/util.py	2014-03-21 17:39:48 +0000
+++ hooks/util.py	2014-08-08 02:34:47 +0000
@@ -23,6 +23,9 @@
     "nova": {"validate": "nova list",
              "detach": "nova volume-detach %s %s"}}
 
+EUCA2_INSTANCE_RESPONSE_LENGTH_V2 = 30
+EUCA2_INSTANCE_RESPONSE_LENGTH_V3 = 32
+
 
 class StorageServiceUtil(object):
     """Interact with an underlying cloud storage provider.
@@ -45,11 +48,6 @@
         self.environment_map = ENVIRONMENT_MAP[provider]
         self.commands = PROVIDER_COMMANDS[provider]
         self.required_config_options = REQUIRED_CONFIG_OPTIONS[provider]
-        if provider == "ec2":
-            import euca2ools.commands.euca.describevolumes as getvolumes
-            import euca2ools.commands.euca.describeinstances as getinstances
-            self.ec2_volume_class = getvolumes.DescribeVolumes
-            self.ec2_instance_class = getinstances.DescribeInstances
 
     def load_environment(self):
         """
@@ -212,83 +210,112 @@
         return
 
     # EC2-specific methods
+    def _run_command(self, command):
+        try:
+            output = subprocess.check_call(command, shell=True)
+        except subprocess.CalledProcessError, e:
+            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
+            sys.exit(1)
+
+        # Split the output by lines
+        lines = []
+        if output:
+            for line in output.split("\n"):
+                line = line.strip()
+                if line:
+                    lines.append(line)
+        return lines
+
     def _ec2_create_tag(self, volume_id, tag_name, tag_value=None):
         """Attach a tag and optional C{tag_value} to the given C{volume_id}"""
         tag_string = tag_name
         if tag_value:
             tag_string += "=%s" % tag_value
-        command = 'euca-create-tags %s --tag "%s"' % (volume_id, tag_string)
+        command = "euca-create-tags %s --tag \"%s\"" % (volume_id, tag_string)
 
-        try:
-            subprocess.check_call(command, shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log(
-                "ERROR: Couldn't add tags to the resource. %s" % str(e),
-                hookenv.ERROR)
-            sys.exit(1)
+        self._run_command(command)
         hookenv.log("Tagged (%s) to %s." % (tag_string, volume_id))
 
     def _ec2_describe_instances(self, instance_id=None):
         """
-        Use euca2ools libraries to describe instances and return a C{dict}
+        Use euca2ools command output to describe instances and return a C{dict}
         """
         result = {}
-        try:
-            command = self.ec2_instance_class()
-            reservations = command.main()
-        except SystemExit:
-            hookenv.log(
-                "ERROR: Couldn't contact EC2 using euca-describe-instances",
-                hookenv.ERROR)
-            sys.exit(1)
-        for reservation in reservations:
-            for inst in reservation.instances:
-                result[inst.id] = {
-                    "ip-address": inst.ip_address, "image-id": inst.image_id,
-                    "instance-type": inst.image_id, "kernel": inst.kernel,
-                    "private-dns-name": inst.private_dns_name,
-                    "public-dns-name": inst.public_dns_name,
-                    "reservation-id": reservation.id,
-                    "state": inst.state, "tags": inst.tags,
-                    "availability_zone": inst.placement}
+
+        lines = self._run_command("euca-describe-instances")
+        for line in lines:
+            response_type = line.split("\t")[0]
+            if response_type != "INSTANCE":
+                continue
+            instance_data = parse_ec2_instance_response(line)
+            result[instance_data["instance_id"]] = instance_data
+
         if instance_id:
             if instance_id in result:
                 return result[instance_id]
             return {}
         return result
 
+    def _ec2_describe_volume_tags(self):
+        """
+        Parse output of C{euca-describe-tags} returning a C{dict} of volume ids
+        and any associated tags.
+        """
+        result = {}
+
+        lines = self._run_command("euca-describe-tags")
+        for line in lines:
+            response_type = line.split("\t")[0]
+            if response_type != "TAG":
+                continue
+            data = parse_ec2_tag_response(line)
+            if not result.get(data["id"]):
+                result[data["id"]] = {"tags": {}}
+            result[data["id"]]["tags"].update({data["key"]: data["value"]})
+        return result
+
     def _ec2_describe_volumes(self, volume_id=None):
         """
-        Use euca2ools libraries to describe volumes and return a C{dict}
+        Parse output of C{euca-describe-volumes} returning a C{dict} of volume
+        ids all related volume information.
         """
         result = {}
-        try:
-            command = self.ec2_volume_class()
-            volumes = command.main()
-        except SystemExit:
-            hookenv.log(
-                "ERROR: Couldn't contact EC2 using euca-describe-volumes",
-                hookenv.ERROR)
-            sys.exit(1)
-        for volume in volumes:
-            result[volume.id] = {
-                "device": "",
-                "instance_id": "",
-                "size": volume.size,
-                "snapshot_id": volume.snapshot_id,
-                "status": volume.status,
-                "tags": volume.tags,
-                "id": volume.id,
-                "availability_zone": volume.zone}
-            if "volume_name" in volume.tags:
-                result[volume.id]["volume_label"] = volume.tags["volume_name"]
-            else:
-                result[volume.id]["tags"]["volume_name"] = ""
-                result[volume.id]["volume_label"] = ""
-            if volume.status == "in-use":
-                result[volume.id]["instance_id"] = (
-                    volume.attach_data.instance_id)
-                result[volume.id]["device"] = volume.attach_data.device
+
+        lines = self._run_command("euca-describe-volumes")
+        for line in lines:
+            response_type = line.split("\t")[0]
+            if response_type == "VOLUME":
+                data = parse_ec2_volume_response(line)
+                data["device"] = ""
+                data["volume_label"] = ""
+                data["instance_id"] = ""
+                data["tags"] = {"volume_name": ""}
+                result[data["id"]] = data
+            elif response_type == "TAG":
+                data = parse_ec2_tag_response(line)
+                tags = result[data["id"]]["tags"]
+                tags.update({data["key"]: data["value"]})
+                volume_name = tags.get("volume_name")
+                if volume_name:
+                    result[data["id"]]["volume_label"] = volume_name
+            elif response_type == "ATTACHMENT":
+                data = parse_ec2_attachment_response(line)
+                volume_data = result[data["volume_id"]]
+                volume_data.update({"instance_id": data["instance_id"]})
+                volume_data.update({"device": data["device"]})
+            else:
+                continue
+
+        volume_tag_data = self._ec2_describe_volume_tags()
+        for tag_volume_id in volume_tag_data.keys():
+            additional_tags = volume_tag_data[tag_volume_id]["tags"]
+            if tag_volume_id not in result:
+                hookenv.log(
+                    "Ignoring tags for volume-id %s that doesn't exist" %
+                    tag_volume_id)
+            else:
+                result[tag_volume_id]["tags"].update(additional_tags)
+
         if volume_id:
             if volume_id in result:
                 return result[volume_id]
@@ -311,19 +338,15 @@
                 (instance_id, config_data["endpoint"]), hookenv.ERROR)
             sys.exit(1)
 
-        try:
-            output = subprocess.check_output(
-                "euca-create-volume -z %s -s %s" %
-                (instance["availability_zone"], size), shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
-            sys.exit(1)
+        command = "euca-create-volume -z %s -s %s" % (
+            instance["availability_zone"], size)
 
-        response_type, volume_id = output.split()[:2]
+        lines = self._run_command(command)
+        response_type, volume_id = lines[:2]
         if response_type != "VOLUME":
             hookenv.log(
                 "ERROR: Didn't get VOLUME response from euca-create-volume. "
-                "Response: %s" % output, hookenv.ERROR)
+                "Response: %s" % "".join(lines), hookenv.ERROR)
             sys.exit(1)
         volume = self.describe_volumes(volume_id.strip())
         if not volume:
@@ -341,13 +364,10 @@
         the device path.
         """
         device = "/dev/xvdc"
-        try:
-            subprocess.check_call(
-                "euca-attach-volume -i %s -d %s %s" %
-                (instance_id, device, volume_id), shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
-            sys.exit(1)
+        command = "euca-attach-volume -i %s -d %s %s" % (
+            instance_id, device, volume_id)
+
+        self._run_command(command)
         return device
 
     # Nova-specific methods
@@ -358,17 +378,9 @@
         """
         from ast import literal_eval
         result = {"tags": {}, "instance_id": "", "device": ""}
-        command = "nova volume-show '%s'" % volume_id
-        try:
-            output = subprocess.check_output(command, shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log(
-                "ERROR: Failed to get nova volume info. %s" % str(e),
-                hookenv.ERROR)
-            sys.exit(1)
-        for line in output.split("\n"):
-            if not line.strip():  # Skip empty lines
-                continue
+
+        lines = self._run_command("nova volume-show '%s'" % volume_id)
+        for line in lines:
             if "+----" in line or "Property" in line:
                 continue
             (_, key, value, _) = line.split("|")
@@ -393,15 +405,9 @@
     def _nova_describe_volumes(self, volume_id=None):
         """Create a C{dict} describing all nova volumes"""
         result = {}
-        command = "nova volume-list"
-        try:
-            output = subprocess.check_output(command, shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
-            sys.exit(1)
-        for line in output.split("\n"):
-            if not line.strip():  # Skip empty lines
-                continue
+
+        lines = self._run_command("nova volume-list")
+        for line in lines:
             if "+----" in line or "ID" in line:
                 continue
             values = line.split("|")
@@ -434,15 +440,12 @@
         Attach a Nova C{volume_id} to the provided C{instance_id} and return
         the device path.
         """
-        try:
-            device = subprocess.check_output(
-                "nova volume-attach %s %s auto | egrep -o \"/dev/vd[b-z]\"" %
-                (instance_id, volume_id), shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
-            sys.exit(1)
-        if device.strip():
-            return device.strip()
+        command = (
+            "nova volume-attach %s %s auto | egrep -o \"/dev/vd[b-z]\"" %
+            (instance_id, volume_id))
+        lines = self._run_ec2_command(command)
+        if lines:
+            return lines[0]
         return ""
 
     def _nova_create_volume(self, size, volume_label, instance_id):
@@ -450,14 +453,11 @@
         hookenv.log(
             "Creating a %sGig volume named (%s) for instance %s" %
             (size, volume_label, instance_id))
-        try:
-            subprocess.check_call(
-                "nova volume-create --display-name '%s' %s" %
-                (volume_label, size), shell=True)
-        except subprocess.CalledProcessError, e:
-            hookenv.log("ERROR: %s" % str(e), hookenv.ERROR)
-            sys.exit(1)
+        command = (
+            "nova volume-create --display-name '%s' %s" %
+            (volume_label, size))
 
+        self._run_command(command)
         volume_id = self.get_volume_id(volume_label)
         if not volume_id:
             hookenv.log(
@@ -467,6 +467,102 @@
         return volume_id
 
 
+def parse_ec2_tag_response(line):
+    """
+    Parses a line of output of response type TAG from euca commands and
+    returns a C{dict} of the values. If not a TAG response, an empty C{dict} is
+    returned.
+    """
+    values = line.split("\t")
+    response_type = values[0]
+    if response_type != "TAG":
+        return {}
+
+    # There are two versions of the TAG response
+    if values[1] == "volume":
+        tag_type = values[1]
+        object_id = values[2]
+    else:
+        tag_type = values[2]
+        object_id = values[1]
+    if tag_type != "volume":
+        hookenv.log(
+            "Ignoring non-volume TAG response: %s." % line, hookenv.WARNING)
+        return {}
+    return {
+        "tag_type": tag_type, "id": object_id, "key": values[3],
+        "value": values[4]}
+
+
+def parse_ec2_attachment_response(line):
+    """
+    Parses a line of output of response type ATTACHMENT from euca commands and
+    returns a C{dict} of the values. If not a ATTACHMENT response, an empty
+    C{dict} is returned.
+    """
+    values = line.split("\t")
+    response_type = values[0]
+    if response_type != "ATTACHMENT":
+        return {}
+
+    return {
+        "volume_id": values[1], "instance_id": values[2], "device": values[3],
+        "attach_status": values[4]}
+
+
+def parse_ec2_volume_response(line):
+    """
+    Parses a line of output of response type VOLUME from euca commands and
+    returns a C{dict} of the values. If not a VOLUME response, an empty C{dict}
+    is returned.
+    """
+    values = line.split("\t")
+    response_type = values[0]
+    if response_type != "VOLUME":
+        return {}
+
+    return {
+        "id": values[1], "size": values[2].strip(),
+        "snapshot_id": values[3], "availability_zone": values[4],
+        "status": values[5]}
+
+
+def parse_ec2_instance_response(line):
+    """
+    Parses a line of output of response type INSTANCE from euca commands and
+    returns a C{dict} of the values. If not an INSTANCE response, and empty
+    C{dict} is returned.
+    """
+    values = line.split("\t")
+    response_type = values[0]
+    if response_type != "INSTANCE":
+        return {}
+
+    if len(values) not in [EUCA2_INSTANCE_RESPONSE_LENGTH_V2,
+                           EUCA2_INSTANCE_RESPONSE_LENGTH_V3]:
+        hookenv.log(
+            "ERROR: Unsupported euca INSTANCE output format version",
+            hookenv.ERROR)
+        sys.exit(1)
+
+    (ec2_instance_id, image_id) = values[1:3]
+    public_dns = values[3]
+    private_dns = values[4]
+    state = values[5]
+    instance_type = values[9]
+    availability_zone = values[11]
+    kernel = values[12]
+    public_ip = values[16]
+    private_ip = values[17]
+
+    return {
+        "ip-address": public_ip, "private-ip-address": private_ip,
+        "image-id": image_id, "instance-type": instance_type,
+        "kernel": kernel, "private-dns-name": private_dns,
+        "public-dns-name": public_dns, "state": state,
+        "availability_zone": availability_zone, "instance_id": ec2_instance_id}
+
+
 def generate_volume_label(remote_unit):
     """Create a volume label for the requesting remote unit"""
     return "%s unit volume" % remote_unit