Merge lp:~viswesn/juju-ci-tools/ensure_provider_cleanup into lp:juju-ci-tools

Proposed by viswesuwara nathan
Status: Merged
Merged at revision: 1967
Proposed branch: lp:~viswesn/juju-ci-tools/ensure_provider_cleanup
Merge into: lp:juju-ci-tools
Prerequisite: lp:~viswesn/juju-ci-tools/juju-ci-cleanup
Diff against target: 508 lines (+434/-3)
4 files modified
deploy_stack.py (+18/-1)
substrate.py (+96/-2)
tests/test_deploy_stack.py (+58/-0)
tests/test_substrate.py (+262/-0)
To merge this branch: bzr merge lp:~viswesn/juju-ci-tools/ensure_provider_cleanup
Reviewer Review Type Date Requested Status
Aaron Bentley (community) Approve
Christopher Lee (community) Needs Fixing
Review via email: mp+315456@code.launchpad.net

Description of the change

Do ensure cleanup for each of the provider and make sure we need to come with the list of resources that were not cleanup during cleanup activity.

To post a comment you must log in.
Revision history for this message
Christopher Lee (veebers) wrote :

I don't think that adding unclean=[] to terminate_instances is the best way to do this. Instead you should make the call to terminate_instances and catch any exception that might occur and log that.
Changing terminate_instances like you have drastically changes how the method responds to error (it use to raise an exception in some places, now it errors silently) which might have a major effect on other parts of existing code that rely on the original behaviour.

Also of note, I don't think having an empty list as a default arg like that is what you want, you might not be aware but it's behaviour might be surprising.
This link can describe it better than I can: http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

Please also see inline comments too.
I plan to get some other eyes on this in case I miss something.

review: Needs Fixing
1833. By viswesuwara nathan

review comments addressed

Revision history for this message
viswesuwara nathan (viswesn) wrote :

Most of the code remains same for ensure_cleanup at this point in time which invokes terminate the instance. The other cleanup activity specific to security group, firewall, etc., were not addressed here at this point in time because that requires contribution from the providers before supporting invoking them in ensure_cleanup.

Revision history for this message
ABHAY (ankatare) wrote :

looks good to me, only one phrase correction required.

Revision history for this message
Aaron Bentley (abentley) wrote :

This branch introduces a bunch of duplicate code. Please avoid that.

I think you probably want an Account-specific list_dirty method and then a non-specific clean_dirty method which operates on that list. If a substrate doesn't support e.g. delete_detached_interfaces, then list_dirty should not return the dirty interfaces.

Whatever you do, don't repeat yourself. Factor out the duplicate code.

Don't catch Exception. Catch the specific exceptions that you expect to be raised.

When you do catch an exception, don't make it impossible to see what the exception was. We need to know *why* it's failing in order to improve the script in the future. You could print it. You could append a tuple of (resource, exception) to unclean_resources.

Are you certain that this script is not going to delete freshly-created resources? For example, could non_instance_groups contain a group created by a different script that is about to be associated to an instance? Please explain why it's safe to delete specific resource types in the docstring or comments.

You are providing a string to terminate_instances. I don't expect this to work; you should be providing a list of strings.

review: Needs Fixing
Revision history for this message
viswesuwara nathan (viswesn) wrote :

>> Whatever you do, don't repeat yourself. Factor out the duplicate code.
<<Viswesn>> Currently we see most of the code is duplicated but then once we have list of resources to be cleaned up in coming days for each substrate then ensure_cleanup is going to be different for each substrate.

>> Don't catch Exception. Catch the specific exceptions that you expect to be raised.
<<Viswesn>> Now I need to address this by sending the unclean=[] to the callee function and then append the resource name + exception something like ["instance_id", "instance-001", "Interface failed"]

------------------------------
delete_detached_interfaces(self, security_group, unclean=None)
    if unclean is None:
          unclean=[]
    ...
    ....
     except Exception as ex:
          unclean.append(["instance_id", instance_name, type(ex).__name__, ex.args])
------------------------------

Chris, let me know your comments on this. We decided not to do so in our last call but then to catch the exact exception then I need to pass unclean to the callee method rather than having like try..catch.. statement on invoking terminate_instance.

>> You are providing a string to terminate_instances. I don't expect this to work;
<<Viswesn>> Yes, I agree and I will address this issue.

>> Are you certain that this script is not going to delete freshly-created resources?
<<Viswesn>> if ensure_cleanup has visibility to those freshly-created resource then yes but seems like it is not at present and it is up the substrate developer to bring this in to picture as part of ensure_cleanup.

I will start working on this code after getting comments from Chirs and Aaron.
Please let me know on this. Thanks.

Revision history for this message
Aaron Bentley (abentley) wrote :

> >> Whatever you do, don't repeat yourself. Factor out the duplicate code.
> <<Viswesn>> Currently we see most of the code is duplicated but then once we
> have list of resources to be cleaned up in coming days for each substrate then
> ensure_cleanup is going to be different for each substrate.

The parts that are the same, like terminate_instances, should not be duplicated. Factor them out.

> >> Don't catch Exception. Catch the specific exceptions that you expect to be
> raised.
> <<Viswesn>> Now I need to address this by sending the unclean=[] to the callee
> function and then append the resource name + exception something like
> ["instance_id", "instance-001", "Interface failed"]
>
> ------------------------------
> delete_detached_interfaces(self, security_group, unclean=None)
> if unclean is None:
> unclean=[]
> ...
> ....
> except Exception as ex:
> unclean.append(["instance_id", instance_name, type(ex).__name__,
> ex.args])
> ------------------------------
>
> Chris, let me know your comments on this. We decided not to do so in our last
> call but then to catch the exact exception then I need to pass unclean to the
> callee method rather than having like try..catch.. statement on invoking
> terminate_instance.

I don't understand why catching a specific exception would require you to pass unclean to the callee method.

> >> Are you certain that this script is not going to delete freshly-created
> resources?
> <<Viswesn>> if ensure_cleanup has visibility to those freshly-created resource
> then yes but seems like it is not at present and it is up the substrate
> developer to bring this in to picture as part of ensure_cleanup.

ensure_cleanup has visibility to those freshly-created resources. It is your problem, not the substrate developer's.

Revision history for this message
Aaron Bentley (abentley) wrote :

I have had a chance to look at the design doc: https://docs.google.com/document/d/1vi94tBGWf5UiWIIoer5Hn8Un9u7PqauX8fnF2iNIQpA/edit#

This feature has two intents:
1. Ensure resources are cleanup so that we have uninterrupted testing
   and do not accrue extra costs
2. We want to report a failure when destroy-controller fails to
   cleanup.

The way security groups and interfaces are being handled does not meet goal 2. It cannot report that destroy-controller failed to clean up for this test run, because it does not know that the security groups were created by this test run.

The doc even suggests "Match security groups/firewalls by controller uuid in the name or in tags/metadata".

Since the doc was written, goal 2 has become more important than goal 1, because our existing cleanup scripts are doing a good job and juju itself is better-behaved.

review: Needs Fixing
1834. By viswesuwara nathan

Code updated

1835. By viswesuwara nathan

Review comments addressed

Revision history for this message
Christopher Lee (veebers) wrote :

A couple of times in discussion I'm stated that we'll concentrate only on AWS for this MP and then do the other providers in follow up MPs.
This MP alters more than just the AWSAccount class, a couple of issues with this:

  - This makes the MP even longer. The diff is 548 lines long, this is too long to comfortably review. As mentioned multiple times before we want to have short MPs (less than 300 lines) this makes reviews quicker, easier and less prone to errors.

  - There are no tests included for the altered *Account classes. There is no assurance that this branch hasn't broken anything.

  - Due to the first points, adding the needed tests would blow the diff count way out.

make lint fails.

Please notice the comments about methods that could be made functions, having functions makes testing a lot easier (no need to create or mock state).

review: Needs Fixing
1836. By viswesuwara nathan

Review comments addressed

1837. By viswesuwara nathan

Review comments addressed

Revision history for this message
viswesuwara nathan (viswesn) wrote :

This branch is created long back for juju-ci-cleanup enhancement work and already we integrated the changes specific to most of the cloud providers and that is the reason for seeing code changes for providers like OpenStack, GCE, etc. As discussed in the call today I kept changes only for AWS and other provider changes were removed.

Please review it and let me know your comments on the updated code base. Thanks

Revision history for this message
Christopher Lee (veebers) wrote :

Some further queries about the data-structures used for storing errors.
I expanded on your question a bit, let me know what you think or if you would like more info.

Some test issues too. Make sure you're mocking the right thing, it's all too easy when mocking to produce false positives in your code that don't match reality when actually being run.

review: Needs Fixing
Revision history for this message
viswesuwara nathan (viswesn) wrote :

Chris, I updated the code by taken up almost all the review comments provided and more over in the current code review I removed the test case "TestGetSecurityGroups" because I need to discuss with you on how to handle "get_all_security_groups" using Mock.

Let us discuss on this on coming Monday call or please let me know by mail how to address "get_all_security_groups" mock in returning the value. It also requires invoking sg.instance() based on the object that it returns. Thanks

1838. By viswesuwara nathan

Review comments addressed

1839. By viswesuwara nathan

Review comments updated

Revision history for this message
Christopher Lee (veebers) wrote :

Hi Vishwa,

Lets talk in our next call re: your query for TestGetSecurityGroups.

Please see comments inline.

review: Needs Fixing
Revision history for this message
viswesuwara nathan (viswesn) wrote :

Hi Chris,
    I updated the review comments and let us discuss more on TestGetSecurityGroups in call. Thanks

1840. By viswesuwara nathan

Review comments addressed

1841. By viswesuwara nathan

GetSecurityGroups changes

1842. By viswesuwara nathan

GetSecurityGroups changes

Revision history for this message
Aaron Bentley (abentley) wrote :

attempt_terminate_instances does not work on AWS.

The tests mock out the direct callees, even though there are alternatives. Once consequence is that you didn't notice that attempt_terminate_instances does not work on AWS.

The tests use assertEqual(0, mock.call_count) where they could use mock.assert_called_once_with.

Also, I don't think it makes sense to reimplement set.issubset as contains_only_known_instances.

There are some formatting suggestions, too. See below.

Also, please merge trunk more frequently. I got merge conflicts with one algorithm.

review: Needs Fixing
Revision history for this message
Aaron Bentley (abentley) wrote :
1843. By viswesuwara nathan

Review comments addressed

1844. By viswesuwara nathan

Merge to trunk

Revision history for this message
viswesuwara nathan (viswesn) wrote :

1. Most of the review comments were addressed.
2. Code merged with trunk
3. Hitting one failure on having patch for terminate_instances - http://paste.ubuntu.com/24188081/
   Please let me know how to resolve this error.
4. Update the comments section on addressing it.

Revision history for this message
Christopher Lee (veebers) wrote :

responded to comments.

Revision history for this message
Christopher Lee (veebers) wrote :

clarified some points and pointed out some test issues.

review: Needs Fixing
1845. By viswesuwara nathan

Review comments addressed

Revision history for this message
Christopher Lee (veebers) wrote :

You haven't answered the questions or given reason for the changes that differ to the requests from the previous review (namely repr vs str and the ability to store better details on why the delete failed). Please respond.

Also you seemed to have ignored the comments surrounding contains_only_known_instances (using issubset and removing the method). Please respond.

I've identified concerns with the testing below, namely patching the wrong then and incorrect asserts.

review: Needs Fixing
Revision history for this message
viswesuwara nathan (viswesn) wrote :

I used repr instead of str because when I read couple of articles in the stackoverflow they recommended to use repr than str hence I decided to go for repr than str.

Now coming to use of issubset; I already asked this on March 16th; Please find the same here

"Getting error as - AttributeError: 'list' object has no attribute 'issubset'
Do I miss something? Please guide me on this.

I also addressed the comments given and I too asked questions on few of them. Please
help me in answering those questions. Thanks

1846. By viswesuwara nathan

Review comments addressed

Revision history for this message
Christopher Lee (veebers) wrote :

Hi Vishwa,

In the future it would be helpful to mention why you made a change different to what was suggested, it removes any need for guessing and/or assuming that there was a misunderstanding.
Also, please state the actual reason for the decision (i.e. x is better because of y, not someone else said to do it this way).

Regarding your question re: issubset, please make sure you read all the comments as I have already answered this. On March 17th I state:
"""
You're missing that possibly_known_ids is a list object and doesn't have the method issubset. You need a set object for that.
The use of issubset was also mentioned in a previous comment.

Aarons suggestion here is also to get rid of the function and just make the call where it's used. With reducing the call down to use issubset this makes sense.
Originally I suggested to have a separate function to ease testing and readability but this has evolved to the point where that's not needed.
"""

I have answered your question in line (sorry for the wall of text).
Please see here: http://paste.ubuntu.com/24226385/ to see an example of changing the test to follow the suggestions I made. Not this test now uncovers an error in the code.
Hint: it's related to the comment about what get_security_groups actually returns.

review: Needs Fixing
Revision history for this message
Christopher Lee (veebers) wrote :

Updated comment.

Revision history for this message
viswesuwara nathan (viswesn) wrote :

Hi All,

>> In the future it would be helpful to mention why you made a change different to what was suggested, it removes any need for guessing and/or assuming that there was a misunderstanding.

<<Viswa>> I definitely do this And I used repr instead of str on reading from link
http://stackoverflow.com/questions/4308182/getting-the-exception-value-in-python

>> Regarding your question re: issubset
<<Viswa>> The function is now modified and it is now single line; I didn't integrated the same in the caller and kept the function contains_only_known_instances as it is.

Major changes in the test code were done to make Mock only for lower level function (client) as discussed and mock were not introduced for the functions that were introduced by us.

1847. By viswesuwara nathan

Code review comments addressed

Revision history for this message
Christopher Lee (veebers) wrote :

One fix, one question that needs answered.

review: Needs Fixing
1848. By viswesuwara nathan

review comments addressed

Revision history for this message
viswesuwara nathan (viswesn) wrote :

Fixed and provided answer to your question. Thanks

Revision history for this message
Aaron Bentley (abentley) wrote :

Some minor issues mentioned inline. Otherwise, I think this is good to merge.

review: Approve
1849. By viswesuwara nathan

Review comments addressed

1850. By viswesuwara nathan

fixed lint errors

Revision history for this message
viswesuwara nathan (viswesn) wrote :

Fixed all the review comments.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'deploy_stack.py'
--- deploy_stack.py 2017-03-01 20:34:01 +0000
+++ deploy_stack.py 2017-03-29 17:54:35 +0000
@@ -518,6 +518,22 @@
518 os.environ['SSO_PASSWORD'], client, tear_down_client)518 os.environ['SSO_PASSWORD'], client, tear_down_client)
519519
520520
521def error_if_unclean(unclean_resources):
522 """List all the resource that were not cleaned up programmatically.
523
524 :param unclean_resources: List of unclean resources
525 """
526 if unclean_resources:
527 logging.critical("Following resource requires manual cleanup")
528 for resources in unclean_resources:
529 resource = resources.get("resource")
530 logging.critical(resource)
531 errors = resources.get("errors")
532 for (id, reason) in errors:
533 reason_string = "\t{}: {}".format(id, reason)
534 logging.critical(reason_string)
535
536
521class CreateController:537class CreateController:
522 """A Controller strategy where the controller is created.538 """A Controller strategy where the controller is created.
523539
@@ -920,7 +936,8 @@
920 if self.has_controller:936 if self.has_controller:
921 self.collect_resource_details()937 self.collect_resource_details()
922 self.tear_down(self.jes_enabled)938 self.tear_down(self.jes_enabled)
923 self.ensure_cleanup()939 unclean_resources = self.ensure_cleanup()
940 error_if_unclean(unclean_resources)
924941
925 # GZ 2016-08-11: Should this method be elsewhere to avoid poking backend?942 # GZ 2016-08-11: Should this method be elsewhere to avoid poking backend?
926 def _should_dump(self):943 def _should_dump(self):
927944
=== modified file 'substrate.py'
--- substrate.py 2017-02-13 15:53:00 +0000
+++ substrate.py 2017-03-29 17:54:35 +0000
@@ -76,6 +76,37 @@
76 subprocess.check_call(command_args, env=environ)76 subprocess.check_call(command_args, env=environ)
7777
7878
79def attempt_terminate_instances(account, instance_ids):
80 """Initiate terminate instance method of specific handler
81
82 :param account: Substrate account object.
83 :param instance_ids: List of instance_ids to terminate
84 :return: List of instance_ids failed to terminate
85 """
86 uncleaned_instances = []
87 for instance_id in instance_ids:
88 try:
89 # We are calling terminate instances for each instances
90 # individually so as to catch any error.
91 account.terminate_instances([instance_id])
92 except Exception as e:
93 # Using too broad exception here because terminate_instances method
94 # is handlers specific
95 uncleaned_instances.append((instance_id, repr(e)))
96 return uncleaned_instances
97
98
99def contains_only_known_instances(known_instance_ids, possibly_known_ids):
100 """Identify instance_id_list only contains ids we know about.
101
102 :param known_instance_ids: The list of instance_ids (superset)
103 :param possibly_known_ids: The list of instance_ids (subset)
104 :return: True if known_instance_ids only contains
105 possibly_known_ids
106 """
107 return set(possibly_known_ids).issubset(set(known_instance_ids))
108
109
79class AWSAccount:110class AWSAccount:
80 """Represent the credentials of an AWS account."""111 """Represent the credentials of an AWS account."""
81112
@@ -160,14 +191,77 @@
160 break191 break
161 return unclean192 return unclean
162193
194 def cleanup_security_groups(self, instances, secgroups):
195 """Destroy any security groups used only by `instances`.
196
197 :param instances: The list of instance_ids
198 :param secgroups: dict of security groups
199 :return: list of failed deleted security groups
200 """
201 failures = []
202 for sg_id, sg_instances in secgroups:
203 if contains_only_known_instances(instances, sg_instances):
204 try:
205 deleted = self.client.delete_security_group(name=sg_id)
206 if not deleted:
207 failures.append((sg_id, "Failed to delete"))
208 except EC2ResponseError as e:
209 failures.append((sg_id, repr(e)))
210
211 return failures
212
213 def get_security_groups(self, instances):
214 """Get AWS configured security group
215 If instances list is specified then get security groups mapped
216 to those instances only.
217
218 :param instances: list of instance names
219 :return: list containing tuples; where each tuples contains security
220 group id as first element and the list of instances mapped to that
221 security group as second element. [(sg_id, [i_id, id2]),
222 (sg_id2, [i_id1])]
223 """
224 group_ids = [sg[0] for sg in self.iter_instance_security_groups(
225 instances)]
226 all_groups = self.client.get_all_security_groups(
227 group_ids=group_ids)
228 secgroups = [(sg.id, [id for id in sg.instances()])
229 for sg in all_groups]
230 return secgroups
231
232 def terminate_instances(self, instance_ids):
233 """Terminate the specified instances."""
234 return self.client.terminate_instances(instance_ids=instance_ids)
235
163 def ensure_cleanup(self, resource_details):236 def ensure_cleanup(self, resource_details):
164 """237 """
165 Do AWS specific clean-up activity.238 Do AWS specific clean-up activity.
166 :param resource_details: The list of resource to be cleaned up239 :param resource_details: The list of resource to be cleaned up
167 :return: list of resources that were not cleaned up240 :return: list of resources that were not cleaned up
168 """241 """
169 uncleaned_resource = []242 uncleaned_resources = []
170 return uncleaned_resource243
244 if not resource_details:
245 return uncleaned_resources
246
247 security_groups = self.get_security_groups(
248 resource_details.get('instances', []))
249
250 uncleaned_instances = attempt_terminate_instances(
251 self, resource_details.get('instances', []))
252
253 uncleaned_security_groups = self.cleanup_security_groups(
254 resource_details.get('instances', []), security_groups)
255
256 if uncleaned_instances:
257 uncleaned_resources.append(
258 {'resource': 'instances',
259 'errors': uncleaned_instances})
260 if uncleaned_security_groups:
261 uncleaned_resources.append(
262 {'resource': 'security groups',
263 'errors': uncleaned_security_groups})
264 return uncleaned_resources
171265
172266
173class OpenStackAccount:267class OpenStackAccount:
174268
=== modified file 'tests/test_deploy_stack.py'
--- tests/test_deploy_stack.py 2017-03-10 00:50:40 +0000
+++ tests/test_deploy_stack.py 2017-03-29 17:54:35 +0000
@@ -51,6 +51,7 @@
51 retain_config,51 retain_config,
52 update_env,52 update_env,
53 wait_for_state_server_to_shutdown,53 wait_for_state_server_to_shutdown,
54 error_if_unclean,
54 )55 )
55from jujupy import (56from jujupy import (
56 EnvJujuClient1X,57 EnvJujuClient1X,
@@ -2673,3 +2674,60 @@
2673 wfp_mock.assert_called_once_with('example.org', 17070, closed=True,2674 wfp_mock.assert_called_once_with('example.org', 17070, closed=True,
2674 timeout=60)2675 timeout=60)
2675 hni_mock.assert_called_once_with(client.env, 'i-255')2676 hni_mock.assert_called_once_with(client.env, 'i-255')
2677
2678
2679class TestErrorIfUnclean(FakeHomeTestCase):
2680 def test_empty_unclean_resources(self):
2681 uncleaned_resources = []
2682 error_if_unclean(uncleaned_resources)
2683 self.assertEquals(self.log_stream.getvalue(), '')
2684
2685 def test_contain_unclean_resources(self):
2686 uncleaned_resources = [
2687 {
2688 'resource': 'instances',
2689 'errors': [('ifoo', 'err-msg'), ('ibar', 'err-msg')]
2690 },
2691 {
2692 'resource': 'security groups',
2693 'errors': [('sg-bar', 'err-msg')]
2694 }
2695 ]
2696 error_if_unclean(uncleaned_resources)
2697 self.assertListEqual(self.log_stream.getvalue().splitlines(), [
2698 "CRITICAL Following resource requires manual cleanup",
2699 "CRITICAL instances",
2700 "CRITICAL \tifoo: err-msg",
2701 "CRITICAL \tibar: err-msg",
2702 "CRITICAL security groups",
2703 "CRITICAL \tsg-bar: err-msg"
2704 ])
2705
2706 def test_unclean_resources_without_sg_error(self):
2707 uncleaned_resources = [
2708 {
2709 'resource': 'instances',
2710 'errors': [('ifoo', 'err-msg'), ('ibar', 'err-msg')]
2711 },
2712 ]
2713 error_if_unclean(uncleaned_resources)
2714 self.assertListEqual(self.log_stream.getvalue().splitlines(), [
2715 "CRITICAL Following resource requires manual cleanup",
2716 "CRITICAL instances",
2717 "CRITICAL \tifoo: err-msg",
2718 "CRITICAL \tibar: err-msg",
2719 ])
2720
2721 def test_unclean_resources_without_instances_error(self):
2722 uncleaned_resources = [
2723 {
2724 'resource': 'security groups',
2725 'errors': [('sg-bar', 'err-msg')]
2726 }
2727 ]
2728 error_if_unclean(uncleaned_resources)
2729 self.assertListEqual(self.log_stream.getvalue().splitlines(), [
2730 "CRITICAL Following resource requires manual cleanup",
2731 "CRITICAL security groups",
2732 "CRITICAL \tsg-bar: err-msg"
2733 ])
26762734
=== modified file 'tests/test_substrate.py'
--- tests/test_substrate.py 2017-02-13 15:53:00 +0000
+++ tests/test_substrate.py 2017-03-29 17:54:35 +0000
@@ -51,6 +51,8 @@
51 stop_libvirt_domain,51 stop_libvirt_domain,
52 terminate_instances,52 terminate_instances,
53 verify_libvirt_domain,53 verify_libvirt_domain,
54 contains_only_known_instances,
55 attempt_terminate_instances,
54 )56 )
55from tests import (57from tests import (
56 FakeHomeTestCase,58 FakeHomeTestCase,
@@ -1677,3 +1679,263 @@
1677 def test_maas_ensure_cleanup(self):1679 def test_maas_ensure_cleanup(self):
1678 substrate_account = MAASAccount('profile', 'url', 'oauth')1680 substrate_account = MAASAccount('profile', 'url', 'oauth')
1679 self.assertEqual([], substrate_account.ensure_cleanup([]))1681 self.assertEqual([], substrate_account.ensure_cleanup([]))
1682
1683
1684class FakeSecurityGroup:
1685 def __init__(self, id, instances):
1686 self.id = id
1687 self._instances = instances
1688
1689 def instances(self):
1690 return self._instances
1691
1692
1693class TestAWSEnsureCleanUp(TestCase):
1694 def test_ensure_cleanup_successfully(self):
1695 client = MagicMock()
1696 resource_details = dict()
1697 resource_details['instances'] = ["i_id1", "i_id2"]
1698 aws = AWSAccount(None, 'myregion', client)
1699 client.get_all_security_groups.return_value = [
1700 FakeSecurityGroup('sg_id1', ['i_id1', 'i_id2'])]
1701 client.delete_security_group.return_value = True
1702 uncleaned_resources = aws.ensure_cleanup(resource_details)
1703 client.delete_security_group.assert_called_once_with(name='sg_id1')
1704 self.assertEqual(client.get_all_instances.call_args_list,
1705 [call(instance_ids=['i_id1', 'i_id2'])])
1706 self.assertEqual(uncleaned_resources, [])
1707 self.assertEqual(
1708 aws.client.terminate_instances.call_args_list,
1709 [call(instance_ids=['i_id1']), call(instance_ids=['i_id2'])])
1710
1711 def test_ensure_cleanup_with_uncleaned_instances(self):
1712 client = MagicMock()
1713 resource_details = dict()
1714 resource_details['instances'] = ["i_id1", "i_id2"]
1715 aws = AWSAccount(None, 'myregion', client)
1716 err_msg = 'Instance error'
1717 client.terminate_instances.side_effect = [
1718 Exception(err_msg), Exception(err_msg)]
1719 client.get_all_security_groups.return_value = [
1720 FakeSecurityGroup('sg_id1', ['i_id1', 'i_id2'])]
1721 client.delete_security_group.return_value = True
1722 uncleaned_resources = aws.ensure_cleanup(resource_details)
1723 self.assertEqual(client.get_all_instances.call_args_list,
1724 [call(instance_ids=['i_id1', 'i_id2'])])
1725 self.assertEqual(uncleaned_resources, [
1726 {'errors': [('i_id1', "Exception('Instance error',)"),
1727 ('i_id2', "Exception('Instance error',)")],
1728 'resource': 'instances'}])
1729
1730 def test_ensure_cleanup_with_uncleaned_sg(self):
1731 client = MagicMock()
1732 resource_details = dict()
1733 resource_details['instances'] = ["i_id1", "i_id2"]
1734 aws = AWSAccount(None, 'myregion', client)
1735 client.terminate_instances.side_effect = ["i_id1", "i_id2"]
1736 client.get_all_security_groups.return_value = [
1737 FakeSecurityGroup('sg_id1', [])]
1738 client.delete_security_group.return_value = False
1739 uncleaned_resources = aws.ensure_cleanup(resource_details)
1740 self.assertEqual(uncleaned_resources, [
1741 {'errors': [('sg_id1', 'Failed to delete')],
1742 'resource': 'security groups'}])
1743 self.assertEqual(client.get_all_instances.call_args_list,
1744 [call(instance_ids=['i_id1', 'i_id2'])])
1745
1746 def test_ensure_cleanup_with_uncleaned_instances_and_sg(self):
1747 client = MagicMock()
1748 resource_details = dict()
1749 resource_details['instances'] = ["i_id1", "i_id2"]
1750 aws = AWSAccount(None, 'myregion', client)
1751 ati_err_msg = 'Instance not found'
1752 client.terminate_instances.side_effect = [
1753 Exception(ati_err_msg), Exception(ati_err_msg)]
1754 client.get_all_security_groups.return_value = [
1755 FakeSecurityGroup('sg_id1', ['i_id1', 'i_id2'])]
1756 client.delete_security_group.side_effect = EC2ResponseError(
1757 400, "Bad Request",
1758 body={
1759 "RequestID": "xxx-yyy-zz",
1760 "Error": {
1761 "Code": "Security group failed to delete",
1762 "Message": "failed"
1763 }
1764 })
1765 uncleaned_resources = aws.ensure_cleanup(resource_details)
1766 self.assertEqual(client.get_all_instances.call_args_list,
1767 [call(instance_ids=['i_id1', 'i_id2'])])
1768 self.assertEqual(
1769 uncleaned_resources,
1770 [{'errors': [
1771 ('i_id1', "Exception('Instance not found',)"),
1772 ('i_id2', "Exception('Instance not found',)")],
1773 'resource': 'instances'},
1774 {'errors': [
1775 ('sg_id1',
1776 "EC2ResponseError: 400 Bad Request\n{"
1777 "'RequestID': 'xxx-yyy-zz', 'Error': {"
1778 "'Message': 'failed', "
1779 "'Code': 'Security group failed to delete'}}")],
1780 'resource': 'security groups'}])
1781
1782
1783class TestAWSCleanUpSecurityGroups(TestCase):
1784 def test_delete_secgroup_not_in_use(self):
1785 secgroup = [("sg-foo", ["foo", "bar"])]
1786 instances = ["foo", "bar"]
1787 client = MagicMock()
1788 aws = AWSAccount(None, 'myregion', client)
1789 failures = aws.cleanup_security_groups(instances, secgroup)
1790 self.assertEqual(failures, [])
1791 self.assertEqual(
1792 client.delete_security_group.call_args, call(name='sg-foo'))
1793
1794 def test_dont_delete_secgroup_in_use(self):
1795 secgroup = [("sg-foo", ["foo", "bar", "baz"])]
1796 instances = ["foo", "bar"]
1797 client = MagicMock()
1798 aws = AWSAccount(None, 'myregion', client)
1799 failures = aws.cleanup_security_groups(instances, secgroup)
1800 self.assertEqual(client.delete_security_group.call_count, 0)
1801 self.assertEqual(failures, [])
1802
1803 def test_return_failure_on_exception(self):
1804 secgroup = [("sg-foo", ["foo", "bar"]), ("sg-bar", ["foo", "bar"])]
1805 instances = ["foo", "bar"]
1806 client = MagicMock(spec=["delete_security_group"])
1807 client.delete_security_group.side_effect = EC2ResponseError(
1808 400, "Bad Request",
1809 body={
1810 "RequestID": "xxx-yyy-zz",
1811 "Error": {
1812 "Code": "InvalidSecurityGroup.NotFound",
1813 "Message": "failed"
1814 }
1815 })
1816 aws = AWSAccount(None, 'myregion', client)
1817 failures = aws.cleanup_security_groups(instances, secgroup)
1818 self.assertEqual(client.delete_security_group.call_args_list,
1819 [call(name='sg-foo'), call(name='sg-bar')])
1820 self.assertListEqual(failures,
1821 [('sg-foo',
1822 "EC2ResponseError: 400 Bad Request\n{"
1823 "'RequestID': 'xxx-yyy-zz', 'Error': {"
1824 "'Message': 'failed',"
1825 " 'Code': 'InvalidSecurityGroup.NotFound'}}"),
1826 ('sg-bar', "EC2ResponseError: 400 Bad Request\n{"
1827 "'RequestID': 'xxx-yyy-zz', 'Error': {"
1828 "'Message': 'failed',"
1829 " 'Code': 'InvalidSecurityGroup.NotFound'}}")])
1830
1831 def test_return_mixed_response(self):
1832 secgroup = [("sg-foo", ["foo", "bar"]), ("sg-bar", ["fooX", "barX"])]
1833 instances = ["foo", "bar", "fooX", "barX"]
1834 client = MagicMock(spec=["delete_security_group"])
1835 client.delete_security_group.side_effect = [
1836 True, False]
1837 aws = AWSAccount(None, 'myregion', client)
1838 failures = aws.cleanup_security_groups(instances, secgroup)
1839 self.assertEqual(failures, [('sg-bar', 'Failed to delete')])
1840 self.assertEqual(client.delete_security_group.call_args_list,
1841 [call(name='sg-foo'), call(name='sg-bar')])
1842
1843 def test_instance_mapped_to_more_than_one_secgroup(self):
1844 # Delete security group only if it has all the mapped instances
1845 # specified in the instances list.
1846 secgroup = [("sg-foo", ["foo", "bar"]), ("sg-bar", ["foo", "baz"])]
1847 instances = ["foo", "bar"]
1848 client = MagicMock()
1849 aws = AWSAccount(None, 'myregion', client)
1850 failures = aws.cleanup_security_groups(instances, secgroup)
1851 self.assertEqual(failures, [])
1852 self.assertEqual(aws.client.delete_security_group.call_count, 1)
1853 self.assertEqual(
1854 client.delete_security_group.call_args, call(name='sg-foo'))
1855
1856
1857class TestContainsOnlyKnownInstances(TestCase):
1858 def test_return_true_when_all_ids_known(self):
1859 instances = ["foo", "bar", "qnx"]
1860 sg_list = ["foo", "bar", "qnx"]
1861 self.assertEqual(
1862 contains_only_known_instances(instances, sg_list), True)
1863
1864 def test_return_true_known_ids_are_subset(self):
1865 instances = ["foo", "bar", "qnx", "foo1"]
1866 sg_list = ["foo", "bar", "qnx"]
1867 self.assertEqual(
1868 contains_only_known_instances(instances, sg_list), True)
1869
1870 def test_return_false_when_some_ids_unknown(self):
1871 instances = ["foo", "qnx"]
1872 sg_list = ["foo", "bar"]
1873 self.assertEqual(
1874 contains_only_known_instances(instances, sg_list),
1875 False)
1876
1877
1878class TestAttemptTerminateInstances(TestCase):
1879 def test_return_error_on_exception(self):
1880 client = MagicMock()
1881 instances = ["foo", "bar"]
1882 err_msg = "Instance not found"
1883 aws = AWSAccount(None, 'myregion', client)
1884 client.terminate_instances.side_effect = Exception(err_msg)
1885 failed = attempt_terminate_instances(aws, instances)
1886 self.assertEqual(failed,
1887 [('foo', "Exception('{}',)".format(err_msg)),
1888 ('bar', "Exception('{}',)".format(err_msg))])
1889
1890 def test_return_with_no_error(self):
1891 client = MagicMock()
1892 instances = ["foo", "bar"]
1893 aws = AWSAccount(None, 'myregion', client)
1894 client.terminate_instances.return_value = ["foo", "bar"]
1895 failed = attempt_terminate_instances(aws, instances)
1896 self.assertEqual(client.terminate_instances.call_args_list,
1897 [call(instance_ids=['foo']),
1898 call(instance_ids=['bar'])])
1899 self.assertEqual(failed, [])
1900
1901 def test_returns_has_some_error(self):
1902 client = MagicMock()
1903 instances = ["foo", "bar"]
1904 err_msg = "Instance not found"
1905 aws = AWSAccount(None, 'myregion', client)
1906 client.terminate_instances.side_effect = ["foo", Exception(err_msg)]
1907 failed = attempt_terminate_instances(aws, instances)
1908 self.assertEqual(client.terminate_instances.call_args_list, [
1909 call(instance_ids=['foo']), call(instance_ids=['bar'])])
1910 self.assertEqual(failed, [
1911 ('bar', "Exception('Instance not found',)")])
1912
1913
1914class TestGetSecurityGroups(TestCase):
1915 def test_instance_managed_by_single_security_group(self):
1916 client = MagicMock()
1917 instance_sec_groups = (('i_id1', 'sg_id1'), ('i_id2', 'sg_id1'))
1918 all_sec_groups = [FakeSecurityGroup('sg_id1', ['i_id1', 'i_id2'])]
1919 aws = AWSAccount(None, 'myregion', client)
1920 client.get_all_security_groups.return_value = all_sec_groups
1921 with patch.object(
1922 aws, 'iter_instance_security_groups',
1923 autospec=True, return_value=instance_sec_groups):
1924 sec_groups = aws.get_security_groups(['i_id1', 'i_id2'])
1925 self.assertEqual(sec_groups, [('sg_id1', ['i_id1', 'i_id2'])])
1926
1927 def test_instance_managed_by_multiple_security_group(self):
1928 client = MagicMock()
1929 instance_sec_groups = (('i_id1', 'sg_id1'), ('i_id2', 'sg_id1'))
1930 all_sec_groups = [FakeSecurityGroup(
1931 'sg_id1', ['i_id1', 'i_id2']),
1932 FakeSecurityGroup('sg_id2', ['i_id1'])]
1933 aws = AWSAccount(None, 'myregion', client)
1934 client.get_all_security_groups.return_value = all_sec_groups
1935 with patch.object(
1936 aws, 'iter_instance_security_groups',
1937 autospec=True, return_value=instance_sec_groups):
1938 sec_groups = aws.get_security_groups(['i_id1', 'i_id2'])
1939 self.assertEqual(sec_groups,
1940 [('sg_id1', ['i_id1', 'i_id2']),
1941 ('sg_id2', ['i_id1'])])

Subscribers

People subscribed via source and target branches