I added other tests that exercise the interfaces between the methods; the tests that actually try to use openssl go beyond them. I'll disable these tests for now and, in a future merge, move them into cloud_test where they can do their thing. I have an objective for the first half of 2019 to improve the cloud-meets-distro testing of cloud-init, so it'll be easy to get the time I need to do it right.

Thanks for all the explanation. I don't suppose there's a doc anyplace that I was supposed to read that explained this stuff; although I'd be embarrassed at jumping in without doing the reading, I'd certainly go back and read it now.

On a related thought: would it be acceptable to pivot to using pyOpenSSL or even pyca/cryptography as a replacement for subp() invocation of command line tools? Not in this merge, but some time in the next few months?

Jason

-----Original Message-----
From: <email address hidden> <email address hidden> On Behalf Of Chad Smith
Sent: Thursday, February 21, 2019 1:42 PM
To: <email address hidden>
Subject: Re: [Merge] ~jasonzio/cloud-init:fingerprint into cloud-init:master

Thanks for the response Jason,

> Do the tests intended for cloud_test live in a separate location from the unittests, or are they marked or flagged so as to run only in that environment?

cloud-init/tests/cloud_tests is the location Ryan was referring to instead of something under tests/unittests. We have individual integration tests all of which are run against a given platform. Integration test documentation is at https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcloudinit.readthedocs.io%2Fen%2Flatest%2Ftopics%2Ftests.html&amp;data=02%7C01%7Cjason.zions%40microsoft.com%7Cb9f00b35607f449fed3e08d698456ae4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636863821245995729&amp;sdata=qLHKaOO5%2F29UD2uBYodEw7bqvlHCQ5PNU7Sh7H9W%2Bx0%3D&amp;reserved=0


It feels like you might be able to create an integration test that exercises some of this logic directly by adding an integration test under tests/cloud_tests/testcases/bugs/lp1811265.(yaml|py).

You'd need a yaml file which can feed #cloud-config and collect_scripts directives into an instance under test and use collect_scripts to grab any response content you expected to see when processing certs with azure_helper.OpenSSLManager.  The python file is used to process the collected_script results and report failure conditions for unexpected content/results.

Additionally you could limit that integration test to only run on a specific platform by raising a SkipTest from the python test module for your specific test case.

        if self.platform != 'lxd':
            raise SkipTest(
                'Skipping lxd instance-data.json on %s' % self.platform)



> To move forward expeditiously with this merge, though, the fastest approach is for me to either disable the tests or delete them. There's no point in mocking out the subp call; there are other tests that do that.

I'd agree that integration tests probably have more value here than the mocked subp calls, but there is still value in exercising some of the interfaces called between your methods and to validate the contained logic. I'm all for simplifying or reducing what your unittests are asserting. If you are able to put an integration test together, I'd wouldn't also be less concerned your unittest coverage.


Cloud-init's unittests are run not just for continuous integration, but also during package builds in restricted environments which do not come with some runtime dependencies like openssl etc. On the other hand, our integration tests are run on "complete" systems across a variety of platforms (instead of just our build/CI environment) and exercise those external dependencies directly on a variety of platforms and distros.

-- 
https://nam06.safelinks.protection.outlook.com/?url=https:%2F%2Fcode.launchpad.net%2F~jasonzio%2Fcloud-init%2F%2Bgit%2Fcloud-init%2F%2Bmerge%2F363445&amp;data=02%7C01%7Cjason.zions%40microsoft.com%7Cb9f00b35607f449fed3e08d698456ae4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636863821246005723&amp;sdata=kUHQU1U%2BuYtWoEisJzlMniHgCTFp%2BYACF2RqIIKWubo%3D&amp;reserved=0
You are the owner of ~jasonzio/cloud-init:fingerprint.