Code review comment for lp:~canonical-platform-qa/ubuntu-system-tests/app-launch-introspect

> The reason for doing this way is to write the tests without knowledge of the
> type of app being used. - Hence using the config value.
>
> Although the app-launch tests could be implemented with snap/deb specific
> knowledge, it wouldn't be good to have a suite of sanity or regression tests
> where we need to have specific test cases for specific app versions.

This is actually my point, with the approach I suggested, the tests do not know if they are dealing with a snap or a deb, they are just dealing with an application. So for example in the case of the app launch test cases, the test would call calculator.launch() or web_browser.launch(), and since they are both extending from different classes, the apps will be launched differently.

Right now, we specify a mode and we skip those tests that do not comply with that mode right? With the approach I suggest, we don't specify a mode at all. Every app (either deb or snap) knows how to launch itself

 - The way
> I see it is you would want to run a sanity suite on a deb image for deb apps
> and then run exactly the same set of tests on same image for snap apps.

There you go...so this is what we need to agree on. The way I see it, you have an image and you want to run the sanity suite against that image, no matter what that image has, either debs or snaps or both, as you said, the test do not care

If we follow the mode approach, we would have to start skipping test cases based on the mode right? just as tests are skipped here. And when a deb is moved to snap we'll need to do the other way around. With the approach that I proposed, we don't skip tests at all and when a deb app is snapped, then we just change the inheritance from Deb to Snap

With
> the current way that would just mean changing the config value and re-running
> the same set of tests on same image. Similarly for a snap based image, you
> would want to run the suite using snap apps only.
>

From my point of view the suite of tests is exactly the same either in a pure snap, pure deb or in a mix of debs and snaps like we have now. The tests do not change, what changes, is how you start ans introspect a app or a snap

> The jenkins jobs I am creating will do this from a single job. See:
> https://code.launchpad.net/~canonical-platform-qa/qa-jenkins-jobs/ust-add-app-
> modes/+merge/319313
>

I was not clear. What I don't want to do is to re run the tests and merge the results when we could run the tests once and avoid the post processing

> I'm not sure if I understood your concerns correctly or not, what do you
> think?

Let's go through this over the phone. It'll be much easier, and we can get everyone opinion on this