test should return error while necessary environ not fulfilled

I don't quite see the point of this - unless maybe you make it so that the job results clearly displays which values weren't set.

As for the tests passing put failing to create the connection, did you create a wireless connection manually before they ran?

Brendan,
I'm not sure if you agree the followings are different:
1) a test is not fulfilled to run
2) a test is not fulfilled to run, but is being run and result in failure(or accidentally success) .

The 2) is what this case is, if some factors(environ here) are not satisfied in a test, then I would say running it makes no sense.

And as an user of checkbox, I have all kind of reason not setting the right configurations before running checkbox(in this case, wireless configurations in the /etc/profile), because I'm an user, a human, I can mess up any configurations easily. So we need the program to check those for us.

Let look at the test result below:
wireless/wireless_connection_open_n certification Pass Must specify an SSID to connect to. Internet connection fully established
The status is "Pass" which is not actually, is what "accidentally success" I'm talking about. If we have picked up the environ error early, then we don't need to worry about it.

Also the test command below:
trap "rm -f /etc/NetworkManager/system-connections/$OPEN_N_SSID" EXIT; ./create_connection $OPEN_N_SSID; ./internet_test --interface=`nmcli dev status | awk '/802-11-wireless/ {print $1}'`

I would suggest to use "nmcli con status" instead of "nmcli dev status".
"nmcli con status" has output like below:
DEVICE TYPE STATE
wlan0 802-11-wireless disconnected
eth0 802-3-ethernet unmanaged

The above information don't provide which AP the system is connected to.

Whereas "nmcli con status" has below output:
NAME UUID DEVICES DEFAULT VPN MASTER-PATH
ubuntu-cert-n-open e8ddb87c-0685-4f72-a476-6c846eafa73b wlan0 yes no --

We can compare the NAME to make sure the AP is what we want to connect to.

Also it might be good to concatenate each sub-command with "&&" to catch the error early.

-------------------
Now get back to your question:
did I create a wireless connection manually before they ran?
I really don't remember if I have done that or not, but it seems like I did, according to the result from "internet_test".
So we may disconnect any present wireless connection first and continue the test.

Unless we update each test to explicitly check the environment variables, there's no universal way of addressing this. I still don't see it as a big problem though, so I've assigned it an importance of Low

Rather than doing this check at runtime, I added a simple test that validates that all jobs that have a user also use environ for all the variables referenced in the command.

This still leaves us uncovered in two cases:

1- A variable is referenced in a script rather than in the job file. This is bad practice as hardcoding environment variable names in scripts is too opaque.

2- User-written jobs. Since the tests work only on the jobs we ship with checkbox, anything outside that will not be tested.

Still, this will prevent us from shipping broken jobs in checkbox itself which I think is an improvement anyway.