wireless/wireless_connection_open_n and wireless/wireless_connection_wpa_n are skipped on some systems

Hi,

These tests need "n: supported" from the 80211_resource.

Are we sure that the systems support 802.11n? Check in the networks menu in Network Manager to see if it detects any of our N-only networks.

Also, to diagnose what checkbox is seeing, can you please run this in one of those systems?

$ /usr/lib/2013.com.canonical.certification\:plainbox-resources/bin/80211_resource

You should get something like:
IBSS: supported
managed: supported
monitor: supported
n: supported
bg: supported
band_5GHz: supported

What the tests are looking for is "n: supported", see this from the failure: Job requirement not met: 'IEEE_80211.n == 'supported''

I used checkbox legacy to do this test, so

ii checkbox 0.17.9.1~ubuntu12.04.1 System testing application
ii checkbox-certification 0.19~ubuntu12.04.1 Checkbox Certification Tests
ii checkbox-certification-client 0.19~ubuntu12.04.1 Client Certification
ii checkbox-certification-tools 0.19~ubuntu12.04.1 Checkbox Certification Tools
ii checkbox-qt 0.17.9.1~ubuntu12.04.1 QT4 interface for checkbox
ii python3-checkbox 0.17.9.1~ubuntu12.04.1 CheckBox python3 library

and from checkbox/script/80211_resource I got
IBSS: supported
managed: supported
bg: supported
band_5GHz: supported

the system 201112-10226, for example, should has the resource because it was certified on 12.04 with
wireless/wireless_connection_open_n test pass
wireless/wireless_connection_wpa_n test pass
https://certification.canonical.com/hardware/201112-10226/submission/4520/test-results/pass/?page=3

I manually click the network-manager indicator could see open/bg n APs, and also could build the connection with them.

Marking back as Confirmed, as Tai provided the requested information

Aha, so according to 80211_resource, n is *not* supported:

IBSS: supported
managed: supported
bg: supported
band_5GHz: supported

Tai, can you please run this and attach the output? it's the raw data from the 'iw' tool, the 80211_resource binary is a miniature version of this. Maybe iw is returning bad information.

iw phy0 info

with iw version 3.2
"iw phy info" output for 201112-10226

iw output does not contain information about HT capabilities, and 80211_resource checks if NL80211_BAND_ATTR_HT_CAPA is true. I think that the Broadcom driver does not report those fields as the band Capabilities are just not here in logs. So I suspect the driver.

Both iw and 80211 did their job but apparently this chip supports 80211n (deducted from the product name). If we can trust the driver I'm wodering how Network manager allow N connections to be set up.

https://certification.canonical.com/hardware/201103-7380/submission/97936/devices/?page=2

More comments associated to comment 6:

The driver version in the previous SRU [1] which automatically run and passed wireless_n testing is
          bcmwl-kernel-source 6.20.155.1+bdcom-0ubuntu0.0.2 [2]

it is the same version in the run which raising this bug. [3]

-----------

For the network-manager package between two SRUs above, they are
earlier SRU [4]: network-manager 0.9.4.0-0ubuntu4.4
SRU with this issue [5]: network-manager 0.9.4.0-0ubuntu4.4.1

[1] https://certification.canonical.com/hardware/201103-7380/submission/96458/test/99/result/7124011/
[2] https://certification.canonical.com/hardware/201103-7380/submission/96458/packages/?term=wl
[3] https://certification.canonical.com/hardware/201103-7380/submission/97936/packages/?term=wl
[4] https://certification.canonical.com/hardware/201103-7380/submission/96458/packages/?term=network-manager
[5] https://certification.canonical.com/hardware/201103-7380/submission/97936/packages/?term=network-manager

201103-7380 with
  * saucy
  * 3.11.0-20-generic
80211_resource could find wireless_n
Thus checkbox passed the wireless n test as well.

201112-10226
  * saucy
  * 3.11.0-20-generic
80211_resource could not find wireless_n. wireless does not be enabled.

network manager may have bug on that

nmcli eth1 wifi list shows
all the APs in the lab RATE is 54MB/s

but I have confirmed the AP settings for wireless n in the lab are correct.

Hm, so apparently as far as network manager is concerned, the chipset doesn't support N; I guess network manager simply doesn't care and just asks the driver to connect to networks. I don't think NM cares about the protocol, really; I don't see that specified anywhere when setting up a connection.

I don't think we can really fix this one if the driver is not reporting its capabilities correctly. We could try coming up with some sort of workaround or heuristic.

We know broadcom chips misreport this, we can compile a list of broadcom PCI IDs that are supposed to support N, and if the card matches one of those, we just blindly report "n". This is better than reporting "n" support for ALL broadcom chipsets.

It's a bit more work for us but it may be the most reliable solution.

Another possible solution, of course, is to remove the "n" requirement from the "n" tests; in this case they will always run (and fail on systems that don't really support them) but at least they will run.

Yet another solution would be to wrap the resource in a shell script that also looks at output of "sudo iwlist wlan0 scanning" and checks if it sees the "n" access points. This is bad because it relies on an assumption that may not be true, but it would allow our tests to run correctly.

Marking as a blocker because without this, the N tests which are required for cert will simply not run.

I reproduced the case which
the wireless n is automatically tested and passed
by checkbox 0.16.8 (bzr rev 2294) on 201112-10226.
The wireless n is automatically launched because
it does not require the 80211 n resource in the job file,
say wireless.txt.in.

The requirement of the 80211 resource is added in the bzr rev 2617,
with a timestamp on 14 Jan 27th.
This is consistent with the date of the last submission
which automatically launched and passed the wireless n testing.
Please refer to https://certification.canonical.com/hardware/201112-10226/submissions/

I also tested elder kernels (3.2.0-29-generic, 3.2.0-59-generic for precise)
together with bcmwl-kernel-source (6.20.155.1+bdcom-0ubuntu0.0.2)
'iw phy0 info' doesn't show the HT is supported.
This means the wireless n protocol of the wireless cards
is not supported very well long time ago and
checkbox does not catch that.
The good news is, it could work for some of wirless cards in saucy kernels, please see comment#8.

So, my conclusion is, our checkbox is on the right way.
We have a more robust wireless testing jobs now,
which requires the necessary n protocol resource, than we did before.
Now checkbox catches the bad wirless support.
The wireless modules, mainly broadcom's product, seem not to support very well.
And it may or may not be caused by the driver bcmwl-kernel-source.
It doesn't report to iw there is n protocol support.
Please see comment#8, the chipset supports protocol n by saucy kernel.

I don't think the skipped wireless n testing is a checkbox bug.
Actully this is the correct behavior
when the wireless module support, including kernels, wl kernel module/driver or hardware,
doesn't work very well.
Checkbox should catch the bad support.

I prefer to
* set the status as invalid.
* file a bug report for bcm driver which does not report correct driver info.
* (optional) file a bug for network-manager which may use a n protocol as a g protocol.
* keep our resource requirement. e.g. we don't need to modify our code.

So, as reported by the resource job, N is not supported, but it can still connect to our router and pass the test?

Can you still connect to the N router by using the Network manager?

the answer for comment #13 is "yes"

I removed the tag since Daniel's branch is now merged (removing the use of the resource job output for N tests)