Merge ~jocave/certification-docs:add-iot-coverage-guide into certification-docs:master

[Jonathan Cave]

Add the IoT coverage guide for UC 22 / 22.04 to the certification-docs repository. I think we would like to move to the devices docs being publishable in html format as well as PDF. The git versioning is also appealing.

I've done a quick conversion of the current google doc to RST based on the server coverage guide and checks it produces a reasonable looking PDF.

I didnt want to mess with the current Makefile targets (e.g. the coverage one) in case you have anything using those and expecting particular output.

[Jeff Lane  (bladernr)]

I think this needs some things, but maybe that can be handled separately from this. Just to log them, what I think we need:

1: additions to the Makefile to build PDF and HTML versions of this doc
2: Packaging updates to build server and devices packages (unless you think it makes sense to just put them into a single monolithic package).

[Rod Smith (rodsmith)]

I concur with Jeff on his point #1. Although the added file builds cleanly to both PDF and HTML when doing a "make all", new Makefile targets to build just this new file are highly desirable. (Without this change, building this one document requires building everything else, and that build could conceivably fail if there's a problem building something else -- say, if a change to rst2pdf causes problems on some documents but not others.) This change could be put into a separate MR, but IMHO it'd be cleaner to just add it to this MR. Given the current Makefile target naming, I'd suggest "iot_coverage" and "iot_coverageh" as target names for PDF and HTML, respectively, but feel free to use what you like.

Changes to packaging rules are a bit iffier. Currently, we build MANIACS and the STG for the Debian package; the other documents are not included in our Debian package. Presumably the IoT documents would not go in our current package, but might go in another one; but this is a matter we might want to discuss to decide on a plan going forward. We currently track changes to the project in debian/changelog, and adding a note about this addition there might be reasonable, although as that file technically tracks changes to the .deb file's contents, if this new document won't go there, omitting changes to this file seems OK. (Note that I've recently submitted another MR [#423024] that makes changes to debian/changelog, so another change to this file will create a conflict unless it's based off, and merged after, my recent MR.) Other packaging file changes would be to debian/control, debian/debinstall, and debian/README.source, if it's decided to add this document to the .deb file.

So to summarize, I'm marking this "Needs Fixing" with the intent that new Makefile entries be added, but packaging changes can wait and will likely either never be made or will create a separate .deb file.

review: Needs Fixing

[Jonathan Cave (jocave)]

Added the Makefile entries

[Rod Smith (rodsmith)]

LGTM!

review: Approve

diff --git a/Coverage_Guide_IoT-22.04.rst b/Coverage_Guide_IoT-22.04.rst
 new file mode 100644
index 0000000..9d5f6f0
--- /dev/null
+++ b/Coverage_Guide_IoT-22.04.rst
@@ -0,0 +1,387 @@
+===========================================================================
+Internet of Things Certified Hardware Coverage for Ubuntu Core 22 and 22.04
+===========================================================================
+
+.. include:: <isonum.txt>
+
+.. header:: |ubuntu_logo|
+
+.. |ubuntu_logo| image:: images/logo-ubuntu_su-white_orange-hex.png
+   :scale: 20%
+
+.. footer:: |canonical_logo|
+
+.. |canonical_logo| image:: images/logo-canonical_no-tm-white-hex.png
+   :scale: 10%
+
+.. raw:: pdf
+
+   PageBreak oneColumn
+
+.. contents::
+
+.. raw:: pdf
+
+   PageBreak
+
+.. role:: green-text
+
+Introduction
+============
+
+This document lists the coverage for certification of Internet of Things devices
+with Ubuntu Core 22 or Ubuntu 22.04. The guide applies to devices submitted to 
+Canonical through one of the following programmes:
+
+- IoT Devices Enablement Programme with Certification
+- IoT ODM Partner Programme
+ 
+The following test categories are specified: 
+
+Blocking
+  Features that are required for certification. If any of the 
+  tests in the required category fails, the certification will fail.
+
+Non-blocking
+  Features that are tested, but that don’t block certification. If any of the 
+  tests under the optional category fail, a note will be added to the 
+  certificate to warn the potential customer or user. 
+
+Untested
+  The items in the Untested category are just reference items.
+  Anything not explicitly called out in the Blocking or Non-blocking categories
+  can be considered part of the Untested category. We will consider adding 
+  more tests as needed.
+
+Note: only categories of hardware are tested and not specific types of 
+hardware. For example, tests are run to verify USB controllers work, but the 
+type of peripheral(s) used during those tests are not specified. Coverage is 
+flexible based on customer requirements (for example, if a device’s use cases 
+don’t require LEDs, then LEDs can be untested)
+
+Full test descriptions can be found in Canonical certification site for 
+partners:
+
+    http://certification.canonical.com
+ 
+
+Blocking
+========
+
+Audio
+-----
+
+Output needs to be undistorted between 0%-100%. Output lines tested:
+
+- Internal speakers
+- 3.5mm headphones
+- HDMI audio output
+- DisplayPort audio output
+
+Input needs to be recorded undistorted between 0%-100%. Input lines tested:
+
+- Internal microphone
+- 3.5mm microphone
+
+Plug detection: when a new audio line input or output is plugged in the system,
+it needs to be recognized.
+
+Bluetooth
+---------
+
+Bluetooth LE (Smart and Smart Ready) is tested for device scanning and pairing.
+Apart from pairing, several profiles are specifically tested and required:
+
+- Eddystone Beacon
+- HID Over GATT Profile (HOGP), Low-Energy keyboard or mouse with basic functionality
+
+CPU
+---
+
+x86_64 and ARM processors are tested to ensure proper functionality. We will 
+test specific features as:
+
+- CPU's performance states (frequency up and down in runtime)
+- CPU's sleep states (cpu on and off in runtime)
+- Running CPU at its maximum frequency
+
+We will also include a general stress test performed for 120 minutes to verify 
+that the system can handle a sustained high load for a period of time. This 
+test uses the tool “stress” available in the Universe repositories.
+
+For Intel CPU’s, the IPDT (Intel Processor Diagnostic Tool) test suite will be 
+run.  The diagnostic checks for brand identification, verifies the processor 
+operating frequency, tests specific processor features, and performs a stress 
+test on the processor.
+
+Ethernet
+--------
+
+Connections are tested for functionality, but not for performance.
+
+Firmware
+--------
+
+The Ubuntu image must be installed using the factory default bootloader 
+firmware (for example BIOS,  UEFI or uboot as applicable) and with the default 
+options (including SecureBoot, if that’s the default setting). Firmware needs 
+to be compliant with Canonical Firmware Test Suite (FWTS).
+
+It is recommended that after running Canonical fwts with the list of tests 
+defined in the Appendix A, ideally, no CRITICAL or HIGH failures should be 
+reported, but those are not automatically certification blockers.
+
+GPIO
+----
+
+We test the functionality of individual GPIO lines when the associated 
+controller driver in the kernel implements a GPIO Sysfs Interface via the 
+gpiolib implementers framework. In such cases, the GPIO system may be tested 
+in two ways:
+
+- Direct:
+
+  - GPIO controllers are exposed through sysfs
+  - GPIO lines are accessible by the user
+
+- Indirect:
+  
+  - Communication with device connected via GPIO
+
+I2C
+---
+
+All devices attached to the I2C bus must be detectable,  This includes:
+
+- Temperature sensors
+- Humidity sensors
+- Accelerometers
+
+LEDs
+----
+
+When LEDs exist, they will be tested by following some basic expectations here.
+The actual behavior may vary depending on the hardware design.  To ensure that 
+the behavior is working as expected, please be sure to test against 
+specifications obtained from OEM, as each OEM may have different defined 
+behavior for LEDs.  The following LEDs are tested:
+
+- Power 
+- Serial Port LEDs (indicating activity)
+
+Media Card readers
+------------------
+
+Media Card readers are tested for read and write for the following type of 
+cards:
+
+- CF
+- MMC
+- MS
+- MSP
+- SD
+- SDHC
+- SDXC
+- XD
+
+Memory
+------
+
+Proper detection of the amount of memory installed is required (the amount of 
+memory installed is the memory seen by the OS).
+
+Monitors
+--------
+
+Each of the available external video ports (supported ports are HDMI, 
+DisplayPort, DVI) are tested one by one. Output to the display must work i.e. 
+a console is presented.
+
+Power Management
+----------------
+
+Warm reboot is tested such that the system must be able to perform the reboot 
+command and services must be restarted such that systemctl does not identify a 
+failed state.
+
+Cold reboot is performed where an RTC is available (see next section). The 
+wakealarm is used to reboot the system after a period of rest and services must 
+be restarted such that systemctl does not identify a failed state.
+
+Real-Time Clock (RTC)
+---------------------
+
+If present on the device, the device must have a working real-time clock. This 
+will be tested by scheduling a wake alarm to bring the system up after a halt.
+
+Serial Ports
+------------
+
+Tests are carried out on ports that provide access via the Linux tty layer. The
+exact tests performed depend on the physical characteristics of the 
+driver/receiver hardware. The possible tests include:
+
+- Ensure expected number of devices are available
+- Looped tests:
+
+  - RS232 Ports: perform loopback test to ensure RX/TX
+  - RS422/485 Ports: connect together to ensure RX/TX
+
+- Machine to Machine tests: confirm that a connection can be made to another 
+  PC device and RX/TX is operational
+
+Internal Storage
+----------------
+
+All internal storage devices are tested to be properly detected. An in-house 
+performance test is run on all disks with read performance of 15MB/s required 
+to pass.
+
+USB controllers
+---------------
+
+USB 2.0
+^^^^^^^
+
+USB storage devices must work on all available USB ports. USB Human Interface 
+Devices (HID), specifically keyboard or mouse, should be working properly on 
+any USB port.
+
+USB 3.0
+^^^^^^^
+
+USB storage devices must work on all available USB ports. USB Human Interface 
+Devices (HID), specifically keyboard or mouse, should be working properly on 
+any USB port.
+
+USB Type C (USB 3.1)
+^^^^^^^^^^^^^^^^^^^^
+
+USB Type C (USB 3.1) supports various types of devices (e.g. Video, Power) 
+through the use of adapters or peripherals. The following adapters/peripherals 
+should work:
+
+- Storage devices
+- Keyboard or mouse (basic functionality)
+- When DisplayPort over USB Type-C is advertised:
+
+  - Display hot plugging and the following display are required to work: 
+    mirrored, extended, internal only, external only.
+  - Audio output needs to be undistorted over this port.
+
+Wireless Networking
+-------------------
+
+Wi-Fi interfaces are tested for connection to access points configured for 
+802.11 b/g/n/ac/ax protocols.
+
+Wireless Wide Area Network
+--------------------------
+
+WWAN interfaces are tested for connection to 3G/4G/LTE services.
+
+Thunderbolt
+-----------
+
+Thunderbolt featues tested:
+
+- Audio output must be undistorted over this port.
+- Storage devices with hot plugging capability should work when BIOS is set to 
+  “No security” option.
+- Monitor hot plugging including different modes (mirrored, extended, internal 
+  only, external only) are required to work.
+- Daisy-chaining devices should work with a storage device and a monitor 
+  chained together. 
+
+TPM
+---
+
+TPM 2.0 functionality will be tested using FWTS. It is required that all 
+commands necessary to support Ubuntu’s Full Disk Encryption functionality are 
+supported.
+
+Non-blocking
+============
+
+CANBus
+------
+
+Devices that support the SocketCAN standard are tested to ensure that the 
+adapter is present and can be communicated with via CANBus configuration 
+commands.  
+
+Media Card readers
+------------------
+
+Media Card readers are also tested for read and write for the following type or
+cards: 
+
+- MMC (before and after suspend)
+
+Power Management
+----------------
+
+Suspend/Resume
+^^^^^^^^^^^^^^
+
+For x86 devices, a 30 cycle suspend/resume stress test is performed using the 
+FWTS. The suspend mode (e.g. S3, S2Idle) used during the test will be the 
+default for the system under test. The test is passed if all 30 cycles complete
+without failure. Any errors reported in the fwts log for the 30 cycle 
+suspend/resume stress test are informational only and do not affect the outcome
+of the test, however, we do recommend examining and fixing any failures noted, 
+as they indicate firmware non-compliance with standards.
+
+In addition a single suspend is performed across which  the following features
+and devices are tested:
+
+- CPU
+- Memory
+- Networking (Wifi, Ethernet)
+- Audio
+- Bluetooth
+- USB controllers
+- Input devices
+- Mediacards
+
+Watchdog Timer
+^^^^^^^^^^^^^^
+
+A test will be performed to verify that any kernel modules needed for watchdog
+timers are loaded and working as expected. 
+
+LEDs
+----
+
+The following LEDS will be tested for functionality:
+
+- Cloud LED
+- Wireless LED
+- Bluetooth LED
+- WWAN LED
+
+Appendix A. FWTS tests
+======================
+
+As part of the certification process, we run a series of firmware tests that 
+are part of the Canonical Firmware Test Suite. In general, any HIGH or CRITICAL
+error found in the fwts log can cause potential errors in the system and should
+be looked at by OEMs/ODMs.
+
+===========   ============   ===========
+Category      Test Item      Description
+===========   ============   ===========
+Information   acpidump       Check ACPI table acpidump
+Information   version        Gather kernel system information
+ACPI          acpitables     ACPI table settings sanity checks
+ACPI          apicinstance   Check for single instance of APIC/MADT table
+ACPI          hpet_check     High Precision Event Timer configuration test
+ACPI          mcfg           MCFG PCI Express* memory mapped config space
+ACPI          method         ACPI DSDT Method Semantic Tests
+CPU           mpcheck        Check Multi Processor tables
+CPU           msr            CPU MSR consistency check
+CPU           mtrr           MTRR validation
+System        apicedge       APIC Edge/Level Check
+System        klog           Scan kernel log for errors and warnings
+===========   ============   ===========
+
diff --git a/Makefile b/Makefile
index 7b98b35..43b4075 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
 SHELL = /bin/sh
 RST2PDF=rst2pdf
 RST2HTML=rst2html
-DOC_NAMES=Test_Case_Guide-20.04 Test_Case_Guide-18.04 Programme_Guide Coverage_Guide Self-Test_Guide MAAS_Advanced_Network_Installation_And_Configuration_Scripted Policy_Guide
+DOC_NAMES=Test_Case_Guide-20.04 Test_Case_Guide-18.04 Programme_Guide Coverage_Guide Self-Test_Guide MAAS_Advanced_Network_Installation_And_Configuration_Scripted Policy_Guide Coverage_Guide_IoT-22.04
 HTML_NAMES=$(DOC_NAMES:=.html)
 PDF_NAMES=$(DOC_NAMES:=.pdf)
 
@@ -23,6 +23,8 @@ stg: Self-Test_Guide.pdf Self-Test_Guide.pdf
 
 coverage: Coverage_Guide.pdf
 
+iot_coverage: Coverage_Guide_IoT-22.04.pdf
+
 policy: Policy_Guide.pdf
 
 programme: Programme_Guide.pdf
@@ -35,6 +37,8 @@ stgh: Self-Test_Guide.html Self-Test_Guide.html
 
 coverageh: Coverage_Guide.html
 
+iot_coverageh: Coverage_Guide_IoT-22.04.html
+
 policyh: Policy_Guide.html
 
 programmeh: Programme_Guide.html