Merge ~robbat2/cirros:gentoo into cirros:master

[Robin H. Johnson]

Merge of Gentoo CirrOS improvements
- PPC64 work: matching strings, more ibm* modules, virtio input, USB input
- Improve layout of src/etc/modules with sections
- More virtio modules

[Scott Moser (smoser)]

Thanks for putting together a merge propsal.
A tiny things:
 - fixes comments are done with:
   LP: #XXXXXX

Generally I dont have interest in just adding lots of modules,
as that makes things bigger. I've not looked at the size change
that would be caused, so I dont know specifically about these
modules.

Specifically for the Hyper-v modules there is an issue that
loading those modules causes 5 second delay even if the hardware
is not there. Unless that is now known fixed, then we can't
really just add them to the list as they have a negative affect
everywhere.

I'm generally OK with the powerpc changes.

the BR_VERSION change is probably not really all that useful in
reality. Its quite a pain to move buildroot versions (more
than just changing that file :)

224059f... by Robin H. Johnson on 2017-11-09

src/etc/modules: config_drive cdrom support.

config_drive can present as a SCSI CDROM, so we need suitable modules
loaded to access it.

Added in a new section for block storage.

There is no growth in size on Ubuntu, because all 3 modules are
compiled-in already, but the module loading is useful for other kernels
where they might be modular.

LP: #1715009
Signed-off-by: Robin H. Johnson <email address hidden>

[Robin H. Johnson (robbat2)]

I'll drop Hyper-V to a seperate branch for now, and produce size stats for you.

ce16a57... by Robin H. Johnson on 2017-11-09

src/etc/modules: add virtio_input

Add support for input devices on the virtio bus.

Size impact:
------------
(Uncompressed modules, size in bytes, on Ubuntu 4.4.0-28.47)
amd64 15134
arm64 18678
armhf 13938
i386 11382
ppc64 19366
ppc64el 18590

Requires: linux>=4.1
Requires: qemu>=2.4
Requires: libvirt>=1.3.0
See: https://lwn.net/Articles/638075/
See: https://www.kraxel.org/blog/2015/06/new-member-in-the-virtio-family-input-devices/
See: https://www.kraxel.org/blog/2016/09/using-virtio-input-with-libvirt/

57933e8... by Robin H. Johnson on 2017-11-09

src/etc/modules: add qemu_fw_cfg

Add support for input devices on the virtio bus.

Size impact:
------------
(Uncompressed modules, size in bytes)
amd64 21374 **4.13.0-17.20
arm64 20475 **4.14.0-0.rc8.git2.2.fc28
armhf N/A
i386 14206 **4.13.0-17.20
ppc64 N/A
ppc64el N/A

Requires: linux>=4.6
Requires: qemu>=2.6
See: http://www.contrib.andrew.cmu.edu/~somlo/QEMU_fw_cfg/

4091d98... by Robin H. Johnson on 2017-11-09

src/etc/modules: add 9pnet_virtio

Support host filesystem passthrough

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 163564
arm64 182604
armhf 126744
i386 110800
ppc64 192188
ppc64el 188580

See: https://www.linux-kvm.org/page/9p_virtio

5c330ad... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvscsi

Newer kernels renamed ibmvscsic to ibmvscsi.
Try to load both names, so that it always works.
No impact in size of included modules.

a2a8cb9... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvnic

PPC64 systems can use either ibmvnic as an alternative to ibmveth, mostly for
higher performance networking in LPAR.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 117310
ppc64el 118758

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters

1236ac0... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvfc

PPC64 systems can use ibmvfc for virtualized fibre channel access in
LPAR environment.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 278228
ppc64el 274804

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters
Signed-off-by: Robin H. Johnson <email address hidden>

[Robin H. Johnson (robbat2)]

I split out many of the module changes individually now, for ease of later review/revert.

Overall growth in raw module bytes:
   amd64 553698
   arm64 632427
   armhf 420524
    i386 373338
   ppc64 1003526
 ppc64el 1027954

[Scott Moser (smoser)]

How does this look:
https://code.launchpad.net/~smoser/cirros/+git/cirros/+ref/robbat2-cherry-picks1

That grabs a fair number of them, the most objectionable change I made
was to just not carry the old 'ibmvscsic' name.

[Robin H. Johnson (robbat2)]

Some comments inline, re ppc* matching.

My objectives was to make this work with the existing system, but also ensure that it was more likely to work with other kernels. Carrying both names of ibmvscsic and ibmvscsi for example. Only one will exist in a given kernel, but we can't tell ahead of time.

Why not merging the USB parts?

[Scott Moser (smoser)]

> Some comments inline, re ppc* matching.
>
> My objectives was to make this work with the existing system, but also ensure
> that it was more likely to work with other kernels. Carrying both names of
> ibmvscsic and ibmvscsi for example. Only one will exist in a given kernel, but
> we can't tell ahead of time.

Yes, but 'ibmvscsic' will only exist in old kernels, those < 4.4.
Do you have an actual use case where you'd use something older than that ?

Unmerged commits

224059f... by Robin H. Johnson on 2017-11-09

src/etc/modules: config_drive cdrom support.

config_drive can present as a SCSI CDROM, so we need suitable modules
loaded to access it.

Added in a new section for block storage.

There is no growth in size on Ubuntu, because all 3 modules are
compiled-in already, but the module loading is useful for other kernels
where they might be modular.

LP: #1715009
Signed-off-by: Robin H. Johnson <email address hidden>

d7fc61b... by Robin H. Johnson on 2017-11-09

src/etc/modules: USB input devices

Add new section for input devices, as used by USB keyboards.

USB mice & tablets are NOT included, as CirrOS has no need of them.

This provides working console input on OSUOSL's POWER OpenStack development
environment (https://osuosl.org/services/powerdev/), which presents as
XHCI/EHCI with USB keyboard.

The large module growth comes mostly from the HID module, which represents ~61%
of the new bytes.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 324584
arm64 328312
armhf 248476
i386 215660
ppc64 359440
ppc64el 353024

Thanks: Lance Albertson <email address hidden>
Signed-off-by: Robin H. Johnson <email address hidden>

9888a8c... by Robin H. Johnson on 2017-11-09

src/etc/modules: USB host controllers

Add new section for USB host controllers, in preparation for USB input devices.

Includes all QEMU-support host controllers: XHCI, EHCI, OHCI, UHCI.
New usage should prefer virtio-input or at the XHCI controller.

This provides working console input on OSUOSL's POWER OpenStack development
environment (https://osuosl.org/services/powerdev/), which presents as
XHCI/EHCI with USB keyboard.

Many systems may have these modules built-in, but these list is complete
to ensure it works.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 13494
arm64 61618
armhf 17154
i386 10046
ppc64 18390
ppc64el 56490

Thanks: Lance Albertson <email address hidden>
Signed-off-by: Robin H. Johnson <email address hidden>

a912bc7... by Robin H. Johnson on 2017-11-09

src/etc/modules: networking tweak.

- add section
- add lots of commented out modules based on bugs filed in LP, do not
  want the size bloat yet.

Signed-off-by: Robin H. Johnson <email address hidden>

9458109... by Robin H. Johnson on 2017-11-09

src/etc/modules: add nls_utf8, nls_ascii

Improve VFAT support by adding two more very common encodings it can
use: UTF-8 and ASCII.

nls_ascii is frequently compiled into kernels, thus this fix is only
applicable to very modular kernels.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 15548
arm64 20740
armhf 14212
i386 11244
ppc64 18604
ppc64el 17708

Signed-off-by: Robin H. Johnson <email address hidden>

04091c3... by Robin H. Johnson on 2017-11-09

src/etc/modules: filesystem tweak.

Split to section for further changes.
Document that nls_* are included for VFAT to work.

Signed-off-by: Robin H. Johnson <email address hidden>

1236ac0... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvfc

PPC64 systems can use ibmvfc for virtualized fibre channel access in
LPAR environment.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 278228
ppc64el 274804

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters
Signed-off-by: Robin H. Johnson <email address hidden>

a2a8cb9... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvnic

PPC64 systems can use either ibmvnic as an alternative to ibmveth, mostly for
higher performance networking in LPAR.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 117310
ppc64el 118758

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters

5c330ad... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvscsi

Newer kernels renamed ibmvscsic to ibmvscsi.
Try to load both names, so that it always works.
No impact in size of included modules.

3cf090f... by Robin H. Johnson on 2017-11-09

src/etc/modules: powerpc tweak.

Split to section, ready for further changes.

Signed-off-by: Robin H. Johnson <email address hidden>

diff --git a/bin/build-release b/bin/build-release
index c947deb..15e1919 100755
--- a/bin/build-release
+++ b/bin/build-release
@@ -28,10 +28,10 @@ if [ "$1" = "daily" ]; then
 fi
 shift
 pre=cirros-$VER
-BR_VER="2015.05"
+BR_VER="${BR_VER:-2015.05}" # WARNING: this may be non-trivial to change
 ARCHES="${ARCHES:-i386 x86_64 arm powerpc aarch64 ppc64 ppc64le}"
-KVER="4.4.0-28.47" # Ubuntu 16.04
-GVER="2.02~beta2-36ubuntu8"
+KVER="${KVER:-4.4.0-28.47}" # Ubuntu 16.04
+GVER="${GVER:-2.02~beta2-36ubuntu8}" # Ubuntu 16.04
 ME=$(readlink -f "$0")
 MY_D=${ME%/*}
 PATH=${MY_D}:$PATH
@@ -93,7 +93,7 @@ build_arch() {
 get_grub_format() {
     local arch=$1
     case "$arch" in
-        powerpc|ppc*) format=ieee;;
+        powerpc*|ppc*) format=ieee;;
         *) format=efi;;
     esac
     echo $format
@@ -102,7 +102,7 @@ get_grub_format() {
 get_img_size() {
     local arch=$1
     case "$arch" in
-        powerpc|ppc*|aarch64) size=96M;;
+        powerpc*|ppc*|aarch64) size=96M;;
         *) size="";;
     esac
     echo $size
diff --git a/bin/bundle b/bin/bundle
index 7d801d6..4bb5083 100755
--- a/bin/bundle
+++ b/bin/bundle
@@ -193,6 +193,9 @@ for x in ${MODULES}; do
         }
     }' mat="/${x}.ko$" p="lib/modules/${kver}" "${mdep}"
 done > "${kern_modules}"
+# NOTE: if you want to inject lots of modules for testing, this is the spot to
+# do it!
+#( cd ${kernel_d} ; find -name '*.ko' ) >>"${kern_modules}"
 sort -u "${kern_modules}" > "${kern_files}"
 vmlinuz=$( cd "${kernel_d}" && [ -f boot/vmlinu?-* ] &&
     echo boot/vmlinu?-* ) && echo "${vmlinuz}" >> "${kern_files}" &&
diff --git a/src/etc/init.d/load-modules b/src/etc/init.d/load-modules
index 8a0a8f9..a1f4871 100755
--- a/src/etc/init.d/load-modules
+++ b/src/etc/init.d/load-modules
@@ -18,7 +18,7 @@ parse_modules() {
 	case "$arch" in
 		i?86) march=",i386,x86,";;
 		amd64|x86_64) march=",x86_64,x86,";;
-		ppc64|powerpc) march=",powerpc,ppc64,";;
+		ppc64*|powerpc*) march=",powerpc,ppc64,";;
 		arm*) march=",arm,$arch,";;
 	esac
 	while read line; do
diff --git a/src/etc/modules b/src/etc/modules
index ef013e1..5f4591b 100644
--- a/src/etc/modules
+++ b/src/etc/modules
@@ -1,16 +1,69 @@
 # modules here are loaded one per line
-ibmvscsic          # arch=powerpc
-ibmveth            # arch=powerpc
-virtio_scsi
-virtio_pci
-virtio_blk
-virtio_net
 e1000
 ne2k-pci           # arch=x86
 8139cp             # arch=x86
 pcnet32            # arch=x86
-ip_tables
+
+## VirtIO
+9pnet_virtio
+qemu_fw_cfg
+virtio_blk
+virtio_input
+virtio_net
+virtio_pci
+virtio_scsi
+
+## PowerPC
+ibmveth            # arch=powerpc
+ibmvfc             # arch=powerpc
+ibmvnic            # arch=powerpc
+ibmvscsi           # arch=powerpc
+ibmvscsic          # arch=powerpc
+
+## Filesystem
+## nls* are needed for VFAT
 isofs
 vfat
+nls_ascii
 nls_cp437
 nls_iso8859-1
+nls_utf8
+
+## Networking
+#8021q
+#gre
+#ipip
+ip_tables
+#ipvlan
+#macvlan
+#macvtap
+#sit
+#veth
+#vxlan
+
+## USB host
+## absolutely critical if your arch depends on USB devices and these are not
+## built-in.
+ehci-hcd
+ehci-pci
+ehci-platform
+ohci-hcd
+ohci-pci
+ohci-platform
+uhci-hcd
+xhci-hcd
+xhci-pci
+xhci-plat-hcd
+
+## Input systems
+## absolutely critical if your arch depends on USB input devices.
+hid
+hid-generic
+usbhid
+usbkbd
+
+## Block storage
+## Present for config_drive support
+cdrom
+sr_mod
+sg