Merge ~robbat2/cirros:gentoo into cirros:master

Proposed by Robin H. Johnson on 2017-11-09
Status: Needs review
Proposed branch: ~robbat2/cirros:gentoo
Merge into: cirros:master
Diff against target: 144 lines (+69/-13)
4 files modified
bin/build-release (+5/-5)
bin/bundle (+3/-0)
src/etc/init.d/load-modules (+1/-1)
src/etc/modules (+60/-7)
Reviewer Review Type Date Requested Status
Scott Moser 2017-11-09 Pending
Review via email: mp+333488@code.launchpad.net

Description of the change

Merge of Gentoo CirrOS improvements
- PPC64 work: matching strings, more ibm* modules, virtio input, USB input
- Improve layout of src/etc/modules with sections
- More virtio modules

To post a comment you must log in.
Scott Moser (smoser) wrote :

Thanks for putting together a merge propsal.
A tiny things:
 - fixes comments are done with:
   LP: #XXXXXX

Generally I dont have interest in just adding lots of modules,
as that makes things bigger. I've not looked at the size change
that would be caused, so I dont know specifically about these
modules.

Specifically for the Hyper-v modules there is an issue that
loading those modules causes 5 second delay even if the hardware
is not there. Unless that is now known fixed, then we can't
really just add them to the list as they have a negative affect
everywhere.

I'm generally OK with the powerpc changes.

the BR_VERSION change is probably not really all that useful in
reality. Its quite a pain to move buildroot versions (more
than just changing that file :)

~robbat2/cirros:gentoo updated on 2017-11-09
224059f... by Robin H. Johnson on 2017-11-09

src/etc/modules: config_drive cdrom support.

config_drive can present as a SCSI CDROM, so we need suitable modules
loaded to access it.

Added in a new section for block storage.

There is no growth in size on Ubuntu, because all 3 modules are
compiled-in already, but the module loading is useful for other kernels
where they might be modular.

LP: #1715009
Signed-off-by: Robin H. Johnson <email address hidden>

Robin H. Johnson (robbat2) wrote :

I'll drop Hyper-V to a seperate branch for now, and produce size stats for you.

~robbat2/cirros:gentoo updated on 2017-11-09
ce16a57... by Robin H. Johnson on 2017-11-09

src/etc/modules: add virtio_input

Add support for input devices on the virtio bus.

Size impact:
------------
(Uncompressed modules, size in bytes, on Ubuntu 4.4.0-28.47)
amd64 15134
arm64 18678
armhf 13938
i386 11382
ppc64 19366
ppc64el 18590

Requires: linux>=4.1
Requires: qemu>=2.4
Requires: libvirt>=1.3.0
See: https://lwn.net/Articles/638075/
See: https://www.kraxel.org/blog/2015/06/new-member-in-the-virtio-family-input-devices/
See: https://www.kraxel.org/blog/2016/09/using-virtio-input-with-libvirt/

57933e8... by Robin H. Johnson on 2017-11-09

src/etc/modules: add qemu_fw_cfg

Add support for input devices on the virtio bus.

Size impact:
------------
(Uncompressed modules, size in bytes)
amd64 21374 **4.13.0-17.20
arm64 20475 **4.14.0-0.rc8.git2.2.fc28
armhf N/A
i386 14206 **4.13.0-17.20
ppc64 N/A
ppc64el N/A

Requires: linux>=4.6
Requires: qemu>=2.6
See: http://www.contrib.andrew.cmu.edu/~somlo/QEMU_fw_cfg/

4091d98... by Robin H. Johnson on 2017-11-09

src/etc/modules: add 9pnet_virtio

Support host filesystem passthrough

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 163564
arm64 182604
armhf 126744
i386 110800
ppc64 192188
ppc64el 188580

See: https://www.linux-kvm.org/page/9p_virtio

5c330ad... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvscsi

Newer kernels renamed ibmvscsic to ibmvscsi.
Try to load both names, so that it always works.
No impact in size of included modules.

a2a8cb9... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvnic

PPC64 systems can use either ibmvnic as an alternative to ibmveth, mostly for
higher performance networking in LPAR.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 117310
ppc64el 118758

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters

1236ac0... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvfc

PPC64 systems can use ibmvfc for virtualized fibre channel access in
LPAR environment.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 278228
ppc64el 274804

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters
Signed-off-by: Robin H. Johnson <email address hidden>

Robin H. Johnson (robbat2) wrote :

I split out many of the module changes individually now, for ease of later review/revert.

Overall growth in raw module bytes:
   amd64 553698
   arm64 632427
   armhf 420524
    i386 373338
   ppc64 1003526
 ppc64el 1027954

Scott Moser (smoser) wrote :

How does this look:
https://code.launchpad.net/~smoser/cirros/+git/cirros/+ref/robbat2-cherry-picks1

That grabs a fair number of them, the most objectionable change I made
was to just not carry the old 'ibmvscsic' name.

Robin H. Johnson (robbat2) wrote :

Some comments inline, re ppc* matching.

My objectives was to make this work with the existing system, but also ensure that it was more likely to work with other kernels. Carrying both names of ibmvscsic and ibmvscsi for example. Only one will exist in a given kernel, but we can't tell ahead of time.

Why not merging the USB parts?

Scott Moser (smoser) wrote :

> Some comments inline, re ppc* matching.
>
> My objectives was to make this work with the existing system, but also ensure
> that it was more likely to work with other kernels. Carrying both names of
> ibmvscsic and ibmvscsi for example. Only one will exist in a given kernel, but
> we can't tell ahead of time.

Yes, but 'ibmvscsic' will only exist in old kernels, those < 4.4.
Do you have an actual use case where you'd use something older than that ?

Unmerged commits

224059f... by Robin H. Johnson on 2017-11-09

src/etc/modules: config_drive cdrom support.

config_drive can present as a SCSI CDROM, so we need suitable modules
loaded to access it.

Added in a new section for block storage.

There is no growth in size on Ubuntu, because all 3 modules are
compiled-in already, but the module loading is useful for other kernels
where they might be modular.

LP: #1715009
Signed-off-by: Robin H. Johnson <email address hidden>

d7fc61b... by Robin H. Johnson on 2017-11-09

src/etc/modules: USB input devices

Add new section for input devices, as used by USB keyboards.

USB mice & tablets are NOT included, as CirrOS has no need of them.

This provides working console input on OSUOSL's POWER OpenStack development
environment (https://osuosl.org/services/powerdev/), which presents as
XHCI/EHCI with USB keyboard.

The large module growth comes mostly from the HID module, which represents ~61%
of the new bytes.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 324584
arm64 328312
armhf 248476
i386 215660
ppc64 359440
ppc64el 353024

Thanks: Lance Albertson <email address hidden>
Signed-off-by: Robin H. Johnson <email address hidden>

9888a8c... by Robin H. Johnson on 2017-11-09

src/etc/modules: USB host controllers

Add new section for USB host controllers, in preparation for USB input devices.

Includes all QEMU-support host controllers: XHCI, EHCI, OHCI, UHCI.
New usage should prefer virtio-input or at the XHCI controller.

This provides working console input on OSUOSL's POWER OpenStack development
environment (https://osuosl.org/services/powerdev/), which presents as
XHCI/EHCI with USB keyboard.

Many systems may have these modules built-in, but these list is complete
to ensure it works.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 13494
arm64 61618
armhf 17154
i386 10046
ppc64 18390
ppc64el 56490

Thanks: Lance Albertson <email address hidden>
Signed-off-by: Robin H. Johnson <email address hidden>

a912bc7... by Robin H. Johnson on 2017-11-09

src/etc/modules: networking tweak.

- add section
- add lots of commented out modules based on bugs filed in LP, do not
  want the size bloat yet.

Signed-off-by: Robin H. Johnson <email address hidden>

9458109... by Robin H. Johnson on 2017-11-09

src/etc/modules: add nls_utf8, nls_ascii

Improve VFAT support by adding two more very common encodings it can
use: UTF-8 and ASCII.

nls_ascii is frequently compiled into kernels, thus this fix is only
applicable to very modular kernels.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
amd64 15548
arm64 20740
armhf 14212
i386 11244
ppc64 18604
ppc64el 17708

Signed-off-by: Robin H. Johnson <email address hidden>

04091c3... by Robin H. Johnson on 2017-11-09

src/etc/modules: filesystem tweak.

Split to section for further changes.
Document that nls_* are included for VFAT to work.

Signed-off-by: Robin H. Johnson <email address hidden>

1236ac0... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvfc

PPC64 systems can use ibmvfc for virtualized fibre channel access in
LPAR environment.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 278228
ppc64el 274804

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters
Signed-off-by: Robin H. Johnson <email address hidden>

a2a8cb9... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvnic

PPC64 systems can use either ibmvnic as an alternative to ibmveth, mostly for
higher performance networking in LPAR.

Size impact:
------------
(Uncompressed modules, size in bytes, Ubuntu 4.4.0-28.47)
ppc64 117310
ppc64el 118758

See: https://chmod666.org/2015/11/a-first-look-at-sriov-vnic-adapters

5c330ad... by Robin H. Johnson on 2017-11-09

src/etc/modules: add ibmvscsi

Newer kernels renamed ibmvscsic to ibmvscsi.
Try to load both names, so that it always works.
No impact in size of included modules.

3cf090f... by Robin H. Johnson on 2017-11-09

src/etc/modules: powerpc tweak.

Split to section, ready for further changes.

Signed-off-by: Robin H. Johnson <email address hidden>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/bin/build-release b/bin/build-release
2index c947deb..15e1919 100755
3--- a/bin/build-release
4+++ b/bin/build-release
5@@ -28,10 +28,10 @@ if [ "$1" = "daily" ]; then
6 fi
7 shift
8 pre=cirros-$VER
9-BR_VER="2015.05"
10+BR_VER="${BR_VER:-2015.05}" # WARNING: this may be non-trivial to change
11 ARCHES="${ARCHES:-i386 x86_64 arm powerpc aarch64 ppc64 ppc64le}"
12-KVER="4.4.0-28.47" # Ubuntu 16.04
13-GVER="2.02~beta2-36ubuntu8"
14+KVER="${KVER:-4.4.0-28.47}" # Ubuntu 16.04
15+GVER="${GVER:-2.02~beta2-36ubuntu8}" # Ubuntu 16.04
16 ME=$(readlink -f "$0")
17 MY_D=${ME%/*}
18 PATH=${MY_D}:$PATH
19@@ -93,7 +93,7 @@ build_arch() {
20 get_grub_format() {
21 local arch=$1
22 case "$arch" in
23- powerpc|ppc*) format=ieee;;
24+ powerpc*|ppc*) format=ieee;;
25 *) format=efi;;
26 esac
27 echo $format
28@@ -102,7 +102,7 @@ get_grub_format() {
29 get_img_size() {
30 local arch=$1
31 case "$arch" in
32- powerpc|ppc*|aarch64) size=96M;;
33+ powerpc*|ppc*|aarch64) size=96M;;
34 *) size="";;
35 esac
36 echo $size
37diff --git a/bin/bundle b/bin/bundle
38index 7d801d6..4bb5083 100755
39--- a/bin/bundle
40+++ b/bin/bundle
41@@ -193,6 +193,9 @@ for x in ${MODULES}; do
42 }
43 }' mat="/${x}.ko$" p="lib/modules/${kver}" "${mdep}"
44 done > "${kern_modules}"
45+# NOTE: if you want to inject lots of modules for testing, this is the spot to
46+# do it!
47+#( cd ${kernel_d} ; find -name '*.ko' ) >>"${kern_modules}"
48 sort -u "${kern_modules}" > "${kern_files}"
49 vmlinuz=$( cd "${kernel_d}" && [ -f boot/vmlinu?-* ] &&
50 echo boot/vmlinu?-* ) && echo "${vmlinuz}" >> "${kern_files}" &&
51diff --git a/src/etc/init.d/load-modules b/src/etc/init.d/load-modules
52index 8a0a8f9..a1f4871 100755
53--- a/src/etc/init.d/load-modules
54+++ b/src/etc/init.d/load-modules
55@@ -18,7 +18,7 @@ parse_modules() {
56 case "$arch" in
57 i?86) march=",i386,x86,";;
58 amd64|x86_64) march=",x86_64,x86,";;
59- ppc64|powerpc) march=",powerpc,ppc64,";;
60+ ppc64*|powerpc*) march=",powerpc,ppc64,";;
61 arm*) march=",arm,$arch,";;
62 esac
63 while read line; do
64diff --git a/src/etc/modules b/src/etc/modules
65index ef013e1..5f4591b 100644
66--- a/src/etc/modules
67+++ b/src/etc/modules
68@@ -1,16 +1,69 @@
69 # modules here are loaded one per line
70-ibmvscsic # arch=powerpc
71-ibmveth # arch=powerpc
72-virtio_scsi
73-virtio_pci
74-virtio_blk
75-virtio_net
76 e1000
77 ne2k-pci # arch=x86
78 8139cp # arch=x86
79 pcnet32 # arch=x86
80-ip_tables
81+
82+## VirtIO
83+9pnet_virtio
84+qemu_fw_cfg
85+virtio_blk
86+virtio_input
87+virtio_net
88+virtio_pci
89+virtio_scsi
90+
91+## PowerPC
92+ibmveth # arch=powerpc
93+ibmvfc # arch=powerpc
94+ibmvnic # arch=powerpc
95+ibmvscsi # arch=powerpc
96+ibmvscsic # arch=powerpc
97+
98+## Filesystem
99+## nls* are needed for VFAT
100 isofs
101 vfat
102+nls_ascii
103 nls_cp437
104 nls_iso8859-1
105+nls_utf8
106+
107+## Networking
108+#8021q
109+#gre
110+#ipip
111+ip_tables
112+#ipvlan
113+#macvlan
114+#macvtap
115+#sit
116+#veth
117+#vxlan
118+
119+## USB host
120+## absolutely critical if your arch depends on USB devices and these are not
121+## built-in.
122+ehci-hcd
123+ehci-pci
124+ehci-platform
125+ohci-hcd
126+ohci-pci
127+ohci-platform
128+uhci-hcd
129+xhci-hcd
130+xhci-pci
131+xhci-plat-hcd
132+
133+## Input systems
134+## absolutely critical if your arch depends on USB input devices.
135+hid
136+hid-generic
137+usbhid
138+usbkbd
139+
140+## Block storage
141+## Present for config_drive support
142+cdrom
143+sr_mod
144+sg

Subscribers

People subscribed via source and target branches