Ubuntu
linux package

Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue

Bug #1657281 reported by Dragan S. on 2017-01-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	linux (Ubuntu)	Fix Released	Undecided	Dragan S.
	Xenial	Fix Released	Undecided	Thadeu Lima de Souza Cascardo

Bug Description

[Impact]

* When running block device I/O on an Amazon i3 system there
   is a performance degradation. A patch with git change id
   87c279e613f848c691111b29 that is in Ubuntu-lts-4.8.0 kernel
   series increases I/O performance.

* Fix should be backported to xenial 4.4 series to avoid more
support issues being filed.

* This change gets rid of uneccessary work being performed.

[Test Case]

* Steps to reproduce below
   1) partition ephemeral disks
      /sbin/parted -s --align optimal /dev/nvme0n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme1n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme2n1 mklabel gpt mkpart primary 0% 100%
      /sbin/parted -s --align optimal /dev/nvme3n1 mklabel gpt mkpart primary 0% 100%

2) create raid array
/sbin/mdadm --create /dev/md0 --assume-clean --chunk=2048 --level=10 --raid-devices=4 /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1 /dev/nvme3n1p1

3) create pv
/sbin/pvcreate --force --metadatasize 4092k /dev/md0

4) create volume group
/sbin/vgcreate RDSVG /dev/md0

5) create lv
/sbin/lvcreate -L 2.5T -n RDSRAIDLV RDSVG

5) test I/O
sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync

[Regression Potential]

* No known regression potential.

[Original Description]

When we were doing testing on i3, we noticed that it is taking significantly longer to perform operations when using software RAID than without, we believe this is resolved in an upstream commit: http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=87c279e613f848c691111b29d49de8df3f4f56da

So stock 4.4.0-59-generic performs ok:
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
sudo: unable to resolve host ip-10-0-85-167
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 54.3711 s, 151 MB/s

and the patch you originally providing works a little bit better (as expected):
$ uname -a
Linux ip-10-0-85-167 4.4.0-57-generic #78hf000000v20170110b0h3199a6e718db-Ubuntu SMP Tue Jan 10 02:53: x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ip-10-0-85-167:~$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
sudo: unable to resolve host ip-10-0-85-167
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 31.4108 s, 261 MB/s

See original description

Tags:

Revision history for this message

Dragan S. (dragan-s) wrote on 2017-01-17:

block: fix plug list flushing for nomerge queues Edit (2.1 KiB, text/plain)

tags:	added: sts
description:	updated
Changed in linux (Ubuntu):
assignee:	nobody → Dragan S. (dragan-s)

Revision history for this message

Brad Figg (brad-figg) wrote on 2017-01-17: Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1657281

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Ubuntu Foundations Team Bug Bot (crichton) on 2017-01-18

tags:

added: patch

Brad Figg (brad-figg) on 2017-01-18

Changed in linux (Ubuntu):
status:	Incomplete → Confirmed

Revision history for this message

Tim Gardner (timg-tpi) wrote on 2017-01-18:

https://lists.ubuntu.com/archives/kernel-team/2017-January/081890.html

Changed in linux (Ubuntu Xenial):
assignee:	nobody → Thadeu Lima de Souza Cascardo (cascardo)
status:	New → In Progress

Thadeu Lima de Souza Cascardo (cascardo) on 2017-01-18

Changed in linux (Ubuntu Xenial):
status:	In Progress → Fix Committed

Revision history for this message

John Donnelly (jpdonnelly) wrote on 2017-01-19:

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags:

added: verification-needed-xenial

Revision history for this message

Dragan S. (dragan-s) wrote on 2017-01-19:

Verified:

4.4.0-59-generic
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 56.7741 s, 144 MB/s

4.4.0-62-generic
$ sudo dd if=/dev/zero of=/dev/RDSVG/RDSRAIDLV bs=8k count=1000000 && sync
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB, 7.6 GiB) copied, 31.3783 s, 261 MB/s

tags:

added: verification-done-xenial
removed: verification-needed-xenial

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-02-02:

Download full text (10.8 KiB)

This bug was fixed in the package linux - 4.4.0-62.83

---------------
linux (4.4.0-62.83) xenial; urgency=low

[ Thadeu Lima de Souza Cascardo ]

* Release Tracking Bug
- LP: #1657430

  * Backport DP MST fixes to i915 (LP: #1657353)
    - SAUCE: i915_bpo: Fix DP link rate math
    - SAUCE: i915_bpo: Validate mode against max. link data rate for DP MST

* Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue (LP: #1657281)
- blk-mq: really fix plug list flushing for nomerge queues

linux (4.4.0-61.82) xenial; urgency=low

[ Thadeu Lima de Souza Cascardo ]

* Release Tracking Bug
- LP: #1656810

* Xen MSI setup code incorrectly re-uses cached pirq (LP: #1656381)
- SAUCE: xen: do not re-use pirq number cached in pci device msi msg data

* nvme drive probe failure (LP: #1626894)
- nvme: revert NVMe: only setup MSIX once

linux (4.4.0-60.81) xenial; urgency=low

[ John Donnelly ]

* Release Tracking Bug
- LP: #1656084

* Couldn't emulate instruction 0x7813427c (LP: #1634129)
- KVM: PPC: Book3S PR: Fix illegal opcode emulation

  * perf: 24x7: Eliminate domain name suffix in event names (LP: #1560482)
    - powerpc/perf/hv-24x7: Fix usage with chip events.
    - powerpc/perf/hv-24x7: Display change in counter values
    - powerpc/perf/hv-24x7: Display domain indices in sysfs
    - powerpc/perf/24x7: Eliminate domain suffix in event names

  * i386 ftrace tests hang on ADT testing (LP: #1655040)
    - ftrace/x86_32: Set ftrace_stub to weak to prevent gcc from using short jumps
      to it

  * VMX module autoloading if available (LP: #1651322)
    - powerpc: Add module autoloading based on CPU features
    - crypto: vmx - Convert to CPU feature based module autoloading

  * ACPI probe support for AD5592/3 configurable multi-channel converter
    (LP: #1654497)
    - SAUCE: iio: dac: ad5592r: Add ACPI support
    - SAUCE: iio: dac: ad5593r: Add ACPI support

This bug was fixed in the package linux - 4.4.0-62.83

---------------
linux (4.4.0-62.83) xenial; urgency=low

[ Thadeu Lima de Souza Cascardo ]

* Release Tracking Bug
    - LP: #1657430

* Backport DP MST fixes to i915 (LP: #1657353)
    - SAUCE: i915_bpo: Fix DP link rate math
    - SAUCE: i915_bpo: Validate mode against max. link data rate for DP MST

* Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue (LP: #1657281)
    - blk-mq: really fix plug list flushing for nomerge queues

linux (4.4.0-61.82) xenial; urgency=low

[ Thadeu Lima de Souza Cascardo ]

* Release Tracking Bug
    - LP: #1656810

* Xen MSI setup code incorrectly re-uses cached pirq (LP: #1656381)
    - SAUCE: xen: do not re-use pirq number cached in pci device msi msg data

* nvme drive probe failure (LP: #1626894)
    - nvme: revert NVMe: only setup MSIX once

linux (4.4.0-60.81) xenial; urgency=low

[ John Donnelly ]

* Release Tracking Bug
    - LP: #1656084

* Couldn't emulate instruction 0x7813427c (LP: #1634129)
    - KVM: PPC: Book3S PR: Fix illegal opcode emulation

* i386 ftrace tests hang on ADT testing (LP: #1655040)
    - ftrace/x86_32: Set ftrace_stub to weak to prevent gcc from using short jumps
      to it

* VMX module autoloading if available (LP: #1651322)
    - powerpc: Add module autoloading based on CPU features
    - crypto: vmx - Convert to CPU feature based module autoloading

* ACPI probe support for AD5592/3 configurable multi-channel converter
    (LP: #1654497)
    - SAUCE: iio: dac: ad5592r: Add ACPI support
    - SAUCE: iio: dac: ad5593r: Add ACPI support

* Xenial update to v4.4.40 stable release (LP: #1654602)
    - btrfs: limit async_work allocation and worker func duration
    - Btrfs: fix tree search logic when replaying directory entry deletes
    - btrfs: store and load values of stripes_min/stripes_max in balance status
      item
    - Btrfs: fix qgroup rescan worker initialization
    - USB: serial: option: add support for Telit LE922A PIDs 0x1040, 0x1041
    - USB: serial: option: add dlink dwm-158
    - USB: serial: kl5kusb105: fix open error path
    - USB: cdc-acm: add device id for GW Instek AFG-125
    - usb: hub: Fix auto-remount of safely removed or ejected USB-3 devices
    - usb: gadget: f_uac2: fix error handling at afunc_bind
    - usb: gadget: composite: correctly initialize ep->maxpacket
    - USB: UHCI: report non-PME wakeup signalling for Intel hardware
    - ALSA: usb-audio: Add QuickCam Communicate Deluxe/S7500 to
      volume_control_quirks
    - ALSA: hiface: Fix M2Tech hiFace driver sampling rate change
    - ALSA: hda/ca0132 - Add quirk for Alienware 15 R2 2016
    - ALSA: hda - ignore the assoc and seq when comparing pin configurations
    - ALSA: hda - fix headset-mic problem on a Dell laptop
    - ALSA: hda - Gate the mic jack on HP Z1 Gen3 AiO
    - ALSA: hda: when comparing pin configurations, ignore assoc in addition to
      seq
    - clk: ti: omap36xx: Work around sprz319 advisory 2.1
    - Btrfs: fix memory leak in reading btree blocks
    - Btrfs: bail out if block group has different mixed flag
    - Btrfs: return gracefully from balance if fs tree is corrupted
    - Btrfs: don't leak reloc root nodes on error
    - Btrfs: fix memory leak in do_walk_down
    - Btrfs: don't BUG() during drop snapshot
    - btrfs: make file clone aware of fatal signals
    - block_dev: don't test bdev->bd_contains when it is not stable
    - ptrace: Capture the ptracer's creds not PT_PTRACE_CAP
    - crypto: caam - fix AEAD givenc descriptors
    - ext4: fix mballoc breakage with 64k block size
    - ext4: fix stack memory corruption with 64k block size
    - ext4: use more strict checks for inodes_per_block on mount
    - ext4: fix in-superblock mount options processing
    - ext4: add sanity checking to count_overhead()
    - ext4: reject inodes with negative size
    - ext4: return -ENOMEM instead of success
    - ext4: do not perform data journaling when data is encrypted
    - f2fs: set ->owner for debugfs status file's file_operations
    - loop: return proper error from loop_queue_rq()
    - mm/vmscan.c: set correct defer count for shrinker
    - fs: exec: apply CLOEXEC before changing dumpable task flags
    - exec: Ensure mm->user_ns contains the execed files
    - usb: gadget: composite: always set ep->mult to a sensible value
    - blk-mq: Do not invoke .queue_rq() for a stopped queue
    - dm flakey: return -EINVAL on interval bounds error in flakey_ctr()
    - dm crypt: mark key as invalid until properly loaded
    - dm space map metadata: fix 'struct sm_metadata' leak on failed create
    - ASoC: intel: Fix crash at suspend/resume without card registration
    - CIFS: Fix a possible memory corruption during reconnect
    - CIFS: Fix missing nls unload in smb2_reconnect()
    - CIFS: Fix a possible memory corruption in push locks
    - kernel/watchdog: use nmi registers snapshot in hardlockup handler
    - kernel/debug/debug_core.c: more properly delay for secondary CPUs
    - tpm xen: Remove bogus tpm_chip_unregister
    - xen/gntdev: Use VM_MIXEDMAP instead of VM_IO to avoid NUMA balancing
    - arm/xen: Use alloc_percpu rather than __alloc_percpu
    - xfs: set AGI buffer type in xlog_recover_clear_agi_bucket
    - driver core: fix race between creating/querying glue dir and its cleanup
    - ppp: defer netns reference release for ppp channel
    - Linux 4.4.40

* igb i210 probe of pci device failed with error -2 (LP: #1639810)
    - SAUCE: igb: Workaround for igb i210 firmware issue.
    - SAUCE: igb: add i211 to i210 PHY workaround

* PowerNV: PCI Slot is invalid after fencedPHB Error injection (LP: #1652018)
    - powerpc/powernv: Call opal_pci_poll() if needed

* mfd: intel-lpss: Add default I2C device properties for Apollo Lake
    (LP: #1635177)
    - mfd: intel-lpss: Add default I2C device properties for Apollo Lake

* Xenial update to v4.4.39 stable release (LP: #1650609)
    - powerpc/eeh: Fix deadlock when PE frozen state can't be cleared
    - parisc: Purge TLB before setting PTE
    - parisc: Remove unnecessary TLB purges from flush_dcache_page_asm and
      flush_icache_page_asm
    - parisc: Fix TLB related boot crash on SMP machines
    - zram: restrict add/remove attributes to root only
    - locking/rtmutex: Prevent dequeue vs. unlock race
    - locking/rtmutex: Use READ_ONCE() in rt_mutex_owner()
    - perf/x86: Fix full width counter, counter overflow
    - crypto: mcryptd - Check mcryptd algorithm compatibility
    - can: raw: raw_setsockopt: limit number of can_filter that can be set
    - can: peak: fix bad memory access and free sequence
    - arm64: futex.h: Add missing PAN toggling
    - m68k: Fix ndelay() macro
    - batman-adv: Check for alloc errors when preparing TT local data
    - hotplug: Make register and unregister notifier API symmetric
    - crypto: rsa - Add Makefile dependencies to fix parallel builds
    - Linux 4.4.39

* Xenial update to v4.4.38 stable release (LP: #1650607)
    - virtio-net: add a missing synchronize_net()
    - net: check dead netns for peernet2id_alloc()
    - ip6_tunnel: disable caching when the traffic class is inherited
    - net: sky2: Fix shutdown crash
    - af_unix: conditionally use freezable blocking calls in read
    - rtnetlink: fix FDB size computation
    - l2tp: fix racy SOCK_ZAPPED flag check in l2tp_ip{,6}_bind()
    - net: dsa: bcm_sf2: Ensure we re-negotiate EEE during after link change
    - net, sched: respect rcu grace period on cls destruction
    - net/sched: pedit: make sure that offset is valid
    - netlink: Call cb->done from a worker thread
    - netlink: Do not schedule work from sk_destruct
    - net/dccp: fix use-after-free in dccp_invalid_packet
    - net: bcmgenet: Utilize correct struct device for all DMA operations
    - sh_eth: remove unchecked interrupts for RZ/A1
    - geneve: avoid use-after-free of skb->data
    - net: ping: check minimum size on ICMP header length
    - sparc32: Fix inverted invalid_frame_pointer checks on sigreturns
    - sparc64: Fix find_node warning if numa node cannot be found
    - sparc64: fix compile warning section mismatch in find_node()
    - constify iov_iter_count() and iter_is_iovec()
    - Don't feed anything but regular iovec's to blk_rq_map_user_iov
    - ipv6: Set skb->protocol properly for local output
    - ipv4: Set skb->protocol properly for local output
    - esp4: Fix integrity verification when ESN are used
    - esp6: Fix integrity verification when ESN are used
    - Linux 4.4.38

* Xenial update to v4.4.37 stable release (LP: #1650604)
    - ARC: Don't use "+l" inline asm constraint
    - zram: fix unbalanced idr management at hot removal
    - kasan: update kasan_global for gcc 7
    - x86/traps: Ignore high word of regs->cs in early_fixup_exception()
    - rcu: Fix soft lockup for rcu_nocb_kthread
    - PCI: Export pcie_find_root_port
    - PCI: Set Read Completion Boundary to 128 iff Root Port supports it (_HPX)
    - mwifiex: printk() overflow with 32-byte SSIDs
    - pwm: Fix device reference leak
    - arm64: cpufeature: Schedule enable() calls instead of calling them via IPI
    - arm64: mm: Set PSTATE.PAN from the cpu_enable_pan() call
    - arm64: suspend: Reconfigure PSTATE after resume from idle
    - Linux 4.4.37

* Xenial update to v4.4.36 stable release (LP: #1650601)
    - iommu/vt-d: Fix PASID table allocation
    - iommu/vt-d: Fix IOMMU lookup for SR-IOV Virtual Functions
    - KVM: x86: check for pic and ioapic presence before use
    - usb: chipidea: move the lock initialization to core file
    - USB: serial: cp210x: add ID for the Zone DPMX
    - USB: serial: ftdi_sio: add support for TI CC3200 LaunchPad
    - Fix USB CB/CBI storage devices with CONFIG_VMAP_STACK=y
    - scsi: mpt3sas: Fix secure erase premature termination
    - tile: avoid using clocksource_cyc2ns with absolute cycle count
    - cfg80211: limit scan results cache size
    - NFSv4.x: hide array-bounds warning
    - parisc: Fix races in parisc_setup_cache_timing()
    - parisc: Fix race in pci-dma.c
    - parisc: Also flush data TLB in flush_icache_page_asm
    - mpi: Fix NULL ptr dereference in mpi_powm() [ver #3]
    - drm/radeon: Ensure vblank interrupt is enabled on DPMS transition to on
    - mei: me: disable driver on SPT SPS firmware
    - mei: me: fix place for kaby point device ids.
    - mei: fix return value on disconnection
    - scsi: mpt3sas: Unblock device after controller reset
    - Linux 4.4.36

* Miscellaneous Ubuntu changes
    - [Debian] consider renames in gen-auto-reconstruct

-- Thadeu Lima de Souza Cascardo <cascardo@canonical.com>  Wed, 18 Jan 2017 09:34:07 -0200

Changed in linux (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Guilherme G. Piccoli (gpiccoli) on 2020-07-15

Changed in linux (Ubuntu):
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Patches

block: fix plug list flushing for nomerge queues Edit

Add patch

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

Ubuntu xenial - 4.4.0-59-generic i3 I/O performance issue

Bug Description

Other bug subscribers

Patches

Remote bug watches

Ubuntu
linux package