Ubuntu
zfs-linux package

Ship arcstat.py and arc_summary.py with zfsutils-linux

Bug #1574342 reported by Hajo Möller
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
Medium
Eric Desrochers
Xenial
Fix Released
Medium
Eric Desrochers
Yakkety
Fix Released
Medium
Eric Desrochers

Bug Description

[Impact]

zfsutils-linux missing some important tools.

Python3 compatible scripts : arc_summary.py, arcstat.py and dbufstat.py found in src_code/cmd/ are not taken into account during compilation.

    * arc_summary.py : Provides a summary of the statistics
https://github.com/zfsonlinux/zfs/blob/master/cmd/arc_summary

    * arcstat.py : Print out ZFS ARC Statistics exported via kstat(1)
https://github.com/zfsonlinux/zfs/tree/master/cmd/arcstat

    * dbufstat.py : Print out statistics for all cached dmu buffers
https://github.com/zfsonlinux/zfs/blob/master/cmd/dbufstat

[Test Case]

# rmadison zfsutils-linux
...
 zfsutils-linux | 0.6.5.6-0ubuntu10 | xenial-updates
...

# List contents of a deb package.
$ dpkg -c zfsutils-linux_0.6.5.6-0ubuntu10_amd64.deb | egrep "arcstat|arc_summary|dbufstat"

# You will notice there are not present.

[Regression Potential]

None expected, debian package[1] has already made the change to build the python scripts.

[1]
Package: zfsutils-linux
Source: zfs-linux
Version: 0.6.5.7-1
Architecture: amd64
Maintainer: Debian ZFS on Linux maintainers <email address hidden>

# List contents of a deb package.
$ dpkg -c zfsutils-linux_0.6.5.7-1_amd64.deb | egrep -i "arc|dbufstat"
-rwxr-xr-x root/root 40751 2016-05-31 03:40 ./bin/arc_summary.py
-rwxr-xr-x root/root 13057 2016-05-31 03:40 ./bin/arcstat.py
-rwxr-xr-x root/root 18718 2016-05-31 03:40 ./bin/dbufstat.py

[Other Info]

* This is based on the following Debian commits :

Debian git commits
===
commit 730989397063955d60a1845003cfb7433eba1e52
Author: Aron Xu <email address hidden>
Date: Tue Apr 26 17:01:29 2016 +0800

Add dh-python to b-d

--

commit 814413118fb88de3ce596a87a8a011f50c225fdd
Author: Aron Xu <email address hidden>
Date: Sun Nov 22 15:44:16 2015 +0800

Add python related dependencies
===

* Despite what Debian did by adding theses script under /bin.
  I'll add them under /usr/sbin, which is more appropriate.

* Unfortunately manpage for theses 3 program seems limited.
  There is none in Debian nor git upstream repo.

The only manpage I found is for arcstat online :

http://linux.die.net/man/1/arcstat

[Original Description]

arcstat.py and arc_summary.py are valuable tools to determine ZFS' ARC usage, it is not obvious why they are not included in zfsutils-linux. As ubuntu-minimal already depends on python3 it should be safe to assume python is available, or am I mistaken here?

arcstat.py gives an iostat-like overview about ARC reads, hit rate, current and target size in regular intervals:
# ./arcstat.py 1
    time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
19:22:33 0 0 0 0 0 0 0 0 0 2.0G 7.8G
19:22:34 3 0 0 0 0 0 0 0 0 2.0G 7.8G
19:22:35 21 0 0 0 0 0 0 0 0 2.0G 7.8G
^C

arc_summary.py shows a more detailed overview of the current ARC status and ZFS tunables:
# ./arc_summary.py

------------------------------------------------------------------------
ZFS Subsystem Report Sun Apr 24 19:23:25 2016
ARC Summary: (HEALTHY)
 Memory Throttle Count: 0

ARC Misc:
 Deleted: 46
 Mutex Misses: 0
 Evict Skips: 0

ARC Size: 25.10% 1.95 GiB
 Target Size: (Adaptive) 100.00% 7.78 GiB
 Min Size (Hard Limit): 0.40% 32.00 MiB
 Max Size (High Water): 248:1 7.78 GiB

ARC Size Breakdown:
 Recently Used Cache Size: 50.00% 3.89 GiB
 Frequently Used Cache Size: 50.00% 3.89 GiB

ARC Hash Breakdown:
 Elements Max: 32.31k
 Elements Current: 99.78% 32.24k
 Collisions: 40.54k
 Chain Max: 3
 Chains: 240

ARC Total accesses: 4.54m
 Cache Hit Ratio: 99.39% 4.51m
 Cache Miss Ratio: 0.61% 27.74k
 Actual Hit Ratio: 98.76% 4.48m

 Data Demand Efficiency: 99.73% 3.23m
 Data Prefetch Efficiency: 11.32% 6.41k

 CACHE HITS BY CACHE LIST:
   Anonymously Used: 0.64% 28.65k
   Most Recently Used: 21.90% 987.29k
   Most Frequently Used: 77.47% 3.49m
   Most Recently Used Ghost: 0.00% 0
   Most Frequently Used Ghost: 0.00% 0

 CACHE HITS BY DATA TYPE:
   Demand Data: 71.40% 3.22m
   Prefetch Data: 0.02% 725
   Demand Metadata: 27.97% 1.26m
   Prefetch Metadata: 0.62% 27.92k

 CACHE MISSES BY DATA TYPE:
   Demand Data: 31.81% 8.82k
   Prefetch Data: 20.48% 5.68k
   Demand Metadata: 21.99% 6.10k
   Prefetch Metadata: 25.72% 7.13k

File-Level Prefetch: (HEALTHY)
DMU Efficiency: 36.39m
 Hit Ratio: 93.36% 33.97m
 Miss Ratio: 6.64% 2.42m

 Colinear: 2.42m
   Hit Ratio: 0.02% 505
   Miss Ratio: 99.98% 2.42m

 Stride: 33.94m
   Hit Ratio: 100.00% 33.94m
   Miss Ratio: 0.00% 14

DMU Misc:
 Reclaim: 2.42m
   Successes: 2.46% 59.51k
   Failures: 97.54% 2.36m

 Streams: 35.39k
   +Resets: 0.05% 18
   -Resets: 99.95% 35.37k
   Bogus: 0

ZFS Tunable:
 metaslab_debug_load 0
 zfs_arc_min_prefetch_lifespan 0
 zfetch_max_streams 8
 zfs_nopwrite_enabled 1
 zfetch_min_sec_reap 2
 zfs_dbgmsg_enable 0
 zfs_dirty_data_max_max_percent 25
 zfs_arc_p_aggressive_disable 1
 spa_load_verify_data 1
 zfs_zevent_cols 80
 zfs_dirty_data_max_percent 10
 zfs_sync_pass_dont_compress 5
 l2arc_write_max 8388608
 zfs_vdev_scrub_max_active 2
 zfs_vdev_sync_write_min_active 10
 zvol_prefetch_bytes 131072
 metaslab_aliquot 524288
 zfs_no_scrub_prefetch 0
 zfs_arc_shrink_shift 0
 zfetch_block_cap 256
 zfs_txg_history 0
 zfs_delay_scale 500000
 zfs_vdev_async_write_active_min_dirty_percent 30
 metaslab_debug_unload 0
 zfs_read_history 0
 zvol_max_discard_blocks 16384
 zfs_recover 0
 l2arc_headroom 2
 zfs_deadman_synctime_ms 1000000
 zfs_scan_idle 50
 zfs_free_min_time_ms 1000
 zfs_dirty_data_max 1670300876
 zfs_vdev_async_read_min_active 1
 zfs_mg_noalloc_threshold 0
 zfs_dedup_prefetch 0
 zfs_vdev_max_active 1000
 l2arc_write_boost 8388608
 zfs_resilver_min_time_ms 3000
 zfs_vdev_async_write_max_active 10
 zil_slog_limit 1048576
 zfs_prefetch_disable 0
 zfs_resilver_delay 2
 metaslab_lba_weighting_enabled 1
 zfs_mg_fragmentation_threshold 85
 l2arc_feed_again 1
 zfs_zevent_console 0
 zfs_immediate_write_sz 32768
 zfs_dbgmsg_maxsize 4194304
 zfs_free_leak_on_eio 0
 zfs_deadman_enabled 1
 metaslab_bias_enabled 1
 zfs_arc_p_dampener_disable 1
 zfs_object_mutex_size 64
 zfs_metaslab_fragmentation_threshold 70
 zfs_no_scrub_io 0
 metaslabs_per_vdev 200
 zfs_dbuf_state_index 0
 zfs_vdev_sync_read_min_active 10
 metaslab_fragmentation_factor_enabled 1
 zvol_inhibit_dev 0
 zfs_vdev_async_write_active_max_dirty_percent 60
 zfs_vdev_cache_size 0
 zfs_vdev_mirror_switch_us 10000
 zfs_dirty_data_sync 67108864
 spa_config_path /etc/zfs/zpool.cache
 zfs_dirty_data_max_max 4175752192
 zfs_arc_lotsfree_percent 10
 zfs_zevent_len_max 128
 zfs_scan_min_time_ms 1000
 zfs_arc_sys_free 0
 zfs_arc_meta_strategy 1
 zfs_vdev_cache_bshift 16
 zfs_arc_meta_adjust_restarts 4096
 zfs_max_recordsize 1048576
 zfs_vdev_scrub_min_active 1
 zfs_vdev_read_gap_limit 32768
 zfs_arc_meta_limit 0
 zfs_vdev_sync_write_max_active 10
 l2arc_norw 0
 zfs_arc_meta_prune 10000
 metaslab_preload_enabled 1
 l2arc_nocompress 0
 zvol_major 230
 zfs_vdev_aggregation_limit 131072
 zfs_flags 0
 spa_asize_inflation 24
 zfs_admin_snapshot 0
 l2arc_feed_secs 1
 zio_taskq_batch_pct 75
 zfs_sync_pass_deferred_free 2
 zfs_disable_dup_eviction 0
 zfs_arc_grow_retry 0
 zfs_read_history_hits 0
 zfs_vdev_async_write_min_active 1
 zfs_vdev_async_read_max_active 3
 zfs_scrub_delay 4
 zfs_delay_min_dirty_percent 60
 zfs_free_max_blocks 100000
 zfs_vdev_cache_max 16384
 zio_delay_max 30000
 zfs_top_maxinflight 32
 spa_slop_shift 5
 zfs_vdev_write_gap_limit 4096
 spa_load_verify_metadata 1
 spa_load_verify_maxinflight 10000
 l2arc_noprefetch 1
 zfs_vdev_scheduler noop
 zfs_expire_snapshot 300
 zfs_sync_pass_rewrite 2
 zil_replay_disable 0
 zfs_nocacheflush 0
 zfs_arc_max 0
 zfs_arc_min 0
 zfs_read_chunk_size 1048576
 zfs_txg_timeout 5
 zfs_pd_bytes_max 52428800
 l2arc_headroom_boost 200
 zfs_send_corrupt_data 0
 l2arc_feed_min_ms 200
 zfs_arc_meta_min 0
 zfs_arc_average_blocksize 8192
 zfetch_array_rd_sz 1048576
 zfs_autoimport_disable 1
 zfs_arc_p_min_shift 0
 zio_requeue_io_start_cut_in_line 1
 zfs_vdev_sync_read_max_active 10
 zfs_mdcomp_disable 0
 zfs_arc_num_sublists_per_state 4

Both tools are well-documented and distributed upstream, see
https://github.com/zfsonlinux/zfs/blob/master/cmd/arcstat/arcstat.py
https://github.com/zfsonlinux/zfs/blob/master/cmd/arc_summary/arc_summary.py

See original description
Revision history for this message
Richard Laager (rlaager) wrote :

I think it is safe to "assume" python for the purposes of "is this pulling in more code?". It's not safe to "assume" python for the purposes of the actual dependencies in debian/control. That is, python will have to be listed as a dependency, but I don't think adding this dependency to zfsutils-linux is a problem.

A few questions/comments:

1) Does arc_summary.py actually use these binaries, as listed in the comment at the top of the file? I couldn't find any direct calls to them.
# Binaries used are:
#
# dc(1), kldstat(8), sed(1), sysctl(8) & vmstat(8)

2) We need man pages for these commands. (Debian/Ubuntu Policy 12.1). I don't think upstream ships them, so we'll have to write them first.

3) The commands should have the .py extensions dropped, at least when they're installed in the PATH on Ubuntu. (Debian/Ubuntu policy 10.4.)

4) Upstream also has dbufstat. Is that worth shipping too?

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in zfs-linux (Ubuntu):
status: New → Confirmed
Colin Ian King (colin-king)
Changed in zfs-linux (Ubuntu):
status: Confirmed → In Progress
importance: Undecided → Medium
assignee: nobody → Colin Ian King (colin-king)
Eric Desrochers (slashd)
Changed in zfs-linux (Ubuntu):
assignee: Colin Ian King (colin-king) → Eric Desrochers (slashd)
Revision history for this message
Eric Desrochers (slashd) wrote :

I looked at the Debian pkg, and it looks like everything is okay on the Debian side.

Package: zfsutils-linux
Source: zfs-linux
Version: 0.6.5.7-1
Architecture: amd64
Maintainer: Debian ZFS on Linux maintainers <email address hidden>

---

$ dpkg -c zfsutils-linux_0.6.5.7-1_amd64.deb | egrep -i "arc|dbufstat"
-rwxr-xr-x root/root 40751 2016-05-31 03:40 ./bin/arc_summary.py
-rwxr-xr-x root/root 13057 2016-05-31 03:40 ./bin/arcstat.py
-rwxr-xr-x root/root 18718 2016-05-31 03:40 ./bin/dbufstat.py

I have identified some Debian git commits that may address the situation :

===
commit 730989397063955d60a1845003cfb7433eba1e52
Author: Aron Xu <email address hidden>
Date: Tue Apr 26 17:01:29 2016 +0800

Add dh-python to b-d

--

commit 814413118fb88de3ce596a87a8a011f50c225fdd
Author: Aron Xu <email address hidden>
Date: Sun Nov 22 15:44:16 2015 +0800

Add python related dependencies

===

I'll be working on a test package, and will ask peoples affected by this to test and provide feedbacks.

Regards,
Eric

Eric Desrochers (slashd)
description: updated
description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

I'll be working on fixing this bugs, and then star the SRU for the fix to make its way to stable release.

Revision history for this message
Richard Laager (rlaager) wrote :

@slashd, did Debian drop the .py extensions per Policy? Did they write man pages?

Revision history for this message
Adam Conrad (adconrad) wrote :

These also should be in /usr/bin (or /usr/sbin), not /bin

Eric Desrochers (slashd)
Changed in zfs-linux (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Eric Desrochers (slashd)
Revision history for this message
Eric Desrochers (slashd) wrote :

@rlaager,

Neither debian src code or upstream (git) src code have manpage for arcstat.py, arc_summary.py and dbufstat.py.

But arcstat have a manpage that can be found online :

http://linux.die.net/man/1/arcstat

@adconrad,

Yes, my plan was to place them under /usr/sbin, and not like Debian did under /bin.
/usr/sbin is more appropriate.

Eric

description: updated
Revision history for this message
Eric Desrochers (slashd) wrote :

@rlageer,

At least the program have a "--help" argument

Example :
---
$ /usr/sbin/arcstat.py --help
---
Usage: arcstat.py [-hvx] [-f fields] [-o file] [-s string] [interval [count]]

  -h : Print this help message
  -v : List all possible field headers and definitions
  -x : Print extended stats
  -f : Specify specific fields to print (see -v)
  -o : Redirect output to the specified file
  -s : Override default field separator with custom character or string

Examples:
 arcstat.py -o /tmp/a.log 2 10
 arcstat.py -s "," -o /tmp/a.log 2 10
 arcstat.py -v
 arcstat.py -f time,hit%,dh%,ph%,mh% 1
---

Eric

Eric Desrochers (slashd)
description: updated
description: updated
Eric Desrochers (slashd)
tags: added: patch sts
Revision history for this message
Colin Ian King (colin-king) wrote :
Download full text (7.5 KiB)

Hi Eric,

I applied these patches, built them in a ppa and then did a clean install of zfsutils-linux from -proposed and then upgraded to the packages in the ppa. However, your patches break the upgrade:

king@ubuntu:~$ dpkg -l | grep zfsutils-linux
ii zfsutils-linux 0.6.5.6-0ubuntu11 amd64 Native OpenZFS management utilities for Linux
king@ubuntu:~$ sudo add-apt-repository ppa:colin-king/zfs-0.6.5.7-lp1574342
[sudo] password for king:

 More info: https://launchpad.net/~colin-king/+archive/ubuntu/zfs-0.6.5.7-lp1574342
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmp3bjyv0sl/secring.gpg' created
gpg: keyring `/tmp/tmp3bjyv0sl/pubring.gpg' created
gpg: requesting key B0F01786 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp3bjyv0sl/trustdb.gpg: trustdb created
gpg: key XXXXXXXXX: public key "Launchpad PPA for Colin King" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
OK
king@ubuntu:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial InRelease [17.6 kB]
Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:5 http://gb.archive.ubuntu.com/ubuntu xenial-proposed InRelease
Hit:6 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:7 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 Packages [3,024 B]
Get:8 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main i386 Packages [3,032 B]
Get:9 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main Translation-en [1,584 B]
Fetched 25.2 kB in 1s (21.5 kB/s)
Reading package lists... Done
king@ubuntu:~$ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  linux-image-extra-4.3.0-7-generic
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed
  libpython-stdlib python python-minimal python2.7 python2.7-minimal
The following packages will be upgraded:
  libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-doc zfs-zed
  zfsutils-linux
7 to upgrade, 5 to newly install, 0 to remove and 0 not to upgrade.
Need to get 2,610 kB of archives.
After this operation, 4,986 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7-minimal amd64 2.7.12-1~16.04 [1,294 kB]
Get:2 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libuutil1linux amd64 0.6.5.6-0ubuntu12 [27.5 kB]
Get:3 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libzpool2linux amd64 0.6.5.6-0ubuntu12 [384 kB]
Get:4 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libzfs2linux amd64 0.6.5.6-0ubuntu12 [106 kB]...

Read more...

Revision history for this message
Eric Desrochers (slashd) wrote :

@colin-king,

I think I found the problem,

instead of putting usr/sbin in zfsutils-linux.install.

I'll need to specify them per script as follow:

usr/sbin/arcstat.py
usr/sbin/arc_sumamry.py
usr/sbin/dbufstat.py

I'll re-do the debdiff for both "Y" and "X".

Eric

Revision history for this message
Eric Desrochers (slashd) wrote :

Debdiff for Xenial V2 (16.10)

Revision history for this message
Eric Desrochers (slashd) wrote :

Debdiff for Xenial V2 (16.04)

Revision history for this message
Eric Desrochers (slashd) wrote :

@colin-king, I confirmed the correction works, zed is not complaining anymore :

# Output of apt-get install #

Fetched 905 kB in 2s (320 kB/s)
Selecting previously unselected package zfs-doc.
(Reading database ... 60137 files and directories currently installed.)
Preparing to unpack .../zfs-doc_0.6.5.6-0ubuntu12_all.deb ...
Unpacking zfs-doc (0.6.5.6-0ubuntu12) ...
Selecting previously unselected package libuutil1linux.
Preparing to unpack .../libuutil1linux_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking libuutil1linux (0.6.5.6-0ubuntu12) ...
Selecting previously unselected package libnvpair1linux.
Preparing to unpack .../libnvpair1linux_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking libnvpair1linux (0.6.5.6-0ubuntu12) ...
Selecting previously unselected package libzpool2linux.
Preparing to unpack .../libzpool2linux_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking libzpool2linux (0.6.5.6-0ubuntu12) ...
Selecting previously unselected package libzfs2linux.
Preparing to unpack .../libzfs2linux_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking libzfs2linux (0.6.5.6-0ubuntu12) ...
Selecting previously unselected package zfsutils-linux.
Preparing to unpack .../zfsutils-linux_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking zfsutils-linux (0.6.5.6-0ubuntu12) ...
Preparing to unpack .../zfs-zed_0.6.5.6-0ubuntu12_amd64.deb ...
Unpacking zfs-zed (0.6.5.6-0ubuntu12) ...
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for initramfs-tools (0.122ubuntu8) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-21-generic
W: plymouth: The plugin label.so is missing, the selected theme might not work as expected.
W: plymouth: You might want to install the plymouth-themes and plymouth-label package to fix this.
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up zfs-doc (0.6.5.6-0ubuntu12) ...
Setting up libuutil1linux (0.6.5.6-0ubuntu12) ...
Setting up libnvpair1linux (0.6.5.6-0ubuntu12) ...
Setting up libzpool2linux (0.6.5.6-0ubuntu12) ...
Setting up libzfs2linux (0.6.5.6-0ubuntu12) ...
Setting up zfsutils-linux (0.6.5.6-0ubuntu12) ...
zfs-import-cache.service is a disabled or a static unit, not starting it.
zfs-import-scan.service is a disabled or a static unit, not starting it.
zfs-mount.service is a disabled or a static unit, not starting it.
Processing triggers for initramfs-tools (0.122ubuntu8) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-21-generic
W: plymouth: The plugin label.so is missing, the selected theme might not work as expected.
W: plymouth: You might want to install the plymouth-themes and plymouth-label package to fix this.
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Setting up zfs-zed (0.6.5.6-0ubuntu12) ...
zed.service is a disabled or a static unit, not starting it.
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu4) ...

Revision history for this message
Colin Ian King (colin-king) wrote :

These tested OK, so lets get them sponsored and uploaded. I don't have upload rights, so I can't do that right now.

Revision history for this message
Andy Whitcroft (apw) wrote : Please test proposed package

Hello Hajo, or anyone else affected,

Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.6.5.6-0ubuntu12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in zfs-linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Adam Conrad (adconrad) wrote :

The scripts shipped to fix this bug still violate Policy 10.4, namely:

"When scripts are installed into a directory in the system PATH, the script name should not include an extension such as .sh or .pl that denotes the scripting language currently used to implement it."

Ideally, this should be fixed and upstreamed to Debian (or to zfslinux upstream) as well.

Revision history for this message
Richard Laager (rlaager) wrote :
Revision history for this message
Richard Laager (rlaager) wrote :

I've attached a .debdiff to fix the filenames to comply with Policy 10.4, as @adconrad and I mentioned.

Revision history for this message
Scott Emmons (lscotte) wrote :

Testing with 0.6.5.6-0ubuntu12 confirms the tools are now available as expected. Thank you!

As far as names ending in .py - while it should be fixed, currently they come that way directly from zfsonlinux.

Revision history for this message
Richard Laager (rlaager) wrote :

Sure, but ZoL is not bound by Ubuntu Policy. Ubuntu is. That said, I'll suggest they be fixed upstream too.

Revision history for this message
Eric Desrochers (slashd) wrote :

I have reported a bug to Debian about 2 topics :

* Python script location /usr/sbin or /usr/bin, but not /bin
* Rename python scripts [arcstat, arc_summary * dbufstat] without .py extension as per Debian Policy 10.4 [1]

[1] - https://www.debian.org/doc/debian-policy/ch-files.html#s-scripts

Eric

Revision history for this message
Eric Desrochers (slashd) wrote :

Debian bug :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=832938

Revision history for this message
Eric Desrochers (slashd) wrote :

@lscotte,

Keep in mind that depending on the discussion I'll have with Debian, the scripts location and naming is susceptible to change from what you see in "0.6.5.6-0ubuntu12".

I'll update the bug when I have more details.

Regards,
Eric

Revision history for this message
Scott Emmons (lscotte) wrote :

To add to the confusion - arcstat.py and friends are already in /usr/sbin in the Debian packages I have on Debian Jessie...

$ dpkg -S /usr/sbin/arcstat.py
zfsutils: /usr/sbin/arcstat.py

$ apt show zfsutils
Package: zfsutils
Source: zfs-linux
Version: 0.6.5.7-8-jessie
Maintainer: Debian ZFS on Linux maintainers <email address hidden>
[...]
APT-Sources: http://archive.zfsonlinux.org/debian/ jessie/main amd64 Packages
[...]

Revision history for this message
Eric Desrochers (slashd) wrote :

Thanks lscotte,

That good to know, not sure if this package is an official Debian package though, or more something ZoL have build to work Debian ?

https://github.com/zfsonlinux/zfs/wiki/Debian

"Although ZFS On Linux is now officially in the Debian GNU/Linux repository, we will continue to provide packages for Wheezy and Jessie (but only in the amd64 architecture). The ZoL packages differ slightly compared to the official Debian GNU/Linux packages, mostly in that it provides new, improved sharenfs, sharesmb and shareiscsi options."

But you are right, let's call it the "ZoL Debian package" is build using /usr/sbin, that should be enough to close the subjet of where theses scripts should be stored.

$ dpkg -c zfsutils_0.6.5.7-8-jessie_amd64.deb | egrep -i "arc|dbufstat"
-rwxr-xr-x root/root 40751 2016-05-19 18:59 ./usr/sbin/arc_summary.py
-rwxr-xr-x root/root 13057 2016-05-19 18:59 ./usr/sbin/arcstat.py
-rwxr-xr-x root/root 18718 2016-05-19 18:59 ./usr/sbin/dbufstat.py

Eric

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.6.5.7-0ubuntu5

---------------
zfs-linux (0.6.5.7-0ubuntu5) yakkety; urgency=medium

  * Change to include some important tools in zfsutils-linux build.
  - Add tools : arcstat.py, arc_summary.py & dbufstat.py in /usr/sbin.
  - Change utilities path (bindir) to /usr/sbin per
    https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1574342/comments/6
  - Add python related dependencies (LP: #1574342)

 -- Eric Desrochers <email address hidden> Thu, 28 Jul 2016 16:22:58 -0400

Changed in zfs-linux (Ubuntu Yakkety):
status: In Progress → Fix Released
Revision history for this message
Eric Desrochers (slashd) wrote :

Following a discussion with a ZFS debian maintainer....

He confirmed that he is going to put the python scripts into /usr/sbin and remove the .py suffix in the Debian package.

Eric

Eric Desrochers (slashd)
tags: added: verification-done
Revision history for this message
Scott Emmons (lscotte) wrote :

That's great news that upstream ZFS will get that fixed up - I didn't see a an issue for it at the ZoL github, are there plans to do so. Would be nice from a tracking perspective so this issue is tied in.

Also, I see the Ubuntu update was published for Yakkety, but I'm not sure where Zenial is - will the same Ubuntu fix go for Xenial, or is it waiting for validation, fix from upstream, or ? Thanks and let me know anything I can do to help.

Revision history for this message
Scott Emmons (lscotte) wrote :

OOps, per my last comment regarding tracking issue upstream - that's the Debian bug you previously mentioned, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=832938, not further upstream from ZoL. Sorry for my misunderstanding. :-/

Revision history for this message
Eric Desrochers (slashd) wrote :

Hi scotte (lscotte),

The xenial patched package can be found in the -proposed archive (AKA Fix Commited : last step before landing into stable release -update) if positive feedbacks from community.

Here's the instruction on how to enable -proposed archive:

https://wiki.ubuntu.com/Testing/EnableProposed

Regards,
Eric

Revision history for this message
Eric Desrochers (slashd) wrote :

@rlaager,

Thanks for the .debdiff, let's wait and see what will be done in Debian by Aron, and then cherry-pick what's needed to remove the .py suffix.

Also instead of using "mv" to rename, I wonder if dh-exec can be put to use for renaming files during the dh_install phase.

example :
usr/sbin/arc_summary.py => usr/sbin/arc_summary
usr/sbin/arcstat.py => usr/sbin/arcstat
usr/sbin/dbufstat.py => usr/sbin/dbufstat

Regards,
Eric

Revision history for this message
Scott Emmons (lscotte) wrote :

@slashd Sorry for not being more clear - I already tested from proposed (see comment #22). As far as I can tell, it hasn't changed since then but it is unclear to me if the renaming from .py from upstream will happen separately or as part of this issue. I'm fine with it either way, it just helps in knowing the plan. Thanks.

Revision history for this message
Eric Desrochers (slashd) wrote :

@scotte (lscotte),

The .py removal will come separately, the package found in -proposed (Xenial) and -update (Yakkety) are for the original issue only: arcstat.py, arc_summary.py & dbufstat.py not ship in the Ubuntu zfsutils-linux DEB package.

Another package version will include the .py suffix removal.

Eric

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu12

---------------
zfs-linux (0.6.5.6-0ubuntu12) xenial; urgency=medium

  * Change to include some important tools in zfsutils-linux build.
  - Add tools : arcstat.py, arc_summary.py & dbufstat.py in /usr/sbin.
  - Change utilities path (bindir) to /usr/sbin per
    https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1574342/comments/6
  - Add python related dependencies (LP: #1574342)

zfs-linux (0.6.5.6-0ubuntu11) xenial; urgency=medium

  [ Richard Laager ]
  * Set PATH in cron.d job to fix monthly scrubs. (LP: #1548009)

 -- Eric Desrochers <email address hidden> Thu, 28 Jul 2016 16:18:16 -0400

Changed in zfs-linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.