Merge lp:~roadmr/ubuntu/quantal/checkbox/0.14.9 into lp:ubuntu/quantal/checkbox

Proposed by Daniel Manrique
Status: Merged
Merge reported by: Luke Yelavich
Merged at revision: not available
Proposed branch: lp:~roadmr/ubuntu/quantal/checkbox/0.14.9
Merge into: lp:ubuntu/quantal/checkbox
Diff against target: 2871 lines (+2175/-238)
14 files modified
checkbox/dbus/__init__.py (+89/-0)
checkbox/dbus/udisks2.py (+479/-0)
checkbox/heuristics/__init__.py (+56/-0)
checkbox/heuristics/udev.py (+44/-0)
checkbox/heuristics/udisks2.py (+62/-0)
checkbox/parsers/udevadm.py (+19/-0)
checkbox/tests/heuristics.py (+40/-0)
checkbox/udev.py (+93/-0)
debian/changelog (+15/-0)
debian/control (+1/-0)
jobs/mediacard.txt.in (+24/-24)
scripts/removable_storage_test (+425/-175)
scripts/removable_storage_watcher (+823/-37)
setup.py (+5/-2)
To merge this branch: bzr merge lp:~roadmr/ubuntu/quantal/checkbox/0.14.9
Reviewer Review Type Date Requested Status
Mathieu Trudel-Lapierre Approve
Ubuntu branches Pending
Review via email: mp+128777@code.launchpad.net

Description of the change

This adds udisks2 support to removable storage tests in checkbox. This encompasses media cards (all variants), USB storage devices, ESATA and FireWire.

The code required to implement this is somewhat extensive, but is necessary because udisks1 is not shipping in Quantal and without this, none of the checkbox removable storage tests will work.

I was told we shouldn't need to request an explicit FFe for this as it needs release team review anyway, but the spirit of this is a "new feature", please take this into account when reviewing it.

I'll add the required build and install logs to the linked bug anyway, in case they're needed for the review.

Thanks and apologies for the late-landing request.

To post a comment you must log in.
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Seems fine to me, I'll merge once the bug https://bugs.launchpad.net/checkbox/+bug/1016035 gets approved by the release team.

review: Approve
Revision history for this message
Ara Pulido (ara) wrote :

The release team is asking to upload this, so they can review it in the queue

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'checkbox/dbus'
2=== added file 'checkbox/dbus/__init__.py'
3--- checkbox/dbus/__init__.py 1970-01-01 00:00:00 +0000
4+++ checkbox/dbus/__init__.py 2012-10-09 17:53:24 +0000
5@@ -0,0 +1,89 @@
6+# This file is part of Checkbox.
7+#
8+# Copyright 2012 Canonical Ltd.
9+# Written by:
10+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
11+#
12+# Checkbox is free software: you can redistribute it and/or modify
13+# it under the terms of the GNU General Public License as published by
14+# the Free Software Foundation, either version 3 of the License, or
15+# (at your option) any later version.
16+#
17+# Checkbox is distributed in the hope that it will be useful,
18+# but WITHOUT ANY WARRANTY; without even the implied warranty of
19+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20+# GNU General Public License for more details.
21+#
22+# You should have received a copy of the GNU General Public License
23+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
24+#
25+"""
26+checkbox.dbus
27+=============
28+
29+Utility modules for working with various things accessible over dbus
30+"""
31+
32+import logging
33+
34+from dbus import SystemBus
35+from dbus.mainloop.glib import DBusGMainLoop
36+from dbus import (Array, Boolean, Byte, Dictionary, Double, Int16, Int32,
37+ Int64, ObjectPath, String, Struct, UInt16, UInt32, UInt64)
38+from gi.repository import GObject
39+
40+
41+def connect_to_system_bus():
42+ """
43+ Connect to the system bus properly.
44+
45+ Returns a tuple (system_bus, loop) where loop is a GObject.MainLoop
46+ instance. The loop is there so that you can listen to signals.
47+ """
48+ # We'll need an event loop to observe signals. We will need the instance
49+ # later below so let's keep it. Note that we're not passing it directly
50+ # below as DBus needs specific API. The DBusGMainLoop class that we
51+ # instantiate and pass is going to work with this instance transparently.
52+ #
53+ # NOTE: DBus tutorial suggests that we should create the loop _before_
54+ # connecting to the bus.
55+ logging.debug("Setting up glib-based event loop")
56+ loop = GObject.MainLoop()
57+ # Let's get the system bus object. We need that to access UDisks2 object
58+ logging.debug("Connecting to DBus system bus")
59+ system_bus = SystemBus(mainloop=DBusGMainLoop())
60+ return system_bus, loop
61+
62+
63+def drop_dbus_type(value):
64+ """
65+ Convert types from the DBus bindings to their python counterparts.
66+
67+ This function is mostly lossless, except for arrays of bytes (DBus
68+ signature "y") that are transparently converted to strings, assuming
69+ an UTF-8 encoded string.
70+
71+ The point of this function is to simplify printing of nested DBus data that
72+ gets displayed in a rather illegible way.
73+ """
74+ if isinstance(value, Array) and value.signature == "y":
75+ # Some other things are reported as array of bytes that are just
76+ # strings but due to Unix heritage the encoding is not known.
77+ # In practice it is better to treat them as UTF-8 strings
78+ return bytes(value).decode("UTF-8", "replace").strip("\0")
79+ elif isinstance(value, (Struct, Array)):
80+ return [drop_dbus_type(item) for item in value]
81+ elif isinstance(value, (Dictionary)):
82+ return {drop_dbus_type(dict_key): drop_dbus_type(dict_value)
83+ for dict_key, dict_value in value.items()}
84+ elif isinstance(value, (String, ObjectPath)):
85+ return str(value)
86+ elif isinstance(value, (Byte, UInt16, UInt32, UInt64,
87+ Int16, Int32, Int64)):
88+ return int(value)
89+ elif isinstance(value, Boolean):
90+ return bool(value)
91+ elif isinstance(value, Double):
92+ return float(value)
93+ else:
94+ return value
95
96=== added file 'checkbox/dbus/udisks2.py'
97--- checkbox/dbus/udisks2.py 1970-01-01 00:00:00 +0000
98+++ checkbox/dbus/udisks2.py 2012-10-09 17:53:24 +0000
99@@ -0,0 +1,479 @@
100+# Copyright 2012 Canonical Ltd.
101+# Written by:
102+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
103+#
104+# This program is free software: you can redistribute it and/or modify
105+# it under the terms of the GNU General Public License version 3,
106+# as published by the Free Software Foundation.
107+#
108+# This program is distributed in the hope that it will be useful,
109+# but WITHOUT ANY WARRANTY; without even the implied warranty of
110+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
111+# GNU General Public License for more details.
112+#
113+# You should have received a copy of the GNU General Public License
114+# along with this program. If not, see <http://www.gnu.org/licenses/>.
115+
116+"""
117+checkbox.dbus.udisks2
118+=====================
119+
120+Module for working with UDisks2 from python.
121+
122+There are two main classes that are interesting here.
123+
124+The first class is UDisksObserver, which is easy to setup and has pythonic API
125+to all of the stuff that happens in UDisks2. It offers simple signal handlers
126+for any changes that occur in UDisks2 that were advertised by DBus.
127+
128+The second class is UDisksModel, that builds on the observer class to offer
129+persistent collection of objects managed by UDisks2.
130+
131+To work with this model you will likely want to look at:
132+ http://udisks.freedesktop.org/docs/latest/ref-dbus.html
133+"""
134+
135+import logging
136+
137+from dbus import Interface, PROPERTIES_IFACE
138+from dbus.exceptions import DBusException
139+
140+from checkbox.dbus import drop_dbus_type
141+
142+__all__ = ['UDisks2Observer', 'UDisks2Model', 'Signal', 'is_udisks2_supported',
143+ 'lookup_udev_device']
144+
145+
146+def is_udisks2_supported(system_bus):
147+ """
148+ Check if udisks2 is available on the system bus.
149+
150+ ..note::
151+ Calling this _may_ trigger activation of the UDisks2 daemon but it
152+ should only happen on systems where it is already expected to run all
153+ the time.
154+ """
155+ observer = UDisks2Observer()
156+ try:
157+ logging.debug("Trying to connect to UDisks2...")
158+ observer.connect_to_bus(system_bus)
159+ except DBusException as exc:
160+ if exc.get_dbus_name() == "org.freedesktop.DBus.Error.ServiceUnknown":
161+ logging.debug("No UDisks2 on the system bus")
162+ return False
163+ else:
164+ raise
165+ else:
166+ logging.debug("Got UDisks2 connection")
167+ return True
168+
169+
170+def map_udisks1_connection_bus(udisks1_connection_bus):
171+ """
172+ Map the value of udisks1 ConnectionBus property to the corresponding values
173+ in udisks2. This a lossy function as some values are no longer supported.
174+
175+ Incorrect values raise LookupError
176+ """
177+ return {
178+ 'ata_serial_esata': '', # gone from udisks2
179+ 'firewire': 'ieee1934', # renamed
180+ 'scsi': '', # gone from udisks2
181+ 'sdio': 'sdio', # as-is
182+ 'usb': 'usb', # as-is
183+ }[udisks1_connection_bus]
184+
185+
186+def lookup_udev_device(udisks2_object, udev_devices):
187+ """
188+ Find the udev_device that corresponds to the udisks2 object
189+
190+ Devices are matched by unix filesystem path of the special file (device).
191+ The udisks2_object must implement the block device interface (so that the
192+ block device path can be determined) or a ValueError is raised.
193+
194+ The udisks2_object must be the dictionary that maps from interface names to
195+ dictionaries of properties. For compatible data see
196+ UDisks2Model.managed_objects The udev_devices must be a list of udev
197+ device, as returned from GUdev.
198+
199+ If there is no match, LookupError is raised with the unix block device
200+ path.
201+ """
202+ try:
203+ block_props = udisks2_object[UDISKS2_BLOCK_INTERFACE]
204+ except KeyError:
205+ raise ValueError("udisks2_object must be a block device")
206+ else:
207+ block_dev = block_props['Device']
208+ for udev_device in udev_devices:
209+ if udev_device.get_device_file() == block_dev:
210+ return udev_device
211+ raise LookupError(block_dev)
212+
213+
214+# The well-known name for the ObjectManager interface, sadly it is not a part
215+# of the python binding along with the rest of well-known names.
216+OBJECT_MANAGER_INTERFACE = "org.freedesktop.DBus.ObjectManager"
217+
218+# The well-known name of the filesystem interface implemented by certain
219+# objects exposed by UDisks2
220+UDISKS2_FILESYSTEM_INTERFACE = "org.freedesktop.UDisks2.Filesystem"
221+
222+# The well-known name of the block (device) interface implemented by certain
223+# objects exposed by UDisks2
224+UDISKS2_BLOCK_INTERFACE = "org.freedesktop.UDisks2.Block"
225+
226+# The well-known name of the drive interface implemented by certain objects
227+# exposed by UDisks2
228+UDISKS2_DRIVE_INTERFACE = "org.freedesktop.UDisks2.Drive"
229+
230+
231+class Signal:
232+ """
233+ Basic signal that supports arbitrary listeners.
234+
235+ While this class can be used directly it is best used with the helper
236+ decorator Signal.define on a member function. The function body is ignored,
237+ apart from the documentation.
238+
239+ The function name then becomes a unique (per encapsulating class instance)
240+ object (an instance of this Signal class) that is created on demand.
241+
242+ In practice you just have a documentation and use
243+ object.signal_name.connect() and object.signal_name(*args, **kwargs) to
244+ fire it.
245+ """
246+
247+ def __init__(self, signal_name):
248+ """
249+ Construct a signal with the given name
250+ """
251+ self._listeners = []
252+ self._signal_name = signal_name
253+
254+ def connect(self, listener):
255+ """
256+ Connect a new listener to this signal
257+
258+ That listener will be called whenever fire() is invoked on the signal
259+ """
260+ self._listeners.append(listener)
261+
262+ def disconnect(self, listener):
263+ """
264+ Disconnect an existing listener from this signal
265+ """
266+ self._listeners.remove(listener)
267+
268+ def fire(self, args, kwargs):
269+ """
270+ Fire this signal with the specified arguments and keyword arguments.
271+
272+ Typically this is used by using __call__() on this object which is more
273+ natural as it does all the argument packing/unpacking transparently.
274+ """
275+ for listener in self._listeners:
276+ listener(*args, **kwargs)
277+
278+ def __call__(self, *args, **kwargs):
279+ """
280+ Call fire() with all arguments forwarded transparently
281+ """
282+ self.fire(args, kwargs)
283+
284+ @classmethod
285+ def define(cls, dummy_func):
286+ """
287+ Helper decorator to define a signal descriptor in a class
288+
289+ The decorated function is never called but is used to get
290+ documentation.
291+ """
292+ return _SignalDescriptor(dummy_func)
293+
294+
295+class _SignalDescriptor:
296+ """
297+ Descriptor for convenient signal access.
298+
299+ Typically this class is used indirectly, when accessed from Signal.define
300+ method decorator. It is used to do all the magic required when accessing
301+ signal name on a class or instance.
302+ """
303+
304+ def __init__(self, dummy_func):
305+ self.signal_name = dummy_func.__name__
306+ self.__doc__ = dummy_func.__doc__
307+
308+ def __repr__(self):
309+ return "<SignalDecorator for signal: %r>" % self.signal_name
310+
311+ def __get__(self, instance, owner):
312+ if instance is None:
313+ return self
314+ # Ensure that the instance has __signals__ property
315+ if not hasattr(instance, "__signals__"):
316+ instance.__signals__ = {}
317+ if self.signal_name not in instance.__signals__:
318+ instance.__signals__[self.signal_name] = Signal(self.signal_name)
319+ return instance.__signals__[self.signal_name]
320+
321+ def __set__(self, instance, value):
322+ raise AttributeError("You cannot overwrite signals")
323+
324+ def __delete__(self, instance):
325+ raise AttributeError("You cannot delete signals")
326+
327+
328+class UDisks2Observer:
329+ """
330+ Class for observing ongoing changes in UDisks2
331+ """
332+
333+ def __init__(self):
334+ """
335+ Create a UDisks2 model.
336+
337+ The model must be connected to a bus before it is first used, see
338+ connect()
339+ """
340+ # Proxy to the UDisks2 object
341+ self._udisks2_obj = None
342+ # Proxy to the ObjectManager interface exposed by UDisks2 object
343+ self._udisks2_obj_manager = None
344+
345+ @Signal.define
346+ def on_initial_objects(self, managed_objects):
347+ """
348+ Signal fired when the initial list of objects becomes available
349+ """
350+
351+ @Signal.define
352+ def on_interfaces_added(self, object_path, interfaces_and_properties):
353+ """
354+ Signal fired when one or more interfaces gets added to a specific
355+ object.
356+ """
357+
358+ @Signal.define
359+ def on_interfaces_removed(self, object_path, interfaces):
360+ """
361+ Signal fired when one or more interface gets removed from a specific
362+ object
363+ """
364+
365+ @Signal.define
366+ def on_properties_changed(self, interface_name, changed_properties,
367+ invalidated_properties, sender=None):
368+ """
369+ Signal fired when one or more property changes value or becomes
370+ invalidated.
371+ """
372+
373+ def connect_to_bus(self, bus):
374+ """
375+ Establish initial connection to UDisks2 on the specified DBus bus.
376+
377+ This will also load the initial set of objects from UDisks2 and thus
378+ fire the on_initial_objects() signal from the model. Please call this
379+ method only after connecting that signal if you want to observe that
380+ event.
381+ """
382+ # Once everything is ready connect to udisks2
383+ self._connect_to_udisks2(bus)
384+ # And read all the initial objects and setup
385+ # change event handlers
386+ self._get_initial_objects()
387+
388+ def _connect_to_udisks2(self, bus):
389+ """
390+ Setup the initial connection to UDisks2
391+
392+ This step can fail if UDisks2 is not available and cannot be
393+ service-activated.
394+ """
395+ # Access the /org/freedesktop/UDisks2 object sitting on the
396+ # org.freedesktop.UDisks2 bus name. This will trigger the necessary
397+ # activation if udisksd is not running for any reason
398+ logging.debug("Accessing main UDisks2 object")
399+ self._udisks2_obj = bus.get_object(
400+ "org.freedesktop.UDisks2", "/org/freedesktop/UDisks2")
401+ # Now extract the standard ObjectManager interface so that we can
402+ # observe and iterate the collection of objects that UDisks2 provides.
403+ logging.debug("Accessing ObjectManager interface on UDisks2 object")
404+ self._udisks2_obj_manager = Interface(
405+ self._udisks2_obj, OBJECT_MANAGER_INTERFACE)
406+ # Connect to the PropertiesChanged signal. Here unlike before we want
407+ # to listen to all signals, regardless of who was sending them in the
408+ # first place.
409+ logging.debug("Setting up DBus signal handler for PropertiesChanged")
410+ bus.add_signal_receiver(
411+ self._on_properties_changed,
412+ signal_name="PropertiesChanged",
413+ dbus_interface=PROPERTIES_IFACE,
414+ # Use the sender_keyword keyword argument to indicate that we wish
415+ # to know the sender of each signal. For consistency with other
416+ # signals we choose to use the 'object_path' keyword argument.
417+ sender_keyword='sender')
418+
419+ def _get_initial_objects(self):
420+ """
421+ Get the initial collection of objects.
422+
423+ Needs to be called before the first signals from DBus are observed.
424+ Requires a working connection to UDisks2.
425+ """
426+ # Having this interface we can now peek at the existing objects.
427+ # We can use the standard method GetManagedObjects() to do that
428+ logging.debug("Accessing GetManagedObjects() on UDisks2 object")
429+ managed_objects = self._udisks2_obj_manager.GetManagedObjects()
430+ managed_objects = drop_dbus_type(managed_objects)
431+ # Fire the public signal for getting initial objects
432+ self.on_initial_objects(managed_objects)
433+ # Connect our internal handles to the DBus signal handlers
434+ logging.debug("Setting up DBus signal handler for InterfacesAdded")
435+ self._udisks2_obj_manager.connect_to_signal(
436+ "InterfacesAdded", self._on_interfaces_added)
437+ logging.debug("Setting up DBus signal handler for InterfacesRemoved")
438+ self._udisks2_obj_manager.connect_to_signal(
439+ "InterfacesRemoved", self._on_interfaces_removed)
440+
441+ def _on_interfaces_added(self, object_path, interfaces_and_properties):
442+ """
443+ Internal callback that is called by DBus
444+
445+ This function is responsible for firing the public signal
446+ """
447+ # Convert from dbus types
448+ object_path = drop_dbus_type(object_path)
449+ interfaces_and_properties = drop_dbus_type(interfaces_and_properties)
450+ # Log what's going on
451+ logging.debug("The object %r has gained the following interfaces and "
452+ "properties: %r", object_path, interfaces_and_properties)
453+ # Call the signal handler
454+ self.on_interfaces_added(object_path, interfaces_and_properties)
455+
456+ def _on_interfaces_removed(self, object_path, interfaces):
457+ """
458+ Internal callback that is called by DBus
459+
460+ This function is responsible for firing the public signal
461+ """
462+ # Convert from dbus types
463+ object_path = drop_dbus_type(object_path)
464+ interfaces = drop_dbus_type(interfaces)
465+ # Log what's going on
466+ logging.debug("The object %r has lost the following interfaces: %r",
467+ object_path, interfaces)
468+ # Call the signal handler
469+ self.on_interfaces_removed(object_path, interfaces)
470+
471+ def _on_properties_changed(self, interface_name, changed_properties,
472+ invalidated_properties, sender=None):
473+ """
474+ Internal callback that is called by DBus
475+
476+ This function is responsible for firing the public signal
477+ """
478+ # Convert from dbus types
479+ interface_name = drop_dbus_type(interface_name)
480+ changed_properties = drop_dbus_type(changed_properties)
481+ invalidated_properties = drop_dbus_type(invalidated_properties)
482+ sender = drop_dbus_type(sender)
483+ # Log what's going on
484+ logging.debug("Some object with the interface %r has changed "
485+ "properties: %r and invalidated properties %r "
486+ "(sender: %s)",
487+ interface_name, changed_properties,
488+ invalidated_properties, sender)
489+ # Call the signal handler
490+ self.on_properties_changed(interface_name, changed_properties,
491+ invalidated_properties, sender)
492+
493+
494+class UDisks2Model:
495+ """
496+ Model for working with UDisks2
497+
498+ This class maintains a persistent model of what UDisks2 knows about, based
499+ on the UDisks2Observer class and the signals it offers.
500+ """
501+
502+ def __init__(self, observer):
503+ """
504+ Create a UDisks2 model.
505+
506+ The model will track changes using the specified observer (which is
507+ expected to be a UDisks2Observer instance)
508+
509+ You should only connect the observer to the bus after creating the
510+ model otherwise the initial objects will not be detected.
511+ """
512+ # Local state, everything that UDisks2 tells us
513+ self._managed_objects = {}
514+ self._observer = observer
515+ # Connect all the signals to the observer
516+ self._observer.on_initial_objects.connect(self._on_initial_objects)
517+ self._observer.on_interfaces_added.connect(self._on_interfaces_added)
518+ self._observer.on_interfaces_removed.connect(
519+ self._on_interfaces_removed)
520+ self._observer.on_properties_changed.connect(
521+ self._on_properties_changed)
522+
523+ @Signal.define
524+ def on_change(self):
525+ """
526+ Signal sent whenever the collection of managed object changes
527+
528+ Note that this signal is fired _after_ the change has occurred
529+ """
530+
531+ @property
532+ def managed_objects(self):
533+ """
534+ A collection of objects that is managed by this model. All changes as
535+ well as the initial state, are reflected here.
536+ """
537+ return self._managed_objects
538+
539+ def _on_initial_objects(self, managed_objects):
540+ """
541+ Internal callback called when we get the initial collection of objects
542+ """
543+ self._managed_objects = drop_dbus_type(managed_objects)
544+
545+ def _on_interfaces_added(self, object_path, interfaces_and_properties):
546+ """
547+ Internal callback called when an interface is added to certain object
548+ """
549+ # Update internal state
550+ if object_path not in self._managed_objects:
551+ self._managed_objects[object_path] = {}
552+ obj = self._managed_objects[object_path]
553+ obj.update(interfaces_and_properties)
554+ # Fire the change signal
555+ self.on_change()
556+
557+ def _on_interfaces_removed(self, object_path, interfaces):
558+ """
559+ Internal callback called when an interface is removed from a certain
560+ object
561+ """
562+ # Update internal state
563+ if object_path in self._managed_objects:
564+ obj = self._managed_objects[object_path]
565+ for interface in interfaces:
566+ if interface in obj:
567+ del obj[interface]
568+ # Fire the change signal
569+ self.on_change()
570+
571+ def _on_properties_changed(self, interface_name, changed_properties,
572+ invalidated_properties, sender=None):
573+ # XXX: This is a workaround the fact that we cannot
574+ # properly track changes to all properties :-(
575+ self._managed_objects = drop_dbus_type(
576+ self._observer._udisks2_obj_manager.GetManagedObjects())
577+ # Fire the change signal()
578+ self.on_change()
579
580=== added directory 'checkbox/heuristics'
581=== added file 'checkbox/heuristics/__init__.py'
582--- checkbox/heuristics/__init__.py 1970-01-01 00:00:00 +0000
583+++ checkbox/heuristics/__init__.py 2012-10-09 17:53:24 +0000
584@@ -0,0 +1,56 @@
585+# This file is part of Checkbox.
586+#
587+# Copyright 2012 Canonical Ltd.
588+# Written by:
589+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
590+#
591+# Checkbox is free software: you can redistribute it and/or modify
592+# it under the terms of the GNU General Public License as published by
593+# the Free Software Foundation, either version 3 of the License, or
594+# (at your option) any later version.
595+#
596+# Checkbox is distributed in the hope that it will be useful,
597+# but WITHOUT ANY WARRANTY; without even the implied warranty of
598+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
599+# GNU General Public License for more details.
600+#
601+# You should have received a copy of the GNU General Public License
602+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
603+
604+"""
605+checkbox.heuristics
606+===================
607+
608+This module contains implementations behind various heuristics used throughout
609+the code. The intent of this module is twofold:
610+
611+ 1) To encourage code reuse so that developers can use one implementation of
612+ "guesswork" that is sometimes needed in our test. This reduces duplicate
613+ bugs where many scripts do similar things in a different way.
614+
615+ 2) To identify missing features in plumbing layer APIs such as
616+ udev/udisks/dbus etc. Ideally no program should have to guess this, the
617+ plumbing layer should be able to provide this meta data to allow
618+ application developers deliver consistent behavior across userspace.
619+
620+Heuristics should be reusable from both python and shell. To make that possible
621+each heuristics needs to be constrained to serializable input and output. This
622+levels the playing field and allows both shell developers and python developers
623+to reuse the same function.
624+
625+Additionally heuristics should try to avoid accessing thick APIs (such as
626+objects returned by various libraries. This is meant to decrease the likelihood
627+that updates to those libraries break this code. As an added side effect this
628+also should make the implementation more explicit and easier to understand.
629+
630+In the long term each heuristic should be discussed with upstream developers of
631+the particular problem area (udev, udisks, etc) to see if that subsystem can
632+provide the required information directly, without us having to guess and fill
633+the gaps.
634+
635+Things to consider when adding entries to this package:
636+
637+ 1) File a bug on the upstream package about missing feature.
638+
639+ 2) File a bug on checkbox to de-duplicate similar heuristics
640+"""
641
642=== added file 'checkbox/heuristics/udev.py'
643--- checkbox/heuristics/udev.py 1970-01-01 00:00:00 +0000
644+++ checkbox/heuristics/udev.py 2012-10-09 17:53:24 +0000
645@@ -0,0 +1,44 @@
646+# This file is part of Checkbox.
647+#
648+# Copyright 2012 Canonical Ltd.
649+# Written by:
650+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
651+#
652+# Checkbox is free software: you can redistribute it and/or modify
653+# it under the terms of the GNU General Public License as published by
654+# the Free Software Foundation, either version 3 of the License, or
655+# (at your option) any later version.
656+#
657+# Checkbox is distributed in the hope that it will be useful,
658+# but WITHOUT ANY WARRANTY; without even the implied warranty of
659+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
660+# GNU General Public License for more details.
661+#
662+# You should have received a copy of the GNU General Public License
663+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
664+
665+"""
666+checkbox.heuristics.dev
667+=======================
668+
669+Heuristics for udev.
670+
671+ Documentation: http://udisks.freedesktop.org/docs/latest/
672+ Source code: http://cgit.freedesktop.org/systemd/systemd/ (src/udev)
673+ Bug tracker: http://bugs.freedesktop.org/ (using systemd product)
674+"""
675+
676+
677+def is_virtual_device(device_file):
678+ """
679+ Given a device name like /dev/ramX, /dev/sdX or /dev/loopX determine if
680+ this is a virtual device. Virtual devices are typically uninteresting to
681+ users. The only exception may be nonempty loopback device.
682+
683+ Possible prior art: gnome-disks, palimpset (precursor, suffering from this
684+ flaw and showing all the /dev/ram devices by default)
685+ """
686+ for part in device_file.split("/"):
687+ if part.startswith("ram") or part.startswith("loop"):
688+ return True
689+ return False
690
691=== added file 'checkbox/heuristics/udisks2.py'
692--- checkbox/heuristics/udisks2.py 1970-01-01 00:00:00 +0000
693+++ checkbox/heuristics/udisks2.py 2012-10-09 17:53:24 +0000
694@@ -0,0 +1,62 @@
695+# This file is part of Checkbox.
696+#
697+# Copyright 2012 Canonical Ltd.
698+# Written by:
699+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
700+#
701+# Checkbox is free software: you can redistribute it and/or modify
702+# it under the terms of the GNU General Public License as published by
703+# the Free Software Foundation, either version 3 of the License, or
704+# (at your option) any later version.
705+#
706+# Checkbox is distributed in the hope that it will be useful,
707+# but WITHOUT ANY WARRANTY; without even the implied warranty of
708+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
709+# GNU General Public License for more details.
710+#
711+# You should have received a copy of the GNU General Public License
712+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
713+
714+"""
715+checkbox.heuristics.udisks2
716+===========================
717+
718+Heuristics for udisks2.
719+
720+ Documentation: http://udisks.freedesktop.org/docs/latest/
721+ Source code: http://cgit.freedesktop.org/systemd/systemd/ (src/udev)
722+ Bug tracker: http://bugs.freedesktop.org/ (using systemd product)
723+"""
724+
725+from checkbox.parsers.udevadm import CARD_READER_RE, GENERIC_RE, FLASH_RE
726+
727+
728+def is_memory_card(vendor, model, udisks2_media):
729+ """
730+ Check if the device seems to be a memory card
731+
732+ The vendor and model arguments are _strings_, not integers.
733+ The udisks2_media argument is the value of org.freedesktop.UDisks2.Drive/
734+
735+
736+ This is rather fuzzy, sadly udev and udisks2 don't do a very good job and
737+ mostly don't specify the "media" property (it has a few useful values, such
738+ as flash_cf, flash_ms, flash_sm, flash_sd, flash_sdhc, flash_sdxc and
739+ flash_mmc but I have yet to see a device that reports such values)
740+ """
741+ # Treat any udisks2_media that contains 'flash' as a memory card
742+ if udisks2_media is not None and FLASH_RE.search(udisks2_media):
743+ return True
744+ # Treat any device that match model name to the following regular
745+ # expression as a memory card reader.
746+ if CARD_READER_RE.search(model):
747+ return True
748+ # Treat any device that contains the word 'Generic' in the vendor string as
749+ # a memory card reader.
750+ #
751+ # XXX: This seems odd but strangely enough seems to gets the job done. I
752+ # guess if I should start filing tons of bugs/patches on udev/udisks2 to
753+ # just have a few more rules and make this rule obsolete.
754+ if GENERIC_RE.search(vendor):
755+ return True
756+ return False
757
758=== modified file 'checkbox/parsers/udevadm.py'
759--- checkbox/parsers/udevadm.py 2012-10-04 02:54:55 +0000
760+++ checkbox/parsers/udevadm.py 2012-10-09 17:53:24 +0000
761@@ -59,6 +59,9 @@
762 r"^scsi:"
763 r"t-0x(?P<type>[%(hex)s]{2})"
764 % {"hex": string.hexdigits})
765+CARD_READER_RE = re.compile(r"SD|MMC|CF|MS|SM|xD|Card", re.I)
766+GENERIC_RE = re.compile(r"Generic", re.I)
767+FLASH_RE = re.compile(r"Flash", re.I)
768
769
770 class UdevadmDevice:
771@@ -208,6 +211,22 @@
772 if test_bit(Input.BTN_MOUSE, bitmask, self._bits):
773 return "MOUSE"
774
775+ if self.driver:
776+ if self.driver.startswith("sdhci"):
777+ return "CARDREADER"
778+
779+ if self.driver.startswith("mmc"):
780+ return "CARDREADER"
781+
782+ if self.driver == "sd" and self.product:
783+ if any(FLASH_RE.search(k) for k in self._environment.keys()):
784+ return "CARDREADER"
785+ if any(d.bus == 'usb' for d in self._stack):
786+ if CARD_READER_RE.search(self.product):
787+ return "CARDREADER"
788+ if GENERIC_RE.search(self.vendor):
789+ return "CARDREADER"
790+
791 if "ID_TYPE" in self._environment:
792 id_type = self._environment["ID_TYPE"]
793
794
795=== added file 'checkbox/tests/heuristics.py'
796--- checkbox/tests/heuristics.py 1970-01-01 00:00:00 +0000
797+++ checkbox/tests/heuristics.py 2012-10-09 17:53:24 +0000
798@@ -0,0 +1,40 @@
799+# This file is part of Checkbox.
800+#
801+# Copyright 2012 Canonical Ltd.
802+# Written by:
803+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
804+#
805+# Checkbox is free software: you can redistribute it and/or modify
806+# it under the terms of the GNU General Public License as published by
807+# the Free Software Foundation, either version 3 of the License, or
808+# (at your option) any later version.
809+#
810+# Checkbox is distributed in the hope that it will be useful,
811+# but WITHOUT ANY WARRANTY; without even the implied warranty of
812+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
813+# GNU General Public License for more details.
814+#
815+# You should have received a copy of the GNU General Public License
816+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
817+
818+"""
819+
820+checkbox.tests.heuristics
821+=========================
822+
823+Unit tests for checkbox.heuristics package
824+"""
825+
826+from unittest import TestCase
827+
828+from checkbox.heuristics.udisks2 import is_memory_card
829+
830+
831+class TestIsMemoryCard(TestCase):
832+
833+ def test_generic(self):
834+ """
835+ Device with vendor string "GENERIC" is a memory card
836+ """
837+ self.assertTrue(
838+ is_memory_card(vendor="Generic", model="", udisks2_media=None))
839
840=== added file 'checkbox/udev.py'
841--- checkbox/udev.py 1970-01-01 00:00:00 +0000
842+++ checkbox/udev.py 2012-10-09 17:53:24 +0000
843@@ -0,0 +1,93 @@
844+# This file is part of Checkbox.
845+#
846+# Copyright 2012 Canonical Ltd.
847+# Written by:
848+# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
849+#
850+# Checkbox is free software: you can redistribute it and/or modify
851+# it under the terms of the GNU General Public License as published by
852+# the Free Software Foundation, either version 3 of the License, or
853+# (at your option) any later version.
854+#
855+# Checkbox is distributed in the hope that it will be useful,
856+# but WITHOUT ANY WARRANTY; without even the implied warranty of
857+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
858+# GNU General Public License for more details.
859+#
860+# You should have received a copy of the GNU General Public License
861+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
862+#
863+"""
864+checkbox.udev
865+=============
866+
867+A collection of utility function sfor interacting with GUdev
868+"""
869+
870+from gi.repository import GUdev
871+
872+from checkbox.heuristics.udev import is_virtual_device
873+
874+
875+def get_interconnect_speed(device):
876+ """
877+ Compute the speed of the USB interconnect of single device
878+
879+ This function traverses up the tree of devices all the way to the root
880+ object and returns the minimum value of the 'speed' sysfs attribute, or
881+ None if there was no speed in any of the devices.
882+ """
883+ # We'll track the actual speed of the interconnect here. The value None
884+ # means that we just don't know
885+ interconnect_speed = None
886+ while device:
887+ # For each udev device that we traverse we attempt to lookup the
888+ # 'speed' attribute. If present it is converted to an ASCII string and
889+ # then to an integer. That integer represents the speed of the
890+ # interconnect in megabits.
891+ #
892+ # Here we use get_sysfs_attr_as_int that does it all for us, returning
893+ # 0 if anything is wrong.
894+ device_speed = device.get_sysfs_attr_as_int('speed')
895+ if device_speed != 0: # Empty values get truncated to 0
896+ # As USB devices can be connected via any number of hubs we
897+ # carefully use the smallest number that is encountered but it
898+ # seems that the Kernel already does the right thing and shows a
899+ # SuperSpeed USB 3.0 device (that normally has speed of 5000Mbit/s)
900+ # which is connected to a HighSpeed USB 2.0 device (that is limited
901+ # to 480Mbit/s) to also have the smaller, 480Mbit/s speed.
902+ if interconnect_speed is not None:
903+ interconnect_speed = min(interconnect_speed, device_speed)
904+ else:
905+ interconnect_speed = device_speed
906+ # We walk up the tree of udev devices looking for any parent that
907+ # belongs to the 'usb' subsystem having device_type of 'usb_device'. I
908+ # have not managed to find any documentation about this (I've yet to
909+ # check Kernel documentation) but casual observation and testing seems
910+ # to indicate that this is what we want.
911+ # TODO: get_parent_with_subsystem('usb', 'usb_device')
912+ device = device.get_parent()
913+ return interconnect_speed
914+
915+
916+def get_udev_block_devices(udev_client):
917+ """
918+ Get a list of all block devices
919+
920+ Returns a list of GUdev.Device objects representing all block devices in
921+ the system. Virtual devices are filtered away using
922+ checkbox.heuristics.udev.is_virtual_device.
923+ """
924+ # setup an enumerator so that we can list devices
925+ enumerator = GUdev.Enumerator(client=udev_client)
926+ # Iterate over block devices only
927+ enumerator.add_match_subsystem('block')
928+ # Convert the enumerator into a plain list and filter-away all
929+ # devices deemed virtual by the heuristic.
930+ devices = [
931+ device for device in enumerator.execute()
932+ if not is_virtual_device(device.get_device_file())]
933+ # Sort the list, this is not needed but makes various debugging dumps
934+ # look better.
935+ devices.sort(key=lambda device: device.get_device_file())
936+ return devices
937
938=== modified file 'debian/changelog'
939--- debian/changelog 2012-10-05 14:04:55 +0000
940+++ debian/changelog 2012-10-09 17:53:24 +0000
941@@ -1,3 +1,18 @@
942+checkbox (0.14.9) quantal; urgency=low
943+
944+ * New upstream release (LP: #1064504)
945+
946+ [Zygmunt Krynicki]
947+ * checkbox/dbus/__init__.py, checkbox/dbus/udisks2.py, checkbox/heuristics/,
948+ checkbox/heuristics/__init__.py, checkbox/heuristics/udev.py,
949+ checkbox/heuristics/udisks2.py, checkbox/tests/heuristics.py,
950+ checkbox/udev.py, checkbox/parsers/udevadm.py, debian/control,
951+ jobs/mediacard.txt.in, scripts/removable_storage_test,
952+ scripts/removable_storage_watcher, setup.py: Added udisks2 support
953+ (LP: #1016035)
954+
955+ -- Daniel Manrique <roadmr@ubuntu.com> Tue, 09 Oct 2012 12:23:41 -0400
956+
957 checkbox (0.14.8) quantal; urgency=low
958
959 * New upstream release (LP: #1061359)
960
961=== modified file 'debian/control'
962--- debian/control 2012-10-04 02:54:55 +0000
963+++ debian/control 2012-10-09 17:53:24 +0000
964@@ -22,6 +22,7 @@
965 Depends: debconf,
966 python3-lxml,
967 udev,
968+ gir1.2-gudev-1.0,
969 udisks2 | udisks,
970 ${misc:Depends},
971 ${python3:Depends},
972
973=== modified file 'jobs/mediacard.txt.in'
974--- jobs/mediacard.txt.in 2012-10-04 02:54:55 +0000
975+++ jobs/mediacard.txt.in 2012-10-09 17:53:24 +0000
976@@ -1,6 +1,6 @@
977 plugin: manual
978 name: mediacard/mmc-insert
979-command: removable_storage_watcher insert sdio usb scsi
980+command: removable_storage_watcher --memorycard insert sdio usb scsi
981 _description:
982 PURPOSE:
983 This test will check that the systems media card reader can
984@@ -16,7 +16,7 @@
985 plugin: shell
986 name: mediacard/mmc-storage
987 depends: mediacard/mmc-insert
988-command: removable_storage_test sdio usb scsi
989+command: removable_storage_test --memorycard sdio usb scsi
990 _description:
991 This test is automated and executes after the mediacard/mmc-insert
992 test is run. It tests reading and writing to the MMC card.
993@@ -24,7 +24,7 @@
994 plugin: manual
995 name: mediacard/mmc-remove
996 depends: mediacard/mmc-storage
997-command: removable_storage_watcher remove sdio usb scsi
998+command: removable_storage_watcher --memorycard remove sdio usb scsi
999 _description:
1000 PURPOSE:
1001 This test will check that the system correctly detects
1002@@ -39,7 +39,7 @@
1003 plugin: manual
1004 name: mediacard/mmc-insert-after-suspend
1005 depends: suspend/suspend_advanced
1006-command: removable_storage_watcher insert sdio usb scsi
1007+command: removable_storage_watcher --memorycard insert sdio usb scsi
1008 _description:
1009 PURPOSE:
1010 This test will check that the systems media card reader can
1011@@ -55,7 +55,7 @@
1012 plugin: shell
1013 name: mediacard/mmc-storage-after-suspend
1014 depends: mediacard/mmc-insert-after-suspend
1015-command: removable_storage_test sdio usb scsi
1016+command: removable_storage_test --memorycard sdio usb scsi
1017 _description:
1018 This test is automated and executes after the mediacard/mmc-insert-after-suspend test
1019 is run. It tests reading and writing to the MMC card after the system has been suspended.
1020@@ -63,7 +63,7 @@
1021 plugin: manual
1022 name: mediacard/mmc-remove-after-suspend
1023 depends: mediacard/mmc-storage-after-suspend
1024-command: removable_storage_watcher remove sdio usb scsi
1025+command: removable_storage_watcher --memorycard remove sdio usb scsi
1026 _description:
1027 PURPOSE:
1028 This test will check that the system correctly detects the removal
1029@@ -77,7 +77,7 @@
1030
1031 plugin: manual
1032 name: mediacard/sd-insert
1033-command: removable_storage_watcher insert sdio usb scsi
1034+command: removable_storage_watcher --memorycard insert sdio usb scsi
1035 _description:
1036 PURPOSE:
1037 This test will check that the systems media card reader can
1038@@ -93,7 +93,7 @@
1039 plugin: shell
1040 name: mediacard/sd-storage
1041 depends: mediacard/sd-insert
1042-command: removable_storage_test sdio usb scsi
1043+command: removable_storage_test --memorycard sdio usb scsi
1044 _description:
1045 This test is automated and executes after the mediacard/sd-insert
1046 test is run. It tests reading and writing to the SD card.
1047@@ -101,7 +101,7 @@
1048 plugin: manual
1049 name: mediacard/sd-remove
1050 depends: mediacard/sd-storage
1051-command: removable_storage_watcher remove sdio usb scsi
1052+command: removable_storage_watcher --memorycard remove sdio usb scsi
1053 _description:
1054 PURPOSE:
1055 This test will check that the system correctly detects
1056@@ -116,7 +116,7 @@
1057 plugin: manual
1058 name: mediacard/sd-insert-after-suspend
1059 depends: suspend/suspend_advanced
1060-command: removable_storage_watcher insert sdio usb scsi
1061+command: removable_storage_watcher --memorycard insert sdio usb scsi
1062 _description:
1063 PURPOSE:
1064 This test will check that the systems media card reader can
1065@@ -132,7 +132,7 @@
1066 plugin: shell
1067 name: mediacard/sd-storage-after-suspend
1068 depends: mediacard/sd-insert-after-suspend
1069-command: removable_storage_test sdio usb scsi
1070+command: removable_storage_test --memorycard sdio usb scsi
1071 _description:
1072 This test is automated and executes after the mediacard/sd-insert-after-suspend test
1073 is run. It tests reading and writing to the SD card after the system has been suspended.
1074@@ -140,7 +140,7 @@
1075 plugin: manual
1076 name: mediacard/sd-remove-after-suspend
1077 depends: mediacard/sd-storage-after-suspend
1078-command: removable_storage_watcher remove sdio usb scsi
1079+command: removable_storage_watcher --memorycard remove sdio usb scsi
1080 _description:
1081 PURPOSE:
1082 This test will check that the system correctly detects
1083@@ -154,7 +154,7 @@
1084
1085 plugin: manual
1086 name: mediacard/sdhc-insert
1087-command: removable_storage_watcher insert sdio usb scsi
1088+command: removable_storage_watcher --memorycard insert sdio usb scsi
1089 _description:
1090 PURPOSE:
1091 This test will check that the systems media card reader can
1092@@ -170,7 +170,7 @@
1093 plugin: shell
1094 name: mediacard/sdhc-storage
1095 depends: mediacard/sdhc-insert
1096-command: removable_storage_test sdio usb scsi
1097+command: removable_storage_test --memorycard sdio usb scsi
1098 _description:
1099 This test is automated and executes after the mediacard/sdhc-insert
1100 test is run. It tests reading and writing to the SDHC card.
1101@@ -178,7 +178,7 @@
1102 plugin: manual
1103 name: mediacard/sdhc-remove
1104 depends: mediacard/sdhc-storage
1105-command: removable_storage_watcher remove sdio usb scsi
1106+command: removable_storage_watcher --memorycard remove sdio usb scsi
1107 _description:
1108 PURPOSE:
1109 This test will check that the system correctly detects
1110@@ -193,7 +193,7 @@
1111 plugin: manual
1112 name: mediacard/sdhc-insert-after-suspend
1113 depends: suspend/suspend_advanced
1114-command: removable_storage_watcher insert sdio usb scsi
1115+command: removable_storage_watcher --memorycard insert sdio usb scsi
1116 _description:
1117 PURPOSE:
1118 This test will check that the systems media card reader can
1119@@ -209,7 +209,7 @@
1120 plugin: shell
1121 name: mediacard/sdhc-storage-after-suspend
1122 depends: mediacard/sdhc-insert-after-suspend
1123-command: removable_storage_test sdio usb scsi
1124+command: removable_storage_test --memorycard sdio usb scsi
1125 _description:
1126 This test is automated and executes after the mediacard/sdhc-insert-after-suspend test
1127 is run. It tests reading and writing to the SDHC card after the system has been suspended.
1128@@ -217,7 +217,7 @@
1129 plugin: manual
1130 name: mediacard/sdhc-remove-after-suspend
1131 depends: mediacard/sdhc-storage-after-suspend
1132-command: removable_storage_watcher remove sdio usb scsi
1133+command: removable_storage_watcher --memorycard remove sdio usb scsi
1134 _description:
1135 PURPOSE:
1136 This test will check that the system correctly detects the removal
1137@@ -231,7 +231,7 @@
1138
1139 plugin: manual
1140 name: mediacard/cf-insert
1141-command: removable_storage_watcher insert sdio usb scsi
1142+command: removable_storage_watcher --memorycard insert sdio usb scsi
1143 _description:
1144 PURPOSE:
1145 This test will check that the systems media card reader can
1146@@ -247,7 +247,7 @@
1147 plugin: shell
1148 name: mediacard/cf-storage
1149 depends: mediacard/cf-insert
1150-command: removable_storage_test sdio usb scsi
1151+command: removable_storage_test --memorycard sdio usb scsi
1152 _description:
1153 This test is automated and executes after the mediacard/cf-insert
1154 test is run. It tests reading and writing to the CF card.
1155@@ -255,7 +255,7 @@
1156 plugin: manual
1157 name: mediacard/cf-remove
1158 depends: mediacard/cf-storage
1159-command: removable_storage_watcher remove sdio usb scsi
1160+command: removable_storage_watcher --memorycard remove sdio usb scsi
1161 _description:
1162 PURPOSE:
1163 This test will check that the system correctly detects
1164@@ -270,7 +270,7 @@
1165 plugin: manual
1166 name: mediacard/cf-insert-after-suspend
1167 depends: suspend/suspend_advanced
1168-command: removable_storage_watcher insert sdio usb scsi
1169+command: removable_storage_watcher --memorycard insert sdio usb scsi
1170 _description:
1171 PURPOSE:
1172 This test will check that the systems media card reader can
1173@@ -286,7 +286,7 @@
1174 plugin: shell
1175 name: mediacard/cf-storage-after-suspend
1176 depends: mediacard/cf-insert-after-suspend
1177-command: removable_storage_test sdio usb scsi
1178+command: removable_storage_test --memorycard sdio usb scsi
1179 _description:
1180 This test is automated and executes after the mediacard/cf-insert-after-suspend test
1181 is run. It tests reading and writing to the CF card after the system has been suspended.
1182@@ -294,7 +294,7 @@
1183 plugin: manual
1184 name: mediacard/cf-remove-after-suspend
1185 depends: mediacard/cf-storage-after-suspend
1186-command: removable_storage_watcher remove sdio usb scsi
1187+command: removable_storage_watcher --memorycard remove sdio usb scsi
1188 _description:
1189 PURPOSE:
1190 This test will check that the system correctly detects the removal
1191
1192=== modified file 'scripts/removable_storage_test'
1193--- scripts/removable_storage_test 2012-08-20 18:13:17 +0000
1194+++ scripts/removable_storage_test 2012-10-09 17:53:24 +0000
1195@@ -1,17 +1,31 @@
1196 #!/usr/bin/env python3
1197
1198+import argparse
1199+import collections
1200 import dbus
1201+import hashlib
1202+import logging
1203+import os
1204+import subprocess
1205 import sys
1206-import random
1207-import os
1208 import tempfile
1209-import hashlib
1210-import argparse
1211-import subprocess
1212-import threading
1213 import time
1214
1215-from shutil import copy2
1216+from gi.repository import GUdev
1217+
1218+from checkbox.dbus import connect_to_system_bus
1219+from checkbox.dbus.udisks2 import UDISKS2_BLOCK_INTERFACE
1220+from checkbox.dbus.udisks2 import UDISKS2_DRIVE_INTERFACE
1221+from checkbox.dbus.udisks2 import UDISKS2_FILESYSTEM_INTERFACE
1222+from checkbox.dbus.udisks2 import UDisks2Model, UDisks2Observer
1223+from checkbox.dbus.udisks2 import is_udisks2_supported
1224+from checkbox.dbus.udisks2 import lookup_udev_device
1225+from checkbox.dbus.udisks2 import map_udisks1_connection_bus
1226+from checkbox.heuristics.udisks2 import is_memory_card
1227+from checkbox.parsers.udevadm import CARD_READER_RE, GENERIC_RE, FLASH_RE
1228+from checkbox.udev import get_interconnect_speed
1229+from checkbox.udev import get_udev_block_devices
1230+
1231
1232 class ActionTimer():
1233 '''Class to implement a simple timer'''
1234@@ -23,105 +37,268 @@
1235 self.stop = time.time()
1236 self.interval = self.stop - self.start
1237
1238-class DiskTest():
1239- ''' Class to contain various methods for testing USB disks '''
1240- def __init__(self):
1241- self.process = None
1242- self.cmd = None
1243- self.timeout = 3
1244- self.returnCode = None
1245- self.rem_disks = {} # mounted before the script running
1246- self.rem_disks_nm = {} # not mounted before the script running
1247
1248- def generate_test_data(self, size):
1249- '''Generate a random data file of a given size'''
1250- min = 100
1251- max = 1000000
1252+class RandomData():
1253+ '''Class to create data files'''
1254+ def __init__(self, size):
1255 self.tfile = tempfile.NamedTemporaryFile(delete=False)
1256+ self.path = ''
1257+ self.name = ''
1258+ self.path, self.name = os.path.split(self.tfile.name)
1259+ self._write_test_data_file(size)
1260+
1261+ def _generate_test_data(self):
1262+ seed = "104872948765827105728492766217823438120"
1263+ phrase = '''
1264+ Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam
1265+ nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat
1266+ volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation
1267+ ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.
1268+ Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse
1269+ molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero
1270+ eros et accumsan et iusto odio dignissim qui blandit praesent luptatum
1271+ zzril delenit augue duis dolore te feugait nulla facilisi.
1272+ '''
1273+ words = phrase.replace('\n', '').split()
1274+ word_deque = collections.deque(words)
1275+ seed_deque = collections.deque(seed)
1276+ while True:
1277+ yield ' '.join(list(word_deque))
1278+ word_deque.rotate(int(seed_deque[0]))
1279+ seed_deque.rotate(1)
1280+
1281+ def _write_test_data_file(self, size):
1282+ data = self._generate_test_data()
1283 while os.path.getsize(self.tfile.name) < size:
1284- self.tfile.write(str(random.randint(min, max)).encode('UTF-8'))
1285-
1286- def md5_hash_file(self, path):
1287- try:
1288- fh = open(path, 'r')
1289- except IOError:
1290- print("ERROR: unable to open file %s" % path, file=sys.stderr)
1291- return 1
1292- md5 = hashlib.md5()
1293- try:
1294+ self.tfile.write(next(data).encode('UTF-8'))
1295+ return self
1296+
1297+
1298+def md5_hash_file(path):
1299+ md5 = hashlib.md5()
1300+ try:
1301+ with open(path, 'rb') as stream:
1302 while True:
1303- data = fh.read(8192)
1304+ data = stream.read(8192)
1305 if not data:
1306 break
1307- md5.update(data.encode('utf-8'))
1308- finally:
1309- fh.close()
1310+ md5.update(data)
1311+ except IOError as exc:
1312+ logging.error("unable to checksum %s: %s", path, exc)
1313+ return None
1314+ else:
1315 return md5.hexdigest()
1316
1317- def copy_file(self, source, dest):
1318- try:
1319- copy2(source, dest)
1320- except IOError:
1321- print("ERROR: Unable to copy the file to %s" % dest,
1322- file=sys.stderr)
1323- return False
1324- else:
1325- return True
1326-
1327- def compare_hash(self, parent, child):
1328- if not parent == child:
1329- return False
1330- else:
1331- return True
1332+
1333+class DiskTest():
1334+ ''' Class to contain various methods for testing removable disks '''
1335+
1336+ def __init__(self, device, memorycard):
1337+ self.rem_disks = {} # mounted before the script running
1338+ self.rem_disks_nm = {} # not mounted before the script running
1339+ self.rem_disks_memory_cards = {}
1340+ self.rem_disks_memory_cards_nm = {}
1341+ self.rem_disks_speed = {}
1342+ self.data = ''
1343+ self.device = device
1344+ self.memorycard = memorycard
1345+ self._probe_disks()
1346+
1347+ def read_file(self, source):
1348+ with open(source, 'rb') as infile:
1349+ try:
1350+ self.data = infile.read()
1351+ except IOError as exc:
1352+ logging.error("Unable to read data from %s: %s", source, exc)
1353+ return False
1354+ else:
1355+ return True
1356+
1357+ def write_file(self, data, dest):
1358+ with open(dest, 'wb', 0) as outfile:
1359+ try:
1360+ outfile.write(self.data)
1361+ except IOError as exc:
1362+ logging.error("Unable to write data to %s: %s", dest, exc)
1363+ return False
1364+ else:
1365+ outfile.flush()
1366+ os.fsync(outfile.fileno())
1367+ return True
1368
1369 def clean_up(self, target):
1370 try:
1371 os.unlink(target)
1372- except:
1373- print("ERROR: Unable to remove tempfile %s" % target,
1374- file=sys.stderr)
1375-
1376- def get_disk_info(self, device):
1377- ''' Probes dbus to find any attached/mounted devices'''
1378- bus = dbus.SystemBus()
1379+ except OSError as exc:
1380+ logging.error("Unable to remove tempfile %s: %s", target, exc)
1381+
1382+ def _probe_disks(self):
1383+ """
1384+ Internal method used to probe for available disks
1385+
1386+ Indirectly sets:
1387+ self.rem_disks{,_nm,_memory_cards,_memory_cards_nm,_speed}
1388+ """
1389+ bus, loop = connect_to_system_bus()
1390+ if is_udisks2_supported(bus):
1391+ self._probe_disks_udisks2(bus)
1392+ else:
1393+ self._probe_disks_udisks1(bus)
1394+
1395+ def _probe_disks_udisks2(self, bus):
1396+ """
1397+ Internal method used to probe / discover available disks using udisks2
1398+ dbus interface using the provided dbus bus (presumably the system bus)
1399+ """
1400+ # We'll need udisks2 and udev to get the data we need
1401+ udisks2_observer = UDisks2Observer()
1402+ udisks2_model = UDisks2Model(udisks2_observer)
1403+ udisks2_observer.connect_to_bus(bus)
1404+ udev_client = GUdev.Client()
1405+ # Get a collection of all udev devices corresponding to block devices
1406+ udev_devices = get_udev_block_devices(udev_client)
1407+ # Get a collection of all udisks2 objects
1408+ udisks2_objects = udisks2_model.managed_objects
1409+ # Let's get a helper to simplify the loop below
1410+
1411+ def iter_filesystems_on_block_devices():
1412+ """
1413+ Generate a collection of UDisks2 object paths that
1414+ have both the filesystem and block device interfaces
1415+ """
1416+ for udisks2_object_path, interfaces in udisks2_objects.items():
1417+ if (UDISKS2_FILESYSTEM_INTERFACE in interfaces
1418+ and UDISKS2_BLOCK_INTERFACE in interfaces):
1419+ yield udisks2_object_path
1420+ # We need to know about all IO candidates,
1421+ # let's iterate over all the block devices reported by udisks2
1422+ for udisks2_object_path in iter_filesystems_on_block_devices():
1423+ # Get interfaces implemented by this object
1424+ udisks2_object = udisks2_objects[udisks2_object_path]
1425+ # Find the path of the udisks2 object that represents the drive
1426+ # this object is a part of
1427+ drive_object_path = (
1428+ udisks2_object[UDISKS2_BLOCK_INTERFACE]['Drive'])
1429+ # Lookup the drive object, if any. This can fail when
1430+ try:
1431+ drive_object = udisks2_objects[drive_object_path]
1432+ except KeyError:
1433+ logging.error(
1434+ "Unable to locate drive associated with %s",
1435+ udisks2_object_path)
1436+ continue
1437+ else:
1438+ drive_props = drive_object[UDISKS2_DRIVE_INTERFACE]
1439+ # Get the connection bus property from the drive interface of the
1440+ # drive object. This is required to filter out the devices we don't
1441+ # want to look at now.
1442+ connection_bus = drive_props["ConnectionBus"]
1443+ desired_connection_buses = set([
1444+ map_udisks1_connection_bus(device)
1445+ for device in self.device])
1446+ # Skip devices that are attached to undesired connection buses
1447+ if connection_bus not in desired_connection_buses:
1448+ continue
1449+ # Lookup the udev object that corresponds to this object
1450+ try:
1451+ udev_device = lookup_udev_device(udisks2_object, udev_devices)
1452+ except LookupError:
1453+ logging.error(
1454+ "Unable to locate udev object that corresponds to: %s",
1455+ udisks2_object_path)
1456+ continue
1457+ # Get the block device pathname,
1458+ # to avoid the confusion, this is something like /dev/sdbX
1459+ dev_file = udev_device.get_device_file()
1460+ # Get the list of mount points of this block device
1461+ mount_points = (
1462+ udisks2_object[UDISKS2_FILESYSTEM_INTERFACE]['MountPoints'])
1463+ # Get the speed of the interconnect that is associated with the
1464+ # block device we're looking at. This is purely informational but
1465+ # it is a part of the required API
1466+ interconnect_speed = get_interconnect_speed(udev_device)
1467+ if interconnect_speed:
1468+ self.rem_disks_speed[dev_file] = (
1469+ interconnect_speed * 10 ** 6)
1470+ else:
1471+ self.rem_disks_speed[dev_file] = None
1472+ # We need to skip-non memory cards if we look for memory cards and
1473+ # vice-versa so let's inspect the drive and use heuristics to
1474+ # detect memory cards (a memory card reader actually) now.
1475+ if self.memorycard != is_memory_card(drive_props['Vendor'],
1476+ drive_props['Model'],
1477+ drive_props['Media']):
1478+ continue
1479+ # The if/else test below simply distributes the mount_point to the
1480+ # appropriate variable, to keep the API requirements. It is
1481+ # confusing as _memory_cards is variable is somewhat dummy.
1482+ if mount_points:
1483+ # XXX: Arbitrarily pick the first of the mount points
1484+ mount_point = mount_points[0]
1485+ self.rem_disks_memory_cards[dev_file] = mount_point
1486+ self.rem_disks[dev_file] = mount_point
1487+ else:
1488+ self.rem_disks_memory_cards_nm[dev_file] = None
1489+ self.rem_disks_nm[dev_file] = None
1490+
1491+ def _probe_disks_udisks1(self, bus):
1492+ """
1493+ Internal method used to probe / discover available disks using udisks1
1494+ dbus interface using the provided dbus bus (presumably the system bus)
1495+ """
1496 ud_manager_obj = bus.get_object("org.freedesktop.UDisks",
1497 "/org/freedesktop/UDisks")
1498 ud_manager = dbus.Interface(ud_manager_obj, 'org.freedesktop.UDisks')
1499- self.rem_disks = {}
1500- self.rem_disks_nm = {}
1501 for dev in ud_manager.EnumerateDevices():
1502 device_obj = bus.get_object("org.freedesktop.UDisks", dev)
1503 device_props = dbus.Interface(device_obj, dbus.PROPERTIES_IFACE)
1504 udisks = 'org.freedesktop.UDisks.Device'
1505 if not device_props.Get(udisks, "DeviceIsDrive"):
1506- if device_props.Get(udisks,
1507- "DriveConnectionInterface") in device:
1508+ dev_bus = device_props.Get(udisks, "DriveConnectionInterface")
1509+ if dev_bus in self.device:
1510+ parent_model = parent_vendor = ''
1511+ if device_props.Get(udisks, "DeviceIsPartition"):
1512+ parent_obj = bus.get_object(
1513+ "org.freedesktop.UDisks",
1514+ device_props.Get(udisks, "PartitionSlave"))
1515+ parent_props = dbus.Interface(
1516+ parent_obj, dbus.PROPERTIES_IFACE)
1517+ parent_model = parent_props.Get(udisks, "DriveModel")
1518+ parent_vendor = parent_props.Get(udisks, "DriveVendor")
1519+ parent_media = parent_props.Get(udisks, "DriveMedia")
1520+ if self.memorycard:
1521+ if (dev_bus != 'sdio'
1522+ and not FLASH_RE.search(parent_media)
1523+ and not CARD_READER_RE.search(parent_model)
1524+ and not GENERIC_RE.search(parent_vendor)):
1525+ continue
1526+ else:
1527+ if (FLASH_RE.search(parent_media)
1528+ or CARD_READER_RE.search(parent_model)
1529+ or GENERIC_RE.search(parent_vendor)):
1530+ continue
1531 dev_file = str(device_props.Get(udisks, "DeviceFile"))
1532-
1533+ dev_speed = str(device_props.Get(udisks,
1534+ "DriveConnectionSpeed"))
1535+ self.rem_disks_speed[dev_file] = dev_speed
1536 if len(device_props.Get(udisks, "DeviceMountPaths")) > 0:
1537 devPath = str(device_props.Get(udisks,
1538 "DeviceMountPaths")[0])
1539 self.rem_disks[dev_file] = devPath
1540+ self.rem_disks_memory_cards[dev_file] = devPath
1541 else:
1542 self.rem_disks_nm[dev_file] = None
1543+ self.rem_disks_memory_cards_nm[dev_file] = None
1544
1545 def mount(self):
1546 passed_mount = {}
1547
1548- for key, dummy in self.rem_disks_nm.items():
1549- file = tempfile.mkdtemp(dir='/tmp')
1550-
1551- result = False
1552- try:
1553- result = self.make_thread(self._mount(key, file))
1554- except:
1555- pass
1556-
1557- # remove those devices fail at mounting
1558- if result:
1559- passed_mount[key] = file
1560+ for key in self.rem_disks_nm:
1561+ temp_dir = tempfile.mkdtemp()
1562+ if self._mount(key, temp_dir) != 0:
1563+ logging.error("can't mount %s", key)
1564 else:
1565- print("ERROR: can't mount %s" % (key), file=sys.stderr)
1566+ passed_mount[key] = temp_dir
1567
1568 if len(self.rem_disks_nm) == len(passed_mount):
1569 self.rem_disks_nm = passed_mount
1570@@ -131,72 +308,69 @@
1571 self.rem_disks_nm = passed_mount
1572 return count
1573
1574- def _mount(self, dev_file, tmp_dir):
1575- cmd = ['/bin/mount', dev_file, tmp_dir]
1576- self.process = subprocess.Popen(cmd)
1577- self.process.communicate()
1578+ def _mount(self, dev_file, mount_point):
1579+ return subprocess.call(['mount', dev_file, mount_point])
1580
1581 def umount(self):
1582 errors = 0
1583-
1584 for disk in self.rem_disks_nm:
1585- result = False
1586- try:
1587- result = self.make_thread(self._umount(disk))
1588- except:
1589+ if not self.rem_disks_nm[disk]:
1590+ continue
1591+ if self._umount(disk) != 0:
1592 errors += 1
1593- pass
1594-
1595- if not result:
1596- print("ERROR: can't umount %s on %s"
1597- % (disk, self.rem_disks_nm[disk]), file=sys.stderr)
1598-
1599+ logging.error("can't umount %s on %s",
1600+ disk, self.rem_disks_nm[disk])
1601 return errors
1602
1603- def _umount(self, dir):
1604+ def _umount(self, mount_point):
1605 # '-l': lazy umount, dealing problem of unable to umount the device.
1606- cmd = ['/bin/umount', '-l', dir]
1607- self.process = subprocess.Popen(cmd)
1608- self.process.communicate()
1609+ return subprocess.call(['umount', '-l', mount_point])
1610
1611 def clean_tmp_dir(self):
1612 for disk in self.rem_disks_nm:
1613+ if not self.rem_disks_nm[disk]:
1614+ continue
1615 if not os.path.ismount(self.rem_disks_nm[disk]):
1616 os.rmdir(self.rem_disks_nm[disk])
1617
1618- def make_thread(self, target):
1619- thread = threading.Thread(target=target)
1620- thread.start()
1621- thread.join(self.timeout)
1622-
1623- if thread.is_alive():
1624- self.process.terminate()
1625- thread.join()
1626-
1627- result = getattr(self.process, 'returncode', 1)
1628-
1629- if result == 0:
1630- return True
1631- else:
1632- return False
1633-
1634
1635 def main():
1636 parser = argparse.ArgumentParser()
1637 parser.add_argument('device',
1638- choices=['usb', 'firewire', 'sdio', 'scsi', 'ata_serial_esata'],
1639+ choices=['usb', 'firewire', 'sdio',
1640+ 'scsi', 'ata_serial_esata'],
1641 nargs='+',
1642 help=("The type of removable media "
1643- "(usb, firewire, sdio, scsi or ata_serial_esata) to test."))
1644+ "(usb, firewire, sdio, scsi or ata_serial_esata)"
1645+ "to test."))
1646 parser.add_argument('-l', '--list',
1647 action='store_true',
1648 default=False,
1649 help="List the removable devices and mounting status")
1650+ parser.add_argument('-m', '--min-speed',
1651+ action='store',
1652+ default=0,
1653+ type=int,
1654+ help="Minimum speed a device must support to be "
1655+ "considered eligible for being tested (bits/s)")
1656+ parser.add_argument('-p', '--pass-speed',
1657+ action='store',
1658+ default=0,
1659+ type=int,
1660+ help="Minimum average throughput from all eligible"
1661+ "devices for the test to pass (MB/s)")
1662+ parser.add_argument('-i', '--iterations',
1663+ action='store',
1664+ default='1',
1665+ type=int,
1666+ help=("The number of test cycles to run. One cycle is"
1667+ "comprised of generating --count data files of "
1668+ "--size bytes and writing them to each device."))
1669 parser.add_argument('-c', '--count',
1670 action='store',
1671 default='1',
1672 type=int,
1673- help='The number of times to run the test')
1674+ help='The number of random data files to generate')
1675 parser.add_argument('-s', '--size',
1676 action='store',
1677 type=int,
1678@@ -208,12 +382,14 @@
1679 default=False,
1680 help=("skip the removable devices "
1681 "which haven't been mounted before the test."))
1682+ parser.add_argument('--memorycard', action="store_true",
1683+ help=("Memory cards devices on bus other than sdio "
1684+ "require this parameter to identify "
1685+ "them as such"))
1686
1687 args = parser.parse_args()
1688
1689- test = DiskTest()
1690-
1691- test.get_disk_info(args.device)
1692+ test = DiskTest(args.device, args.memorycard)
1693
1694 errors = 0
1695 # If we do have removable drives attached and mounted
1696@@ -221,24 +397,38 @@
1697 if args.list: # Simply output a list of drives detected
1698 print('-' * 20)
1699 print("Removable devices currently mounted:")
1700- if len(test.rem_disks) > 0:
1701- for disk, mount_point in test.rem_disks.items():
1702- print("%s : %s" % (disk, mount_point))
1703- else:
1704- print("None")
1705-
1706- print("Removable devices currently not mounted:")
1707- if len(test.rem_disks_nm) > 0:
1708- for disk, dummy in test.rem_disks_nm.items():
1709- print(disk)
1710- else:
1711- print("None")
1712+ if args.memorycard:
1713+ if len(test.rem_disks_memory_cards) > 0:
1714+ for disk, mnt_point in test.rem_disks_memory_cards.items():
1715+ print("%s : %s" % (disk, mnt_point))
1716+ else:
1717+ print("None")
1718+
1719+ print("Removable devices currently not mounted:")
1720+ if len(test.rem_disks_memory_cards_nm) > 0:
1721+ for disk in test.rem_disks_memory_cards_nm:
1722+ print(disk)
1723+ else:
1724+ print("None")
1725+ else:
1726+ if len(test.rem_disks) > 0:
1727+ for disk, mnt_point in test.rem_disks.items():
1728+ print("%s : %s" % (disk, mnt_point))
1729+ else:
1730+ print("None")
1731+
1732+ print("Removable devices currently not mounted:")
1733+ if len(test.rem_disks_nm) > 0:
1734+ for disk in test.rem_disks_nm:
1735+ print(disk)
1736+ else:
1737+ print("None")
1738
1739 print('-' * 20)
1740
1741 return 0
1742
1743- else: # Create a file, copy to USB and compare hashes
1744+ else: # Create a file, copy to disk and compare hashes
1745 if args.skip_not_mount:
1746 disks_all = test.rem_disks
1747 else:
1748@@ -255,72 +445,132 @@
1749
1750 if len(disks_all) > 0:
1751 print("Found the following mounted %s partitions:"
1752- % args.device)
1753+ % ', '.join(args.device))
1754
1755 for disk, mount_point in disks_all.items():
1756- print(" %s : %s" % (disk, mount_point))
1757+ supported_speed = test.rem_disks_speed[disk]
1758+ print(" %s : %s : %s bits/s" %
1759+ (disk, mount_point, supported_speed),
1760+ end="")
1761+ if (args.min_speed
1762+ and int(args.min_speed) > int(supported_speed)):
1763+ print(" (Will not test it, speed is below %s bits/s)" %
1764+ args.min_speed, end="")
1765+
1766+ print("")
1767
1768 print('-' * 20)
1769
1770- write_times = []
1771+ disks_eligible = {disk: disks_all[disk] for disk in disks_all
1772+ if not args.min_speed or
1773+ int(test.rem_disks_speed[disk])
1774+ >= int(args.min_speed)}
1775+ write_sizes = []
1776+ test_files = {}
1777+ # Generate our data file(s)
1778+ for count in range(args.count):
1779+ test_files[count] = RandomData(args.size)
1780+ write_sizes.append(os.path.getsize(
1781+ test_files[count].tfile.name))
1782+ total_write_size = sum(write_sizes)
1783+
1784 try:
1785- for iteration in range(args.count):
1786- test.generate_test_data(args.size)
1787- parent_hash = test.md5_hash_file(test.tfile.name)
1788- for disk, mount_point in disks_all.items():
1789- with ActionTimer() as timer:
1790- if not test.copy_file(test.tfile.name, mount_point):
1791- print("ERROR: Failed to copy %s to %s" %
1792- (test.tfile.name, mount_point), file=sys.stderr)
1793+ for disk, mount_point in disks_eligible.items():
1794+ print("%s (Total Data Size / iteration: %0.4f MB):" %
1795+ (disk, (total_write_size / 1024 / 1024)))
1796+ iteration_write_size = (
1797+ total_write_size * args.iterations) / 1024 / 1024
1798+ iteration_write_times = []
1799+ for iteration in range(args.iterations):
1800+ target_file_list = []
1801+ write_times = []
1802+ for file_index in range(args.count):
1803+ parent_file = test_files[file_index].tfile.name
1804+ parent_hash = md5_hash_file(parent_file)
1805+ target_filename = (
1806+ test_files[file_index].name +
1807+ '.%s' % iteration)
1808+ target_path = mount_point
1809+ target_file = os.path.join(target_path,
1810+ target_filename)
1811+ target_file_list.append(target_file)
1812+ test.read_file(parent_file)
1813+ with ActionTimer() as timer:
1814+ if not test.write_file(test.data,
1815+ target_file):
1816+ logging.error(
1817+ "Failed to copy %s to %s",
1818+ parent_file, target_file)
1819+ errors += 1
1820+ continue
1821+ write_times.append(timer.interval)
1822+ child_hash = md5_hash_file(target_file)
1823+ if parent_hash != child_hash:
1824+ logging.warning(
1825+ "[Iteration %s] Parent and Child"
1826+ " copy hashes mismatch on %s!",
1827+ iteration, target_file)
1828+ logging.warning(
1829+ "\tParent hash: %s", parent_hash)
1830+ logging.warning(
1831+ "\tChild hash: %s", child_hash)
1832 errors += 1
1833- continue
1834- print("[Iteration %s] Copy took %0.3f secs." %
1835- (iteration, timer.interval))
1836- write_times.append(timer.interval)
1837-
1838- target = os.path.join(mount_point,
1839- os.path.basename(test.tfile.name))
1840- child_hash = test.md5_hash_file(target)
1841- if not test.compare_hash(parent_hash, child_hash):
1842- print("ERROR: [Iteration %s] Parent and Child"
1843- "copy hashes mismatch on %s!" %
1844- (iteration,mount_point), file=sys.stderr)
1845- print("\tParent hash: %s" % parent_hash,
1846- file=sys.stderr)
1847- print("\tChild hash: %s" % child_hash,
1848- file=sys.stderr)
1849- errors += 1
1850- test.clean_up(target)
1851- test.clean_up(test.tfile.name)
1852+ for file in target_file_list:
1853+ test.clean_up(file)
1854+ total_write_time = sum(write_times)
1855+ avg_write_time = total_write_time / args.count
1856+ avg_write_speed = ((
1857+ total_write_size / total_write_time)
1858+ / 1024 / 1024)
1859+ iteration_write_times.append(total_write_time)
1860+ print("\t[Iteration %s] Average Speed: %0.4f"
1861+ % (iteration, avg_write_speed))
1862+ for iteration in range(args.iterations):
1863+ iteration_write_time = sum(iteration_write_times)
1864+ print("\tSummary:")
1865+ print("\t\tTotal Data Written: %0.4f MB"
1866+ % iteration_write_size)
1867+ print("\t\tTotal Time to write: %0.4f secs"
1868+ % iteration_write_time)
1869+ print("\t\tAverage Write Time: %0.4f secs" %
1870+ (iteration_write_time / args.iterations))
1871+ print("\t\tAverage Write Speed: %0.4f MB/s" %
1872+ (iteration_write_size / iteration_write_time))
1873 finally:
1874+ for key in range(args.count):
1875+ test.clean_up(test_files[key].tfile.name)
1876 if (len(test.rem_disks_nm) > 0):
1877 if test.umount() != 0:
1878 errors += 1
1879 test.clean_tmp_dir()
1880
1881 if errors > 0:
1882- print("ERROR: Completed %s test iterations, "
1883- "but there were errors" % args.count, file=sys.stderr)
1884- return 1
1885+ logging.warning(
1886+ "Completed %s test iterations, but there were"
1887+ " errors", args.count)
1888+ return 1
1889+ elif len(disks_eligible) == 0:
1890+ logging.error(
1891+ "No %s disks with speed higher than %s bits/s",
1892+ args.device, args.min_speed)
1893+ return 1
1894+
1895 else:
1896- print("Successfully completed %s %s file transfer "
1897- "test iterations"
1898- % (args.count, args.device))
1899- total_write_time = sum(write_times)
1900- avg_write_time = total_write_time / args.count
1901- avg_write_speed = (args.size / avg_write_time) / 1024
1902- print("Total time writing data: %0.4f" % total_write_time)
1903- print("Average time per write: %0.4f" % avg_write_time)
1904- print("Average write speed: %0.4f KB/s" %
1905- avg_write_speed)
1906- return 0
1907+ #Pass is not assured!
1908+ if (not args.pass_speed or
1909+ avg_write_speed >= args.pass_speed):
1910+ return 0
1911+ else:
1912+ print("FAIL: Average speed was lower than desired "
1913+ "pass speed of %s MB/s" % args.pass_speed)
1914+ return 1
1915 else:
1916- print("ERROR: No device being mounted successfully for testing, "
1917- "aborting", file=sys.stderr)
1918+ logging.error("No device being mounted successfully "
1919+ "for testing, aborting")
1920 return 1
1921
1922 else: # If we don't have removable drives attached and mounted
1923- print("ERROR: No removable drives were detected, aborting", file=sys.stderr)
1924+ logging.error("No removable drives were detected, aborting")
1925 return 1
1926
1927 if __name__ == '__main__':
1928
1929=== modified file 'scripts/removable_storage_watcher'
1930--- scripts/removable_storage_watcher 2012-08-20 18:13:17 +0000
1931+++ scripts/removable_storage_watcher 2012-10-09 17:53:24 +0000
1932@@ -1,67 +1,217 @@
1933 #!/usr/bin/env python3
1934
1935+import argparse
1936+import collections
1937+import copy
1938+import dbus
1939+import logging
1940 import sys
1941-import dbus
1942-import argparse
1943-
1944-from gi.repository import GObject
1945-from dbus.mainloop.glib import DBusGMainLoop
1946-
1947-
1948-class StorageDeviceListener:
1949-
1950- def __init__(self, action, devices):
1951+
1952+from gi.repository import GObject, GUdev
1953+
1954+from checkbox.dbus import connect_to_system_bus
1955+from checkbox.dbus.udisks2 import UDisks2Model, UDisks2Observer
1956+from checkbox.dbus.udisks2 import is_udisks2_supported
1957+from checkbox.dbus.udisks2 import lookup_udev_device
1958+from checkbox.dbus.udisks2 import map_udisks1_connection_bus
1959+from checkbox.heuristics.udisks2 import is_memory_card
1960+from checkbox.parsers.udevadm import CARD_READER_RE, GENERIC_RE, FLASH_RE
1961+from checkbox.udev import get_interconnect_speed, get_udev_block_devices
1962+
1963+# Record representing properties of a UDisks1 Drive object needed by the
1964+# UDisks1 version of the watcher implementation
1965+UDisks1DriveProperties = collections.namedtuple(
1966+ 'UDisks1DriveProperties', 'file bus speed model vendor media')
1967+
1968+# Delta record that encapsulates difference:
1969+# delta_dir -- directon of the difference, either DELTA_DIR_PLUS or
1970+# DELTA_DIR_MINUS
1971+# value -- the actual value being removed or added, either InterfaceDelta or
1972+# PropertyDelta instance, see below
1973+DeltaRecord = collections.namedtuple("DeltaRecord", "delta_dir value")
1974+
1975+# Delta value for representing interface changes
1976+InterfaceDelta = collections.namedtuple(
1977+ "InterfaceDelta",
1978+ "delta_type object_path iface_name")
1979+
1980+# Delta value for representing property changes
1981+PropertyDelta = collections.namedtuple(
1982+ "PropertyDelta",
1983+ "delta_type object_path iface_name prop_name prop_value")
1984+
1985+# Tokens that encode additions and removals
1986+DELTA_DIR_PLUS = '+'
1987+DELTA_DIR_MINUS = '-'
1988+
1989+# Tokens that encode interface and property deltas
1990+DELTA_TYPE_IFACE = 'i'
1991+DELTA_TYPE_PROP = 'p'
1992+
1993+
1994+def format_bytes(size):
1995+ """
1996+ Format size to be easily read by humans
1997+
1998+ The result is disk-size compatible (using multiples of 10
1999+ rather than 2) string like "4.5GB"
2000+ """
2001+ for index, prefix in enumerate(" KMGTPEZY", 0):
2002+ factor = 10 ** (index * 3)
2003+ if size // factor <= 1000:
2004+ break
2005+ return "{}{}B".format(size // factor, prefix.strip())
2006+
2007+
2008+class UDisks1StorageDeviceListener:
2009+
2010+ def __init__(self, system_bus, loop, action, devices, minimum_speed,
2011+ memorycard):
2012 self._action = action
2013 self._devices = devices
2014- self._bus = dbus.SystemBus(mainloop=DBusGMainLoop())
2015- self._loop = GObject.MainLoop()
2016+ self._minimum_speed = minimum_speed
2017+ self._memorycard = memorycard
2018+ self._bus = system_bus
2019+ self._loop = loop
2020 self._error = False
2021+ self._change_cache = []
2022
2023 def check(self, timeout):
2024 udisks = 'org.freedesktop.UDisks'
2025 if self._action == 'insert':
2026+ signal = 'DeviceAdded'
2027+ logging.debug("Adding signal listener for %s.%s", udisks, signal)
2028 self._bus.add_signal_receiver(self.add_detected,
2029- signal_name='DeviceAdded',
2030+ signal_name=signal,
2031 dbus_interface=udisks)
2032 elif self._action == 'remove':
2033+ signal = 'DeviceRemoved'
2034+ logging.debug("Adding signal listener for %s.%s", udisks, signal)
2035 self._bus.add_signal_receiver(self.remove_detected,
2036- signal_name='DeviceRemoved',
2037+ signal_name=signal,
2038 dbus_interface=udisks)
2039
2040+ self._starting_devices = self.get_existing_devices()
2041+ logging.debug("Starting with the following devices: %r",
2042+ self._starting_devices)
2043+
2044 def timeout_callback():
2045 print("%s seconds have expired "
2046 "waiting for the device to be inserted." % timeout)
2047 self._error = True
2048 self._loop.quit()
2049
2050+ logging.debug("Adding timeout listener, timeout=%r", timeout)
2051 GObject.timeout_add_seconds(timeout, timeout_callback)
2052+ logging.debug("Starting event loop...")
2053 self._loop.run()
2054
2055 return self._error
2056
2057+ def verify_device_change(self, changed_devices, message=""):
2058+ logging.debug("Verifying device change: %s", changed_devices)
2059+ # Filter the applicable bus types, as provided on the command line
2060+ # (values of self._devices can be 'usb', 'firewire', etc)
2061+ desired_bus_devices = [
2062+ device
2063+ for device in changed_devices
2064+ if device.bus in self._devices]
2065+ logging.debug("Desired bus devices: %s", desired_bus_devices)
2066+ for dev in desired_bus_devices:
2067+ if self._memorycard:
2068+ if (dev.bus != 'sdio'
2069+ and not FLASH_RE.search(dev.media)
2070+ and not CARD_READER_RE.search(dev.model)
2071+ and not GENERIC_RE.search(dev.vendor)):
2072+ logging.debug("The device does not seem to be a memory"
2073+ " card (bus: %r, model: %r), skipping",
2074+ dev.bus, dev.model)
2075+ return
2076+ print(message % {'bus': 'memory card', 'file': dev.file})
2077+ else:
2078+ if (FLASH_RE.search(dev.media)
2079+ or CARD_READER_RE.search(dev.model)
2080+ or GENERIC_RE.search(dev.vendor)):
2081+ logging.debug("The device seems to be a memory"
2082+ " card (bus: %r (model: %r), skipping",
2083+ dev.bus, dev.model)
2084+ return
2085+ print(message % {'bus': dev.bus, 'file': dev.file})
2086+ if self._minimum_speed:
2087+ if dev.speed >= self._minimum_speed:
2088+ print("with speed of %(speed)s bits/s "
2089+ "higher than %(min_speed)s bits/s" %
2090+ {'speed': dev.speed,
2091+ 'min_speed': self._minimum_speed})
2092+ else:
2093+ print("ERROR: speed of %(speed)s bits/s lower "
2094+ "than %(min_speed)s bits/s" %
2095+ {'speed': dev.speed,
2096+ 'min_speed': self._minimum_speed})
2097+ self._error = True
2098+ logging.debug("Device matches requirements, exiting event loop")
2099+ self._loop.quit()
2100+
2101 def job_change_detected(self, devices, job_in_progress, job_id,
2102 job_num_tasks, job_cur_task_id,
2103 job_cur_task_percentage):
2104- if job_id == "FilesystemMount" and self.is_device_inserted():
2105- print("Expected device %s inserted" % self._device)
2106- self._loop.quit()
2107+ logging.debug("UDisks1 reports a job change has been detected:"
2108+ " devices: %s, job_in_progress: %s, job_id: %s,"
2109+ " job_num_tasks: %s, job_cur_task_id: %s,"
2110+ " job_cur_task_percentage: %s",
2111+ devices, job_in_progress, job_id, job_num_tasks,
2112+ job_cur_task_id, job_cur_task_percentage)
2113+ if job_id == "FilesystemMount":
2114+ if devices in self._change_cache:
2115+ logging.debug("Ignoring filesystem mount,"
2116+ " the device is present in change cache")
2117+ return
2118+ logging.debug("Adding devices to change cache: %r", devices)
2119+ self._change_cache.append(devices)
2120+ logging.debug("Starting devices were: %s", self._starting_devices)
2121+ current_devices = self.get_existing_devices()
2122+ logging.debug("Current devices are: %s", current_devices)
2123+ inserted_devices = list(set(current_devices) -
2124+ set(self._starting_devices))
2125+ logging.debug("Computed inserted devices: %s", inserted_devices)
2126+ if self._memorycard:
2127+ message = "Expected memory card device %(file)s inserted"
2128+ else:
2129+ message = "Expected %(bus)s device %(file)s inserted"
2130+ self.verify_device_change(inserted_devices,
2131+ message=message)
2132
2133 def add_detected(self, added_path):
2134+ logging.debug("UDisks1 reports device has been added: %s", added_path)
2135+ logging.debug("Resetting change_cache to []")
2136+ self._change_cache = []
2137+ signal_name = 'DeviceJobChanged'
2138+ dbus_interface = 'org.freedesktop.UDisks'
2139+ logging.debug("Adding signal listener for %s.%s",
2140+ dbus_interface, signal_name)
2141 self._bus.add_signal_receiver(self.job_change_detected,
2142- signal_name='DeviceJobChanged',
2143- dbus_interface='org.freedesktop.UDisks')
2144+ signal_name=signal_name,
2145+ dbus_interface=dbus_interface)
2146
2147 def remove_detected(self, removed_path):
2148- if not self.is_device_inserted():
2149- print("Removable storage device has been removed")
2150- #TODO: figure out a way to get the DriveConnectionInterface of the
2151- #device that was just removed.
2152- self._loop.quit()
2153-
2154- def is_device_inserted(self):
2155+ logging.debug("UDisks1 reports device has been removed: %s",
2156+ removed_path)
2157+
2158+ logging.debug("Starting devices were: %s", self._starting_devices)
2159+ current_devices = self.get_existing_devices()
2160+ logging.debug("Current devices are: %s", current_devices)
2161+ removed_devices = list(set(self._starting_devices) -
2162+ set(current_devices))
2163+ logging.debug("Computed removed devices: %s", removed_devices)
2164+ self.verify_device_change(removed_devices,
2165+ message="Removable %(bus)s device %(file)s has been removed")
2166+
2167+ def get_existing_devices(self):
2168+ logging.debug("Getting existing devices from UDisks1")
2169 ud_manager_obj = self._bus.get_object("org.freedesktop.UDisks",
2170 "/org/freedesktop/UDisks")
2171 ud_manager = dbus.Interface(ud_manager_obj, 'org.freedesktop.UDisks')
2172+ existing_devices = []
2173 for dev in ud_manager.EnumerateDevices():
2174 try:
2175 device_obj = self._bus.get_object("org.freedesktop.UDisks",
2176@@ -69,30 +219,666 @@
2177 device_props = dbus.Interface(device_obj,
2178 dbus.PROPERTIES_IFACE)
2179 udisks = 'org.freedesktop.UDisks.Device'
2180- self._device = device_props.Get(udisks,
2181- "DriveConnectionInterface")
2182+ _device_file = device_props.Get(udisks,
2183+ "DeviceFile")
2184+ _bus = device_props.Get(udisks,
2185+ "DriveConnectionInterface")
2186+ _speed = device_props.Get(udisks,
2187+ "DriveConnectionSpeed")
2188+ _parent_model = ''
2189+ _parent_media = ''
2190+ _parent_vendor = ''
2191+
2192+ if device_props.Get(udisks, "DeviceIsPartition"):
2193+ parent_obj = self._bus.get_object(
2194+ "org.freedesktop.UDisks",
2195+ device_props.Get(udisks, "PartitionSlave"))
2196+ parent_props = dbus.Interface(
2197+ parent_obj, dbus.PROPERTIES_IFACE)
2198+ _parent_model = parent_props.Get(udisks, "DriveModel")
2199+ _parent_vendor = parent_props.Get(udisks, "DriveVendor")
2200+ _parent_media = parent_props.Get(udisks, "DriveMedia")
2201+
2202 if not device_props.Get(udisks, "DeviceIsDrive"):
2203- if self._device in self._devices:
2204- return True
2205+ device = UDisks1DriveProperties(
2206+ file=str(_device_file),
2207+ bus=str(_bus),
2208+ speed=int(_speed),
2209+ model=str(_parent_model),
2210+ vendor=str(_parent_vendor),
2211+ media=str(_parent_media))
2212+ existing_devices.append(device)
2213+
2214 except dbus.DBusException:
2215 pass
2216
2217- return False
2218+ return existing_devices
2219+
2220+
2221+def udisks2_objects_delta(old, new):
2222+ """
2223+ Compute the delta between two snapshots of udisks2 objects
2224+
2225+ The objects are encoded as {s:{s:{s:v}}} where the first dictionary maps
2226+ from DBus object path to a dictionary that maps from interface name to a
2227+ dictionary that finally maps from property name to property value.
2228+
2229+ The result is a generator of DeltaRecord objects that encodes the changes:
2230+ * the 'delta_dir' is either DELTA_DIR_PLUS or DELTA_DIR_MINUS
2231+ * the 'value' is a tuple that differs for interfaces and properties.
2232+ Interfaces use the format (DELTA_TYPE_IFACE, object_path, iface_name)
2233+ while properties use the format (DELTA_TYPE_PROP, object_path,
2234+ iface_name, prop_name, prop_value)
2235+
2236+ Interfaces are never "changed", they are only added or removed. Properties
2237+ can be changed and this is encoded as removal followed by an addition where
2238+ both differ only by the 'delta_dir' and the last element of the 'value'
2239+ tuple.
2240+ """
2241+ # Traverse all objects, old or new
2242+ all_object_paths = set()
2243+ all_object_paths |= old.keys()
2244+ all_object_paths |= new.keys()
2245+ for object_path in sorted(all_object_paths):
2246+ old_object = old.get(object_path, {})
2247+ new_object = new.get(object_path, {})
2248+ # Traverse all interfaces of each object, old or new
2249+ all_iface_names = set()
2250+ all_iface_names |= old_object.keys()
2251+ all_iface_names |= new_object.keys()
2252+ for iface_name in sorted(all_iface_names):
2253+ if iface_name not in old_object and iface_name in new_object:
2254+ # Report each ADDED interface
2255+ assert iface_name in new_object
2256+ delta_value = InterfaceDelta(
2257+ DELTA_TYPE_IFACE, object_path, iface_name)
2258+ yield DeltaRecord(DELTA_DIR_PLUS, delta_value)
2259+ # Report all properties ADDED on that interface
2260+ for prop_name, prop_value in new_object[iface_name].items():
2261+ delta_value = PropertyDelta(DELTA_TYPE_PROP, object_path,
2262+ iface_name, prop_name,
2263+ prop_value)
2264+ yield DeltaRecord(DELTA_DIR_PLUS, delta_value)
2265+ elif iface_name not in new_object and iface_name in old_object:
2266+ # Report each REMOVED interface
2267+ assert iface_name in old_object
2268+ delta_value = InterfaceDelta(
2269+ DELTA_TYPE_IFACE, object_path, iface_name)
2270+ yield DeltaRecord(DELTA_DIR_MINUS, delta_value)
2271+ # Report all properties REMOVED on that interface
2272+ for prop_name, prop_value in old_object[iface_name].items():
2273+ delta_value = PropertyDelta(DELTA_TYPE_PROP, object_path,
2274+ iface_name, prop_name,
2275+ prop_value)
2276+ yield DeltaRecord(DELTA_DIR_MINUS, delta_value)
2277+ else:
2278+ # Analyze properties of each interface that existed both in old
2279+ # and new object trees.
2280+ assert iface_name in new_object
2281+ assert iface_name in old_object
2282+ old_props = old_object[iface_name]
2283+ new_props = new_object[iface_name]
2284+ all_prop_names = set()
2285+ all_prop_names |= old_props.keys()
2286+ all_prop_names |= new_props.keys()
2287+ # Traverse all properties, old or new
2288+ for prop_name in sorted(all_prop_names):
2289+ if prop_name not in old_props and prop_name in new_props:
2290+ # Report each ADDED property
2291+ delta_value = PropertyDelta(
2292+ DELTA_TYPE_PROP, object_path, iface_name,
2293+ prop_name, new_props[prop_name])
2294+ yield DeltaRecord(DELTA_DIR_PLUS, delta_value)
2295+ elif prop_name not in new_props and prop_name in old_props:
2296+ # Report each REMOVED property
2297+ delta_value = PropertyDelta(
2298+ DELTA_TYPE_PROP, object_path, iface_name,
2299+ prop_name, old_props[prop_name])
2300+ yield DeltaRecord(DELTA_DIR_MINUS, delta_value)
2301+ else:
2302+ old_value = old_props[prop_name]
2303+ new_value = new_props[prop_name]
2304+ if old_value != new_value:
2305+ # Report each changed property
2306+ yield DeltaRecord(DELTA_DIR_MINUS, PropertyDelta(
2307+ DELTA_TYPE_PROP, object_path, iface_name,
2308+ prop_name, old_value))
2309+ yield DeltaRecord(DELTA_DIR_PLUS, PropertyDelta(
2310+ DELTA_TYPE_PROP, object_path, iface_name,
2311+ prop_name, new_value))
2312+
2313+
2314+class UDisks2StorageDeviceListener:
2315+ """
2316+ Implementation of the storage device listener concept for UDisks2 backend.
2317+ Loosely modeled on the UDisks-based implementation above.
2318+
2319+ Implementation details
2320+ ^^^^^^^^^^^^^^^^^^^^^^
2321+
2322+ The class, once configured reacts to asynchronous events from the event
2323+ loop. Those are either DBus signals or GLib timeout.
2324+
2325+ The timeout, if reached, terminates the test and fails with an appropriate
2326+ end-user message. The user is expected to manipulate storage devices while
2327+ the test is running.
2328+
2329+ DBus signals (that correspond to UDisks2 DBus signals) cause callbacks into
2330+ this code. Each time a signal is reported "delta" is computed and verified
2331+ to determine if there was a successful match. The delta contains a list or
2332+ DeltaRecord objects that encode difference (either addition or removal) and
2333+ the value of the difference (interface name or interface property value).
2334+ This delta is computed by udisks2_objects_delta(). The delta is then passed
2335+ to _validate_delta() which has a chance to end the test but also prints
2336+ diagnostic messages in verbose mode. This is very useful for understanding
2337+ what the test actually sees occurring.
2338+
2339+ Insertion/removal detection strategy
2340+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2341+
2342+ Compared to initial state, the following changes objects need to be
2343+ detected
2344+
2345+ * At least one UDisks2 object with the following _all_ interfaces:
2346+ * UDisks2.Partition
2347+ (because we want a partitioned device)
2348+ * UDisks2.Block
2349+ (because we want that device to have a block device that users can
2350+ format)
2351+ - having IdUsage == 'filesystem'
2352+ (because it should not be a piece of raid or lvm)
2353+ - having Size > 0
2354+ (because it should not be and empty removable storage reader)
2355+ * UDisks2.Filesystem
2356+ (because we want to ensure that a filesystem gets mounted)
2357+ - having MountPoints != []
2358+ - as a special exception this rule is REMOVED from eSATA and SATA
2359+ devices as they are not automatically mounted anymore.
2360+
2361+ This object must be traceable to an UDisks.Drive object:
2362+ (because we need the medium to be inserted somewhere)
2363+ - having ConnectionBus in (desired_connection_buses)
2364+ - as a special exception this rule is weakened for eSATA because
2365+ for such devices the ConnectionBus property is empty.
2366+ """
2367+
2368+ # Name of the DBus interface exposed UDisks2 for various drives
2369+ UDISKS2_DRIVE_INTERFACE = "org.freedesktop.UDisks2.Drive"
2370+
2371+ # Name of the DBus property provided by the "Drive" interface above
2372+ UDISKS2_DRIVE_PROPERTY_CONNECTION_BUS = "ConnectionBus"
2373+
2374+ def __init__(self, system_bus, loop, action, devices, minimum_speed,
2375+ memorycard):
2376+ # Store the desired minimum speed of the device in Mbit/s. The argument
2377+ # is passed as the number of bits per second so let's fix that.
2378+ self._desired_minimum_speed = minimum_speed / 10 ** 6
2379+ # Compute the allowed UDisks2.Drive.ConnectionBus value based on the
2380+ # legacy arguments passed from the command line.
2381+ self._desired_connection_buses = set([
2382+ map_udisks1_connection_bus(device) for device in devices])
2383+ # Check if we are explicitly looking for memory cards
2384+ self._desired_memory_card = memorycard
2385+ # Store the desired "delta" direction depending on
2386+ # whether we test for insertion or removal
2387+ if action == "insert":
2388+ self._desired_delta_dir = DELTA_DIR_PLUS
2389+ elif action == "remove":
2390+ self._desired_delta_dir = DELTA_DIR_MINUS
2391+ else:
2392+ raise ValueError("Unsupported action: {}".format(action))
2393+ # Store DBus bus object as we need to pass it to UDisks2 observer
2394+ self._bus = system_bus
2395+ # Store event loop object
2396+ self._loop = loop
2397+ # Setup UDisks2Observer class to track changes published by UDisks2
2398+ self._udisks2_observer = UDisks2Observer()
2399+ # Set the initial value of reference_objects.
2400+ # The actual value is only set once in check()
2401+ self._reference_objects = None
2402+ # As above, just initializing in init for sake of consistency
2403+ self._is_reference = None
2404+ # Setup UDisks2Model to know what the current state is. This is needed
2405+ # when remove events are reported as they don't carry enough state for
2406+ # the program to work correctly. Since UDisks2Model only applies the
2407+ # changes _after_ processing the signals from UDisks2Observer we can
2408+ # reliably check all of the properties of the removed object / device.
2409+ self._udisks2_model = UDisks2Model(self._udisks2_observer)
2410+ # Whenever anything changes call our local change handler
2411+ # This handler always computes the full delta (versus the
2412+ # reference state) and decides if we have a match or not
2413+ self._udisks2_model.on_change.connect(self._on_change)
2414+ # We may need an udev context for checking the speed of USB devices
2415+ self._udev_client = GUdev.Client()
2416+ # A snapshot of udev devices, set in check()
2417+ self._reference_udev_devices = None
2418+ # Assume the test passes, this is changed when timeout expires or when
2419+ # an incorrect device gets inserted.
2420+ self._error = False
2421+
2422+ def _dump_reference_udisks_objects(self):
2423+ logging.debug("Reference UDisks2 objects:")
2424+ for udisks2_object in self._reference_objects:
2425+ logging.debug(" - %s", udisks2_object)
2426+
2427+ def _dump_reference_udev_devices(self):
2428+ logging.debug("Reference udev devices:")
2429+ for udev_device in self._reference_udev_devices:
2430+ interconnect_speed = get_interconnect_speed(udev_device)
2431+ if interconnect_speed:
2432+ logging.debug(" - %s (USB %dMBit/s)",
2433+ udev_device.get_device_file(),
2434+ interconnect_speed)
2435+ else:
2436+ logging.debug(" - %s", udev_device.get_device_file())
2437+
2438+ def check(self, timeout):
2439+ """
2440+ Run the configured test and return the result
2441+
2442+ The result is False if the test has failed. The timeout, when
2443+ non-zero, will make the test fail after the specified seconds have
2444+ elapsed without conclusive result.
2445+ """
2446+ # Setup a timeout if requested
2447+ if timeout > 0:
2448+ GObject.timeout_add_seconds(timeout, self._on_timeout_expired)
2449+ # Connect the observer to the bus. This will start giving us events
2450+ # (actually when the loop starts later below)
2451+ self._udisks2_observer.connect_to_bus(self._bus)
2452+ # Get the reference snapshot of available devices
2453+ self._reference_objects = copy.deepcopy(self._current_objects)
2454+ self._dump_reference_udisks_objects()
2455+ # Mark the current _reference_objects as ... reference, this is sadly
2456+ # needed by _summarize_changes() as it sees the snapshot _after_ a
2457+ # change has occurred and cannot determine if the slope of the 'edge'
2458+ # of the change. It is purely needed for UI in verbose mode
2459+ self._is_reference = True
2460+ # A collection of objects that we gladly ignore because we already
2461+ # reported on them being somehow inappropriate
2462+ self._ignored_objects = set()
2463+ # Get the reference snapshot of available udev devices
2464+ self._reference_udev_devices = get_udev_block_devices(
2465+ self._udev_client)
2466+ self._dump_reference_udev_devices()
2467+ # Start the loop and wait. The loop will exit either when:
2468+ # 1) A proper device has been detected (either insertion or removal)
2469+ # 2) A timeout (optional) has expired
2470+ self._loop.run()
2471+ # Return the outcome of the test
2472+ return self._error
2473+
2474+ def _on_timeout_expired(self):
2475+ """
2476+ Internal function called when the timer expires.
2477+
2478+ Basically it's just here to tell the user the test failed or that the
2479+ user was unable to alter the device during the allowed time.
2480+ """
2481+ print("You have failed to perform the required manipulation in time")
2482+ # Fail the test when the timeout was reached
2483+ self._error = True
2484+ # Stop the loop now
2485+ self._loop.quit()
2486+
2487+ def _on_change(self):
2488+ """
2489+ Internal method called by UDisks2Model whenever a change had occurred
2490+ """
2491+ # Compute the changes that had occurred since the reference point
2492+ delta_records = list(self._get_delta_records())
2493+ # Display a summary of changes when we are done
2494+ self._summarize_changes(delta_records)
2495+ # If the changes are what we wanted stop the loop
2496+ matching_devices = self._get_matching_devices(delta_records)
2497+ if matching_devices:
2498+ print("Expected device manipulation complete: {}".format(
2499+ ', '.join(matching_devices)))
2500+ # And call it a day
2501+ self._loop.quit()
2502+
2503+ def _get_matching_devices(self, delta_records):
2504+ """
2505+ Internal method called that checks if the delta records match the type
2506+ of device manipulation we were expecting. Only called from _on_change()
2507+
2508+ Returns a set of paths of block devices that matched
2509+ """
2510+ # Results
2511+ results = set()
2512+ # Group changes by DBus object path
2513+ grouped_records = collections.defaultdict(list)
2514+ for record in delta_records:
2515+ grouped_records[record.value.object_path].append(record)
2516+ # Create another snapshot od udev devices so that we don't do it over
2517+ # and over in the loop below (besides, if we did that then results
2518+ # could differ each time).
2519+ current_udev_devices = get_udev_block_devices(self._udev_client)
2520+ # Iterate over all UDisks2 objects and their delta records
2521+ for object_path, records_for_object in grouped_records.items():
2522+ # Skip objects we already ignored and complained about before
2523+ if object_path in self._ignored_objects:
2524+ continue
2525+ needs = set(('block-fs', 'partition', 'non-empty', 'mounted'))
2526+ # As a special exception when the ConnectionBus is allowed to be
2527+ # empty, as is the case with eSATA devices, do not require the
2528+ # filesystem to be mounted as gvfs may choose not to mount it
2529+ # automatically.
2530+ if "" in self._desired_connection_buses:
2531+ needs.remove('mounted')
2532+ found = set()
2533+ drive_object_path = None
2534+ object_block_device = None
2535+ for record in records_for_object:
2536+ # Skip changes opposite to the ones we need
2537+ if record.delta_dir != self._desired_delta_dir:
2538+ continue
2539+ # Detect block devices designated for filesystems
2540+ if (record.value.iface_name ==
2541+ "org.freedesktop.UDisks2.Block"
2542+ and record.value.delta_type == DELTA_TYPE_PROP
2543+ and record.value.prop_name == "IdUsage"
2544+ and record.value.prop_value == "filesystem"):
2545+ found.add('block-fs')
2546+ # Memorize the block device path
2547+ elif (record.value.iface_name ==
2548+ "org.freedesktop.UDisks2.Block"
2549+ and record.value.delta_type == DELTA_TYPE_PROP
2550+ and record.value.prop_name == "PreferredDevice"):
2551+ object_block_device = record.value.prop_value
2552+ # Ensure the device is a partition
2553+ elif (record.value.iface_name ==
2554+ "org.freedesktop.UDisks2.Partition"
2555+ and record.value.delta_type == DELTA_TYPE_IFACE):
2556+ found.add('partition')
2557+ # Ensure the device is not empty
2558+ elif (record.value.iface_name ==
2559+ "org.freedesktop.UDisks2.Block"
2560+ and record.value.delta_type == DELTA_TYPE_PROP
2561+ and record.value.prop_name == "Size"
2562+ and record.value.prop_value > 0):
2563+ found.add('non-empty')
2564+ # Ensure the filesystem is mounted
2565+ elif (record.value.iface_name ==
2566+ "org.freedesktop.UDisks2.Filesystem"
2567+ and record.value.delta_type == DELTA_TYPE_PROP
2568+ and record.value.prop_name == "MountPoints"
2569+ and record.value.prop_value != []):
2570+ found.add('mounted')
2571+ # Finally memorize the drive the block device belongs to
2572+ elif (record.value.iface_name ==
2573+ "org.freedesktop.UDisks2.Block"
2574+ and record.value.delta_type == DELTA_TYPE_PROP
2575+ and record.value.prop_name == "Drive"):
2576+ drive_object_path = record.value.prop_value
2577+ logging.debug("Finished analyzing %s, found: %s,"
2578+ " drive_object_path: %s", object_path, found,
2579+ drive_object_path)
2580+ if needs != found or drive_object_path is None:
2581+ continue
2582+ # We've found our candidate, let's look at the drive it belongs
2583+ # to. We need to do this as some properties are associated with
2584+ # the drive, not the filesystem/block device and the drive may
2585+ # not have been inserted at all.
2586+ try:
2587+ drive_object = self._current_objects[drive_object_path]
2588+ except KeyError:
2589+ # The drive may be removed along with the device, let's check
2590+ # if we originally saw it
2591+ try:
2592+ drive_object = self._reference_objects[drive_object_path]
2593+ except KeyError:
2594+ logging.error(
2595+ "A block device belongs to a drive we could not find")
2596+ logging.error("missing drive: %r", drive_object_path)
2597+ continue
2598+ try:
2599+ drive_props = drive_object["org.freedesktop.UDisks2.Drive"]
2600+ except KeyError:
2601+ logging.error(
2602+ "A block device belongs to an object that is not a Drive")
2603+ logging.error("strange object: %r", drive_object_path)
2604+ continue
2605+ # Ensure the drive is on the appropriate bus
2606+ connection_bus = drive_props["ConnectionBus"]
2607+ if connection_bus not in self._desired_connection_buses:
2608+ logging.warning("The object %r belongs to drive %r that"
2609+ " is attached to the bus %r but but we are"
2610+ " looking for one of %r so it cannot match",
2611+ object_block_device, drive_object_path,
2612+ connection_bus,
2613+ ", ".join(self._desired_connection_buses))
2614+ # Ignore this object so that we don't spam the user twice
2615+ self._ignored_objects.add(object_path)
2616+ continue
2617+ # Ensure it is a media card reader if this was explicitly requested
2618+ drive_is_reader = is_memory_card(
2619+ drive_props['Vendor'], drive_props['Model'],
2620+ drive_props['Media'])
2621+ if self._desired_memory_card and not drive_is_reader:
2622+ logging.warning(
2623+ "The object %s belongs to drive %s that does not seem to"
2624+ " be a media reader", object_block_device,
2625+ drive_object_path)
2626+ # Ignore this object so that we don't spam the user twice
2627+ self._ignored_objects.add(object_path)
2628+ continue
2629+ # Ensure the desired minimum speed is enforced
2630+ if self._desired_minimum_speed:
2631+ # We need to discover the speed of the UDisks2 object that is
2632+ # about to be matched. Sadly UDisks2 no longer supports this
2633+ # property so we need to poke deeper and resort to udev.
2634+ #
2635+ # The UDisks2 object that we are interested in implements a
2636+ # number of interfaces, most notably
2637+ # org.freedesktop.UDisks2.Block, that has the Device property
2638+ # holding the unix filesystem path (like /dev/sdb1). We already
2639+ # hold a reference to that as 'object_block_device'
2640+ #
2641+ # We take this as a start and attempt to locate the udev Device
2642+ # (don't confuse with UDisks2.Device, they are _not_ the same)
2643+ # that is associated with that path.
2644+ if self._desired_delta_dir == DELTA_DIR_PLUS:
2645+ # If we are looking for additions then look at _current_
2646+ # collection of udev devices
2647+ udev_devices = current_udev_devices
2648+ udisks2_object = self._current_objects[object_path]
2649+ else:
2650+ # If we are looking for removals then look at referece
2651+ # collection of udev devices
2652+ udev_devices = self._reference_udev_devices
2653+ udisks2_object = self._reference_objects[object_path]
2654+ try:
2655+ # Try to locate the corresponding udev device among the
2656+ # collection we've selected. Use the drive object as the
2657+ # key -- this looks for the drive, not partition objects!
2658+ udev_device = lookup_udev_device(udisks2_object,
2659+ udev_devices)
2660+ except LookupError:
2661+ logging.error("Unable to map UDisks2 object %s to udev",
2662+ object_block_device)
2663+ # Ignore this object so that we don't spam the user twice
2664+ self._ignored_objects.add(object_path)
2665+ continue
2666+ interconnect_speed = get_interconnect_speed(udev_device)
2667+ # Now that we know the speed of the interconnect we can try to
2668+ # validate it against our desired speed.
2669+ if interconnect_speed is None:
2670+ logging.warning("Unable to determine interconnect speed of"
2671+ " device %s", object_block_device)
2672+ # Ignore this object so that we don't spam the user twice
2673+ self._ignored_objects.add(object_path)
2674+ continue
2675+ elif interconnect_speed < self._desired_minimum_speed:
2676+ logging.warning(
2677+ "Device %s is connected via an interconnect that has"
2678+ " the speed of %dMbit/s but the required speed was"
2679+ " %dMbit/s", object_block_device, interconnect_speed,
2680+ self._desired_minimum_speed)
2681+ # Ignore this object so that we don't spam the user twice
2682+ self._ignored_objects.add(object_path)
2683+ continue
2684+ else:
2685+ logging.info("Device %s is connected via an USB"
2686+ " interconnect with the speed of %dMbit/s",
2687+ object_block_device, interconnect_speed)
2688+ # Yay, success
2689+ results.add(object_block_device)
2690+ return results
2691+
2692+ @property
2693+ def _current_objects(self):
2694+ return self._udisks2_model.managed_objects
2695+
2696+ def _get_delta_records(self):
2697+ """
2698+ Internal method used to compute the delta between reference devices and
2699+ current devices. The result is a generator of DeltaRecord objects.
2700+ """
2701+ assert self._reference_objects is not None, "Only usable after check()"
2702+ old = self._reference_objects
2703+ new = self._current_objects
2704+ return udisks2_objects_delta(old, new)
2705+
2706+ def _summarize_changes(self, delta_records):
2707+ """
2708+ Internal method used to summarize changes (compared to reference state)
2709+ called whenever _on_change() gets called. Only visible in verbose mode
2710+ """
2711+ # Filter out anything but interface changes
2712+ flat_records = [record
2713+ for record in delta_records
2714+ if record.value.delta_type == DELTA_TYPE_IFACE]
2715+ # Group changes by DBus object path
2716+ grouped_records = collections.defaultdict(list)
2717+ for record in flat_records:
2718+ grouped_records[record.value.object_path].append(record)
2719+ # Bail out quickly when nothing got changed
2720+ if not flat_records:
2721+ if not self._is_reference:
2722+ logging.info("You have returned to the reference state")
2723+ self._is_reference = True
2724+ return
2725+ else:
2726+ self._is_reference = False
2727+ # Iterate over grouped delta records for all objects
2728+ logging.info("Compared to the reference state you have:")
2729+ for object_path in sorted(grouped_records.keys()):
2730+ records_for_object = sorted(
2731+ grouped_records[object_path],
2732+ key=lambda record: record.value.iface_name)
2733+ # Skip any job objects as they just add noise
2734+ if any((record.value.iface_name == "org.freedesktop.UDisks2.Job"
2735+ for record in records_for_object)):
2736+ continue
2737+ logging.info("For object %s", object_path)
2738+ for record in records_for_object:
2739+ # Ignore property changes for now
2740+ if record.value.delta_type != DELTA_TYPE_IFACE:
2741+ continue
2742+ # Get the name of the interface that was affected
2743+ iface_name = record.value.iface_name
2744+ # Get the properties for that interface (for removals get the
2745+ # reference values, for additions get the current values)
2746+ if record.delta_dir == DELTA_DIR_PLUS:
2747+ props = self._current_objects[object_path][iface_name]
2748+ action = "inserted"
2749+ else:
2750+ props = self._reference_objects[object_path][iface_name]
2751+ action = "removed"
2752+ # Display some human-readable information associated with each
2753+ # interface change
2754+ if iface_name == "org.freedesktop.UDisks2.Drive":
2755+ logging.info("\t * %s a drive", action)
2756+ logging.info("\t vendor and name: %r %r",
2757+ props['Vendor'], props['Model'])
2758+ logging.info("\t bus: %s", props['ConnectionBus'])
2759+ logging.info("\t size: %s", format_bytes(props['Size']))
2760+ logging.info("\t is media card: %s", is_memory_card(
2761+ props['Vendor'], props['Model'], props['Media']))
2762+ logging.info("\t current media: %s",
2763+ props['Media'] or "???" if
2764+ props['MediaAvailable'] else "N/A")
2765+ elif iface_name == "org.freedesktop.UDisks2.Block":
2766+ logging.info("\t * %s block device", action)
2767+ logging.info("\t from drive: %s", props['Drive'])
2768+ logging.info("\t having device: %s", props['Device'])
2769+ logging.info("\t having usage, type and version:"
2770+ " %s %s %s", props['IdUsage'],
2771+ props['IdType'], props['IdVersion'])
2772+ logging.info("\t having label: %s", props['IdLabel'])
2773+ elif iface_name == "org.freedesktop.UDisks2.PartitionTable":
2774+ logging.info("\t * %s partition table", action)
2775+ logging.info("\t having type: %r", props['Type'])
2776+ elif iface_name == "org.freedesktop.UDisks2.Partition":
2777+ logging.info("\t * %s partition", action)
2778+ logging.info("\t from partition table: %s",
2779+ props['Table'])
2780+ logging.info("\t having size: %s",
2781+ format_bytes(props['Size']))
2782+ logging.info("\t having name: %r", props['Name'])
2783+ elif iface_name == "org.freedesktop.UDisks2.Filesystem":
2784+ logging.info("\t * %s file system", action)
2785+ logging.info("\t having mount points: %r",
2786+ props['MountPoints'])
2787
2788
2789 def main():
2790 description = "Wait for the specified device to be inserted or removed."
2791 parser = argparse.ArgumentParser(description=description)
2792 parser.add_argument('action', choices=['insert', 'remove'])
2793- parser.add_argument('device',
2794- choices=['usb', 'sdio', 'firewire',
2795- 'scsi', 'ata_serial_esata'],
2796- nargs=argparse.REMAINDER)
2797+ parser.add_argument('device', choices=['usb', 'sdio', 'firewire', 'scsi',
2798+ 'ata_serial_esata'], nargs="+")
2799+ memorycard_help = ("Memory cards devices on bus other than sdio require "
2800+ "this parameter to identify them as such")
2801+ parser.add_argument('--memorycard', action="store_true",
2802+ help=memorycard_help)
2803 parser.add_argument('--timeout', type=int, default=20)
2804+ min_speed_help = ("Will only accept a device if its connection speed "
2805+ "attribute is higher than this value "
2806+ "(in bits/s)")
2807+ parser.add_argument('--minimum_speed', '-m', help=min_speed_help,
2808+ type=int, default=0)
2809+ parser.add_argument('--verbose', action='store_const', const=logging.INFO,
2810+ dest='logging_level', help="Enable verbose output")
2811+ parser.add_argument('--debug', action='store_const', const=logging.DEBUG,
2812+ dest='logging_level', help="Enable debugging")
2813+ parser.set_defaults(logging_level=logging.WARNING)
2814 args = parser.parse_args()
2815
2816- listener = StorageDeviceListener(args.action, args.device)
2817- return(listener.check(args.timeout))
2818+ # Configure logging as requested
2819+ # XXX: This may be incorrect as logging.basicConfig() fails after any other
2820+ # call to logging.log(). The proper solution is to setup a verbose logging
2821+ # configuration and I didn't want to do it now.
2822+ logging.basicConfig(
2823+ level=args.logging_level,
2824+ format='[%(asctime)s] %(levelname)s:%(name)s:%(message)s')
2825+
2826+ # Connect to the system bus, we also get the event
2827+ # loop as we need it to start listening for signals.
2828+ system_bus, loop = connect_to_system_bus()
2829+
2830+ # Check if system bus has the UDisks2 object
2831+ if is_udisks2_supported(system_bus):
2832+ # Construct the listener with all of the arguments provided on the
2833+ # command line and the explicit system_bus, loop objects.
2834+ logging.debug("Using UDisks2 interface")
2835+ listener = UDisks2StorageDeviceListener(
2836+ system_bus, loop,
2837+ args.action, args.device, args.minimum_speed, args.memorycard)
2838+ else:
2839+ # Construct the listener with all of the arguments provided on the
2840+ # command line and the explicit system_bus, loop objects.
2841+ logging.debug("Using UDisks1 interface")
2842+ listener = UDisks1StorageDeviceListener(
2843+ system_bus, loop,
2844+ args.action, args.device, args.minimum_speed, args.memorycard)
2845+ # Run the actual listener and wait till it either times out of discovers
2846+ # the appropriate media changes
2847+ try:
2848+ return listener.check(args.timeout)
2849+ except KeyboardInterrupt:
2850+ return 1
2851
2852 if __name__ == "__main__":
2853 sys.exit(main())
2854
2855=== modified file 'setup.py'
2856--- setup.py 2012-10-04 02:54:55 +0000
2857+++ setup.py 2012-10-09 17:53:24 +0000
2858@@ -233,8 +233,11 @@
2859 ("share/apport/package-hooks/", ["apport/source_checkbox.py"]),
2860 ("share/apport/general-hooks/", ["apport/checkbox.py"])],
2861 scripts = ["bin/checkbox-cli", "bin/checkbox-gtk", "bin/checkbox-urwid", "bin/checkbox-qt"],
2862- packages = ["checkbox", "checkbox.contrib", "checkbox.lib", "checkbox.parsers",
2863- "checkbox.reports", "checkbox_cli", "checkbox_gtk", "checkbox_urwid", "checkbox_qt"],
2864+
2865+ packages=[
2866+ "checkbox", "checkbox.contrib", "checkbox.dbus", "checkbox.lib",
2867+ "checkbox.parsers", "checkbox.reports", "checkbox.heuristics",
2868+ "checkbox_cli", "checkbox_gtk", "checkbox_urwid", "checkbox_qt"],
2869 package_data = {
2870 "": ["cputable"]},
2871 cmdclass = {

Subscribers

People subscribed via source and target branches