Merge lp:~aptdaemon-developers/aptdaemon/future-status into lp:aptdaemon

[Sebastian Heinlein]

Add support for simulating a transaction and to preview changes which take into account all currently running and queued transactions.

[lp:~aptdaemon-developers/aptdaemon/future-status]

378. By Sebastian Heinlein on 2010-04-18

Fix showing the download size

379. By Sebastian Heinlein on 2010-04-18

CONSOLE: make use of ngettext for the package listing

380. By Sebastian Heinlein on 2010-04-18

CONSOLE: take APT::Get::Assume-Yes into account

381. By Sebastian Heinlein on 2010-04-18

CONSOLE: Add full-upgrade and safe-upgrade options (see aptitude)

382. By Sebastian Heinlein on 2010-04-18

CORE: Use seconds for the timeout

383. By Sebastian Heinlein on 2010-04-18

CORE: set the idle timeout of a transaction to 5 minutes

384. By Sebastian Heinlein on 2010-04-18

CORE: Allow to modify the packages attribute of a transaction after creation

385. By Sebastian Heinlein on 2010-04-18

Worker: Add Support for simulating file installs

386. By Sebastian Heinlein on 2010-04-18

Console: Allow to install package files by --install PATH

387. By Sebastian Heinlein on 2010-04-18

Console: Fix transaction confirmation

388. By Sebastian Heinlein on 2010-04-18

Core: Fix packages signal (copy paste error)

389. By Sebastian Heinlein on 2010-04-18

Console: Fix packages order

390. By Sebastian Heinlein on 2010-04-18

Worker: Show the package file as install if not installed before and as reinstall in the other case

391. By Sebastian Heinlein on 2010-04-18

WORKER: Fix local install by unlocking the cache

392. By Sebastian Heinlein on 2010-04-19

Improve status messages in the console client

393. By Sebastian Heinlein on 2010-04-20

Worker: Check for skipped updates in the simulation

394. By Sebastian Heinlein on 2010-04-20

Add an enum for the package group index in the Transaction.depends and Transaction.packages lists

395. By Sebastian Heinlein on 2010-04-20

Use the same order in Transaction.packages and Transaction.depends and extend depends accordingly. Use the new enums.

396. By Sebastian Heinlein on 2010-04-20

Do not ierate on the main loop during sensible actions like opening the cache or forking. This could lead to a cache race with the transaction simulation

397. By Sebastian Heinlein on 2010-04-20

Reuse the fixed DaemonOpenProgress again in the worker.

=== modified file 'aptdaemon/client.py'
--- aptdaemon/client.py	2010-04-01 05:22:25 +0000
+++ aptdaemon/client.py	2010-04-20 17:40:54 +0000
@@ -82,6 +82,21 @@
     __gsignals__ = {"finished": (gobject.SIGNAL_RUN_FIRST,
                                  gobject.TYPE_NONE,
                                  (gobject.TYPE_INT,)),
+                    "dependencies-changed": (gobject.SIGNAL_RUN_FIRST,
+                                             gobject.TYPE_NONE,
+                                             (gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT,
+                                              gobject.TYPE_PYOBJECT)),
+                    "download-changed": (gobject.SIGNAL_RUN_FIRST,
+                                         gobject.TYPE_NONE,
+                                         (gobject.TYPE_INT,)),
+                    "space-changed": (gobject.SIGNAL_RUN_FIRST,
+                                      gobject.TYPE_NONE,
+                                      (gobject.TYPE_INT,)),
                     "error": (gobject.SIGNAL_RUN_FIRST,
                               gobject.TYPE_NONE,
                               (gobject.TYPE_INT, gobject.TYPE_STRING)),
@@ -168,9 +183,12 @@
         self.progress = 0
         self.paused = False
         self.http_proxy = None
+        self.dependencies = [[], [], [], [], [], [], []]
         self.packages = [[], [], [], [], []]
         self.meta_data = {}
         self.remove_obsoleted_depends = False
+        self.download = 0
+        self.space = 0
         self._locale = ""
         self._exit_handler = None
         self._method = None
@@ -217,6 +235,9 @@
             self.emit("allow-unauthenticated-changed", value)
         elif property_name == "Terminal":
             self.emit("terminal-changed", value)
+        elif property_name == "Dependencies":
+            self.dependencies = value
+            self.emit("dependencies-changed", *value)
         elif property_name == "Packages":
             self.packages = value
             self.emit("packages-changed", *value)
@@ -255,6 +276,12 @@
         elif property_name == "ProgressDetails":
             self.progress_details = value
             self.emit("progress-details-changed", *value)
+        elif property_name == "Download":
+            self.download = value
+            self.emit("download-changed", value)
+        elif property_name == "Space":
+            self.space = value
+            self.emit("space-changed", value)
         elif property_name == "HttpProxy":
             self.http_proxy = value
             self.emit("http-proxy-changed", value)
@@ -315,6 +342,25 @@
             else:
                 raise
 
+    def simulate(self, reply_handler=None, error_handler=None):
+        """Simulate the transaction to calculate the dependencies, the
+        required download size and the required disk space.
+
+        The corresponding properties of the transaction will be updated.
+
+        Also TransactionFailed exceptions could be raised, if e.g. a
+        requested package could not be installed or the cache is currently
+        broken.
+
+        Keyword arguments:
+        reply_handler - callback function. If specified in combination with
+            error_handler the method will be called asynchrounsouly.
+        error_handler - in case of an error the given callback gets the
+            corresponding DBus exception instance
+        """
+        self._iface.Simulate(reply_handler=reply_handler,
+                             error_handler=error_handler)
+
     def cancel(self, reply_handler=None, error_handler=None):
         """Cancel the running transaction.
 

=== modified file 'aptdaemon/console.py'
--- aptdaemon/console.py	2010-03-30 19:48:19 +0000
+++ aptdaemon/console.py	2010-04-20 17:40:54 +0000
@@ -24,9 +24,12 @@
 import array
 import fcntl
 from gettext import gettext as _
+from gettext import ngettext
+import locale
 from optparse import OptionParser
 import os
 import pty
+import re
 import termios
 import time
 import tty
@@ -47,6 +50,7 @@
 ANSI_BOLD = chr(27) + "[1m"
 ANSI_RESET = chr(27) + "[0m"
 
+locale.setlocale(locale.LC_ALL, "")
 
 class ConsoleClient:
     """
@@ -132,9 +136,10 @@
         self._client.update_cache(reply_handler=self._run_transaction,
                                   error_handler=self._on_exception)
 
-    def upgrade_system(self):
+    def upgrade_system(self, safe_mode):
         """Upgrade system"""
-        self._client.upgrade_system(reply_handler=self._run_transaction,
+        self._client.upgrade_system(safe_mode,
+                                    reply_handler=self._run_transaction,
                                     error_handler=self._on_exception)
 
     def run(self):
@@ -249,6 +254,30 @@
                          "%3.3s%% " % percent +
                          "%-*.*s" % (text_width, text_width, text) + "\r")
 
+    def _update_custom_progress(self, msg, percent=None, spin=True):
+        """Update the progress bar with a custom status message."""
+        text = ANSI_BOLD + msg + ANSI_RESET
+        text_width = self._terminal_width - 9
+        # Spin the progress line (maximum 5 times a second)
+        if spin:
+            self._spin_cur = (self._spin_cur + 1) % len(self._spin_elements)
+            self._spin_stamp = time.time()
+            spinner = self._spin_elements[self._spin_cur]
+        else:
+            spinner = "+"
+        # Show progress information if available
+        if percent is None:
+            percent = "---"
+        sys.stderr.write("[%s] " % spinner +
+                         "%3.3s%% " % percent +
+                         "%-*.*s" % (text_width, text_width, text) + "\r")
+        return True
+
+    def _stop_custom_progress(self):
+        """Stop the spinner which shows non trans status messages."""
+        if self._progress_id is not None:
+            gobject.source_remove(self._progress_id)
+
     def _clear_progress(self):
         """Clear progress information on stderr."""
         sys.stderr.write("%-*.*s\r" % (self._terminal_width,
@@ -257,7 +286,8 @@
 
     def _on_cancel_signal(self, signum, frame):
         """Callback for a cancel signal."""
-        if self._transaction:
+        if self._transaction and \
+           self._transaction.status != enums.STATUS_SETTING_UP:
             self._transaction.cancel()
         else:
             self._loop.quit()
@@ -330,8 +360,119 @@
     def _run_transaction(self, trans):
         """Callback which runs a requested transaction."""
         self._set_transaction(trans)
-        trans.run(reply_handler=lambda: True,
-                  error_handler=self._on_exception)
+        self._stop_custom_progress()
+        if self._transaction.role in [enums.ROLE_UPDATE_CACHE,
+                                      enums.ROLE_ADD_VENDOR_KEY_FILE,
+                                      enums.ROLE_REMOVE_VENDOR_KEY,
+                                      enums.ROLE_FIX_INCOMPLETE_INSTALL]:
+            #TRANSLATORS: status message
+            self._progress_id = \
+                    gobject.timeout_add(250, self._update_custom_progress,
+                                        _("Queuing"))
+            self._transaction.run(error_handler=self._on_exception,
+                     reply_handler=lambda: self._stop_custom_progress())
+        else:
+            #TRANSLATORS: status message
+            self._progress_id = \
+                    gobject.timeout_add(250, self._update_custom_progress,
+                                        _("Resolving dependencies"))
+            self._transaction.simulate(reply_handler=self._show_changes,
+                                       error_handler=self._on_exception)
+
+    def _show_changes(self):
+        def show_packages(pkgs):
+            """Format the pkgs in a nice way."""
+            line = " "
+            pkgs.sort()
+            for pkg in pkgs:
+                if len(line) + 1 + len(pkg) > self._terminal_width and \
+                   line != " ":
+                    print line
+                    line = " "
+                line += " %s" % pkg
+            if line != " ":
+                print line
+        self._stop_custom_progress()
+        self._clear_progress()
+        installs, reinstalls, removals, purges, upgrades  = \
+                self._transaction.packages
+        dep_installs, dep_reinstalls, dep_removals, dep_purges, dep_upgrades, \
+                dep_downgrades, dep_kepts = self._transaction.dependencies
+        installs.extend(dep_installs)
+        upgrades.extend(dep_upgrades)
+        removals.extend(purges)
+        removals.extend(dep_removals)
+        removals.extend(dep_purges)
+        reinstalls.extend(dep_reinstalls)
+        #FIXME: should be change when supported by CommitPackages
+        downgrades = dep_downgrades
+        kepts = dep_kepts
+        if installs:
+            #TRANSLATORS: %s is the number of packages
+            print ngettext("The following NEW package will be installed (%s):",
+                           "The following NEW packages will be installed (%s):",
+                           len(installs)) % len(installs)
+            show_packages(installs)
+        if upgrades:
+            #TRANSLATORS: %s is the number of packages
+            print ngettext("The following package will be upgraded (%s):",
+                           "The following packages will be upgraded (%s):",
+                           len(upgrades)) % len(upgrades)
+            show_packages(upgrades)
+        if removals:
+            #TRANSLATORS: %s is the number of packages
+            print ngettext("The following package will be REMOVED (%s):",
+                           "The following packages will be REMOVED (%s):",
+                           len(removals)) % len(removals)
+            #FIXME: mark purges
+            show_packages(removals)
+        if downgrades:
+            #TRANSLATORS: %s is the number of packages
+            print ngettext("The following package will be DOWNGRADED (%s):",
+                           "The following packages will be DOWNGRADED (%s):",
+                           len(downgrades)) % len(downgrades)
+            show_packages(downgrades)
+        if reinstalls:
+            #TRANSLATORS: %s is the number of packages
+            print ngettext("The following package will be reinstalled (%s):",
+                           "The following packages will be reinstalled (%s):",
+                           len(reinstalls)) % len(reinstalls)
+            show_packages(reinstalls)
+        if kepts:
+            print ngettext("The following package has been kept back (%s):",
+                           "The following packages have been kept back (%s):",
+                           len(kepts)) % len(kepts)
+            show_packages(kepts)
+
+        if self._transaction.download:
+            print _("Need to get %sB of archives.") % \
+                    apt_pkg.size_to_str(self._transaction.download)
+        if self._transaction.space > 0:
+            print _("After this operation, %sB of additional disk space "
+                    "will be used.") % \
+                            apt_pkg.size_to_str(self._transaction.space)
+        elif self._transaction.space < 0:
+            print _("After this operation, %sB of additional disk space "
+                    "will be freed.") % \
+                            apt_pkg.size_to_str(self._transaction.space)
+        if not apt_pkg.config.find_b("APT::Get::Assume-Yes"):
+            try:
+                cont = raw_input(_("Do you want to continue [Y/n]?"))
+            except EOFError:
+                cont = "n"
+            #FIXME: Listen to changed dependencies!
+            if not re.match(locale.nl_langinfo(locale.YESEXPR), cont) and \
+               cont != "":
+                msg = enums.get_exit_string_from_enum(enums.EXIT_CANCELLED)
+                self._update_custom_progress(msg, None, False)
+                self._loop.quit()
+                sys.exit(1)
+        #TRANSLATORS: status message
+        self._progress_id = gobject.timeout_add(250,
+                                                self._update_custom_progress,
+                                                _("Queuing"))
+        self._transaction.run(error_handler=self._on_exception,
+                           reply_handler=lambda: self._stop_custom_progress())
 
 
 def main():
@@ -367,9 +508,16 @@
     parser.add_option("-u", "--upgrade", default="",
                       action="store", type="string", dest="upgrade",
                       help=_("Install the given packages"))
-    parser.add_option("", "--upgrade-system", default="",
-                      action="store_true", dest="dist_upgrade",
-                      help=_("Upgrade the system"))
+    parser.add_option("", "--upgrade-system",
+                      action="store_true", dest="safe_upgrade",
+                      help=_("Deprecated: Please use --safe-upgrade"))
+    parser.add_option("", "--safe-upgrade",
+                      action="store_true", dest="safe_upgrade",
+                      help=_("Upgrade the system in a safe way"))
+    parser.add_option("", "--full-upgrade",
+                      action="store_true", dest="full_upgrade",
+                      help=_("Upgrade the system, possibly installing and "
+                             "removing packages"))
     parser.add_option("", "--add-vendor-key", default="",
                       action="store", type="string", dest="add_vendor_key",
                       help=_("Add the vendor to the trusted ones"))
@@ -396,14 +544,21 @@
     (options, args) = parser.parse_args()
     con = ConsoleClient(show_terminal=not options.hide_terminal,
                            allow_unauthenticated=options.allow_unauthenticated)
-    if options.dist_upgrade:
-        con.upgrade_system()
+    #TRANSLATORS: status message
+    con._progress_id = gobject.timeout_add(250, con._update_custom_progress,
+                                           _("Waiting for authentication"))
+    if options.safe_upgrade:
+        con.upgrade_system(True)
+    elif options.full_upgrade:
+        con.upgrade_system(False)
     elif options.refresh:
         con.update_cache()
     elif options.fix_install:
         con.fix_incomplete_install()
     elif options.fix_depends:
         con.fix_broken_depends()
+    elif options.install and options.install.endswith(".deb"):
+        con.install_file(options.install)
     elif options.install or options.reinstall or options.remove or \
          options.purge or options.upgrade:
         con.commit_packages(options.install.split(),

=== modified file 'aptdaemon/core.py'
--- aptdaemon/core.py	2010-03-31 08:52:50 +0000
+++ aptdaemon/core.py	2010-04-20 17:40:54 +0000
@@ -40,6 +40,7 @@
 import Queue
 import signal
 import sys
+import tempfile
 import time
 import threading
 import uuid
@@ -73,10 +74,10 @@
 APTDAEMON_IDLE_TIMEOUT = 5 * 60
 
 # Maximum allowed time between the creation of a transaction and its queuing
-TRANSACTION_IDLE_TIMEOUT = 30 * 1000
+TRANSACTION_IDLE_TIMEOUT = 300
 # Keep the transaction for the given time alive on the bus after it has
 # finished
-TRANSACTION_DEL_TIMEOUT = 5 * 1000
+TRANSACTION_DEL_TIMEOUT = 5
 
 # Setup the DBus main loop
 dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
@@ -163,22 +164,28 @@
         self.cancelled = False
         self.paused = False
         self._meta_data = dbus.Dictionary(signature="ss")
-        self.packages = [pkgs or dbus.Array([], signature=dbus.Signature('s')) \
-                         for pkgs in packages]
+        self._dpkg_status = None
+        self._download = 0
+        self._space = 0
+        self._depends = [dbus.Array([], signature=dbus.Signature('s')) \
+                         for i in range(7)]
+        self._packages = [pkgs or dbus.Array([], signature=dbus.Signature('s'))\
+                          for pkgs in packages]
         # Add a timeout which removes the transaction from the bus if it
         # hasn't been setup and run for the TRANSACTION_IDLE_TIMEOUT period
-        self._idle_watch = gobject.timeout_add(
+        self._idle_watch = gobject.timeout_add_seconds(
             TRANSACTION_IDLE_TIMEOUT, self._remove_from_connection_no_raise)
 
     def _remove_from_connection_no_raise(self):
         """Version of remove_from_connection that does not raise if the
         object isn't exported.
         """
+        log_trans.debug("Removing transaction")
         try:
             self.remove_from_connection()
         except LookupError, error:
-            logging.debug("remove_from_connection() raised LookupError: "
-                          "'%s'" % error)
+            log_trans.debug("remove_from_connection() raised LookupError: "
+                            "'%s'" % error)
 
     def _set_meta_data(self, data):
         # Perform some checks
@@ -251,8 +258,8 @@
         self.status = STATUS_FINISHED
         # Remove the transaction from the Bus after it is complete. A short
         # timeout helps lazy clients
-        gobject.timeout_add(TRANSACTION_DEL_TIMEOUT,
-                            self._remove_from_connection_no_raise)
+        gobject.timeout_add_seconds(TRANSACTION_DEL_TIMEOUT,
+                                    self._remove_from_connection_no_raise)
 
     def _get_exit(self):
         return self._exit
@@ -260,6 +267,50 @@
     exit = property(_get_exit, _set_exit,
                     doc="The exit state of the transaction.")
 
+    def _get_download(self):
+        return self._download
+
+    def _set_download(self, size):
+        self.PropertyChanged("Download", size)
+        self._download = size
+
+    download = property(_get_download, _set_download,
+                        doc="The download size of the transaction.")
+
+    def _get_space(self):
+        return self._space
+
+    def _set_space(self, size):
+        self.PropertyChanged("Space", size)
+        self._space = size
+
+    space = property(_get_space, _set_space,
+                     doc="The required disk space of the transaction.")
+
+    def _get_packages(self):
+        return self._packages
+
+    def _set_packages(self, packages):
+        self._packages = [dbus.Array(pkgs, signature=dbus.Signature('s')) \
+                          for pkgs in packages]
+        self.PropertyChanged("Packages", self._packages)
+
+    packages = property(_get_packages, _set_packages,
+                        doc="Packages which will be explictly install, "
+                            "upgraded, removed, purged or reinstalled.")
+
+    def _get_depends(self):
+        return self._depends
+
+    def _set_depends(self, depends):
+        self._depends = [dbus.Array(deps, signature=dbus.Signature('s')) \
+                         for deps in depends]
+        self.PropertyChanged("Dependencies", self._depends)
+
+    depends = property(_get_depends, _set_depends,
+                       doc="The additional dependencies: installs, removals, "
+                           "upgrades and downgrades.")
+
     def _get_status(self):
         return self._status
 
@@ -493,6 +544,24 @@
             return
         raise errors.APTDError("Could not cancel transaction")
 
+    @dbus_deferred_method(APTDAEMON_TRANSACTION_DBUS_INTERFACE,
+                          in_signature="", out_signature="",
+                          sender_keyword="sender")
+    def Simulate(self, sender):
+        """Simulate a transaction to update its dependencies, future status,
+        download size and required disk space.
+
+        Call this method if you want to show changes before queuing the
+        transaction.
+        """
+        def helper():
+            self.depends, self._dpkg_status, self.download, self.space = \
+                self.queue.worker.simulate(self, self.queue.future_status)
+        log_trans.info("Simulate was called")
+        deferred = self._check_foreign_user(sender)
+        deferred.add_callback(lambda x: helper())
+        return deferred
+
     def _set_terminal(self, ttyname):
         """Set the controlling terminal.
 
@@ -691,6 +760,9 @@
                     "HttpProxy": self.http_proxy,
                     "Packages": self.packages,
                     "MetaData": self.meta_data,
+                    "Dependencies": self.depends,
+                    "Download": self.download,
+                    "Space": self.space
                     }
         else:
             return {}
@@ -715,6 +787,9 @@
 
     __gsignals__ = {"queue-changed":(gobject.SIGNAL_RUN_FIRST,
                                      gobject.TYPE_NONE,
+                                     ()),
+                    "future-status-changed":(gobject.SIGNAL_RUN_FIRST,
+                                     gobject.TYPE_NONE,
                                      ())}
 
     def __init__(self, dummy):
@@ -726,6 +801,8 @@
             self.worker = DummyWorker()
         else:
             self.worker = AptWorker()
+        self.future_status = None
+        self.future_status_fd = None
         self.worker.connect("transaction-done", self._on_transaction_done)
 
     def __len__(self):
@@ -736,33 +813,65 @@
         log.debug("emitting queue changed")
         self.emit("queue-changed")
 
-    def put(self, transaction):
+    def _emit_future_status_changed(self):
+        """Emit the future-status-changed signal."""
+        log.debug("emitting future-status changed")
+        self.emit("future-status-changed")
+        #FIXME: All not yet queued transactions should listen to this signal
+        #       and update be re-simulated if already done so
+
+    def put(self, trans):
         """Add an item to the queue."""
-        if transaction._idle_watch is not None:
-            gobject.source_remove(transaction._idle_watch)
+        #FIXME: Add a timestamp to check if the future status of the trans
+        #       is really the later one
+        # Simulate the new transaction if this has not been done before:
+        if trans._dpkg_status is None:
+            trans.depends, trans._dpkg_status, trans.download, trans.space = \
+                self.worker.simulate(trans, self.future_status)
+        # Replace the old future status with the new one
+        if self.future_status_fd is not None:
+            os.close(self.future_status_fd)
+        self.future_status_fd, self.future_status = \
+            tempfile.mkstemp(prefix="future-status-")
+        os.write(self.future_status_fd, trans._dpkg_status)
+        self._emit_future_status_changed()
+
+        if trans._idle_watch is not None:
+            gobject.source_remove(trans._idle_watch)
         if self.worker.trans:
-            self._queue.append(transaction)
+            self._queue.append(trans)
         else:
-            self.worker.run(transaction)
+            self.worker.run(trans)
         self._emit_queue_changed()
 
     def _on_transaction_done(self, worker, tid):
         """Mark the last item as done and request a new item."""
+        #FIXME: Check if the transaction failed because of a broken system or
+        #       if dpkg journal is dirty. If so allready queued transactions
+        #       except the repair transactions should be removed from the queue
         try:
             next = self._queue.popleft()
         except IndexError:
             log.debug("There isn't any queued transaction")
+            # Reset the future status to the system one
+            if self.future_status_fd is not None:
+                os.close(self.future_status_fd)
+            self.future_status_fd = None
+            self.future_status = None
+            self._emit_future_status_changed()
         else:
             self.worker.run(next)
         self._emit_queue_changed()
 
     def remove(self, transaction):
         """Remove the specified item from the queue."""
+        # FIXME: handle future status
         self._queue.remove(transaction)
         self._emit_queue_changed()
 
     def clear(self):
         """Remove all items from the queue."""
+        # FIXME: handle future status
         for transaction in self._queue:
             transaction._remove_from_connection_no_raise()
         self._queue.clear()

=== modified file 'aptdaemon/enums.py'
--- aptdaemon/enums.py	2010-04-01 05:52:48 +0000
+++ aptdaemon/enums.py	2010-04-20 17:40:54 +0000
@@ -23,6 +23,15 @@
 def _(msg):
    return gettext.dgettext("aptdaemon", msg)
 
+# Index of package groups in the depends and packages property
+(PKGS_INSTALL,
+ PKGS_REINSTALL,
+ PKGS_REMOVE,
+ PKGS_PURGE,
+ PKGS_UPGRADE,
+ PKGS_DOWNGRADE,
+ PKGS_KEEP) = range(7)
+
 # Finish states
 (EXIT_SUCCESS,
  EXIT_CANCELLED,

=== modified file 'aptdaemon/progress.py'
--- aptdaemon/progress.py	2010-02-10 10:03:26 +0000
+++ aptdaemon/progress.py	2010-04-20 17:40:54 +0000
@@ -82,8 +82,6 @@
             if not self.quiet:
                 self._transaction.progress = progress
             self.progress = progress
-        while gobject.main_context_default().pending():
-            gobject.main_context_default().iteration()
 
     def done(self):
         """Callback after completing a step.
@@ -299,11 +297,6 @@
         """Fork and create a master/slave pty pair by which the forked process
         can be controlled.
         """
-        # process all pending events in the main loop, since we will quit
-        # the loop in the child process
-        context = gobject.main_context_default()
-        while context.pending():
-            context.iteration()
         pid, self.master_fd = os.forkpty()
         if pid == 0:
             mainloop.quit()

=== added file 'aptdaemon/test/test_future_status.py'
--- aptdaemon/test/test_future_status.py	1970-01-01 00:00:00 +0000
+++ aptdaemon/test/test_future_status.py	2010-04-20 17:40:54 +0000
@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""Tests the debconf forwarding"""
+
+import logging
+import os
+import subprocess
+import sys
+import tempfile
+import unittest
+
+import apt
+import apt_pkg
+
+sys.path.insert(0, "../..")
+import aptdaemon.worker
+import aptdaemon.enums
+
+DEBUG=False
+
+class MockTrans(object):
+    def __init__(self, packages):
+        self.packages = packages
+        self.role = aptdaemon.enums.ROLE_COMMIT_PACKAGES
+        self.tid = "12123"
+
+class FutureStatusTest(unittest.TestCase):
+
+    def setUp(self):
+        self.worker = aptdaemon.worker.AptWorker()
+        self.worker._cache = apt.Cache()
+        self.status_orig = apt_pkg.config.get("Dir::State::status")
+
+    def testInstall(self):
+        # Create a transaction which installs a package which has got
+        # uninstalled dependencies
+        for pkg in self.worker._cache:
+            if not pkg.is_installed and pkg.candidate:
+                deps = self._get_uninstalled_deps(pkg)
+                if deps:
+                    break
+                try:
+                    pkg.mark_install()
+                except SystemError:
+                    self.worker._cache.clear()
+                    continue
+        trans = MockTrans([[pkg.name], [], [], [], []])
+        # The test
+        deps, status, download, space = self.worker.simulate(trans)
+        # Check if the package is installed in the new status file
+        status_file, status_path = tempfile.mkstemp(prefix="future-status-")
+        os.write(status_file, status)
+        apt_pkg.config.set("Dir::State::status", status_path)
+        apt_pkg.init_system()
+        self.worker._cache.open()
+        self.assertTrue(self.worker._cache[trans.packages[0][0]].is_installed,
+                        "The package is not installed in the future")
+        if self.worker._cache.broken_count:
+            broken = [pkg for pkg in self.worker._cache if pkg.is_now_broken]
+            self.fail("The following packages are broken: " % " ".join(broken))
+
+    def _get_uninstalled_deps(self, pkg):
+        deps = []
+        for dep in pkg.candidate.dependencies:
+            if len(dep.or_dependencies) > 1:
+                return None
+            for base_dep in dep.or_dependencies:
+                try:
+                    dep = self.worker._cache[base_dep.name]
+                except KeyError:
+                    return None
+                if not dep.is_installed and dep.candidate and \
+                   apt_pkg.CheckDep(dep.candidate.version,
+                                    base_dep.relation,
+                                    base_dep.version):
+                    deps.append(dep)
+                else:
+                    return None
+        return deps
+
+    def tearDown(self):
+        pass
+
+if __name__ == "__main__":
+    if DEBUG:
+        logging.basicConfig(level=logging.DEBUG)
+    unittest.main()
+
+# vim: ts=4 et sts=4

=== modified file 'aptdaemon/worker.py'
--- aptdaemon/worker.py	2010-04-17 07:39:07 +0000
+++ aptdaemon/worker.py	2010-04-20 17:40:54 +0000
@@ -73,6 +73,7 @@
         self.trans = None
         self.last_action_timestamp = time.time()
         self._cache = None
+        self._status_orig = apt_pkg.config.find_file("Dir::State::status")
         self._lock_fd = -1
 
     def run(self, transaction):
@@ -106,7 +107,7 @@
             self._lock_cache()
             if self.trans.role == ROLE_FIX_INCOMPLETE_INSTALL:
                 self.fix_incomplete_install()
-            elif not is_dpkg_journal_clean():
+            elif not self.is_dpkg_journal_clean():
                 raise TransactionFailed(ERROR_INCOMPLETE_INSTALL)
             self._open_cache()
             # Process transaction which can handle a broken dep cache
@@ -302,7 +303,7 @@
             dpkg_range = (64, 99)
             self._commit_changes(fetch_range=(5, 33),
                                  install_range=(34, 63))
-            self._lock_cache()
+        self._unlock_cache()
         # Install the dpkg file
         if deb.install(DaemonDpkgInstallProgress(self.trans,
                                                  begin=64, end=95)):
@@ -486,6 +487,8 @@
         quiet -- if True do no report any progress
         """
         self.trans.status = STATUS_LOADING_CACHE
+        apt_pkg.config.set("Dir::State::status", self._status_orig)
+        apt_pkg.init_system()
         try:
             progress = DaemonOpenProgress(self.trans, begin=begin, end=end,
                                           quiet=quiet)
@@ -499,7 +502,7 @@
     def _lock_cache(self):
         """Lock the APT cache."""
         try:
-            self._lock_fd = lock_pkg_system()
+            self._lock_fd = self.lock_pkg_system()
         except LockFailedError, error:
             logging.error("Failed to lock the cache")
             self.trans.paused = True
@@ -518,7 +521,7 @@
     def _watch_lock(self):
         """Unpause the transaction if the lock can be obtained."""
         try:
-            lock_pkg_system()
+            self.lock_pkg_system()
         except LockFailedError:
             return True
         self.trans.paused = False
@@ -530,6 +533,68 @@
             os.close(self._lock_fd)
             self._lock_fd = -1
 
+    def lock_pkg_system(self):
+        """Lock the package system and provide information if this cannot be
+        done.
+
+        This is a reemplemenataion of apt_pkg.PkgSystemLock(), since we want to
+        handle an incomplete dpkg run separately.
+        """
+        def get_lock_fd(lock_path):
+            """Return the file descriptor of the lock file or raise
+            LockFailedError if the lock cannot be obtained.
+            """
+            fd_lock = apt_pkg.get_lock(lock_path)
+            if fd_lock < 0:
+                process = None
+                try:
+                    # Get the pid of the locking application
+                    fd_lock_read = open(lock_path, "r")
+                    flk = struct.pack('hhQQi', fcntl.F_WRLCK, os.SEEK_SET, 0,
+                                      0, 0)
+                    flk_ret = fcntl.fcntl(fd_lock_read, fcntl.F_GETLK, flk)
+                    pid = struct.unpack("hhQQi", flk_ret)[4]
+                    # Get the command of the pid
+                    fd_status = open("/proc/%s/status" % pid, "r")
+                    try:
+                        for key, value in (line.split(":") for line in \
+                                           fd_status.readlines()):
+                            if key == "Name":
+                                process = value.strip()
+                                break
+                    finally:
+                        fd_status.close()
+                except:
+                    pass
+                finally:
+                    fd_lock_read.close()
+                raise LockFailedError(lock_path, process)
+            else:
+                return fd_lock
+
+        # Try the lock in /var/cache/apt/archive/lock first
+        # this is because apt-get install will hold it all the time
+        # while the dpkg lock is briefly given up before dpkg is
+        # forked off. this can cause a race (LP: #437709)
+        lock_archive = os.path.join(apt_pkg.config.find_dir("Dir::Cache::Archives"),
+                                    "lock")
+        lock_fd_archive = get_lock_fd(lock_archive)
+        os.close(lock_fd_archive)
+        # Then the status lock
+        lock_sys = os.path.join(os.path.dirname(self._status_orig), "lock")
+        return get_lock_fd(lock_sys)
+
+    def is_dpkg_journal_clean(self):
+        """Return False if there are traces of incomplete dpkg status
+        updates."""
+        status_updates = os.path.join(os.path.dirname(self._status_orig),
+                                      "updates/")
+        for dentry in os.listdir(status_updates):
+            if dentry.isdigit():
+                return False
+        return True
+
+
     def _commit_changes(self, fetch_range=(5, 50), install_range=(50, 90)):
         """Commit previously marked changes to the cache.
 
@@ -581,6 +646,182 @@
             raise TransactionFailed(ERROR_PACKAGE_MANAGER_FAILED,
                                     "%s: %s" % (excep, output))
 
+    def simulate(self, trans, status_path=None):
+        """Return the dependencies which will be installed by the transaction,
+        the content of the dpkg status file after the transaction would have
+        been applied, the download size and the required disk space.
+
+        Keyword arguments:
+        trans -- the transaction which should be simulated
+        status_path -- the path to a dpkg status file on which the transaction
+                 should be applied
+        """
+        log.info("Simulating trans: %s" % trans.tid)
+        try:
+            return self._simulate_helper(trans, status_path)
+        except TransactionFailed, excep:
+            trans.error = excep
+        except Exception, excep:
+            trans.error = TransactionFailed(ERROR_UNKNOWN,
+                                            traceback.format_exc())
+        trans.exit = EXIT_FAILED
+        trans.progress = 100
+        self.last_action_timestamp = time.time()
+
+    def _simulate_helper(self, trans, status_path):
+        #FIXME: A lot of redundancy
+        #FIXME: Add checks for obsolete dependencies and unauthenticated
+        def get_base_records(sec, additional=[]):
+            records = ["Priority", "Installed-Size", "Architecture",
+                       "Version", "Replaces", "Depends", "Conflicts",
+                       "Breaks", "Recommends", "Suggests", "Provides",
+                       "Pre-Depends", "Essential"]
+            records.extend(additional)
+            ret = ""
+            for record in records:
+                try:
+                    ret += "%s: %s\n" % (record, sec[record])
+                except KeyError:
+                    pass
+            return ret
+
+        status = ""
+        depends = [[], [], [], [], [], [], []]
+        skip_pkgs = []
+        size = 0
+        installs = reinstalls = removals = purges = upgrades = downgrades = \
+                kepts = upgradables = []
+
+        # Only handle transaction which change packages
+        #FIXME: Add support for ROLE_FIX_INCOMPLETE_INSTALL,
+        #       ROLE_FIX_BROKEN_DEPENDS
+        if trans.role not in [ROLE_INSTALL_PACKAGES, ROLE_UPGRADE_PACKAGES,
+                              ROLE_UPGRADE_SYSTEM, ROLE_REMOVE_PACKAGES,
+                              ROLE_COMMIT_PACKAGES, ROLE_INSTALL_FILE]:
+            return depends, status, 0, 0
+
+        if not self.trans and not self.is_dpkg_journal_clean():
+            raise TransactionFailed(ERROR_INCOMPLETE_INSTALL)
+
+        # Fast forward the cache
+        if not status_path:
+            status_path = self._status_orig
+        apt_pkg.config.set("Dir::State::status", status_path)
+        apt_pkg.init_system()
+        #FIXME: open cache in background after startup
+        if not self._cache:
+            self._cache = apt.cache.Cache()
+        else:
+            self._cache.open()
+
+        if self._cache.broken_count:
+            broken = [pkg.name for pkg in self._cache if pkg.is_now_broken]
+            raise TransactionFailed(ERROR_CACHE_BROKEN, " ".join(broken))
+
+        # Mark the changes and apply
+        if trans.role == ROLE_UPGRADE_SYSTEM:
+            #FIXME: Should be part of python-apt to avoid using private API
+            upgradables = [self._cache[pkgname] \
+                           for pkgname in self._cache._set \
+                           if self._cache._depcache.is_upgradable(\
+                                   self._cache._cache[pkgname])]
+            upgradables = [pkg for pkg in self._cache if pkg.is_upgradable]
+            self._cache.upgrade(not trans.kwargs["safe_mode"])
+        elif trans.role == ROLE_INSTALL_FILE:
+            deb = apt.debfile.DebPackage(trans.kwargs["path"], self._cache)
+            if not deb.check():
+                raise TransactionFailed(ERROR_DEP_RESOLUTION_FAILED,
+                                        deb._failure_string)
+            status += "Package: %s\n" % deb.pkgname
+            status += "Status: install ok installed\n"
+            status += get_base_records(deb)
+            status += "\n"
+            skip_pkgs.append(deb.pkgname)
+            try:
+                size = int(deb["Installed-Size"]) * 1024
+            except (KeyError, AttributeError):
+                pass
+            try:
+                pkg = self._cache[deb.pkgname]
+            except KeyError:
+                trans.packages[PKGS_INSTALL] = [deb.pkgname]
+            else:
+                if pkg.is_installed:
+                    # if we failed to get the size from the deb file do nor
+                    # try to get the delta
+                    if size != 0:
+                        size -= pkg.installed.installed_size
+                    trans.packages[PKGS_REINSTALL] = [deb.pkgname]
+                else:
+                    trans.packages[PKGS_INSTALL] = [deb.pkgname]
+            installs, reinstalls, removal, purges, upgrades = trans.packages
+        else:
+            ac = self._cache.actiongroup()
+            installs, reinstalls, removals, purges, upgrades = trans.packages
+            resolver = apt.cache.ProblemResolver(self._cache)
+            self._mark_packages_for_installation(installs, resolver)
+            self._mark_packages_for_installation(reinstalls, resolver,
+                                                 reinstall=True)
+            self._mark_packages_for_removal(removals, resolver)
+            self._mark_packages_for_removal(purges, resolver, purge=True)
+            self._mark_packages_for_upgrade(upgrades, resolver)
+            self._resolve_depends(resolver)
+            ac.release()
+        changes = self._cache.get_changes()
+        changes_names = []
+
+        # get the additional dependencies
+        for pkg in changes:
+            if pkg.marked_upgrade and pkg.is_installed and \
+               not pkg.name in upgrades:
+                depends[PKGS_UPGRADE].append(pkg.name)
+            elif pkg.marked_reinstall and not pkg.name in reinstalls:
+                depends[PKGS_REINSTALL].append(pkg.name)
+            elif pkg.marked_downgrade and not pkg.name in downgrades:
+                depends[PKGS_DOWNGRADE].append(pkg.name)
+            elif pkg.marked_install and not pkg.name in installs:
+                depends[PKGS_INSTALL].append(pkg.name)
+            elif pkg.marked_delete and not pkg.name in removals:
+                depends[PKGS_REMOVE].append(pkg.name)
+            #FIXME: add support for purges
+            changes_names.append(pkg.name)
+        # Check for skipped upgrades
+        for pkg in upgradables:
+            if not pkg in changes or not pkg.marked_upgrade:
+                depends[PKGS_KEEP].append(pkg.name)
+
+        # merge the changes into the dpkg status
+        for sec in apt_pkg.TagFile(open(status_path)):
+            pkg_name = sec["Package"]
+            if pkg_name in skip_pkgs:
+                continue
+            status += "Package: %s\n" % pkg_name
+            if pkg_name in changes_names:
+                pkg = self._cache[sec["Package"]]
+                if pkg.marked_delete:
+                    status += "Status: deinstall ok config-files\n"
+                    version = pkg.installed
+                else:
+                    # Install, Upgrade, downgrade and reinstall all use the
+                    # candidate version
+                    version = pkg.candidate
+                    status += "Status: install ok installed\n"
+                status += get_base_records(version.record)
+                changes.remove(pkg)
+            else:
+                status += get_base_records(sec, ["Status"])
+            status += "\n"
+        # Add changed and not yet known (installed) packages to the status
+        for pkg in changes:
+            version = pkg.candidate
+            status += "Package: %s\n" % pkg.name
+            status += "Status: install ok installed\n"
+            status += get_base_records(pkg.candidate.record)
+            status += "\n"
+
+        return depends, status, self._cache.required_download, \
+               size + self._cache.required_space
+
 
 class DummyWorker(AptWorker):
 
@@ -663,63 +904,4 @@
         return False
 
 
-def lock_pkg_system():
-    """Lock the package system and provide information if this cannot be done.
-
-    This is a reemplemenataion of apt_pkg.PkgSystemLock(), since we want to
-    handle an incomplete dpkg run separately.
-    """
-    def get_lock_fd(lock_path):
-        """Return the file descriptor of the lock file or raise
-        LockFailedError if the lock cannot be obtained.
-        """
-        fd_lock = apt_pkg.get_lock(lock_path)
-        if fd_lock < 0:
-            process = None
-            try:
-                # Get the pid of the locking application
-                fd_lock_read = open(lock_path, "r")
-                flk = struct.pack('hhQQi', fcntl.F_WRLCK, os.SEEK_SET, 0, 0, 0)
-                flk_ret = fcntl.fcntl(fd_lock_read, fcntl.F_GETLK, flk)
-                pid = struct.unpack("hhQQi", flk_ret)[4]
-                # Get the command of the pid
-                fd_status = open("/proc/%s/status" % pid, "r")
-                try:
-                    for key, value in (line.split(":") for line in \
-                                       fd_status.readlines()):
-                        if key == "Name":
-                            process = value.strip()
-                            break
-                finally:
-                    fd_status.close()
-            except:
-                pass
-            finally:
-                fd_lock_read.close()
-            raise LockFailedError(lock_path, process)
-        else:
-            return fd_lock
-
-    # Try the lock in /var/cache/apt/archive/lock first
-    # this is because apt-get install will hold it all the time
-    # while the dpkg lock is briefly given up before dpkg is
-    # forked off. this can cause a race (LP: #437709)
-    lock_archive = os.path.join(apt_pkg.config.find_dir("Dir::Cache::Archives"),
-                                "lock")
-    lock_fd_archive = get_lock_fd(lock_archive)
-    os.close(lock_fd_archive)
-    # Then the status lock
-    status_file = apt_pkg.config.find_file("Dir::State::status")
-    lock_sys = os.path.join(os.path.dirname(status_file), "lock")
-    return get_lock_fd(lock_sys)
-
-def is_dpkg_journal_clean():
-    """Return False if there are traces of incomplete dpkg status updates."""
-    status_file = apt_pkg.config.find_file("Dir::State::status")
-    status_updates = os.path.join(os.path.dirname(status_file), "updates/")
-    for dentry in os.listdir(status_updates):
-        if dentry.isdigit():
-            return False
-    return True
-
 # vim:ts=4:sw=4:et