Merge lp:~apw/britney/britney1-ubuntu--fix-bug-blocks into lp:britney

[Andy Whitcroft]

When searching against a distribution we only actually will find tasks on
bugs open against development. This leads us to miss bugs which either
have a closed development task or have no development task at all. This
affects bugs in then development series which have no general development
task, and those nominated to older series and have a closed development
task. In particular these forms are seen with kernel tracking bugs.

Enhance the blocking bug search to search in both the distribution and
the associated series.

This has been run in unit-test against development and produces the same
block list there. It has been tested against the current live stable
releases and seems to produce believeable block lists, and significantly
longer ones in those releases.

Unmerged revisions

329. By Andy Whitcroft on 2020-03-12

britney: identify bug blocks against the nominated series as well as devel

When searching against a distribution we only actually will find tasks on
bugs open against development. This leads us to miss bugs which either
have a closed development task or have no development task at all. This
affects bugs in then development series which have no general development
task, and those nominated to older series and have a closed development
task. In particular these forms are seen with kernel tracking bugs.

Enhance the blocking bug search to search in both the distribution and the
associated series.

This has been run in unit-test against development and produces the same
block list there. It has been tested against the current live stable
releases and seems to produce believeable block lists, and significantly
longer ones in those releases.

Signed-off-by: Andy Whitcroft <email address hidden>

328. By Adam Conrad on 2019-10-18

eoan -> focal

327. By Adam Conrad on 2019-09-17

[rbalint, r=adconrad] List bugs tagged as update-excuse(-<release>) in ExcuseBugs

326. By Łukasz Zemczak on 2019-08-29

Once again enable sru-regression e-mails.

325. By Iain Lane on 2019-07-17

turn off sru-regression emails

they are sending false positives and Łukasz is going to look into it

324. By Iain Lane on 2019-07-02

fauxpkg.py: Check all directories for packages

If a package exists in the release pocket, we should not write a faux package
just because it doesn't exist in proposed.

323. By Iain Lane on 2019-07-02

fauxpkg.py: Don't look at source names

Faux packages are *binary* package names. When considering if there are
duplicates, we should only look at the binaries in the input Packages file and
not the sources as we have been previously doing.

322. By Iain Lane on 2019-07-02

fauxpkg.py: Print the full path when skipping FauxPackage entries

there's something wrong here - we're skipping things we shouldn't be; would be
helpful to see what's going wrong

321. By Iain Lane on 2019-07-02

Put the gnome-shell FauxPackage back

We still need this on eoan. It shouldn't break other releases any more, since
r319. If it does, let's fix that instead.

320. By Łukasz Zemczak on 2019-06-25

Switch off the dry-run enforcement for sru regression notification.

=== added file 'README'
--- README	1970-01-01 00:00:00 +0000
+++ README	2020-03-12 14:58:23 +0000
@@ -0,0 +1,104 @@
+Britney is run from /org/release.debian.org/britney. Some notes about
+how it works, what to do and what not:
+
+  * the code is in code/b1, which is a git repo. Changes must be
+    approved by Release Managers first, and all changes must be commited
+    (i.e. nothing uncommited is allowed there).
+
+    Commit emails will be send to [e-mail], you can
+    subscribe to that if you want them (along with cron mail from
+    britney).
+
+  * the main script is code/b1/britney. It accepts a list of "actions"
+    to perform. See ~release/etc/crontab.release for the default list,
+    and see the "Actions" and "Re-running" sections below.
+
+  * input/ contains FauxPackages, edit if if you need, and an urgencies/
+    subdirectory where dinstall places a summary of the uploads of that
+    day.
+
+  * Heidi/, ssh/: see the "save" action below.
+
+  * d-i/: udeb handling is done by the release team too. See d-i/README
+    for details (when Dato writes it; Dato takes care of udebs until then).
+
+
+Actions
+=======
+
+The main actions (read, the ones that are invoked from cron) are:
+
+  * urgencies: updates var/data/testing/Urgency with the data provided
+    by dinstall.
+
+  * bugs: fetches bug counts from bugs.debian.org.
+
+  * pkglists: generates all needed Packages files according to the
+    latest dinstall.
+
+  * run: the main action, running the update_out.py script. Takes
+    hints, packages files, and tries to migrate stuff. When done, it
+    writes a HeidiResult file used by "save" below *and rewrites the
+    var/data/testing/Packags_* files*. That is important.
+
+  * save: commits the HeidiResult provided by update_out above. This is
+    done by:
+
+      - taking HeidiResult, stripping unwanted stuff (faux packages),
+	and put the result into a file under Heidi/set, named after the
+	current timestamp.
+
+      - changing the "current" symlink under Heidi/set to point to this
+	new file, gzipping the previous current.
+
+      - signaling ftpmaster scripts to import the new file. This is done
+	with a ssh trigger, whose key is in ssh/.
+
+	IF YOU NEED TO UNDO ONE IMPORT: gunzip the previous file, point
+	the "current" symlink to it, and:
+
+	  % ssh -2 -i % ~release/britney/ssh/ftp-master_trigger_key [e-mail]
+    
+  * stats: copies the resulting update.OUTPUT and update.EXCUSES files
+    to www/britney, and checks for uninstallabilities.
+
+
+Re-running
+==========
+
+Re-runs can be incremental or not:
+
+  * incremental means that you run britney against the results of the
+    previous run. This is done by just issuing `./britney run`.
+
+    WARNING: even if you don't save the results of an incremental
+    re-run, following re-runs will start with the state the previous one
+    left. This means it's a very very bad idea to use incremental
+    re-runs to test force-hint hints; please always use a private
+    britney instead.
+
+  * non-incremental means that you start from scratch against the state
+    of the previous install. To do this, you need to do "pkglists" prior
+    to "run".
+
+    IF THERE WERE ANY SUCCESSFUL HINTS IN THE PREVIOUS RUN, YOU MUST
+    CHECK WITH THEIR OWNERS WHETHER IT'S OK TO RUN FROM SCRATCH. Or, in
+    the worst case, you should check whether they have cleaned up their
+    hints already, since in that case their work will be lost.
+
+    (For this reason it's not very wise to clean your hints before your
+    stuff is dinstalled.)
+
+If you're satisfied with the result of your re-run, run `./britney save`.
+
+
+Private britneys
+================
+
+It is ok to run private britneys if you nice them, though Ganneff
+prefers if you don't do it while dinstall is running.
+
+Do NOT run your britney against britney/var/data directly, use a
+copy of your own.
+
+TODO(dato): Some HOWTO here.

=== added file 'britney'
--- britney	1970-01-01 00:00:00 +0000
+++ britney	2020-03-12 14:58:23 +0000
@@ -0,0 +1,584 @@
+#!/bin/bash
+
+set -e
+set -u
+
+ulimit -d 8000000 -m 8000000 -v 8000000
+umask 002
+
+
+OPTIONS="$@"
+qoption () {
+    for a in $OPTIONS; do if [ "$a" = "$1" ]; then return 0; fi; done
+    return 1
+}
+
+option () {
+    for a in $OPTIONS; do if [ "$a" = "$1" ]; then date -uR; return 0; fi; done
+    return 1
+}
+
+LP_SERVICE="${LP_SERVICE:-production}"
+DISTRIBUTION="${DISTRIBUTION:-ubuntu}"
+DEFAULT_SERIES=focal
+SERIES="${SERIES:-$DEFAULT_SERIES}"
+
+# DATE
+NOW=`date +"%F/%T" -u`
+
+# Dirs:
+BASE=/home/ubuntu-archive/proposed-migration
+DAK_LOCKDIR=/srv/ftp-master.debian.org/lock
+FTP_MIRROR=/home/ubuntu-archive/mirror/$DISTRIBUTION
+
+D_I=$BASE/d-i
+VAR=$BASE/var
+SSH=$BASE/ssh
+HEIDI=$BASE/Heidi
+INPUT=$BASE/input
+CODE=$BASE/code/b1
+
+DATA=$VAR/data
+LOCKDIR=$VAR/lock
+HEIDI_SET=$HEIDI/set
+SCRIPTS=$CODE/scripts
+FAUXDIR=$CODE/fauxpkg
+UPDATE_OUT=$CODE/update_out
+URGENCIES=$INPUT/urgencies
+
+LOCK=$LOCKDIR/britney-$DISTRIBUTION-$SERIES.lock
+DAK_LOCK=$DAK_LOCKDIR/daily.lock
+DAK_STOP=$DAK_LOCKDIR/archive.stop
+
+TESTING=$DATA/$SERIES
+UNSTABLE=$DATA/$SERIES-proposed
+TPU=$DATA/testing-proposed-updates
+
+FAUXPKG_SCRIPT=$FAUXDIR/fauxpkg.py
+SSH_KEY=$SSH/ftp-master_trigger_key
+
+HTML=/home/ubuntu-archive/public_html/proposed-migration
+if [ "$DISTRIBUTION" != ubuntu ]; then
+    HTML="$HTML/$DISTRIBUTION"
+fi
+BOOTSTRAP_DIR=/home/ubuntu-archive/public_html/bootstrap
+
+# Britney 2
+DATA_B2=$VAR/data-b2
+CODE_B2=$BASE/code/b2
+B2_OUTPUT=$DATA_B2/output
+B2_CONFIG=${B2_CONFIG:-$CODE_B2/britney.conf}
+B2_CONFIG_NOBREAKALL=${B2_CONFIG_NOBREAKALL:-$CODE_B2/britney_nobreakall.conf}
+B2_AMQP_PASSWORD="$(cat $BASE/amqp-password.txt)"
+
+if [ -f $DAK_STOP ]; then
+  echo "$DAK_STOP exists, not running"
+  exit 1
+fi
+
+# 10800 seconds = 3 hours
+# 129600 seconds = 36 hours
+if ! lockfile -r0 -l129600 $LOCK 2>/dev/null >/dev/null; then
+    echo "Could not get britney lockfile!"
+    ls -l "$LOCK"
+    exit 1
+fi
+
+trap exit_function 0
+exit_function () {
+    rm -f $LOCK
+}
+
+#eval $( dak admin config db-shell )
+
+suite_info () {
+    python -c '
+from __future__ import print_function
+
+import os
+import sys
+
+from launchpadlib.launchpad import Launchpad as _Launchpad
+
+# Work around non-multiple-instance-safety of launchpadlib (bug #459418).
+class Launchpad(_Launchpad):
+    @classmethod
+    def _get_paths(cls, service_root, launchpadlib_dir=None):
+        service_root, launchpadlib_dir, cache_path, service_root_dir = (
+            _Launchpad._get_paths(
+                service_root, launchpadlib_dir=launchpadlib_dir))
+        cache_path += "-proposed-migration"
+        if not os.path.exists(cache_path):
+            os.makedirs(cache_path, 0o700)
+        return service_root, launchpadlib_dir, cache_path, service_root_dir
+
+launchpad = Launchpad.login_anonymously("proposed-migration", sys.argv[1])
+distro = launchpad.distributions[sys.argv[2]]
+series_name = sys.argv[3].split("-")[0]
+series = distro.getSeries(name_or_version=series_name)
+'"$1" "$LP_SERVICE" "$DISTRIBUTION" "$2"
+}
+
+suite_arches () {
+    suite_info 'print(" ".join(arch.architecture_tag for arch in series.architectures))' "$1"
+}
+
+suite_archindep () {
+    suite_info 'print(series.nominatedarchindep.architecture_tag)' "$1"
+}
+
+if ! qoption allowdaklock; then
+	while [ -f $DAK_LOCK ]; do
+		echo `date` $DAK_LOCK exists.  Sleeping in 10 more minutes.
+		sleep 600
+	done
+fi
+
+urgencies () {
+  echo URGENCIES:
+  cd $BASE
+  for u in $URGENCIES/install-urgencies-*; do
+    [ -e "$u" ] || continue
+    cat "$u" >> $1/$SERIES/Urgency # XXX I'd rather not have this in $DATA --dato
+    rm "$u"
+  done
+}
+
+if option urgencies; then
+  urgencies $DATA_B2
+fi
+
+bugs () {
+  for suite in $SERIES $SERIES-proposed; do
+    x="$1/${suite}/BugsV"
+    wget --quiet -O "${x}.new" http://bugs.debian.org/release-critical/britney/${suite}-nr
+    if [ -s ${x}.new ]; then mv ${x}.new $x; else rm $x.new; exit 1; fi
+  done
+}
+if option bugs; then
+  echo BUGS:
+  bugs $DATA_B2
+fi
+
+if option bugs_b1; then
+  echo BUGS_B1:
+  bugs $DATA
+fi
+
+if option bugs_b2; then
+  echo BUGS_B2:
+  bugs $DATA_B2
+fi
+
+excuse_bugs() {
+  tagged_bugs $1 update-excuse ExcuseBugs
+}
+
+blocks () {
+  tagged_bugs $1 block-proposed Blocks
+}
+
+tagged_bugs () {
+  local tag_template tags output_file
+  tag_template=$2
+  output_file=$3
+  [ "$DISTRIBUTION" = ubuntu ] || return 0
+  mkdir -p "$1/$SERIES-proposed"
+  if [ "$SERIES" = "$DEFAULT_SERIES" ]; then
+    tags="$tag_template $tag_template-$SERIES"
+  else
+    tags="$tag_template-$SERIES"
+  fi
+  python -c '
+from __future__ import print_function
+
+import calendar
+import os
+import sys
+try:
+    from urllib.parse import urlsplit
+except ImportError:
+    from urlparse import urlsplit
+
+from launchpadlib.launchpad import Launchpad as _Launchpad
+
+# Work around non-multiple-instance-safety of launchpadlib (bug #459418).
+class Launchpad(_Launchpad):
+    @classmethod
+    def _get_paths(cls, service_root, launchpadlib_dir=None):
+        service_root, launchpadlib_dir, cache_path, service_root_dir = (
+            _Launchpad._get_paths(
+                service_root, launchpadlib_dir=launchpadlib_dir))
+        cache_path += "-proposed-migration"
+        if not os.path.exists(cache_path):
+            os.makedirs(cache_path, 0o700)
+        return service_root, launchpadlib_dir, cache_path, service_root_dir
+
+launchpad = Launchpad.login_with("proposed-migration", sys.argv[1])
+distro = launchpad.distributions[sys.argv[2]]
+tags = sys.argv[3].split()
+series = distro.getSeries(name_or_version=sys.argv[4])
+blocks = set()
+tasks = list(series.searchTasks(omit_targeted=False, tags=tags))
+tasks += list(distro.searchTasks(omit_targeted=False, tags=tags))
+block_detail = {}
+for task in tasks:
+    target = task.target
+    bug = task.bug
+    if urlsplit(target.resource_type_link).fragment in (
+            "distribution_source_package", "source_package"):
+        date = block_detail.get(bug.id)
+        if date is None:
+            for action in reversed(
+                    [a for a in bug.activity if a.whatchanged == "tags"]):
+                oldtags = action.oldvalue.split()
+                newtags = action.newvalue.split()
+                gained_block = False
+                for tag in tags:
+                    if tag not in oldtags and tag in newtags:
+                        gained_block = True
+                        break
+                if gained_block:
+                    date = action.datechanged
+                    break
+            else:
+                date = bug.date_created
+            block_detail[bug.id] = date
+        blocks.add("%s %d %d" %
+              (os.path.basename(target.self_link), bug.id,
+               calendar.timegm(date.timetuple())))
+print("\n".join(sorted(blocks)))
+' "$LP_SERVICE" "$DISTRIBUTION" "$tags" "$SERIES" >"$1/$SERIES-proposed/$output_file.new"
+  mv "$1/$SERIES-proposed/$output_file.new" "$1/$SERIES-proposed/$output_file"
+}
+if option blocks; then
+  echo BLOCKS:
+  blocks $DATA_B2
+fi
+if option excuse_bugs; then
+  echo EXCUSE BUGS:
+  excuse_bugs $DATA_B2
+fi
+
+pkg_lists () {
+  for suite in $SERIES $SERIES-proposed; do
+      mkdir -p $1/$suite
+      suite_dirs=$FTP_MIRROR/dists/$suite
+      if [ "$suite" = "$SERIES" ]; then
+          suite_dirs="$suite_dirs $FTP_MIRROR/dists/$SERIES-updates"
+      fi
+      >$1/$suite/Sources
+      for suite_dir in $suite_dirs; do
+          gzip -t $suite_dir/{main,restricted,universe,multiverse}/source/Sources.gz
+          zcat $suite_dir/{main,restricted,universe,multiverse}/source/Sources.gz >>$1/$suite/Sources
+      done
+      for arch in `suite_arches $suite`; do
+          >$1/$suite/Packages_$arch
+          for suite_dir in $suite_dirs; do
+              gzip -t $suite_dir/{main,restricted,universe,multiverse}/binary-$arch/Packages.gz
+              zcat $suite_dir/{main,restricted,universe,multiverse}/binary-$arch/Packages.gz >>$1/$suite/Packages_$arch
+              gzip -t $suite_dir/{main,restricted,universe,multiverse}/debian-installer/binary-$arch/Packages.gz
+              zcat $suite_dir/{main,restricted,universe,multiverse}/debian-installer/binary-$arch/Packages.gz >>$1/$suite/Packages_$arch
+          done
+      done
+  done
+  $FAUXPKG_SCRIPT generate $1/$SERIES $1/$SERIES-proposed
+}
+
+lite_sync () {
+  cat <<EOF | rsync -a --files-from=- $1 $2
+$SERIES/Dates
+$SERIES/BugsV
+$SERIES/Urgency
+$SERIES-proposed/BugsV
+$SERIES-proposed/Hints
+EOF
+
+}
+
+if option pkglists; then
+  echo PKGLISTS:
+  cd $DATA_B2
+  pkg_lists $DATA_B2
+fi
+
+if option sync_b2; then
+  rsync -ar $DATA/ $DATA_B2
+fi
+
+# Give options to run a b2 from scratch after b1 has finished
+if option pkglists_b2; then
+  echo PKGLISTS_B2:
+  cd $DATA_B2
+  pkg_lists $DATA_B2
+fi
+
+if option pkglists_b1; then
+  echo PKGLISTS_B1:
+  cd $DATA
+  pkg_lists $DATA
+fi
+
+if option sync_b2_lite; then
+  lite_sync $DATA $DATA_B2
+fi
+
+if option sync_b1; then
+  rsync -ar $DATA_B2/ $DATA
+fi
+
+if option sync_b1_lite; then
+  lite_sync $DATA_B2 $DATA
+fi
+
+make_hints_branch () {
+  if [ -d "$1" ]; then
+    bzr pull -q -d "$1" --remember --overwrite "$2"
+  else
+    bzr branch "$2" "$1"
+  fi
+}
+
+if option hints_bzr; then
+  echo HINTS_BZR:
+  if [ "$DISTRIBUTION" = ubuntu ]; then
+    if [ "$SERIES" = "$DEFAULT_SERIES" ]; then
+      branch="lp:~ubuntu-release/britney/hints-ubuntu"
+    else
+      branch="lp:~ubuntu-sru/britney/hints-ubuntu-$SERIES"
+    fi
+    make_hints_branch "$DATA_B2/$SERIES-proposed/Hints" "$branch"
+    if [ "$SERIES" = "$DEFAULT_SERIES" ]; then
+      make_hints_branch \
+        "$DATA_B2/$SERIES-proposed/Hints/ubuntu-touch" \
+        "lp:~ubuntu-touch-release/britney/hints-ubuntu-touch"
+    else
+      rm -rf "$DATA_B2/$SERIES-proposed/Hints/ubuntu-touch"
+    fi
+  elif [ "$DISTRIBUTION" = ubuntu-rtm ]; then
+    make_hints_branch \
+      "$DATA_B2/$SERIES-proposed/Hints" \
+      lp:~ubuntu-release/britney/hints-ubuntu-rtm
+  fi
+fi
+
+run_b1 () {
+  cd $UPDATE_OUT
+  ./update_out.py $TESTING $UNSTABLE $TPU
+}
+
+b2_diff () {
+    tmp_dir=`mktemp -d`
+    master="b2"
+    [ "$master" = "b1" ] && secondary="b2" || secondary="b1"
+    (cd $tmp_dir
+
+    sort $TESTING/HeidiResult >HeidiResult_b1
+    sort $B2_OUTPUT/$SERIES/HeidiResult >HeidiResult_b2
+
+    sed -e '/^<p>Generated: /d' $CODE/update_out/update.EXCUSES_py >excuses_b1.html
+    sed -e '/^<p>Generated: /d' $B2_OUTPUT/$SERIES/excuses.html >excuses_b2.html
+
+    touch -m -r $TESTING/HeidiResult HeidiResult_b1
+    touch -m -r $B2_OUTPUT/$SERIES/HeidiResult HeidiResult_b2
+
+    touch -m -r $CODE/update_out/update.EXCUSES_py excuses_b1.html
+    touch -m -r $B2_OUTPUT/$SERIES/excuses.html excuses_b2.html
+
+    # NB: If you remove the "head" below, add a "|| true"; otherwise
+    # set -e may not allow the HeidiResult diff to execute.
+    diff -u excuses_${master}.html excuses_${secondary}.html | head -n 1000
+    diff -u HeidiResult_${master} HeidiResult_${secondary} | head -n 1000)
+    rm -rf $tmp_dir
+}
+
+b2_diff_with_mail () {
+  b2_diff
+  b2_diff | ifne mail -s "britney2 diff `date -u +'%Y-%m-%d %H:%M'`" \
+    -a "Reply-To: [e-mail]" [e-mail]
+}
+
+make_b2_config () {
+    local arches archindep
+    arches=`suite_arches $SERIES`
+    archindep=`suite_archindep $SERIES`
+    sed -e "s/^\\(ARCHITECTURES  *= \\).*/\\1$arches/" \
+        -e "s/^\\(NOBREAKALL_ARCHES  *= \\).*/\\1$archindep/" \
+        -e "/^ADT_AMQP/ s@:password@:${B2_AMQP_PASSWORD//@/\\@}@" \
+        "$1" >"$2"
+    # precise does not yet have ppc64el
+    if [ "$SERIES" = precise ]; then
+        sed -i '/^ADT_ARCHES/ s/ppc64el//' "$2"
+    fi
+    # only do the email thing for the default series
+    #
+    # and
+    #
+    # only do SRU regression emails for the non default series
+    if [ "$SERIES" != "$DEFAULT_SERIES" ]; then
+        sed -i '/^EMAIL_ENABLE/ s/=.*$/= no/' "$2"
+        sed -i '/^SRUREGRESSIONEMAIL_ENABLE/ s/=.*$/= yes/' "$2"
+    fi
+    case $arches in
+        *s390x*)
+            sed -i -e'/^ADT_ARCHES/ s/$/ s390x/' "$2"
+            ;;
+    esac
+}
+
+make_b2_notest_config () {
+    sed -e '/^NONINST_STATUS/ s/ *$/-notest/' \
+        -e '/^EXCUSES_.*OUTPUT/ s%=.*$%= /dev/null%' \
+        -e '/^UPGRADE_OUTPUT/ s/output.txt/output_notest.txt/' \
+        -e '/^HEIDI_OUTPUT/ s/=.*$/=/' \
+        -e '/^ADT_ENABLE/ s/yes/no/' \
+        -e '/^EMAIL_ENABLE/ s/=.*$/= no/' \
+        "$1" > "$2"
+}
+
+run_b2 () {
+  cd $BASE
+  make_b2_config "$B2_CONFIG" "$B2_CONFIG.$DISTRIBUTION.$SERIES"
+  $CODE_B2/britney.py -c "$B2_CONFIG.$DISTRIBUTION.$SERIES" -v --distribution=$DISTRIBUTION --series=$SERIES
+
+  if [ "$SERIES" = "$DEFAULT_SERIES" ]; then
+      # for the devel series we also want an output.txt without tests, to make
+      # it easier to untangle library transitions
+      make_b2_notest_config "$B2_CONFIG.$DISTRIBUTION.$SERIES" "$B2_CONFIG.$DISTRIBUTION.$SERIES.notest"
+      $CODE_B2/britney.py -c "$B2_CONFIG.$DISTRIBUTION.$SERIES.notest" --distribution=$DISTRIBUTION --series=$SERIES >/dev/null
+  else
+      rm -f $DATA_B2/output/$SERIES/output_notest.txt
+  fi
+}
+
+if option run; then
+  echo RUN:
+  run_b2
+fi
+
+if option run_b1; then
+  echo RUN_B1:
+  run_b1
+  # assume that this will have been run after a "run" which is b2
+  b2_diff_with_mail
+fi
+
+if option run_b2; then
+  echo RUN_B2:
+  run_b2
+  b2_diff_with_mail
+fi
+
+if option b2_diff; then
+    b2_diff
+fi
+
+lp_import() {
+  # tell LP about the new delta
+  promote-to-release -l "$LP_SERVICE" -d "$DISTRIBUTION" -s "$SERIES" -v "$1"
+}
+
+save () {
+  cd $BASE
+  echo RESULTS:
+  # process a delta
+  if [ "$1" = "b1" ]; then
+    DELTA="$TESTING/HeidiResultDelta"
+  else
+    DELTA="$B2_OUTPUT/$SERIES/HeidiResultDelta"
+  fi
+  echo Using data from $DELTA
+  lp_import "$DELTA"
+  printf " done\n"
+}
+
+if option save_b1; then
+  save b1
+fi
+if option save || option save_b2; then
+  save b2
+fi
+
+create_uninst_report () {
+    $CODE_B2/britney.py -c $1 --distribution=$DISTRIBUTION --series=$SERIES --print-uninst >$HTML/$2.new
+    echo -e "\n# Generated: `date -uR`" >>$HTML/$2.new
+    mv $HTML/$2.new $HTML/$2
+}
+
+create_ood_report () {
+    $SCRIPTS/ood_report.py $1 >$HTML/$2.new
+    echo -e "\n# Generated: `date -uR`" >>$HTML/$2.new
+    mv $HTML/$2.new $HTML/$2
+}
+
+stats () {
+  echo STATS:
+  mkdir -p "$HTML/$SERIES"
+  if [ "$1" = "b1" ]; then
+    cp $UPDATE_OUT/update.EXCUSES_py $HTML/$SERIES/update_excuses.html
+    cp $UPDATE_OUT/update.OUTPUT_py $HTML/$SERIES/update_output.txt
+  else
+    cp $DATA_B2/output/$SERIES/excuses.html $HTML/$SERIES/update_excuses.html
+    cp $DATA_B2/output/$SERIES/excuses.yaml $HTML/$SERIES/update_excuses.yaml
+    cp $DATA_B2/output/$SERIES/output.txt $HTML/$SERIES/update_output.txt
+    if [ -e $DATA_B2/output/$SERIES/output_notest.txt ]; then
+        cp $DATA_B2/output/$SERIES/output_notest.txt $HTML/$SERIES/update_output_notest.txt
+    else
+        rm -f $HTML/$SERIES/update_output_notest.txt
+    fi
+    cp $DATA_B2/$SERIES-proposed/Blocks $HTML/$SERIES/blocks.txt
+    cp $DATA_B2/$SERIES-proposed/ExcuseBugs $HTML/$SERIES/excuse-bugs.txt || true
+  fi
+  #gzip -9 < $HTML/$SERIES/update_excuses.html > $HTML/$SERIES/update_excuses.html.gz
+  #gzip -9 < $HTML/$SERIES/update_output.txt > $HTML/$SERIES/update_output.txt.gz
+  mkdir -p "$HTML/update_output/$SERIES/${NOW%/*}"
+  mkdir -p "$HTML/update_excuses/$SERIES/${NOW%/*}"
+  gzip -9c $HTML/$SERIES/update_output.txt > $HTML/update_output/$SERIES/$NOW.txt.gz
+  gzip -9c $HTML/$SERIES/update_excuses.html > $HTML/update_excuses/$SERIES/$NOW.html.gz
+  gzip -9c $HTML/$SERIES/update_excuses.yaml > $HTML/update_excuses/$SERIES/$NOW.yaml.gz
+
+  # TODO: {stable,unstable}_uninst.txt -- needs support in b2, talk to Fabio
+  # TODO: this creates uninstallability reports against b2 results, not b1's
+  make_b2_config "$B2_CONFIG" "$B2_CONFIG.$DISTRIBUTION.$SERIES"
+  make_b2_config "$B2_CONFIG_NOBREAKALL" "$B2_CONFIG_NOBREAKALL.$DISTRIBUTION.$SERIES"
+  create_uninst_report "$B2_CONFIG.$DISTRIBUTION.$SERIES" ${SERIES}_uninst.txt
+  create_uninst_report "$B2_CONFIG_NOBREAKALL.$DISTRIBUTION.$SERIES" ${SERIES}_uninst_full.txt
+
+  #if grep -q -e '-meta-faux' $HTML/testing_uninst.txt; then
+  #  echo >&2 'Warning! Some -meta-faux package is uninstallable!'
+  #fi
+
+  #create_ood_report $DATA_B2/stable stable_outdate.txt
+  #create_ood_report $DATA_B2/testing testing_outdate.txt
+
+  $SCRIPTS/backlog-report -o $HTML/$SERIES/update_excuses.csv $HTML/$SERIES/update_excuses.html
+}
+
+if option stats_b1; then
+  stats b1
+fi
+if option stats || option stats_b2; then
+  stats b2
+fi
+
+if option summary; then
+  cd $BASE
+  echo "Out of dates holding up testing:"
+  TERM=vt100 w3m -dump $HTML/$SERIES/update_excuses.html | sed -n 's/^ *[^ ] *[^ ]*out of date on \([^ ]*\): .*$/\1/p' | sort | uniq -c | sort -n
+  echo "Uninstallables holding up testing:"
+  sed < $HTML/$SERIES/update_excuses.html -n 's/^ *<li>[^ ]* (\([^, ]*\),.*) uninstallable.*$/\1/p' | sort | uniq -c | sort -n
+  #echo "wanna-build stats:"
+  #for a in alpha arm hppa hurd-i386 i386 ia64 m68k mips mipsel powerpc s390 sparc; do
+  #  printf "  %-12s " "$a:"
+  #  /srv/wanna-build/bin/wanna-build-statistics --database=$a/build-db | 
+  #    grep "if also counting" || echo "-"
+  #done
+fi
+
+if option archivedata; then
+  echo ARCHIVE:
+  cd $VAR
+  tar czf data_bkup/data-$(date +"%Y.%m.%d").tgz data/{stable,testing,unstable,testing-proposed-updates}
+fi
+
+echo -n "Finished at: "; date -uR

=== added directory 'fauxpkg'
=== added file 'fauxpkg/FauxPackages'
--- fauxpkg/FauxPackages	1970-01-01 00:00:00 +0000
+++ fauxpkg/FauxPackages	2020-03-12 14:58:23 +0000
@@ -0,0 +1,69 @@
+Package: wine1.4-i386
+Source: wine1.6
+Version: ${unstable-version}
+Architecture: amd64
+Provides: wine-i386
+Multi-Arch: foreign
+
+Package: wine1.6-i386
+Source: wine1.6
+Version: ${unstable-version}
+Architecture: amd64
+Provides: wine-i386
+Multi-Arch: foreign
+
+Package: ocaml-mingw-w64-x86-64
+Source: mingw-ocaml
+Version: ${unstable-version}
+Architecture: i386
+Multi-Arch: foreign
+
+Package: claws-mail-extra-plugins-dbg
+Version: 1:1
+
+Package: claws-mail-html2-viewer
+Version: 1:1
+
+Package: ubuntu-emulator-images
+Source: android
+Version: ${unstable-version}
+Architecture: amd64 armhf
+Multi-Arch: foreign
+
+Package: ubuntu-emulator-runtime
+Source: android
+Version: ${unstable-version}
+Architecture: amd64 armhf
+Multi-Arch: foreign
+
+Package: libnss-mdns-i386
+Source: nss-mdns
+Version: ${unstable-version}
+Architecture: amd64
+Multi-Arch: foreign
+
+Package: unity8
+Version: ${unstable-version}
+Architecture: powerpc ppc64el s390x
+
+Package: liboxideqt-qmlplugin
+Source: oxide-qt
+Version: ${unstable-version}
+Architecture: powerpc ppc64el s390x
+
+Package: fenix
+Version: ${unstable-version}
+Architecture: amd64
+
+Package: fenix-plugins-system
+Version: ${unstable-version}
+Architecture: amd64
+
+Package: steam
+Version: ${unstable-version}
+Architecture: amd64
+Multi-Arch: foreign
+
+Package: gnome-shell
+Version: ${unstable-version}
+Architecture: s390x

=== added file 'fauxpkg/README'
--- fauxpkg/README	1970-01-01 00:00:00 +0000
+++ fauxpkg/README	2020-03-12 14:58:23 +0000
@@ -0,0 +1,50 @@
+Files and scripts in this directory are in charge on creating a list of
+"faux" packages for britney. These are packages that do not exist in the
+archive, but which we want britney to think they exist. At the moment
+they are used for:
+
+  - allowing packages in contrib to migrate even if their dependencies
+    are not satisfied within the archive. This is done by creating a
+    faux package for each missing dependency.
+
+  - ensure some packages do not get accidentally removed, because they
+    are needed by parts of testing which don't declare a Depends on
+    them. In particular, to ensure packages needed by the installer and
+    tasksel are not removed. This is done by creating faux packages that
+    do depend on all packages which are not to be removed.
+
+To create packages of the first kind, just append suitable entries to
+the toplevel FauxPackages file. These should have at least a Package and
+Version fields. You can restrict their existance to certain architectures
+by providing an Architecture field with a list of architectures. Other
+fields are propagated verbatim to britney, eg. Provides.
+
+As for the second kind, you don't directly create faux packages, but
+just append the packages which shouldn't be removed to the appropriate
+file under the "noremove.d" directory. Those files just contain a list
+of *binary* packages, one per line. A couple of important notes:
+
+  - if the binary package is not in testing, do not add it: wait until
+    it migrates.
+
+  - if the binary package is not available on i386 (being arch:all would
+    suffice), you have to qualify it with [arch], mentioning one arch in
+    which it is available.
+
+  - the names of the files under noremove.d must be valid as package
+    names, e.g. not contain underscores.
+
+The noremove.d/tasksel file is special: it gets generated automatically
+by the fauxpkg.py script, by running the script with the "update-tasksel"
+command. (Currently is it to be run by hand whenever the tasksel
+maintainer notices our file is out of date.)
+
+One further caveat: if one of these faux meta-packages becomes uninstallable,
+they stop working (since the removal of their dependencies does not increase
+the uninstallability count). The britney script has a mechanism in place to
+warn in case they become uninstallable. When adding packages to the lists,
+make sure to watch the next britney mail, in case you added conflicting
+packages. (In that case, you want to add such packages in different files
+under noremove.d.)
+
+Please don't forget to commit your changes to files in this directory!

=== added file 'fauxpkg/fauxpkg.py'
--- fauxpkg/fauxpkg.py	1970-01-01 00:00:00 +0000
+++ fauxpkg/fauxpkg.py	2020-03-12 14:58:23 +0000
@@ -0,0 +1,193 @@
+#! /usr/bin/python
+## encoding: utf-8
+#
+# Copyright (c) 2008 Adeodato Simó ([e-mail])
+# Licensed under the terms of the MIT license.
+
+"""Handle the creation of britney faux packages.
+
+This program gets called from the "britney" script in order to append to the
+Packages_<arch> files a list of faux packages. This is done with the "generate"
+command, passing a list of britney suite directories:
+
+  % fauxpkg.py generate /home/release/britney/var/data/{testing,unstable}
+
+This automatically appeds to the Packages files the list of faux packages. See
+the README file in this directory for the input files from which such list is
+generated.
+"""
+
+import os
+import re
+import sys
+import glob
+import tempfile
+import subprocess
+
+from collections import defaultdict
+
+import apt_pkg
+apt_pkg.init()
+
+##
+
+BASEDIR = os.path.dirname(__file__)
+
+NOREMOVE_DIR =  os.path.join(BASEDIR, 'noremove.d')
+FAUX_PACKAGES = os.path.join(BASEDIR, 'FauxPackages')
+
+DEFAULT_NOREMOVE_ARCH = 'amd64'
+
+##
+
+def main():
+    if not sys.argv[1:]:
+        print >>sys.stderr, 'Usage: %s <generate | update-tasksel> [ britney_suite_dir1 ... ]' % (
+                os.path.basename(sys.argv[0]))
+        sys.exit(1)
+    else:
+        command = sys.argv.pop(1)
+
+    if command == 'generate':
+        if not sys.argv[1:]:
+            print >>sys.stderr, 'E: need at least one britney suite directory'
+            sys.exit(1)
+        else:
+            do_generate(sys.argv[1:])
+    elif command == 'update-tasksel':
+        if sys.argv[1:]:
+            print >>sys.stderr, 'E: extra arguments not allowed'
+            sys.exit(1)
+        else:
+            do_update_tasksel()
+    else:
+        print >>sys.stderr, 'E: unknown command %s' % (command,)
+
+##
+
+def do_generate(directories):
+    arches = set()
+    allfaux = {}
+
+    for dir_ in directories:
+        arches.update([ re.sub(r'^.*/Packages_', '', x)
+            for x in glob.glob(os.path.join(dir_, 'Packages_*')) ])
+
+    unstable_versions = {}
+    all_packages = defaultdict(set)
+    for dir_ in directories:
+        with open(os.path.join(dir_, 'Sources')) as f:
+            parser = apt_pkg.TagFile(f)
+            for section in parser:
+                if 'Package' in section and 'Version' in section:
+                    unstable_versions[section['Package']] = section['Version']
+        for packages_file in glob.glob(os.path.join(dir_, 'Packages_*')):
+            arch = re.sub(r'^.*/Packages_', '', packages_file)
+            with open(packages_file, 'r') as f:
+                with apt_pkg.TagFile(f) as tf:
+                    for section in tf:
+                        all_packages[arch] |= {section.get('Package')}
+
+    # First, FauxPackages
+    try:
+        parser = apt_pkg.TagFile(file(FAUX_PACKAGES))
+        step = parser.step
+        section = parser.section
+    except AttributeError, e:
+        parser = apt_pkg.ParseTagFile(file(FAUX_PACKAGES))
+        step = parser.Step
+        section = parser.Section
+    while step():
+        d = dict(section)
+        d['Section'] = 'faux' # crucial; britney filters HeidiResult based on section
+
+        if not d.has_key('Architecture'):
+            these_arches = arches
+        else:
+            these_arches = set(re.split(r'[, ]+', d['Architecture']))
+
+        d['Architecture'] = 'all' # same everywhere
+
+        if d.get('Version') == '${unstable-version}':
+            source = d.get('Source', d.get('Package'))
+            if source in unstable_versions:
+                d['Version'] = unstable_versions[source]
+
+        for arch in these_arches:
+            allfaux.setdefault(arch, []).append(d)
+
+    # Now, noremove.d
+    for f in glob.glob(os.path.join(NOREMOVE_DIR, '*.list')):
+        pkgs = {}
+        basename = re.sub(r'.+/(.+)\.list', r'\1', f)
+
+        for line in file(f):
+            line = line.strip()
+            if re.match(r'^#', line):
+                continue
+            elif re.match(r'\S+$', line):
+                pkg = line
+                arch = DEFAULT_NOREMOVE_ARCH
+            else:
+                m = re.match(r'(\S+)\s+\[(.+)\]', line)
+                if m:
+                    pkg, arch = m.groups()
+                else:
+                    print >>sys.stderr, 'W: could not parse line %r' % (line,)
+
+            arch = re.split(r'[, ]+', arch)[0] # just in case
+            pkgs.setdefault(arch, set()).add(pkg)
+
+        for arch in pkgs.keys():
+            d = { 'Package': '%s-meta-faux' % (basename,), 'Version': '1',
+                  'Section': 'faux', 'Architecture': '%s' % (arch,),
+                  'Depends': ', '.join(pkgs[arch]) }
+            allfaux.setdefault(arch, []).append(d)
+
+    # Write the result
+    for arch in arches:
+        if arch not in allfaux:
+            continue
+        for dir_ in directories:
+            f = os.path.join(dir_, 'Packages_' + arch)
+            if not os.path.exists(f):
+                continue
+            else:
+                f = file(f, 'a')
+                for d in allfaux[arch]:
+                    if d['Package'] in all_packages[arch]:
+                        print >>sys.stderr, 'W: skipping %s/%s (%s) as it is already known' % (d['Package'], arch, f.name)
+                        continue
+                    f.write('\n'.join('%s: %s' % (k, v) for k, v in d.iteritems()) + '\n\n')
+
+##
+
+def do_update_tasksel():
+    p = subprocess.Popen('dak ls -f control-suite -s unstable -a source tasksel',
+            shell=True, stdout=subprocess.PIPE)
+    p.wait()
+    version = p.stdout.readline().split()[1]
+
+    p = subprocess.Popen('dak ls -f control-suite -s unstable -S -a i386,all tasksel',
+            shell=True, stdout=subprocess.PIPE)
+    p.wait()
+    tasks = []
+
+    for line in p.stdout:
+        pkg = line.split()[0]
+
+        if pkg.startswith('task-'):
+            tasks.append(pkg)
+
+    # Write the new file
+    tmpfd, tmpname = tempfile.mkstemp(dir=NOREMOVE_DIR)
+    os.write(tmpfd, '# Generated from tasksel-data %s\n' % (version,))
+    os.write(tmpfd, '\n'.join(sorted(tasks)) + '\n')
+    os.close(tmpfd)
+    os.chmod(tmpname, 0644)
+    os.rename(tmpname, os.path.join(NOREMOVE_DIR, 'tasksel.list'))
+
+##
+
+if __name__ == '__main__':
+    main()

=== added directory 'fauxpkg/noremove.d'
=== added file 'fauxpkg/noremove.d/README'
--- fauxpkg/noremove.d/README	1970-01-01 00:00:00 +0000
+++ fauxpkg/noremove.d/README	2020-03-12 14:58:23 +0000
@@ -0,0 +1,3 @@
+Packages listed in these files are protected against accidental removals
+from testing. Please mail [e-mail] if you think
+some package should be added to any of these lists.

=== added directory 'scripts'
=== added file 'scripts/backlog-report'
--- scripts/backlog-report	1970-01-01 00:00:00 +0000
+++ scripts/backlog-report	2020-03-12 14:58:23 +0000
@@ -0,0 +1,95 @@
+#! /usr/bin/python
+#
+# Generate a report of the backlog in -proposed.
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+import calendar
+import csv
+import gzip
+import io
+from optparse import OptionParser
+import os
+import re
+import sys
+import time
+
+
+generated_re = re.compile(r"^<p>Generated: (.*)</p>$")
+age_re = re.compile(r"^<li>([0-9]+) days old$")
+
+
+def import_excuses(csv_writer, path):
+    generated_time = None
+    valid_candidates = 0
+    not_considered = 0
+    ages = []
+    backlog = 0
+
+    if path.endswith(".gz"):
+        binary = gzip.open(path)
+    else:
+        binary = io.open(path, mode="rb")
+    try:
+        with io.BufferedReader(binary) as buffered:
+            with io.TextIOWrapper(buffered) as text:
+                for line in text:
+                    match = generated_re.match(line)
+                    age_match = age_re.match(line)
+                    if match:
+                        generated_time = time.strptime(
+                            match.group(1), "%Y.%m.%d %H:%M:%S +0000")
+                    elif age_match:
+                        ages.append(int(age_match.group(1)))
+                        backlog += max(int(age_match.group(1)) - 3, 0)
+                    elif line.startswith("<li>Valid candidate"):
+                        valid_candidates += 1
+                    elif line.startswith("<li>Not considered"):
+                        not_considered += 1
+    finally:
+        binary.close()
+
+    assert generated_time is not None
+
+    csv_writer.writerow({
+        "time": calendar.timegm(generated_time) * 1000,
+        "valid candidates": valid_candidates,
+        "not considered": not_considered,
+        "total": valid_candidates + not_considered,
+        "median age": ages[len(ages)/2],
+        "backlog": backlog,
+        })
+
+
+def main():
+    parser = OptionParser(description="Generate a backlog report.")
+    parser.add_option(
+        "-o", "--output", help="Write output to this file (default: stdout).")
+    options, args = parser.parse_args()
+
+    if options.output is not None:
+        csv_is_new = not os.path.exists(options.output)
+        if sys.version < "3":
+            output = open(options.output, "ab")
+        else:
+            output = open(options.output, "a", newline="")
+    else:
+        csv_is_new = True
+        output = sys.stdout
+
+    try:
+        csv_writer = csv.DictWriter(
+            output, ["time", "valid candidates", "not considered", "total", "median age", "backlog"])
+        if csv_is_new:
+            csv_writer.writeheader()
+        for arg in args:
+            import_excuses(csv_writer, arg)
+    finally:
+        if options.output is not None:
+            output.close()
+
+
+if __name__ == "__main__":
+    main()

=== added file 'scripts/ood_report.py'
--- scripts/ood_report.py	1970-01-01 00:00:00 +0000
+++ scripts/ood_report.py	2020-03-12 14:58:23 +0000
@@ -0,0 +1,101 @@
+#! /usr/bin/python
+## encoding: utf-8
+#
+# Copyright (c) 2008 Adeodato Simó ([e-mail])
+# Licensed under the terms of the MIT license.
+
+"""Create a report of packages that are out-of-date in each architecture.
+
+It expects a single "directory" argument, that should be a britney directory
+for a distribution, i.e. containing Packages_<arch> files and Sources.
+"""
+
+import os
+import re
+import sys
+import glob
+
+import apt_pkg
+apt_pkg.init()
+
+##
+
+def main():
+    if len(sys.argv) != 2:
+        print >>sys.stderr, 'Usage: %s <directory>'
+        sys.exit(1)
+    else:
+        directory = sys.argv[1]
+
+    pkgfiles = glob.glob(os.path.join(directory, 'Packages_*'))
+    versions = get_src_versions(os.path.join(directory, 'Sources'))
+
+    ood = {} # { arch1: { srcpk1: (oldver, [binpkg1, binpkg2, ...]), ... }, ... }
+
+    for pkgfile in pkgfiles:
+        arch = re.sub(r'^.*/Packages_', '', pkgfile)
+        try:
+            parser = apt_pkg.TagFile(file(pkgfile))
+            step = parser.step
+            get_section = parser.section
+            get_field = parser.section.get
+        except AttributeError, e:
+            parser = apt_pkg.ParseTagFile(file(pkgfile))
+            step = parser.Step
+            get_section = parser.Section
+            get_field = parser.Section.get
+        d = ood[arch] = {}
+
+        while step():
+            pkg = get_section['Package']
+            src = get_field('Source') or pkg
+
+            if ' ' in src:
+                m = re.match(r'(\S+) \((\S+)\)$', src)
+                src = m.group(1)
+                ver = m.group(2)
+            else:
+                ver = re.sub(r'\+b\d+$', '', get_section['Version'])
+
+            try:
+                distver = versions[src]
+            except KeyError:
+                pass # faux package
+            else:
+                if ver != distver:
+                    d.setdefault(src, (ver, []))[1].append(pkg)
+
+    arches = sorted(ood.keys())
+
+    for arch in arches:
+        print '* %s' % (arch,)
+        for src, (oldver, binpkgs) in sorted(ood[arch].iteritems()):
+            # do not print binpkgs, I think it clutters the view too much
+            print '  %s (%s)' % (src, oldver)
+        print
+
+    print '* summary'
+    print '\n'.join(map(lambda x: '%4d %s' % (len(ood[x]), x), arches))
+
+##
+
+def get_src_versions(sources_file):
+    """Return a dict { srcname: version, ... }."""
+    mydict = {}
+    try:
+        parser = apt_pkg.TagFile(file(sources_file))
+
+        while parser.step():
+            mydict[parser.section['Package']] = parser.section['Version']
+    except AttributeError, e:
+        parser = apt_pkg.ParseTagFile(file(sources_file))
+
+        while parser.Step():
+            mydict[parser.Section['Package']] = parser.Section['Version']
+
+    return mydict
+
+##
+
+if __name__ == '__main__':
+    main()

=== added file 'scripts/udeb-report'
--- scripts/udeb-report	1970-01-01 00:00:00 +0000
+++ scripts/udeb-report	2020-03-12 14:58:23 +0000
@@ -0,0 +1,49 @@
+#!/bin/sh
+
+# Generate a report with out-of-date udebs. Queries come from the
+# original "d-i" script by Jeroen van Wolffelaar. Modified not to emit
+# output for queries that return 0 rows by Adeodato Simó.
+
+# TODO(dato): rewrite these queries.
+
+export PAGER="cat"
+
+eval $(dak admin config db-shell)
+
+##
+
+maybe_print () {
+    if ! echo "$2" | grep -q -E '^\(0 rows\)$'; then
+	echo "$1"
+	echo "$2"
+	echo
+    fi
+}
+
+maybe_print "udeb's in testing that don't (anymore) correspond to any testing source:" \
+"`psql -c \"select b.package, b.version, (SELECT arch_string from
+architecture where b.architecture=architecture.id) as arch, s.source from
+bin_associations ba LEFT JOIN binaries b on (ba.bin=b.id) LEFT JOIN source s
+on (b.source=s.id) WHERE ba.suite=4 AND s.id NOT IN (SELECT source from
+src_associations WHERE suite=4) AND b.type = 'udeb' ORDER BY s.source,
+b.package, b.architecture;\"`"
+
+maybe_print "udeb's in unstable that should be in testing too:" \
+"`psql -c \"select b.package, b.version, (SELECT arch_string from
+architecture where b.architecture=architecture.id) as arch, s.source from
+bin_associations ba LEFT JOIN binaries b on (ba.bin=b.id) LEFT JOIN source s
+on (b.source=s.id) WHERE ba.suite=5 AND NOT EXISTS (SELECT 1 FROM
+bin_associations ba2 WHERE ba2.suite=4 AND ba2.bin=ba.bin) AND s.id IN (SELECT
+source from src_associations WHERE suite=4) AND b.type = 'udeb' AND
+b.architecture IN (SELECT architecture from suite_architectures where suite = 4)
+ORDER BY s.source, b.package, b.architecture;\"`"
+
+maybe_print "udeb's in t-p-u that should be in testing too:" \
+"`psql -c \"select b.package, b.version, (SELECT arch_string from
+architecture where b.architecture=architecture.id) as arch, s.source from
+bin_associations ba LEFT JOIN binaries b on (ba.bin=b.id) LEFT JOIN source s
+on (b.source=s.id) WHERE ba.suite=3 AND NOT EXISTS (SELECT 1 FROM
+bin_associations ba2 WHERE ba2.suite=4 AND ba2.bin=ba.bin) AND s.id IN (SELECT
+source from src_associations WHERE suite=4) AND b.type = 'udeb' AND
+b.architecture IN (SELECT architecture from suite_architectures where suite = 4)
+ORDER BY s.source, b.package, b.architecture;\"`"

=== added directory 'update_out'
=== added file 'update_out/.gitignore'
--- update_out/.gitignore	1970-01-01 00:00:00 +0000
+++ update_out/.gitignore	2020-03-12 14:58:23 +0000
@@ -0,0 +1,6 @@
+*.o
+*.so
+/oldstuff
+/Makefile.dep
+/update.EXCUSES_py
+/update.OUTPUT_py

=== added file 'update_out/Makefile'
--- update_out/Makefile	1970-01-01 00:00:00 +0000
+++ update_out/Makefile	2020-03-12 14:58:23 +0000
@@ -0,0 +1,33 @@
+
+CC = gcc
+CXX = g++
+CFLAGS = -fPIC -Wall -W -O2 -DNDEBUG -DMDEBUG0 -g -p # -DDIAGNOSE
+CXXFLAGS = $(CFLAGS)
+
+all : britneymodule.so # libajdpkg.a aptvercmp checklib freelist 
+
+clean :
+	rm -f *.so *.o *~ Makefile.dep gmon.out 
+	rm -f freelist aptvercmp checklib libajdpkg.a
+
+checklib : checklib.o dpkg.o dpkg-lib.o memory3.o freelist.o assert.o
+	$(CC) $(CFLAGS) -o checklib $^ -lapt-pkg # -lccmalloc -ldl
+
+aptvercmp : dpkg-lib.cpp
+	$(CXX) $(CFLAGS) -DTESTBIN -o aptvercmp dpkg-lib.cpp -lapt-pkg
+
+freelist : freelist.c assert.o
+	$(CC) $(CFLAGS) -DTESTBIN -o $@ $^
+
+#libajdpkg.a : dpkg.o dpkg-lib.o memory3.o freelist.o assert.o
+#	ar rv $@ $^
+#	ranlib $@
+
+britneymodule.so : britney-py.o dpkg.o dpkg-lib.o memory3.o freelist.o assert.o
+	$(CXX) -shared -o britneymodule.so $^ -lapt-pkg
+
+Makefile.dep :
+	@gcc -MM *.c *.cpp > Makefile.dep
+	@echo Makefile.dep : Makefile *.c *.h >> Makefile.dep
+
+-include Makefile.dep

=== added file 'update_out/README'
--- update_out/README	1970-01-01 00:00:00 +0000
+++ update_out/README	2020-03-12 14:58:23 +0000
@@ -0,0 +1,32 @@
+
+BUILDING
+========
+
+Install libapt-pkg-dev
+
+testing/ $ make
+testing/ $ mkdir old cur out
+testing/ $ cd testing
+testing/testing/ $ perl Makefile.PL
+testing/testing/ $ make
+
+Add old and new packages files into old and cur, respectively.
+
+testing/ $ ./checklib i386 alpha
+
+Will generate some test stuff in out/
+
+TODO
+====
+
+Need some way of actually updating archives.
+Need some way of judging differences between Packages files.
+  (so I can see what hasn't been updated and work out why;
+   so I can check that Packages.gz matches dpkg-scanpackages output)
+Need some way of automatically explaining why packages aren't upgraded.
+  (shouldn't be hard?)
+
+BUGS
+====
+
+out/ directory must exist for checklib, or segfault

=== added file 'update_out/assert.c'
--- update_out/assert.c	1970-01-01 00:00:00 +0000
+++ update_out/assert.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,11 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+int _myassertbug(int line, char *file, char *err) {
+        fprintf(stderr, "Assertion failed: %s:%d: %s\n", file, line, err);
+        fprintf(stderr, "I HATE YOU!!!");
+        ((void(*)())0)();
+        abort();
+        return 0;
+}
+

=== added file 'update_out/britney-py.c'
--- update_out/britney-py.c	1970-01-01 00:00:00 +0000
+++ update_out/britney-py.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,871 @@
+#include <python2.7/Python.h>
+
+#include "dpkg.h"
+
+#define MAKE_PY_LIST(L,S,E,I,V)                 \
+	do {                                    \
+		L = PyList_New(0);              \
+		if (!L) break;                  \
+		for (S; E; I) {                 \
+			PyObject *EL;           \
+			EL = Py_BuildValue V;   \
+			if (!EL) {              \
+				Py_DECREF(L);   \
+				L = NULL;       \
+				break;          \
+			}                       \
+			PyList_Append(L, EL);   \
+			Py_DECREF(EL);          \
+		}                               \
+		if (L) PyList_Sort(L);          \
+	} while(0)
+
+/**************************************************************************
+ * britney.Packages -- dpkg_packages wrapper
+ *******************************************/
+
+typedef enum { DONTFREE, FREE } dpkgpackages_freeme;
+typedef struct {
+	PyObject_HEAD
+	dpkg_packages		*pkgs;
+	PyObject 		*ref; /* object packages are "in" */
+	dpkgpackages_freeme	freeme; /* free pkgs when deallocing? */
+} dpkgpackages;
+
+staticforward PyTypeObject Packages_Type;
+
+static PyObject *dpkgpackages_new(dpkg_packages *pkgs, 
+                                  dpkgpackages_freeme freeme, PyObject *ref) 
+{
+	dpkgpackages *res;
+
+	res = PyObject_NEW(dpkgpackages, &Packages_Type);
+	if (res == NULL) return NULL;
+
+	res->pkgs = pkgs;
+	res->ref = ref;	Py_INCREF(res->ref);
+	res->freeme = freeme;
+
+	return (PyObject *) res;
+}
+
+static void dpkgpackages_dealloc(dpkgpackages *self) {
+	if (self->freeme == FREE) free_packages(self->pkgs);
+	Py_XDECREF(self->ref);
+	self->pkgs = NULL;
+	self->ref = NULL;
+	PyObject_DEL(self);
+}
+
+
+static dpkg_collected_package *dpkgpackages_lookuppkg(dpkgpackages *self,
+                                                      char *pkgname)
+{
+	dpkg_collected_package *cpkg = NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (!cpkg) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid package");
+	}
+	return cpkg;
+}
+
+static PyObject *dpkgpackages_ispresent(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	return cpkg ? Py_BuildValue("i", 1) : Py_BuildValue("i", 0);
+}
+
+static PyObject *dpkgpackages_getversion(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (cpkg) return Py_BuildValue("s", cpkg->pkg->version);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_getsource(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (cpkg) return Py_BuildValue("s", cpkg->pkg->source);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_getsourcever(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (cpkg) return Py_BuildValue("s", cpkg->pkg->source_ver);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_isarchall(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (cpkg) return Py_BuildValue("i", cpkg->pkg->arch_all);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_isntarchall(dpkgpackages *self, PyObject *args) {
+	dpkg_collected_package *cpkg;
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	cpkg = lookup_packagetbl(self->pkgs->packages, pkgname);
+	if (cpkg) return Py_BuildValue("i", !cpkg->pkg->arch_all);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_getfield(dpkgpackages *self, PyObject *args) {
+	char *field;
+	char *pkgname;
+	int i;
+	dpkg_collected_package *cpkg;
+	dpkg_paragraph *para;
+	if (!PyArg_ParseTuple(args, "ss", &pkgname, &field)) return NULL;
+	cpkg = dpkgpackages_lookuppkg(self, pkgname);
+	if (!cpkg) return NULL;
+	para = cpkg->pkg->details;
+	for (i = 0; i < para->n_entries; i++) {
+		if (strcasecmp(para->entry[i].name, field) == 0) {
+			return Py_BuildValue("s", para->entry[i].value);
+		}
+	}
+	return Py_BuildValue("");
+}
+static PyObject *dpkgpackages_isinstallable(dpkgpackages *self, PyObject *args)
+{
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	if (checkinstallable2(self->pkgs, pkgname)) {
+		return Py_BuildValue("i", 1);
+	} else {
+		return Py_BuildValue("");
+	}
+}
+static PyObject *dpkgpackages_isuninstallable(dpkgpackages *self, 
+                                              PyObject *args)
+{
+	char *pkgname;
+	if (!PyArg_ParseTuple(args, "s", &pkgname)) return NULL;
+	if (!checkinstallable2(self->pkgs, pkgname)) {
+		return Py_BuildValue("i", 1);
+	} else {
+		return Py_BuildValue("");
+	}
+}
+static PyObject *dpkgpackages_unsatdeps(dpkgpackages *self, PyObject *args) {
+	/* arguments are:
+	 * 	testingpkgs[arch].unsatisfiable_deps(unstablepkgs[arch], "netbase", "Depends")
+	 * exciting, huh?
+	 */
+
+	dpkgpackages *pkgpkgs;
+	char *pkgname, *fieldname;
+	dpkg_collected_package *cpkg;
+	int fieldidx;
+	int buflen = 100;
+	char *buf = malloc(buflen);
+	const char *fields[] = { "Pre-Depends", "Depends", "Recommends", 
+				"Suggests", NULL };
+	satisfieddeplist *unsatdeps, *dl;
+	PyObject *res = Py_BuildValue("[]");
+
+	if (!PyArg_ParseTuple(args, "O!ss", &Packages_Type, &pkgpkgs, &pkgname, &fieldname)) return NULL;
+
+	cpkg = lookup_packagetbl(pkgpkgs->pkgs->packages, pkgname);
+	if (!cpkg) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid package");
+		return NULL;
+	}
+
+	for (fieldidx = 0; fields[fieldidx]; fieldidx++) {
+		if (strcmp(fields[fieldidx], fieldname) == 0) break;
+	}
+	if (!fields[fieldidx]) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid dependency field");
+		return NULL;
+	}
+
+	unsatdeps = checkunsatisfiabledeps(self->pkgs, 
+	                                   cpkg->pkg->depends[fieldidx]);
+	for (dl = unsatdeps; dl != NULL; dl = dl->next) {
+		int len;
+		packagelist *it;
+		PyObject *pkglist;
+		deplist *depl;
+		dependency *dep;
+
+		len = 0;
+		buf[0] = '\0';
+		for (depl = dl->value->depl; depl; depl = depl->next) {
+			dep = depl->value;
+			len += strlen(dep->package) + 4;
+			/* 4 = strlen(" | ") + 1 */
+			if (dep->op != dr_NOOP) {
+				len += strlen(dep->version) + 6;
+				/* 6 = strlen(" (>= )") */
+			}
+			if (len >= buflen) {
+				char *newbuf;
+				newbuf = realloc(buf, len + 100);
+				if (newbuf == NULL) {
+					free_satisfieddeplist(unsatdeps);
+					free(buf);
+					Py_DECREF(res);
+					PyErr_SetFromErrno(PyExc_MemoryError);
+					return NULL;
+				}
+				buf = newbuf;
+				buflen = len + 100;
+			}
+			if (buf[0] != '\0') strcat(buf, " | ");
+			strcat(buf, dep->package);
+			if (dep->op != dr_NOOP) {
+				sprintf(buf + strlen(buf), " (%s %s)",
+					dependency_relation_sym[dep->op],
+					dep->version);
+			}
+		}
+
+		MAKE_PY_LIST(pkglist, it = dl->value->pkgs, it, it = it->next, 
+				("s", it->value->package)
+			    );
+
+		{
+			PyObject *depel = Py_BuildValue("(sN)", buf, pkglist);
+			PyList_Append(res, depel);
+			Py_DECREF(depel);
+		}
+	}
+
+	free_satisfieddeplist(unsatdeps);
+	free(buf);
+
+	return res;
+}
+
+static PyObject *dpkgpackages_getattr(dpkgpackages *self, char *name) {
+	static struct PyMethodDef dpkgsources_methods[] = {
+		{ "is_present", (binaryfunc) dpkgpackages_ispresent, 
+			METH_VARARGS, NULL },
+		{ "get_version", (binaryfunc) dpkgpackages_getversion, 
+			METH_VARARGS, NULL },
+		{ "get_source", (binaryfunc) dpkgpackages_getsource, 
+			METH_VARARGS, NULL },
+		{ "get_sourcever", (binaryfunc) dpkgpackages_getsourcever, 
+			METH_VARARGS, NULL },
+		{ "is_arch_all", (binaryfunc) dpkgpackages_isarchall, 
+			METH_VARARGS, NULL },
+		{ "isnt_arch_all", (binaryfunc) dpkgpackages_isntarchall, 
+			METH_VARARGS, NULL },
+		{ "get_field", (binaryfunc) dpkgpackages_getfield, 
+			METH_VARARGS, NULL },
+		{ "is_installable", (binaryfunc) dpkgpackages_isinstallable, 
+			METH_VARARGS, NULL },
+		{ "is_uninstallable", (binaryfunc)dpkgpackages_isuninstallable, 
+			METH_VARARGS, NULL },
+		{ "unsatisfiable_deps", (binaryfunc) dpkgpackages_unsatdeps, 
+			METH_VARARGS, NULL },
+		{ NULL, NULL, 0, NULL }
+	};
+
+	if (strcmp(name, "packages") == 0) {
+		PyObject *packages;
+		packagetbl_iter it;
+		MAKE_PY_LIST(packages, 
+		             it = first_packagetbl(self->pkgs->packages),
+		             !done_packagetbl(it), it = next_packagetbl(it),
+			     ("s", it.k)
+			    );
+		return packages;
+	}
+
+	return Py_FindMethod(dpkgsources_methods, (PyObject *)self, name);
+}
+
+static PyTypeObject Packages_Type = {
+	PyObject_HEAD_INIT(&PyType_Type)
+
+	0,                     /* ob_size (0) */
+	"Packages",            /* type name */
+	sizeof(dpkgpackages),  /* basicsize */
+	0,                     /* itemsize (0) */
+
+	(destructor)  dpkgpackages_dealloc,
+	(printfunc)   0,
+	(getattrfunc) dpkgpackages_getattr,
+	(setattrfunc) 0,
+	(cmpfunc)     0,
+	(reprfunc)    0,
+
+	0,                     /* number methods */
+	0,                     /* sequence methods */
+	0,                     /* mapping methods */
+
+	(hashfunc)    0,       /* dict[x] ?? */
+	(ternaryfunc) 0,       /* x() */
+	(reprfunc)    0        /* str(x) */
+};
+
+/**************************************************************************
+ * britney.Sources -- dpkg_sources wrapper
+ *****************************************/
+
+typedef struct {
+	PyObject_HEAD
+	dpkg_sources	*srcs;
+} dpkgsources;
+
+staticforward PyTypeObject Sources_Type;
+
+static PyObject *dpkgsources_new(PyObject *self, PyObject *args) {
+	dpkgsources *res = NULL;
+	char *dir;
+	PyObject *arches;
+	char **archesStr = NULL;
+	int i, count;
+
+	(void)self; /* unused */
+
+	if (!PyArg_ParseTuple(args, "sO!", &dir, &PyList_Type, &arches)) {
+		goto end;
+	}
+
+	count = PyList_Size(arches);
+	if (count <= 0) {
+		PyErr_SetString(PyExc_TypeError, "No architectures specified");
+		goto end;
+	}
+
+	archesStr = malloc(sizeof(char *) * count);
+	if (!archesStr) {
+		PyErr_SetFromErrno(PyExc_MemoryError);
+		goto end;
+	}
+
+	for (i = 0; i < count; i++) {
+		PyObject *arch = PyList_GetItem(arches, i);
+		if (!PyString_Check(arch)) {
+			goto end;
+		}
+		archesStr[i] = PyString_AsString(arch);
+	}
+
+	res = PyObject_NEW(dpkgsources, &Sources_Type);
+	if (res == NULL) goto end;
+
+	res->srcs = read_directory(dir, count, archesStr);
+	if (!res->srcs) {
+		Py_DECREF(res);
+		res = NULL;
+		goto end;
+	}
+
+end:
+	if (archesStr) free(archesStr);
+	return (PyObject *) res;
+}
+
+static void dpkgsources_dealloc(dpkgsources *self) {
+	free_sources(self->srcs);
+	self->srcs = NULL;
+	PyObject_DEL(self);
+}
+
+static PyObject *dpkgsources_packages(dpkgsources *self, PyObject *args)
+{
+	char *arch;
+	dpkg_packages *pkgs;
+	if (!PyArg_ParseTuple(args, "s", &arch)) return NULL;
+	pkgs = get_architecture(self->srcs, arch);
+	return dpkgpackages_new(pkgs, FREE, (PyObject *) self);
+}
+
+static PyObject *dpkgsources_isfake(dpkgsources *self, PyObject *args) {
+	char *srcname;
+	dpkg_source *src;
+
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	src = lookup_sourcetbl(self->srcs->sources, srcname);
+	if (src) return Py_BuildValue("i", src->fake);
+	else return Py_BuildValue("");
+}
+
+static PyObject *dpkgsources_getversion(dpkgsources *self, PyObject *args) {
+	char *srcname;
+	dpkg_source *src;
+
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	src = lookup_sourcetbl(self->srcs->sources, srcname);
+	if (src) return Py_BuildValue("s", src->version);
+	else return Py_BuildValue("");
+}
+
+static PyObject *dpkgsources_getfield(dpkgsources *self, PyObject *args) {
+	char *srcname, *field;
+	dpkg_source *src;
+	int i;
+	dpkg_paragraph *para;
+
+	if (!PyArg_ParseTuple(args, "ss", &srcname, &field)) return NULL;
+	src = lookup_sourcetbl(self->srcs->sources, srcname);
+	if (!src) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid source package");
+		return NULL;
+	}
+	para = src->details;
+	if (para) {
+		for (i = 0; i < para->n_entries; i++) {
+			if (strcasecmp(para->entry[i].name, field) == 0) {
+				return Py_BuildValue("s", para->entry[i].value);
+			}
+		}
+	}
+	return Py_BuildValue("");
+}
+
+static PyObject *dpkgsources_ispresent(dpkgsources *self, PyObject *args) {
+	char *srcname;
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	if (lookup_sourcetbl(self->srcs->sources, srcname)) {
+		return Py_BuildValue("i", 1);
+	} else {
+		return Py_BuildValue("i", 0);
+	}
+}
+
+static PyObject *dpkgsources_binaries(dpkgsources *self, PyObject *args) {
+	char *srcname, *arch;
+	int archnum;
+	dpkg_source *src;
+	PyObject *res;
+	ownedpackagelist *p;
+
+	if (!PyArg_ParseTuple(args, "ss", &srcname, &arch)) return NULL;
+
+	for (archnum = 0; archnum < self->srcs->n_arches; archnum++) {
+		if (strcmp(arch, self->srcs->archname[archnum]) == 0) break;
+	}
+	if (archnum == self->srcs->n_arches) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid architecture");
+		return NULL;
+	}
+
+	src = lookup_sourcetbl(self->srcs->sources, srcname);
+	if (src == NULL) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid source package");
+		return NULL;
+	}
+
+	MAKE_PY_LIST(res, p = src->packages[archnum], p, p = p->next,
+		     ("s", p->value->package)
+		    );
+	return res;
+}
+
+static PyObject *dpkgsources_getattr(dpkgsources *self, char *name) {
+	static struct PyMethodDef dpkgsources_methods[] = {
+		{ "Packages", (binaryfunc) dpkgsources_packages, 
+			METH_VARARGS, NULL },
+		{ "is_fake", (binaryfunc) dpkgsources_isfake, 
+			METH_VARARGS, NULL },
+		{ "get_version", (binaryfunc) dpkgsources_getversion, 
+			METH_VARARGS, NULL },
+		{ "get_field", (binaryfunc) dpkgsources_getfield, 
+			METH_VARARGS, NULL },
+		{ "is_present", (binaryfunc) dpkgsources_ispresent, 
+			METH_VARARGS, NULL },
+		{ "binaries", (binaryfunc) dpkgsources_binaries, 
+			METH_VARARGS, NULL },
+		{ NULL, NULL, 0, NULL }
+	};
+
+	if (strcmp(name, "arches") == 0) {
+		PyObject *arches;
+		int i;
+		MAKE_PY_LIST(arches, i = 0, i < self->srcs->n_arches, i++, 
+			     ("s", self->srcs->archname[i])
+			    );
+		return arches;
+	} else if (strcmp(name, "sources") == 0) {
+		PyObject *sources;
+		sourcetbl_iter it;
+		MAKE_PY_LIST(sources, 
+		             it = first_sourcetbl(self->srcs->sources),
+		             !done_sourcetbl(it), it = next_sourcetbl(it),
+			     ("s", it.k)
+			    );
+		return sources;
+	}
+
+	return Py_FindMethod(dpkgsources_methods, (PyObject *)self, name);
+}
+
+static PyTypeObject Sources_Type = {
+	PyObject_HEAD_INIT(&PyType_Type)
+
+	0,                     /* ob_size (0) */
+	"Sources",             /* type name */
+	sizeof(dpkgsources),   /* basicsize */
+	0,                     /* itemsize (0) */
+
+	(destructor)  dpkgsources_dealloc,
+	(printfunc)   0,
+	(getattrfunc) dpkgsources_getattr,
+	(setattrfunc) 0,
+	(cmpfunc)     0,
+	(reprfunc)    0,
+
+	0,                     /* number methods */
+	0,                     /* sequence methods */
+	0,                     /* mapping methods */
+
+	(hashfunc)    0,       /* dict[x] ?? */
+	(ternaryfunc) 0,       /* x() */
+	(reprfunc)    0        /* str(x) */
+};
+
+/**************************************************************************
+ * britney.SourcesNote -- dpkg_sourcesnote wrapper
+ *************************************************/
+
+typedef struct {
+	PyObject_HEAD
+	dpkg_sources_note	*srcsn;
+	PyObject 		*refs; /* list of referenced dpkgsources */
+} dpkgsrcsn;
+
+staticforward PyTypeObject SourcesNote_Type;
+
+static PyObject *dpkgsrcsn_new(PyObject *self, PyObject *args) {
+	dpkgsrcsn *res = NULL;
+	PyObject *arches;
+	char **archesStr = NULL;
+	int i, count;
+
+	(void)self; /* unused */
+
+	if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &arches)) {
+		goto end;
+	}
+
+	count = PyList_Size(arches);
+	if (count <= 0) {
+		PyErr_SetString(PyExc_TypeError, "No architectures specified");
+		goto end;
+	}
+
+	archesStr = malloc(sizeof(char *) * count);
+	if (!archesStr) {
+		PyErr_SetFromErrno(PyExc_MemoryError);
+		goto end;
+	}
+
+	for (i = 0; i < count; i++) {
+		PyObject *arch = PyList_GetItem(arches, i);
+		if (!PyString_Check(arch)) {
+			goto end;
+		}
+		archesStr[i] = PyString_AsString(arch);
+	}
+
+	res = PyObject_NEW(dpkgsrcsn, &SourcesNote_Type);
+	if (res == NULL) goto end;
+
+	res->refs = PyList_New(0);
+	res->srcsn = new_sources_note(count, archesStr);
+	if (!res->refs || !res->srcsn) {
+		Py_DECREF(res);
+		res = NULL;
+		goto end;
+	}
+
+end:
+	if (archesStr) free(archesStr);
+	return (PyObject *) res;
+}
+
+static void dpkgsrcsn_dealloc(dpkgsrcsn *self) {
+	if (self->srcsn) free_sources_note(self->srcsn);
+	self->srcsn = NULL;
+	Py_XDECREF(self->refs);
+	self->refs = NULL;
+
+	PyObject_DEL(self);
+}
+
+static PyObject *dpkgsrcsn_removesource(dpkgsrcsn *self, PyObject *args) {
+	char *name;
+	if (!PyArg_ParseTuple(args, "s", &name)) return NULL;
+	remove_source(self->srcsn, name);
+	return Py_BuildValue("");
+}
+static PyObject *dpkgsrcsn_upgradesource(dpkgsrcsn *self, PyObject *args) {
+	char *name;
+	dpkgsources *srcs;
+	dpkg_source *src;
+	if (!PyArg_ParseTuple(args, "O!s", &Sources_Type, &srcs, &name)) 
+		return NULL;
+	src = lookup_sourcetbl(srcs->srcs->sources, name);
+	if (!src) {
+		PyErr_SetString(PyExc_ValueError, "Source does not exist");
+		return NULL;
+	}
+	if (!PySequence_In(self->refs, (PyObject *)srcs))
+		PyList_Append(self->refs, (PyObject *)srcs);
+	upgrade_source(self->srcsn, src);
+	return Py_BuildValue("");
+}
+static PyObject *dpkgsrcsn_upgradearch(dpkgsrcsn *self, PyObject *args) {
+	char *name, *arch;
+	dpkgsources *srcs;
+	dpkg_source *src;
+	if (!PyArg_ParseTuple(args, "O!ss", &Sources_Type, &srcs, &name, &arch))
+		return NULL;
+	src = lookup_sourcetbl(srcs->srcs->sources, name);
+	if (!src) {
+		PyErr_SetString(PyExc_ValueError, "Source does not exist");
+		return NULL;
+	}
+	if (!PySequence_In(self->refs, (PyObject *)srcs))
+		PyList_Append(self->refs, (PyObject *)srcs);
+	upgrade_arch(self->srcsn, src, arch);
+	return Py_BuildValue("");
+}
+
+static PyObject *dpkgsrcsn_undochange(dpkgsrcsn *self, PyObject *args) {
+	if (!PyArg_ParseTuple(args, "")) return NULL;
+	undo_change(self->srcsn);
+	return Py_BuildValue("");
+}
+
+static PyObject *dpkgsrcsn_commitchanges(dpkgsrcsn *self, PyObject *args) {
+	if (!PyArg_ParseTuple(args, "")) return NULL;
+	commit_changes(self->srcsn);
+	return Py_BuildValue("");
+}
+
+static PyObject *dpkgsrcsn_writenotes(dpkgsrcsn *self, PyObject *args) {
+	char *dir;
+	if (!PyArg_ParseTuple(args, "s", &dir)) return NULL;
+	write_notes(dir, self->srcsn);
+	return Py_BuildValue("");
+}
+
+static PyObject *dpkgsrcsn_packages(dpkgsrcsn *self, PyObject *args) {
+	char *arch;
+	int archnum;
+	if (!PyArg_ParseTuple(args, "s", &arch)) return NULL;
+	for (archnum = 0; archnum < self->srcsn->n_arches; archnum++) {
+		if (strcmp(arch, self->srcsn->archname[archnum]) == 0) break;
+	}
+	if (archnum == self->srcsn->n_arches) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid architecture");
+		return NULL;
+	}
+	return dpkgpackages_new(self->srcsn->pkgs[archnum], DONTFREE,
+	                        (PyObject *) self);
+}
+
+static PyObject *dpkgsrcsn_getversion(dpkgsrcsn *self, PyObject *args) {
+	char *srcname;
+	dpkg_source_note *srcn;
+
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	srcn = lookup_sourcenotetbl(self->srcsn->sources, srcname);
+	if (srcn) return Py_BuildValue("s", srcn->source->version);
+	else return Py_BuildValue("");
+}
+static PyObject *dpkgsrcsn_getfield(dpkgsrcsn *self, PyObject *args) {
+	char *srcname, *field;
+	dpkg_source_note *srcn;
+	int i;
+	dpkg_paragraph *para;
+
+	if (!PyArg_ParseTuple(args, "ss", &srcname, &field)) return NULL;
+	srcn = lookup_sourcenotetbl(self->srcsn->sources, srcname);
+	if (!srcn) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid source package");
+		return NULL;
+	}
+	para = srcn->source->details;
+	if (para) {
+		for (i = 0; i < para->n_entries; i++) {
+			if (strcasecmp(para->entry[i].name, field) == 0) {
+				return Py_BuildValue("s", para->entry[i].value);
+			}
+		}
+	}
+	return Py_BuildValue("");
+}
+static PyObject *dpkgsrcsn_ispresent(dpkgsrcsn *self, PyObject *args) {
+	char *srcname;
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	if (lookup_sourcenotetbl(self->srcsn->sources, srcname)) {
+		return Py_BuildValue("i", 1);
+	} else {
+		return Py_BuildValue("i", 0);
+	}
+}
+
+static PyObject *dpkgsrcsn_isfake(dpkgsrcsn *self, PyObject *args) {
+	char *srcname;
+	dpkg_source_note *srcn;
+
+	if (!PyArg_ParseTuple(args, "s", &srcname)) return NULL;
+	srcn = lookup_sourcenotetbl(self->srcsn->sources, srcname);
+	if (srcn) return Py_BuildValue("i", srcn->source->fake);
+	else return Py_BuildValue("");
+}
+
+static PyObject *dpkgsrcsn_binaries(dpkgsrcsn *self, PyObject *args) {
+	char *srcname, *arch;
+	int archnum;
+	dpkg_source_note *srcn;
+	PyObject *res;
+	packagelist *p;
+
+	if (!PyArg_ParseTuple(args, "ss", &srcname, &arch)) return NULL;
+
+	for (archnum = 0; archnum < self->srcsn->n_arches; archnum++) {
+		if (strcmp(arch, self->srcsn->archname[archnum]) == 0) break;
+	}
+	if (archnum == self->srcsn->n_arches) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid architecture");
+		return NULL;
+	}
+
+	srcn = lookup_sourcenotetbl(self->srcsn->sources, srcname);
+	if (srcn == NULL) {
+		PyErr_SetString(PyExc_ValueError, "Not a valid source package");
+		return NULL;
+	}
+
+	MAKE_PY_LIST(res, p = srcn->binaries[archnum], p, p = p->next,
+		     ("s", p->value->package)
+		    );
+	return res;
+}
+
+static PyObject *dpkgsrcsn_getattr(dpkgsrcsn *self, char *name) {
+	static struct PyMethodDef dpkgsrcsn_methods[] = {
+		{ "remove_source", (binaryfunc) dpkgsrcsn_removesource, 
+			METH_VARARGS, NULL },
+		{ "upgrade_source", (binaryfunc) dpkgsrcsn_upgradesource, 
+			METH_VARARGS, NULL },
+		{ "upgrade_arch", (binaryfunc) dpkgsrcsn_upgradearch, 
+			METH_VARARGS, NULL },
+
+		{ "undo_change", (binaryfunc) dpkgsrcsn_undochange, 
+			METH_VARARGS, NULL },
+		{ "commit_changes", (binaryfunc) dpkgsrcsn_commitchanges, 
+			METH_VARARGS, NULL },
+
+		{ "write_notes", (binaryfunc) dpkgsrcsn_writenotes, 
+			METH_VARARGS, NULL },
+
+		{ "Packages", (binaryfunc) dpkgsrcsn_packages, 
+			METH_VARARGS, NULL },
+
+		{ "get_version", (binaryfunc) dpkgsrcsn_getversion, 
+			METH_VARARGS, NULL },
+		{ "get_field", (binaryfunc) dpkgsrcsn_getfield, 
+			METH_VARARGS, NULL },
+		{ "is_present", (binaryfunc) dpkgsrcsn_ispresent, 
+			METH_VARARGS, NULL },
+		{ "is_fake", (binaryfunc) dpkgsrcsn_isfake, 
+			METH_VARARGS, NULL },
+		{ "binaries", (binaryfunc) dpkgsrcsn_binaries, 
+			METH_VARARGS, NULL },
+		{ NULL, NULL, 0, NULL }
+	};
+
+	if (strcmp(name, "arches") == 0) {
+		PyObject *arches;
+		int i;
+		MAKE_PY_LIST(arches, i = 0, i < self->srcsn->n_arches, i++, 
+			     ("s", self->srcsn->archname[i])
+			    );
+		return arches;
+	} else if (strcmp(name, "sources") == 0) {
+		PyObject *sources;
+		sourcenotetbl_iter it;
+		MAKE_PY_LIST(sources, 
+		             it = first_sourcenotetbl(self->srcsn->sources),
+		             !done_sourcenotetbl(it),
+			     it = next_sourcenotetbl(it),
+			     ("s", it.k)
+			    );
+		return sources;
+	} else if (strcmp(name, "can_undo") == 0) {
+		if (can_undo(self->srcsn)) {
+			return Py_BuildValue("i", 1);
+		} else {
+			return Py_BuildValue("");
+		}
+	}
+
+	return Py_FindMethod(dpkgsrcsn_methods, (PyObject *)self, name);
+}
+
+static PyTypeObject SourcesNote_Type = {
+	PyObject_HEAD_INIT(&PyType_Type)
+
+	0,                     /* ob_size (0) */
+	"SourcesNote",         /* type name */
+	sizeof(dpkgsrcsn), /* basicsize */
+	0,                     /* itemsize (0) */
+
+	(destructor)  dpkgsrcsn_dealloc,
+	(printfunc)   0,
+	(getattrfunc) dpkgsrcsn_getattr,
+	(setattrfunc) 0,
+	(cmpfunc)     0,
+	(reprfunc)    0,
+
+	0,                     /* number methods */
+	0,                     /* sequence methods */
+	0,                     /* mapping methods */
+
+	(hashfunc)    0,       /* dict[x] ?? */
+	(ternaryfunc) 0,       /* x() */
+	(reprfunc)    0        /* str(x) */
+};
+
+/**************************************************************************
+ * britney.versioncmp() -- apt version compare function
+ ******************************************************/
+
+static PyObject *apt_versioncmp(PyObject *self, PyObject *args) {
+	char *l, *r;
+	int res;
+
+	(void)self; /* unused */
+
+	if (!PyArg_ParseTuple(args, "ss", &l, &r)) {
+		return NULL;
+	}
+
+	res = versioncmp(l,r);
+	return Py_BuildValue("i", res);
+}
+
+/**************************************************************************
+ * module initialisation
+ ***********************/
+
+static PyMethodDef britneymethods[] = {
+	{ "Sources", dpkgsources_new, METH_VARARGS, NULL },
+	{ "SourcesNote", dpkgsrcsn_new, METH_VARARGS, NULL },
+
+	{ "versioncmp", apt_versioncmp, METH_VARARGS, NULL },
+
+	{ NULL, NULL, 0, NULL }
+};
+
+void initbritney(void) {
+	Py_InitModule("britney", britneymethods);
+}
+

=== added file 'update_out/checklib.c'
--- update_out/checklib.c	1970-01-01 00:00:00 +0000
+++ update_out/checklib.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,185 @@
+#include <stdlib.h>
+#include <unistd.h>
+
+#include <assert.h>
+
+#include "dpkg.h"
+
+#if 0
+static void checknewsrc(sourcetbl *srcstbl, dpkg_source *cur, void *data) {
+    dpkg_sources *oldsrc = data;
+    dpkg_source *old;
+    old = lookup_sourcetbl(oldsrc->sources, cur->package);
+    if (old == NULL) {
+	printf("New: %s (%s)\n", cur->package, cur->version );
+    } else if (strcmp(old->version, cur->version) != 0) {
+	printf("Updated: %s (%s, was %s)\n",
+	       cur->package, cur->version, old->version );
+    } else {
+	dpkg_source *src2;
+	src2 = remove_sourcetbl(srcstbl, cur->package);
+	assert(cur == src2);
+	free_source(cur);
+    }	
+}
+
+static void checkoldsrc(sourcetbl *oldsrctbl, dpkg_source *old, void *data) {
+    dpkg_sources *src = data;
+    dpkg_source *cur;
+    (void)oldsrctbl;
+    cur = lookup_sourcetbl(src->sources, old->package);
+    if (cur == NULL) {
+	printf("Removed: %s (was %s)\n", old->package, old->version );
+    }
+}
+
+static void checkuptodate(sourcetbl *srctbl, dpkg_source *src, void *data) {
+    int i;
+    int remove;
+    ownedpackagelist **p;
+    dpkg_sources *srcs = data;
+
+    (void)srctbl;
+
+    remove = 0;
+    for (i = 0; i < srcs->n_arches; i++) {
+	p = &src->packages[i];
+	while(*p != NULL) {
+	    if (strcmp((*p)->value->source_ver, src->version) != 0) {
+		if (cmpversions((*p)->value->source_ver, GT, src->version)) {
+		    printf("ALERT: old source: ");
+		} else {
+		    printf("WARN: out of date: ");
+		}
+		printf("%s %s: %s binary: %s %s from %s\n",
+		       src->package, src->version, srcs->archname[i],
+		       (*p)->value->package, (*p)->value->version,
+		       (*p)->value->source_ver);
+		delete_ownedpackagelist(p);
+	    } else {
+		p = &(*p)->next;
+	    }
+	}
+	if (src->packages[i] == NULL) {
+	    printf("%s missing uptodate binaries for %s\n",
+		   src->package, srcs->archname[i]);
+	    remove = 1;
+	}
+    }
+    if (remove) {
+	dpkg_source *src2;
+	src2 = remove_sourcetbl(srcs->sources, src->package);
+	assert(src == src2);
+	free_source(src);
+    }
+}
+#endif 
+
+static void upgrade(sourcetbl *srctbl, dpkg_source *src, void *data) {
+    static int i = 0;
+    dpkg_sources_note *srcsn = data;
+    (void)srctbl;
+    i++; i %= 1000;
+    if (can_undo(srcsn)) {
+        if (i % 29 == 1 || i % 31 == 1 || i % 7 == 5)
+            undo_change(srcsn);
+        if (i % 33 == 0) commit_changes(srcsn);
+    }
+    upgrade_source(data, src);
+}
+
+static void checkpkgs(packagetbl *pkgtbl, dpkg_collected_package *cpkg, 
+		      void *data) 
+{
+    dpkg_packages *pkgs = data;
+    assert(pkgs->packages == pkgtbl);
+    printf("Trying %s (%s, %s)\n", cpkg->pkg->package, cpkg->pkg->version, pkgs->arch);
+    if (!checkinstallable2(pkgs, cpkg->pkg->package)) {
+ 	printf("Package: %s (%s, %s) is uninstallable\n",
+	       cpkg->pkg->package, cpkg->pkg->version, pkgs->arch);
+    }
+}
+
+void print_memblock_summary(void);
+
+int main(int argc, char **argv) {
+    dpkg_sources *src = NULL, *oldsrc = NULL;
+    dpkg_sources_note *srcsn;
+    dpkg_source *srcpkg;
+    dpkg_packages *pkgs[10];
+    int n_pkgs;
+    int i,j;
+    int reps;
+    
+    if (argc < 3) {
+	printf("Usage: %s <reps> <arch>...\n", argv[0]);
+	exit(EXIT_FAILURE);
+    }
+
+    reps = atoi(argv[1]);
+    if (reps < 1) {
+        printf("reps must be >= 1\n");
+        exit(EXIT_FAILURE);
+    }
+
+    src = read_directory("cur", argc - 2, argv + 2);
+    oldsrc = read_directory("old", argc - 2, argv + 2); 
+    srcsn = new_sources_note(argc - 2, argv + 2);
+
+    printf("FINISHED LOADING\n"); fflush(stdout); /* sleep(5); */
+
+#if 0
+    iterate_sourcetbl(oldsrc->sources, checkoldsrc, src);
+
+    printf("FIRST\n");
+    iterate_sourcetbl(src->sources, checkuptodate, src);
+    printf("SECOND\n");
+    iterate_sourcetbl(src->sources, checkuptodate, src);
+    printf("END\n");
+
+    iterate_sourcetbl(src->sources, checknewsrc, oldsrc);
+#endif
+
+    n_pkgs = 0;
+    for (i = argc - 1; i > 1; i--) {
+	pkgs[n_pkgs++] = get_architecture(oldsrc, argv[i]);
+    }
+    for (j = 0; j < reps; j++) {
+	printf("Round %d/%d starting...\n", j + 1, reps);
+        for (i = 0; i < n_pkgs; i++) {
+            iterate_packagetbl(pkgs[i]->packages, checkpkgs, pkgs[i]);
+        }
+	printf("Round %d ended.\n", j+1);
+    }
+    iterate_sourcetbl(src->sources, upgrade, srcsn);
+    iterate_sourcetbl(oldsrc->sources, upgrade, srcsn);
+
+    for (i = 0; i < n_pkgs; i++) {
+	free_packages(pkgs[i]);
+    }
+
+    srcpkg = lookup_sourcetbl(oldsrc->sources, "omirr");
+    if (srcpkg != NULL) {
+        printf("Adding old\n");
+	upgrade_source(srcsn, srcpkg);
+    }
+    srcpkg = lookup_sourcetbl(src->sources, "omirr");
+    if (srcpkg != NULL) {
+        printf("Adding cur\n");
+	upgrade_source(srcsn, srcpkg);
+    }
+
+    printf("FINISHED PROCESSING\n"); fflush(stdout); /* sleep(5); */
+
+    write_directory("out", oldsrc);
+
+    printf("FINISHED WRITING\n"); fflush(stdout); /* sleep(5); */
+
+    free_sources_note(srcsn);
+    free_sources(src);
+    free_sources(oldsrc);
+
+    DEBUG_ONLY( print_memblock_summary(); )
+
+    return 0;
+}

=== added file 'update_out/dpkg-lib.cpp'
--- update_out/dpkg-lib.cpp	1970-01-01 00:00:00 +0000
+++ update_out/dpkg-lib.cpp	2020-03-12 14:58:23 +0000
@@ -0,0 +1,34 @@
+
+#include <apt-pkg/debversion.h>
+
+extern "C" {
+
+#include "dpkg.h"
+
+int versioncmp(char *left, char *right) { 
+	return debVS.CmpVersion(left, right);
+}
+
+int cmpversions(char *left, int op, char *right) {
+	int i = debVS.CmpVersion(left, right);
+
+	switch(op) {
+		case dr_LT:    return i <  0;
+		case dr_LTEQ:  return i <= 0;
+		case dr_EQ:    return i == 0;
+		case dr_GTEQ:  return i >= 0;
+		case dr_GT:    return i >  0;
+	}
+	return 0;
+}
+
+}
+
+#ifdef TESTBIN
+int main(int argc, char **argv) {
+	if (argc != 3) { printf("Usage: %s <ver> <ver>\n", argv[0]); exit(1); }
+
+	printf("%d\n", versioncmp(argv[1], argv[2]));
+	return 0;
+}
+#endif

=== added file 'update_out/dpkg.c'
--- update_out/dpkg.c	1970-01-01 00:00:00 +0000
+++ update_out/dpkg.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,2013 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <ctype.h>
+#include <errno.h>
+
+#include "dpkg.h"
+#include "memory.h"
+
+
+// enlarge this is britney has issues to parse packages
+#define SIZEOFHASHMAP 16
+
+#define insert_packagenamelist(x,y) insert_l_packagenamelist(x,y,__LINE__)
+
+static void free_dependency(dependency *dep);
+static void free_package(dpkg_package *pkg);
+static void free_collected_package(dpkg_collected_package *pkg);
+static dpkg_paragraph *read_paragraph( FILE *f );
+static dpkg_package *read_package( FILE *f );
+static collpackagelist **get_matching_low(collpackagelist **addto, 
+		                          dpkg_packages *pkgs, dependency *dep, int line);
+static collpackagelist *get_matching(dpkg_packages *pkgs, deplist *depopts, int line);
+static deplist *read_dep_and(char *buf);
+static deplistlist *read_dep_andor(char *buf);
+static ownedpackagenamelist *read_packagenames(char *buf);
+static dpkg_sources *read_sources_file(char *filename, int n_arches);
+static dpkg_source *new_source(dpkg_sources *owner);
+static dpkg_source *read_source(FILE *f, dpkg_sources *owner);
+static deplist *read_deplist(char **buf, char sep, char end);
+static dependency *read_dependency(char **buf, char *end);
+static void add_virtualpackage(virtualpkgtbl *vpkgs, char *package, 
+                               char *version, dpkg_collected_package *cpkg);
+static void remove_virtualpackage(virtualpkgtbl *vpkgs, char *pkgname,
+			          dpkg_collected_package *cpkg);
+static char *read_packagename(char **buf, char *end);
+static char *read_until_char(char **buf, char *end);
+static void add_package(dpkg_packages *pkgs, dpkg_package *pkg);
+static void remove_package(dpkg_packages *pkgs, dpkg_collected_package *pkg);
+static dpkg_source_note *copy_source_note(dpkg_source_note *srcn);
+
+#if 0
+static inline void *bm(size_t s, int n) { void *res = block_malloc(s); fprintf(stderr, "ALLOCED: %d %p %lu\n", n, res, s); return res; }
+static inline void bf(void *p, size_t s, int n) { block_free(p,s); fprintf(stderr, "FREED: %d %p %lu\n", n, p, s); }
+
+#define block_malloc(s) bm(s,__LINE__)
+#define block_free(p,s) bf(p,s,__LINE__)
+#endif
+
+#define block_malloc(s) block_malloc2(s, __LINE__)
+
+static char *priorities[] = {
+    "required",
+    "important",
+    "standard",
+    "optional",
+    "extra",
+    NULL
+};
+
+static char *dependency_title[] = {
+    "Pre-Depends", "Depends", "Recommends", "Suggests", NULL
+};
+
+static int dependency_counts[] = { 1, 1, 0, 0 };
+
+char *dependency_relation_sym[] = {"*", "<<", "<=", "=", ">=", ">>"};
+
+#define SMB_SIZE (1<<22)
+struct stringmemblock {
+	struct stringmemblock *next;
+	size_t last;
+	char mem[SMB_SIZE];
+};
+static struct stringmemblock *stringmemory = NULL;
+static int stringmemorycount = 0;
+static const unsigned long stringmemblocksizekib = (unsigned long) sizeof(struct stringmemblock) / 1024;
+
+char *my_strdup(char *foo) {
+	struct stringmemblock *which;
+	size_t len;
+
+	if (!foo) return NULL;
+
+	len = strlen(foo) + 1;
+
+	if (len > SMB_SIZE) return strdup(foo);
+
+	for (which = stringmemory; which; which = which->next) {
+		if (SMB_SIZE - which->last > len + 1) {
+			break;
+		}
+	}
+	if (!which) {
+		which = malloc(sizeof(struct stringmemblock));
+		if (!which) return NULL;
+		MDEBUG1_ONLY(fprintf(stderr, 
+			"ALLOC: string memblock %d (%lu KiB, %lu KiB total)\n", 
+			stringmemorycount, stringmemblocksizekib,
+                        (stringmemorycount+1) * stringmemblocksizekib));
+		memset(which->mem, 0, SMB_SIZE);
+		which->last = 0;
+		which->next = stringmemory;
+		stringmemory = which;
+		stringmemorycount++;
+	}
+	strcpy(&which->mem[which->last], foo);
+	foo = &which->mem[which->last];
+	which->last += len;
+	return foo;
+}
+
+char *my_rep_strdup(char *foo) {
+    static char *repeats[1000] = {0};
+    int i;
+
+    for (i = 0; i < 1000; i++) {
+	if (repeats[i] == NULL) {
+	    DEBUG_ONLY(fprintf(stderr, "REPEAT NR %d\n", i+1); )
+	    return repeats[i] = my_strdup(foo);
+	}
+	if (strcmp(repeats[i], foo) == 0) return repeats[i];
+    }
+
+    return my_strdup(foo);
+}
+
+/* DIE **/
+
+static void die(char *orig_msg) {
+        char *msg = my_strdup(orig_msg);
+        if (*msg && msg[strlen(msg)-1] == ':') {
+                msg[strlen(msg)-1] = '\0';
+                perror(msg);
+        } else {
+                printf("%s\n", msg);
+        }
+        abort();
+}
+
+/*************************************************************************
+ * Dpkg Control/Packages/etc Operations
+ */
+
+static dpkg_paragraph *read_paragraph( FILE *f ) {
+    dpkg_paragraph *result;
+    static int line_size = 0;
+    static char *line = NULL;
+    char *pch;
+   
+    dpkg_entry *c_entry = NULL;
+    char *c_value = NULL;
+   
+    if (line == NULL) {
+        line_size = 10;
+	line = malloc(line_size);
+	if (line == NULL) die("read_paragraph alloc 0:");
+    }
+ 
+    result = block_malloc( sizeof(dpkg_paragraph) );
+    if (result == NULL) die("read_paragraph alloc 1:");
+    
+    result->n_entries = 0;
+    result->entry = NULL;
+    result->n_allocated = 0;
+    
+    while(fgets(line, line_size, f)) {
+	while (!feof(f) && *line && line[strlen(line)-1] != '\n') {
+	    line = realloc(line, line_size * 2);
+	    if (!line) die("read_paragraph realloc:");
+	    fgets(line + strlen(line), line_size, f);
+	    line_size *= 2;
+        }
+
+	if (line[0] == '\n') break;
+	
+	if (isspace(line[0])) {
+	    if (c_value == NULL)
+		die("read_paragraph early spaces");
+
+	    if (c_entry == NULL) {
+		/* no need to bother */
+	    } else {
+	        /* extend the line */
+	        c_value = realloc(c_value, strlen(c_value) + strlen(line) + 1);
+	        if (c_value == NULL)
+		    die("read_paragraph realloc c_value:");
+	    
+	        strcat(c_value, line);
+	    }
+	} else {
+	    if (c_entry) {
+		    c_entry->value = my_strdup(c_value);
+		    c_value[0] = '\0';
+		    free(c_value); 
+		    c_value = NULL;
+	    } else if (c_value) {
+		    free(c_value);
+		    c_value = NULL;
+	    }
+	    pch = strchr(line, ':');
+	    if (pch == NULL) {
+                fprintf(stderr, "the line was: \"%s\"\n", line);
+		die("read_paragraph: no colon");
+            }
+	    
+	    *pch = '\0';
+	    while(isspace(*++pch));
+
+	    if (strcmp(line, "Description") == 0) {
+		c_value = strdup(pch);
+		c_entry = NULL;
+	    } else {
+	        assert(result->n_entries <= result->n_allocated);
+	        if (result->n_entries >= result->n_allocated) {
+		    result->n_allocated += 10;
+		    result->entry = realloc( result->entry,
+					     sizeof(dpkg_entry)
+					     * result->n_allocated);
+		    if (result->entry == NULL)
+		        die("read_paragraph realloc entry:");
+	        }
+
+	        c_entry = &result->entry[result->n_entries++];
+	        c_entry->name = my_rep_strdup(line);
+	        c_value = strdup(pch);
+	    }
+	}
+    }
+
+    if (c_entry) {
+	c_entry->value = my_strdup(c_value);
+	c_value[0] = '\0';
+	free(c_value);
+	c_value = NULL;
+    }
+
+    if (result->n_entries == 0) {
+	if (result->entry) free(result->entry);
+	block_free(result, sizeof(dpkg_paragraph));
+	return NULL;
+    } else {
+        result->entry = realloc(result->entry, 
+	    sizeof(result->entry[0]) * result->n_entries);
+	result->n_allocated = result->n_entries;
+    }
+
+    return result;
+}
+
+static void write_paragraph(FILE *f, dpkg_paragraph *p) {
+    int i;
+
+    for (i = 0; i < p->n_entries; i++) {
+	fprintf(f, "%s: %s", p->entry[i].name, p->entry[i].value);
+    }
+    fprintf(f, "\n");
+}
+
+
+static void free_paragraph(dpkg_paragraph *p) {
+    int i;
+    
+    if (p == NULL) return;
+    
+    for (i = 0; i < p->n_entries; i++) {
+	/* block_free(p->entry[i].name); */
+	/* block_free(p->entry[i].value); */
+    }
+    free(p->entry);
+    block_free(p, sizeof(dpkg_paragraph));
+}
+
+/*************************************************************************
+ * Basic Package Operations
+ */
+
+static dpkg_package *new_package(void) {
+    dpkg_package *result;
+    
+    result = block_malloc(sizeof(dpkg_package));
+    if (result == NULL) die("new_package alloc:");
+    
+    result->package    = NULL;
+    result->version    = NULL;
+
+    result->priority   = 0;
+    result->arch_all   = 0;
+    
+    result->source     = NULL;
+    result->source_ver = NULL;
+    
+    result->depends[0] = NULL;
+    result->depends[1] = NULL;
+    result->depends[2] = NULL;
+    result->depends[3] = NULL;
+    
+    result->conflicts  = NULL;
+    
+    result->provides   = NULL;
+    result->details    = NULL;
+    
+    return result;
+}
+
+static dpkg_collected_package *new_collected_package(dpkg_package *pkg) {
+    dpkg_collected_package *result;
+
+    result = block_malloc(sizeof(dpkg_collected_package));
+    if (result == NULL) die("new_collected_package alloc:");
+
+    result->pkg = pkg;
+
+    result->installed  = 0;
+    result->conflicted = 0;
+
+    result->installable = UNKNOWN;
+    result->mayaffect = NULL;
+
+    return result;
+}
+
+static void free_collected_package(dpkg_collected_package *cpkg) {
+    if (cpkg == NULL) return;
+    cpkg->pkg = NULL;
+    free_packagenamelist(cpkg->mayaffect);
+    cpkg->mayaffect = NULL;
+    block_free(cpkg, sizeof(dpkg_collected_package));
+}   
+
+static void free_package(dpkg_package *pkg) {
+    int i;
+    if (pkg == NULL) return;
+    
+    /* block_free(pkg->package);
+     * block_free(pkg->version);
+     * block_free(pkg->source);
+     * block_free(pkg->source_ver); */
+
+    for (i = 0; i < 4; i++)
+	free_deplistlist(pkg->depends[i]);
+
+    free_deplist(pkg->conflicts);
+    free_ownedpackagenamelist(pkg->provides);
+    
+    free_paragraph(pkg->details);
+    
+    block_free(pkg, sizeof(dpkg_package));
+}
+
+static dpkg_package *read_package( FILE *f ) {
+    dpkg_package *result;
+    dpkg_paragraph *para;
+    dpkg_entry *e;
+    int i;
+
+    para = read_paragraph(f);
+    if (para == NULL) return NULL;
+    
+    result = new_package();
+    result->details = para;
+    
+    for (e = &para->entry[0]; e < &para->entry[para->n_entries]; e++) {
+	if (strcasecmp("Package", e->name) == 0) {
+	    result->package = my_strdup(e->value);
+	    if (result->package == NULL)
+		die("read_package my_strdup:");
+	    result->package[strlen(result->package)-1] = '\0';
+	}
+	
+	if (strcasecmp("Version", e->name) == 0) {
+	    result->version = my_strdup(e->value);
+	    if (result->version == NULL)
+		die("read_package my_strdup:");
+	    result->version[strlen(result->version)-1] = '\0';
+	}
+	
+	if (strcasecmp("Priority", e->name) == 0) {
+	    int i;
+	    for (i = 0; priorities[i] != NULL; i++) {
+		if (strcasecmp(priorities[i], e->value))
+		    break;
+	    }
+	    result->priority = i;
+	    if (priorities[i] == NULL) {
+		die("read_package: unknown priority");
+	    }
+	}
+
+	if (strcasecmp("Architecture", e->name) == 0) {
+	    if (strncasecmp(e->value, "all", 3) == 0) {
+		if (!e->value[3] || isspace(e->value[3])) {
+	            result->arch_all = 1;
+		}
+	    }
+	}
+	
+	for (i = 0; dependency_title[i] != NULL; i++) {
+	    if (strcasecmp(dependency_title[i], e->name) == 0) {
+		result->depends[i] = read_dep_andor(e->value);
+	    }
+	}
+	
+	/*
+	 * Support Breaks by treating them as Conflicts.
+	 */
+	if (strcasecmp("Conflicts", e->name) == 0
+	    || strcasecmp("Breaks", e->name) == 0) {
+	    deplist *pkgs = read_dep_and(e->value);
+	    if (result->conflicts == NULL)
+		result->conflicts = pkgs;
+	    else {
+		// TODO(dato): create extend_deplist() in templates.h?
+		deplist *end = result->conflicts;
+		while (end->next)
+		    end = end->next;
+		end->next = pkgs;
+	    }
+	}
+	
+	if (strcasecmp("Provides", e->name) == 0)
+	    result->provides = read_packagenames(e->value);
+	
+	if (strcasecmp("source", e->name) == 0) {
+	    char *pch = e->value;
+	    
+	    assert(result->source == NULL);
+	    assert(result->source_ver == NULL);
+	    
+	    result->source = my_strdup(read_packagename(&pch, "("));
+	    if (result->source == NULL)
+		die("read_package: bad source header");
+	    
+	    while(isspace(*pch)) pch++;
+	    if (*pch == '(') {
+		pch++;
+		result->source_ver = my_strdup(read_until_char(&pch, ")"));
+		if (result->source_ver == NULL)
+		    die("read_package: bad source version");
+		while(isspace(*pch)) pch++;
+		if (*pch != ')')
+		    die("read_package: unterminated ver");
+	    }
+	}
+    }
+
+    if (result->source == NULL) {
+	assert(result->source_ver == NULL);
+	result->source = my_strdup(result->package);
+    }
+    if (result->source_ver == NULL) {
+	result->source_ver = my_strdup(result->version);
+    }
+    
+    return result;
+}
+
+static void freesize(void *p, size_t s) { (void)s; free(p); }
+
+LIST_IMPL(deplist, dependency*, free_dependency, block_malloc, block_free);
+LIST_IMPL(deplistlist, deplist*, free_deplist, block_malloc, block_free);
+
+LIST_IMPLX(packagenamelist, char*, KEEP(char*));
+
+LIST_IMPL(ownedpackagenamelist, char*, KEEP(char*), block_malloc, block_free);
+	/* ownedpackagenamelist stores the packagename in the string store */
+
+static int packagecmp(dpkg_package *l, dpkg_package *r) {
+    if (l->priority < r->priority) return -1;
+    if (l->priority > r->priority) return +1;
+    return strcmp(l->package, r->package);
+}
+
+/* container for owned pkgs */
+LIST_IMPL(ownedpackagelist, dpkg_package *, free_package, 
+		block_malloc, block_free);
+
+/* container for existing pkgs */
+LIST_IMPL(packagelist, dpkg_package *, KEEP(dpkg_package *), block_malloc, block_free);
+
+LIST_IMPLX(collpackagelist, dpkg_collected_package *, 
+	  KEEP(dpkg_collected_package *))
+#define insert_collpackagelist(x,y) insert_l_collpackagelist(x,y,__LINE__)
+
+/*************************************************************************
+ * Operations on distributions (collections of packages)
+ */
+
+dpkg_packages *new_packages(char *arch) {
+    dpkg_packages *result;
+
+    result = block_malloc(sizeof(dpkg_packages));
+    if (result == NULL) die("new_packages alloc:");
+    
+    result->arch = my_strdup(arch);
+    result->packages = new_packagetbl();
+    result->virtualpkgs = new_virtualpkgtbl();
+
+    return result;
+}
+
+static void add_package(dpkg_packages *pkgs, dpkg_package *pkg) 
+{
+    ownedpackagenamelist *v;
+    dpkg_collected_package *cpkg;
+
+    if (lookup_packagetbl(pkgs->packages, pkg->package) != NULL) 
+	return;
+
+    cpkg = new_collected_package(pkg);
+
+    add_packagetbl(pkgs->packages, cpkg->pkg->package, cpkg);
+	
+    add_virtualpackage(pkgs->virtualpkgs, cpkg->pkg->package, 
+		       cpkg->pkg->version, cpkg);
+    for (v = cpkg->pkg->provides; v != NULL; v = v->next) {
+	add_virtualpackage(pkgs->virtualpkgs, v->value, NULL, cpkg);
+    }
+}
+
+static void remove_package(dpkg_packages *pkgs, dpkg_collected_package *cpkg) {
+    ownedpackagenamelist *v;
+    packagenamelist *aff;
+    dpkg_collected_package *p;
+
+    for (aff = cpkg->mayaffect; aff != NULL; aff = aff->next) {
+	p = lookup_packagetbl(pkgs->packages, aff->value);
+	if (p == NULL) continue;
+	p->installable = UNKNOWN;
+    }
+
+    p = remove_packagetbl(pkgs->packages, cpkg->pkg->package);
+    if (p != cpkg) return;
+	
+    remove_virtualpackage(pkgs->virtualpkgs, cpkg->pkg->package, cpkg);
+    for (v = cpkg->pkg->provides; v != NULL; v = v->next) {
+	remove_virtualpackage(pkgs->virtualpkgs, v->value, cpkg);
+    }
+
+    free_collected_package(cpkg);
+}
+
+dpkg_packages *get_architecture(dpkg_sources *srcs, char *arch) {
+    int i, arch_index;
+    dpkg_packages *result;
+    sourcetbl_iter srci;
+    ownedpackagelist *p;
+    
+    arch_index = -1;
+    for (i = 0; i < srcs->n_arches; i++) {
+	if (strcmp(srcs->archname[i], arch) == 0) {
+	    arch_index = i;
+	    break;
+	}
+    }
+    if (arch_index == -1) die("get_architecture: unknown arch");
+    
+    result = new_packages(arch);
+
+    for (srci = first_sourcetbl(srcs->sources);
+	 !done_sourcetbl(srci);
+	 srci = next_sourcetbl(srci))
+    {
+	for (p = srci.v->packages[arch_index]; p != NULL; p = p->next) {
+	    add_package(result, p->value);
+	}
+    }
+    
+    return result;
+}
+
+void free_packages(dpkg_packages *pkgs) {
+    if (pkgs == NULL) return;
+    /* block_free(pkgs->arch); */
+    free_packagetbl(pkgs->packages);
+    free_virtualpkgtbl(pkgs->virtualpkgs);
+    block_free(pkgs, sizeof(dpkg_packages));
+}
+
+
+HASH_IMPL(packagetbl, char *, dpkg_collected_package *, SIZEOFHASHMAP, strhash, strcmp,
+	  KEEP(char*),free_collected_package);
+HASH_IMPL(virtualpkgtbl, char *, virtualpkg *, SIZEOFHASHMAP, strhash, strcmp,
+	  KEEP(char*), free_virtualpkg);
+
+/* dpkg_provision refers to memory allocated elsewhere */
+LIST_IMPL(virtualpkg, dpkg_provision, KEEP(dpkg_provision), block_malloc, block_free);
+
+static void remove_virtualpackage(virtualpkgtbl *vpkgs, char *pkgname,
+			          dpkg_collected_package *cpkg)
+{
+    virtualpkg *list;
+    virtualpkg **where;
+    list = lookup_virtualpkgtbl(vpkgs, pkgname);
+    assert(list != NULL);
+
+    where = &list;
+    while((*where)->value.pkg != cpkg) {
+	where = &(*where)->next;
+	assert(*where != NULL);
+    }
+    
+    delete_virtualpkg(where);
+
+    if (list == NULL) {
+	remove_virtualpkgtbl(vpkgs, pkgname);
+    } else {
+	replace_virtualpkgtbl(vpkgs, pkgname, list);
+    }
+}
+
+static void add_virtualpackage(virtualpkgtbl *vpkgs, char *package, 
+                               char *version, dpkg_collected_package *cpkg)
+{
+    dpkg_provision value;
+    virtualpkg *list, **addto;
+    int shouldreplace;
+   
+    value.pkg = cpkg;
+    value.version = version;
+    
+    list = lookup_virtualpkgtbl(vpkgs, package);
+    shouldreplace = (list != NULL);
+
+    addto = &list;
+    while (*addto != NULL
+	   && packagecmp(cpkg->pkg, (*addto)->value.pkg->pkg) >= 0) 
+    {
+	addto = &(*addto)->next;
+    }
+    insert_virtualpkg(addto, value);
+
+    if (shouldreplace) {
+	replace_virtualpkgtbl(vpkgs, package, list);
+	/* old list is included in new list, so we don't need to free */
+    } else {
+	add_virtualpkgtbl(vpkgs, package, list);
+    }
+}
+
+/*************************************************************************
+ * Parsing Helper Functions
+ */
+
+static ownedpackagenamelist *read_packagenames(char *buf) {
+    ownedpackagenamelist *result = NULL;
+    ownedpackagenamelist **addto = &result;
+
+    DEBUG_ONLY( char *strend = buf + strlen(buf); )
+    
+    char *sub;
+    
+    while ((sub = my_strdup(read_packagename(&buf, ",")))) {
+	insert_ownedpackagenamelist(addto, sub);
+	addto = &(*addto)->next;
+	
+	while(isspace(*buf)) buf++;
+	if (*buf == ',') {
+	    buf++;
+	    continue;
+	}
+	if (*buf == '\0') {
+	    break;
+	}
+	
+	die("read_packagenames no/bad seperator");
+    }
+    
+    DEBUG_ONLY( assert(buf <= strend); )
+    
+    return result;
+}
+
+static char *read_until_char(char **buf, char *end) {
+    static char *result = NULL;
+    char *start;
+    DEBUG_ONLY( char *strend = *buf + strlen(*buf); )
+    int n;
+    
+    while(isspace(**buf)) (*buf)++;
+    
+    start = *buf;
+    while (**buf && !isspace(**buf) && strchr(end, **buf) == NULL) {
+	(*buf)++;
+    }
+    
+    n = *buf - start;
+    if (n == 0) return NULL;
+    
+    result = realloc(result, n + 1);
+    if (result == NULL) die("read_until_char alloc:");
+    
+    strncpy(result, start, n);
+    result[n] = '\0';
+    
+    while(isspace(**buf)) (*buf)++;
+    
+    DEBUG_ONLY( assert(*buf <= strend); )
+    
+    return result;
+}
+
+static char *read_packagename(char **buf, char *end) {
+    return read_until_char(buf, end);
+}
+
+static deplist *read_dep_and(char *buf) {
+    return read_deplist(&buf, ',', '\0'); 
+}
+
+static deplist *read_deplist(char **buf, char sep, char end) {
+    deplist *result = NULL;
+    deplist **addto = &result;
+    
+    char separs[3] = { sep, end, '\0' };
+    
+    DEBUG_ONLY( char *strend = *buf + strlen(*buf); )
+    
+    dependency *sub;
+    
+    while ((sub = read_dependency(buf, separs))) {
+	insert_deplist(addto, sub);
+	addto = &(*addto)->next;
+	
+	while(isspace(**buf)) (*buf)++;
+	if (**buf == sep) {
+	    (*buf)++;
+	    continue;
+	}
+	if (**buf == '\0' || **buf == end) {
+	    break;
+	}
+
+	die("read_deplist no/bad seperator");
+    }
+    
+    DEBUG_ONLY( assert(*buf <= strend); )
+    
+    return result;
+}
+
+static deplistlist *read_dep_andor(char *buf) {
+    deplistlist *result = NULL;
+    deplistlist **addto = &result;
+    
+    deplist *sub;
+    
+    DEBUG_ONLY( char *strend = buf + strlen(buf); )
+    
+    while ((sub = read_deplist(&buf, '|', ','))) {
+	insert_deplistlist(addto, sub);
+	addto = &(*addto)->next;
+	
+	if (*buf == ',') buf++;
+    }
+    
+    DEBUG_ONLY( assert(buf <= strend); )
+    
+    return result;
+}
+
+static dependency *read_dependency(char **buf, char *end) {
+    dependency *dep;
+    char *name;
+    char newend[10];
+    DEBUG_ONLY( char *strend = *buf + strlen(*buf); )
+    
+    assert(strlen(end) <= 8);
+    newend[0] = '('; strcpy(newend + 1, end);
+    
+    name = my_strdup(read_until_char(buf, newend));
+    if (name == NULL) return NULL;
+    
+    dep = block_malloc(sizeof(dependency));
+    if (dep == NULL) die("read_dependency alloc 1:");
+    
+    dep->package = name;
+    
+    while(isspace(**buf)) (*buf)++;
+    
+    if (**buf != '(') {
+	dep->op = dr_NOOP;
+	dep->version = NULL;
+    } else {
+	(*buf)++;
+	while(isspace(**buf)) (*buf)++;
+	/* << , <= , = , >= , >> */
+	if (**buf == '<') {
+	    (*buf)++;
+	    if (**buf == '<') {
+		dep->op = dr_LT;
+		(*buf)++;
+	    } else if (**buf == '=') {
+		dep->op = dr_LTEQ;
+		(*buf)++;
+	    } else {
+		/* The forms `<' and `>' were used to mean earlier/later or 
+		 * equal, rather than strictly earlier/later, so they should 
+		 * not appear in new packages (though `dpkg' still supports 
+		 * them).
+		 */
+		dep->op = dr_LTEQ;
+	    }
+	} else if (**buf == '>') {
+	    (*buf)++;
+	    if (**buf == '>') {
+		dep->op = dr_GT;
+		(*buf)++;
+	    } else if (**buf == '=') {
+		dep->op = dr_GTEQ;
+		(*buf)++;
+	    } else {
+		dep->op = dr_GTEQ;
+	    }
+	} else if (**buf == '=') {
+	    dep->op = dr_EQ;
+	    (*buf)++;
+	    if (**buf == '>') {
+		dep->op = dr_GTEQ;
+		(*buf)++;
+	    } else if (**buf == '<') {
+		dep->op = dr_LTEQ;
+		(*buf)++;
+	    }
+	} else {
+	    /* treat it as an implicit = :( */
+	    dep->op = dr_EQ;
+	    /* would prefer to: die("read_dependency unknown version op"); */
+	}
+	
+	while (isspace(**buf)) (*buf)++;
+	newend[0] = ')';
+	dep->version = my_strdup(read_until_char(buf, newend));
+	while (isspace(**buf)) (*buf)++;
+	
+	if (dep->version == NULL) die("read_dependency: no version");
+	if (**buf != ')') die("read_dependency: unterminated version");
+	(*buf)++;
+    }
+    
+    DEBUG_ONLY( assert(*buf <= strend); )
+    
+    return dep;
+}
+
+static void free_dependency(dependency *dep) {
+    if (dep == NULL) return;
+    /* block_free(dep->package); */
+    /* if (dep->version) block_free(dep->version); */
+    block_free(dep, sizeof(dependency));
+}
+
+/*************************************************************************
+ * Installability Checking
+ */
+
+static collpackagelist **get_matching_low(collpackagelist **addto, 
+		                          dpkg_packages *pkgs, dependency *dep, int line)
+{
+    virtualpkg *vpkg;
+    for (vpkg = lookup_virtualpkgtbl(pkgs->virtualpkgs, dep->package);
+	 vpkg != NULL;
+	 vpkg = vpkg->next)
+    {
+	int add;
+
+	add = 0;
+	if (dep->op == dr_NOOP) {
+	    add = 1;
+	} else if (vpkg->value.version != NULL) {
+	    if (cmpversions(vpkg->value.version, dep->op, dep->version)) {
+		add = 1;
+	    }
+	}
+
+	if (add) {
+	    insert_l_collpackagelist(addto, vpkg->value.pkg, line);
+	    addto = &(*addto)->next;
+	}
+    }
+
+    return addto;
+}
+
+static collpackagelist *get_matching(dpkg_packages *pkgs, deplist *depopts, int line) {
+    collpackagelist *list = NULL;
+    collpackagelist **addto = &list;
+    
+    for(; depopts != NULL; depopts = depopts->next) {
+	addto = get_matching_low(addto, pkgs, depopts->value, line);
+    }
+
+    return list;
+}
+
+typedef struct instonelist instonelist;
+struct instonelist {
+    collpackagelist *curX;
+    collpackagelist *instoneX;
+    int expandedX;
+    struct instonelist *nextX, *prevX, *cutoffX;
+};
+
+#define I1CUR(i1)      ((i1)->curX)
+#define I1INSTONE(i1)  ((i1)->instoneX)
+#define I1CUTOFF(i1)   ((i1)->cutoffX)
+#define I1NEXT(i1)     ((i1)->nextX) /* can be modified ! */
+#define I1PREV(i1)     ((i1)->prevX)
+#define I1EXPANDED(i1) ((i1)->expandedX)
+
+static instonelist *insert_instonelist(instonelist *where, collpackagelist *instone);
+static void trim_instonelist_after(instonelist *first);
+static void free_instonelist(instonelist *l);
+
+static int max_instone_len = 0;
+static int cur_instone_len = 0;
+
+static instonelist *insert_instonelist(instonelist *old, collpackagelist *instone)
+{
+    instonelist *n = block_malloc(sizeof(instonelist));
+    if (n == NULL)
+        die("insert_instonelist alloc:");
+
+    n->curX = NULL;
+    n->instoneX = instone;
+    n->cutoffX = NULL;
+    n->nextX = (old ? old->nextX : NULL);
+    n->prevX = old;
+    n->expandedX = 0;
+
+    if (old) old->nextX = n;
+    if (n->nextX) n->nextX->prevX = n;
+
+    cur_instone_len++;
+    if (cur_instone_len > max_instone_len) max_instone_len = cur_instone_len;
+
+    return n;
+}
+
+static void trim_instonelist_after(instonelist *first) {
+    if (!first->nextX) return;
+    first->nextX->prevX = NULL;
+    free_instonelist(first->nextX);
+    first->nextX = NULL;
+}
+
+static void free_instonelist(instonelist *l) {
+    instonelist *p, *k;
+    if (!l) return;
+    for (p = l; p->nextX; p = p->nextX);
+    do {
+        k = p;
+        p = k->prevX;
+        free_collpackagelist(k->instoneX);
+        block_free(k, sizeof(instonelist));
+        cur_instone_len--;
+    } while (k != l);
+}
+
+static int caninstall(dpkg_packages *pkgs, dpkg_collected_package *cpkg) {
+    collpackagelist *conflicts;
+    collpackagelist *conf;
+    int okay;
+
+    if (cpkg->installed > 0) return 1;
+    if (cpkg->conflicted > 0) return 0;
+
+    conflicts = get_matching(pkgs, cpkg->pkg->conflicts, __LINE__);
+
+    okay = 1;
+    for (conf = conflicts; conf != NULL; conf = conf->next) {
+	if (conf->value->installed > 0) {
+	    okay = 0;
+	    break;
+	}
+    }
+    free_collpackagelist(conflicts);
+    return okay;
+}
+
+static int cur_install =0;
+static int max_install = 0;
+static int cur_conflicted = 0;
+static int max_conflicted = 0;
+
+static void install(dpkg_packages *pkgs, dpkg_collected_package *cpkg) {
+    if (cpkg->installed == 0) {
+	collpackagelist *conflicts = get_matching(pkgs, cpkg->pkg->conflicts, __LINE__);
+	collpackagelist *conf;
+	for (conf = conflicts; conf != NULL; conf = conf->next) {
+	    if (conf->value == cpkg) continue;
+	    assert(conf->value->installed == 0);
+	    conf->value->conflicted++;
+	    if (conf->value->conflicted == 1) {
+	    	cur_conflicted++;
+	    	if (max_conflicted < cur_conflicted)
+			max_conflicted = cur_conflicted;
+	    }
+	}
+	free_collpackagelist(conflicts);
+	cur_install++;
+	if (max_install < cur_install) max_install = cur_install;
+    }
+    assert(cpkg->conflicted == 0);
+    cpkg->installed++;
+}
+
+static void uninstall(dpkg_packages *pkgs, dpkg_collected_package *cpkg) {
+    assert(cpkg->installed > 0);
+    assert(cpkg->conflicted == 0);
+    cpkg->installed--;
+    if (cpkg->installed == 0) {
+	collpackagelist *conflicts = get_matching(pkgs, cpkg->pkg->conflicts, __LINE__);
+	collpackagelist *conf;
+	for (conf = conflicts; conf != NULL; conf = conf->next) {
+	    if (conf->value == cpkg) continue;
+	    assert(conf->value->installed == 0);
+	    assert(conf->value->conflicted > 0);
+	    conf->value->conflicted--;
+	    if (conf->value->conflicted == 0) cur_conflicted--;
+	}
+	free_collpackagelist(conflicts);
+	cur_install--;
+    }
+}
+
+satisfieddep *new_satisfieddep(void) {
+	satisfieddep *sd = block_malloc(sizeof(satisfieddep));
+	if (!sd) die("new_satisfieddep alloc:");
+	return sd;
+}
+
+void free_satisfieddep(satisfieddep *sd) {
+	if (!sd) return;
+	free_packagelist(sd->pkgs);
+	block_free(sd, sizeof(satisfieddep));
+}
+
+LIST_IMPL(satisfieddeplist, satisfieddep *, free_satisfieddep, block_malloc, block_free);
+
+packagelist *collpkglist2pkglist(collpackagelist *l) {
+	packagelist *r = NULL;
+	packagelist **addto = &r;
+
+	for (; l != NULL; l = l->next) {
+		insert_packagelist(addto, l->value->pkg);
+		addto = &(*addto)->next;
+	}
+
+	return r;
+}
+
+satisfieddeplist *checkunsatisfiabledeps(dpkg_packages *pkgs, 
+					 deplistlist *deps) {
+	satisfieddeplist *unsatisfiable = NULL;
+	satisfieddeplist **addto = &unsatisfiable;
+	satisfieddep *sd;
+	collpackagelist *deppkgs;
+
+	for (; deps != NULL; deps = deps->next) {
+		/* deplist *dep; */
+		/* for (dep = deps->value; dep != NULL; dep = dep->next) { */
+			sd = new_satisfieddep();
+			/* sd->dep = dep->value; */
+			sd->depl = deps->value;
+
+			deppkgs = NULL;
+			/* get_matching_low(&deppkgs, pkgs, dep->value); */
+			deppkgs = get_matching(pkgs, deps->value, __LINE__);
+			sd->pkgs = collpkglist2pkglist(deppkgs);
+			free_collpackagelist(deppkgs);
+
+			insert_satisfieddeplist(addto, sd);
+			addto = &(*addto)->next;
+		/* } */
+	}
+
+	return unsatisfiable;
+}
+
+int checkinstallable2(dpkg_packages *pkgs, char *pkgname) {
+    dpkg_collected_package *cpkg = lookup_packagetbl(pkgs->packages, pkgname);
+    collpackagelist *cpl = NULL;
+
+    if (cpkg == NULL) return 0;
+
+    insert_collpackagelist(&cpl, cpkg);
+    /* cpl gets freed in checkinstallable :-/ */
+    return checkinstallable(pkgs, cpl);
+}
+
+static void debug_checkinstallable(FILE *out, instonelist *list, 
+	instonelist *last, instonelist *pointer) 
+{
+    instonelist *l;
+    fprintf(out, "Status:");
+
+    /* codes:   | = multiple options here
+     *          @ = no options can satisfy this dep
+     *          + = dependencies that can be expanded have been
+     *          * = nothing selected yet
+     *          > = where pointer points
+     *          ^ = the cut point for where we are
+     */
+
+    for (l = list; ; l = I1NEXT(l)) {
+	fprintf(out, " ");
+	if (l == pointer)           fprintf(out, ">");
+	if (l == I1CUTOFF(pointer)) fprintf(out, "^");
+	if (I1INSTONE(l) == NULL) {
+	    fprintf(out, "@");
+	} else {
+	    if (I1INSTONE(l)->next != NULL) {
+		fprintf(out, "|");
+	    }
+	    if (I1EXPANDED(l)) {
+		fprintf(out, "+");
+	    }
+	    if (I1CUR(l) == NULL) {
+	        fprintf(out, "*%s", I1INSTONE(l)->value->pkg->package);
+	    } else {
+		fprintf(out, "%s", I1CUR(l)->value->pkg->package);
+	    }
+	}
+	if (l == last) break;
+    }
+    fprintf(out, " ###\n");
+    fflush(out);
+}    
+
+int checkinstallable(dpkg_packages *pkgs, collpackagelist *instoneof) {
+    /* We use pkg->installed, pkg->conflicted to note how many
+     * times we've used this pkg to satisfy a dependency or installed
+     * a package that conflicts with it.
+     *    Thus: pkg->installed == 0, or pkg->conflicted == 0
+     *
+     * We assume these are okay initially, aren't being played with
+     * concurrently elsewhere, and make sure they're still okay when
+     * we return.
+     */
+   
+    instonelist *list;
+    instonelist *last;
+    
+    instonelist *pointer;
+
+    unsigned long counter = 500000000;
+
+    int last_max_inst = max_install;
+    int last_max_instone = max_instone_len;
+    int last_max_conflict = max_conflicted;
+
+    {
+	collpackagelist *cpkg;
+        for (cpkg = instoneof; cpkg; cpkg = cpkg->next) {
+	    if (cpkg->value->installable == YES) {
+		free_collpackagelist(instoneof);
+		return 1;
+	    }
+	}
+    }
+    
+    list = insert_instonelist(NULL, instoneof);
+
+    last = list;
+    pointer = list;
+    
+    while(--counter > 0 && pointer) {
+	deplistlist *dep;
+	dpkg_collected_package *instpkg; /* convenient alias */
+	int i;
+
+#ifndef NDEBUG
+	{
+	    instonelist *p;
+	    for (p = list; p != pointer; p = I1NEXT(p)) {
+		assert(p != NULL);
+		assert(I1CUR(p) != NULL);
+		assert(I1CUR(p)->value != NULL);
+		assert(I1CUR(p)->value->installed > 0);
+		assert(I1CUR(p)->value->conflicted == 0);
+	    }
+	    if (I1NEXT(pointer) == NULL) {
+		assert(pointer == last);
+	    } else {
+		for (p = I1NEXT(pointer); p; p = I1NEXT(p)) {
+		    if (I1NEXT(p) == NULL) {
+ 			assert(p == last);
+		    }
+		    assert(I1CUR(p) == NULL);
+		}
+	    }
+	}
+#endif
+
+#ifdef DIAGNOSE
+        debug_checkinstallable(stdout, list, last, pointer);
+#endif
+
+	if (I1CUR(pointer) == NULL) {
+	    I1CUR(pointer) = I1INSTONE(pointer);
+	    /* try to choose an already installed package if there is one */
+	    while (I1CUR(pointer) != NULL) {
+		if (I1CUR(pointer)->value->installed != 0) {
+		    break;
+		}
+		I1CUR(pointer) = I1CUR(pointer)->next;
+	    }
+	    if (I1CUR(pointer) == NULL) {
+		I1CUR(pointer) = I1INSTONE(pointer);
+	    }
+	    assert(I1CUR(pointer) || !I1INSTONE(pointer));
+
+	    I1CUTOFF(pointer) = last;
+	} else {
+	    uninstall(pkgs, I1CUR(pointer)->value);
+	    trim_instonelist_after(I1CUTOFF(pointer));
+	    last = I1CUTOFF(pointer);
+	    
+	    if (I1CUR(pointer)->value->installed > 0) {
+		/* this dependency isn't the issue -- even doing
+		 * nothing to satisfy it (ie, using an already
+		 * installed package) doesn't do any good. So give up.  
+		 */
+		I1CUR(pointer) = NULL;
+	    } else {
+		I1CUR(pointer) = I1CUR(pointer)->next;
+	    }
+	}
+	
+	while(I1CUR(pointer) && !caninstall(pkgs, I1CUR(pointer)->value)) {
+	    I1CUR(pointer) = I1CUR(pointer)->next;
+	}
+	
+	if (I1CUR(pointer) == NULL) {
+	    if (I1PREV(pointer) == NULL) break;
+	    pointer = I1PREV(pointer);
+	    continue;
+	}
+	
+	instpkg = I1CUR(pointer)->value;
+	
+	install(pkgs, instpkg);
+	
+	assert(instpkg->installed > 0);
+	if (instpkg->installed == 1) {
+            /* if it's been installed exactly once, then this must've been
+	     * the first time it was touched, so we need to look at the 
+	     * dependencies. If it's the second or later, then we don't care 
+	     * about them.
+	     */
+
+	    /* if any of the deps can't be satisfied, don't move on */
+	    int bother = 1;
+
+	    int expanded = I1EXPANDED(pointer);
+
+	    for (i = 0; i < 4; i++) {
+		if (!dependency_counts[i]) continue;
+		for (dep = instpkg->pkg->depends[i];
+		     dep != NULL; dep = dep->next)
+		{
+		    collpackagelist *thisdep = get_matching(pkgs, dep->value, __LINE__);
+
+		    if (thisdep == NULL)  {
+			bother = 0;
+
+		    } else if (thisdep != NULL && thisdep->next == NULL) {
+			collpackagelist *x;
+
+			/* if there's only one way of fulfilling this dep,
+			 * do it "ASAP"
+			 */
+
+			/* optimisation: if thisdep == foo, but the parent
+			 * was foo|bar, then we already know "foo" is not going
+			 * to work in this combination, and we can skip it.
+			 *
+			 * This deals with cases like X deps: Y|bar, bar deps: Y
+			 * where bar is a virtual package; cf xlibs
+			 */
+			for (x = I1INSTONE(pointer); x != I1CUR(pointer); x = x->next) {
+			    if (x->value == thisdep->value) {
+			        bother = 0;
+			        break;
+			    }
+			}
+
+		        if (I1INSTONE(pointer)->next == NULL) {
+		            /* the parent of this entry essentially depends 
+			     * on this too, so we'll get it out of the way 
+			     * ASAP, to reduce the degree of exponentiation 
+			     * in bad cases.
+			     *
+			     * _However_ we only want to do this _once_ for
+			     * any particular node.
+			     */
+			    if (expanded) {
+				/* thisdep isn't used! */
+				free_collpackagelist(thisdep);
+			    } else {
+				insert_instonelist(pointer, thisdep);
+	    			I1EXPANDED(pointer) = 1;
+			    }
+			} else {
+			    insert_instonelist(I1CUTOFF(pointer), thisdep);
+			}
+			if (I1NEXT(last)) last = I1NEXT(last);
+			assert(!I1NEXT(last));
+
+		    } else {
+			/* otherwise it's a multi possibility dep, so do it
+			 * at the end
+			 */
+
+		        last = insert_instonelist(last, thisdep);
+		    }
+		}
+	    }
+	    if (!bother) {
+		/* stay where we are, and try the next possibility */
+		continue;
+	    }
+	}
+	
+	pointer = I1NEXT(pointer);
+    }
+
+    if (max_install>last_max_inst || max_instone_len>last_max_instone || max_conflicted >last_max_conflict) {
+        printf("MAX: i:%d / c:%d / i1:%d %s\n", max_install,max_conflicted,max_instone_len,instoneof->value->pkg->package);
+    }
+
+    if (counter == 0) {
+	fprintf(stderr, "AIEEE: counter overflow:");
+	assert(pointer != NULL);
+	if (I1CUR(pointer) == NULL || I1CUR(pointer)->value == NULL) {
+	    /* we're not guaranteed that pointer will make sense here */
+	    pointer = I1PREV(pointer);
+	}
+	for (; pointer != NULL; pointer = I1PREV(pointer)) {
+	    if (I1CUR(pointer) == NULL) {
+		/* should only happen at pointer, so not here */
+		fprintf(stderr, " >> eep, no packages at pointer <<");
+		continue;
+	    }
+	    if (I1CUR(pointer)->value == NULL) {
+		/* should never happen */
+		fprintf(stderr, " >> eep, no package selected <<");
+		continue;
+	    }
+	    fprintf(stderr, " %s%s", 
+		(I1INSTONE(pointer)->next == NULL ? "" : "|"),
+		I1CUR(pointer)->value->pkg->package);
+	    uninstall(pkgs, I1CUR(pointer)->value);
+	}	
+	fprintf(stderr, ".\n");
+	free_instonelist(list);
+	return 0;
+    }
+
+    if (pointer == NULL) {
+	dpkg_collected_package *cpkg = I1CUR(list)->value;
+	assert(cpkg->installable != YES);
+	cpkg->installable = YES;
+	for (pointer = last; pointer != NULL; pointer = I1PREV(pointer)) {
+	    if (I1CUR(pointer)->value->installed == 1) {
+		packagenamelist **p = &I1CUR(pointer)->value->mayaffect;
+#if 0
+		while ( *p && (*p)->value < cpkg->pkg->package ) {
+		    p = &(*p)->next;
+		}
+                if (*p == NULL || (*p)->value > cpkg->pkg->package)
+#endif
+		{
+		    insert_packagenamelist(p, cpkg->pkg->package);
+		}
+	    }
+	    uninstall(pkgs, I1CUR(pointer)->value);
+	}
+	free_instonelist(list);
+	return 1;
+    } else {
+	assert(I1CUR(list) == NULL);
+	free_instonelist(list);
+	return 0;
+    }
+}
+
+/******************/
+
+HASH_IMPL(sourcetbl, char *, dpkg_source *, SIZEOFHASHMAP, strhash, strcmp,
+	  KEEP(char*), free_source);
+
+static dpkg_sources *read_sources_file(char *filename, int n_arches) {
+    FILE *f; 
+    dpkg_sources *result;
+    dpkg_source *src;
+    int i;
+
+    f = fopen(filename, "r");
+    if (f == NULL && errno != ENOENT) {
+	die("read_sources_file: couldn't open file:");
+    }
+
+    result = block_malloc(sizeof(dpkg_sources));
+    if (result == NULL) die("read_sources_file alloc 1:");
+
+    result->n_arches = n_arches;
+    result->archname = block_malloc(sizeof(char*) * n_arches);
+    if (result->archname == NULL) die("read_sources_file alloc 2:");
+    for (i = 0; i < n_arches; i++) result->archname[i] = NULL;
+    result->unclaimedpackages = block_malloc(sizeof(ownedpackagelist*) 
+					     * n_arches);
+    if (result->unclaimedpackages == NULL) die("read_sources_file alloc 3:");
+    for (i = 0; i < n_arches; i++) result->unclaimedpackages[i] = NULL;
+
+    result->sources = new_sourcetbl();
+
+    if (f != NULL) {
+        while ((src = read_source(f, result))) {
+	    dpkg_source *old = lookup_sourcetbl(result->sources, src->package);
+	    if (old == NULL) {
+	        add_sourcetbl(result->sources, src->package, src);
+            } else {
+                if (versioncmp(old->version, src->version) || 1) {
+                    int i;
+		    old = replace_sourcetbl(result->sources, src->package, src);
+                    for (i = 0; i < old->owner->n_arches; i++) {
+	                assert(old->packages[i] == NULL);
+                    }
+		    free_source(old);
+		} else {
+                    int i;
+                    for (i = 0; i < src->owner->n_arches; i++) {
+	                assert(src->packages[i] == NULL);
+                    }
+		    free_source(src);
+		}
+	    }
+        }
+        fclose(f);
+    }
+    
+    return result;    
+}
+
+void free_sources(dpkg_sources *s) {
+    int i;
+    if (s == NULL) return;
+    free_sourcetbl(s->sources);
+    for (i = 0; i < s->n_arches; i++) {
+	/* block_free(s->archname[i]); */
+	free_ownedpackagelist(s->unclaimedpackages[i]);
+    }
+    block_free(s->archname, s->n_arches * sizeof(char*));
+    block_free(s->unclaimedpackages, s->n_arches * sizeof(ownedpackagelist*));
+    block_free(s, sizeof(dpkg_sources));
+}
+
+static dpkg_source *new_source(dpkg_sources *owner) {
+    dpkg_source *result;
+    int i;
+    
+    result = block_malloc(sizeof(dpkg_source));
+    if (result == NULL) die("new_source alloc 1:");
+    
+    result->package = NULL;
+    result->version = NULL;
+    result->details = NULL;
+    result->fake = 0;
+
+    result->owner = owner;
+    result->packages = block_malloc(sizeof(packagelist*) * owner->n_arches);
+    if (result->packages == NULL) die("new_source alloc 2:");
+    for (i = 0; i < owner->n_arches; i++) {
+	result->packages[i] = NULL;
+    }
+
+    return result;
+}
+
+static dpkg_source *read_source(FILE *f, dpkg_sources *owner) {
+    dpkg_source *result;
+    dpkg_paragraph *para;
+    dpkg_entry *e;
+
+    para = read_paragraph(f);
+    if (para == NULL) return NULL;
+    
+    result = new_source(owner);
+    result->details = para;
+    
+    for (e = &para->entry[0]; e < &para->entry[para->n_entries]; e++) {
+	if (strcmp("Package", e->name) == 0) {
+	    result->package = my_strdup(e->value);
+	    if (result->package == NULL)
+		die("read_source strdup:");
+	    result->package[strlen(result->package)-1] = '\0';
+	}
+	
+	if (strcmp("Version", e->name) == 0) {
+	    result->version = my_strdup(e->value);
+	    if (result->version == NULL)
+		die("read_source strdup:");
+	    result->version[strlen(result->version)-1] = '\0';
+	}
+    }
+
+    return result;
+}
+
+void free_source(dpkg_source *s) {
+    int i;
+    if (s == NULL) return;
+    assert(s->owner != NULL); /* shouldn't have allocated it */
+    /* block_free(s->package); */
+    /* block_free(s->version); */
+    free_paragraph(s->details);
+    for (i = 0; i < s->owner->n_arches; i++) {
+	free_ownedpackagelist(s->packages[i]);
+    }
+    block_free(s->packages, s->owner->n_arches * sizeof(ownedpackagelist*));
+    block_free(s, sizeof(dpkg_source));
+}
+
+/******************************/
+
+dpkg_sources *read_directory(char *dir, int n_arches, char *archname[]) {
+    char buf[1000];
+    dpkg_sources *srcs;
+    int i;
+    
+    snprintf(buf, 1000, "%s/Sources", dir);
+    srcs = read_sources_file(buf, n_arches);
+
+    for (i = 0; i < n_arches; i++) {
+	FILE *f;
+	dpkg_package *pkg;
+
+	srcs->archname[i] = my_strdup(archname[i]);
+
+	snprintf(buf, 1000, "%s/Packages_%s", dir, archname[i]);
+	f = fopen(buf, "r");
+	if (f == NULL && errno != ENOENT) die("load_dirctory fopen:");
+        if (f != NULL) {
+	    while ((pkg = read_package(f))) {
+	        dpkg_source *src = lookup_sourcetbl(srcs->sources, pkg->source);
+	        if (src == NULL) {
+		    src = new_source(srcs);
+		    src->fake = 1;
+		    src->package = my_strdup(pkg->source);
+		    src->version = my_strdup(pkg->source_ver);
+		    add_sourcetbl(srcs->sources, src->package, src);
+		} 
+                insert_ownedpackagelist(&src->packages[i], pkg);
+	    }
+	    fclose(f);
+	}
+    }	
+
+    return srcs;
+}
+
+void write_directory(char *dir, dpkg_sources *srcs) {
+    FILE *src;
+    FILE *archfile[100];
+    char buf[1000];
+    int i;
+    sourcetbl_iter srciter;
+
+    snprintf(buf, 1000, "%s/Sources", dir);
+    src = fopen(buf, "w");
+    if (!src) die("write_directory: Couldn't open Sources file for output");
+
+    for (i = 0; i < srcs->n_arches; i++) {
+	snprintf(buf, 1000, "%s/Packages_%s", dir, srcs->archname[i]);
+	archfile[i] = fopen(buf, "w");
+    }
+
+    for (srciter = first_sourcetbl(srcs->sources);
+	 !done_sourcetbl(srciter);
+	 srciter = next_sourcetbl(srciter))
+    {
+	ownedpackagelist *p;
+	int i;
+
+	if (!srciter.v->fake) 
+		write_paragraph(src, srciter.v->details);
+	
+	for (i = 0; i < srcs->n_arches; i++) {
+	    for (p = srciter.v->packages[i]; p != NULL; p = p->next) {
+		write_paragraph(archfile[i], p->value->details);
+	    }
+	}
+    }
+    
+    fclose(src);
+    for (i = 0; i < srcs->n_arches; i++) {
+	fclose(archfile[i]);
+    }
+}
+
+/*********************/
+
+HASH_IMPL(sourcenotetbl, char *, dpkg_source_note *, SIZEOFHASHMAP, strhash, strcmp,
+	  KEEP(char*), free_source_note);
+
+dpkg_source_note *new_source_note(dpkg_source *src, int n_arches) {
+    dpkg_source_note *result = block_malloc(sizeof(dpkg_source_note));
+    int i;
+
+    if (result == NULL) die("new_source_note alloc 1:");    
+    result->source = src;
+    result->n_arches = n_arches;
+    result->binaries = block_malloc(n_arches * sizeof(packagelist*));
+    if (result->binaries == NULL) die("new_source_note alloc 2:");
+    for (i = 0; i < n_arches; i++) {
+	result->binaries[i] = NULL;
+    }
+    return result;
+} 
+
+void free_source_note(dpkg_source_note *srcn) {
+    int i;
+    
+    if (srcn == NULL) return;
+
+    if (srcn->binaries != NULL) {
+	for (i = 0; i < srcn->n_arches; i++) {
+	    free_packagelist(srcn->binaries[i]);
+	}
+	block_free(srcn->binaries, sizeof(packagelist*) * srcn->n_arches);
+    }
+    block_free(srcn, sizeof(dpkg_source_note));
+}    
+
+#ifdef DEBUG
+static int is_sources_note(dpkg_sources_note *srcsn) {
+    int i;
+
+    assert(srcsn != NULL);
+    assert(srcsn->magic == 0xa1eebabe);
+    assert(srcsn->pkgs != NULL);
+    for (i = 0; i < srcsn->n_arches; i++) {
+        assert(srcsn->pkgs[i] != NULL && srcsn->archname[i] != NULL);
+        assert(strcmp(srcsn->archname[i], srcsn->pkgs[i]->arch) == 0);
+    }
+
+    return 1;
+}
+#endif
+
+dpkg_sources_note *new_sources_note(int n_arches, char **archname) {
+    dpkg_sources_note *result = block_malloc(sizeof(dpkg_sources_note));
+    int i;
+
+    if (result == NULL) die("new_sources_note alloc 1:");
+    result->magic = 0xA1EEBABE;
+    result->sources = new_sourcenotetbl();
+    result->pkgs = block_malloc(n_arches * sizeof(dpkg_packages*));
+    if (result->pkgs == NULL) die("new_sources_note alloc 2:");
+    result->archname = block_malloc(n_arches * sizeof(char*));
+    if (result->archname == NULL) die("new_sources_note alloc 3:");
+
+    result->n_arches = n_arches;
+    for (i = 0; i < n_arches; i++) {
+	result->archname[i] = my_strdup(archname[i]);
+	result->pkgs[i] = new_packages(result->archname[i]);
+    }
+    result->undo = NULL;
+    return result;
+}
+
+void free_sources_note(dpkg_sources_note *srcsn) {
+    int i;
+    if (srcsn == NULL) return;
+    assert(is_sources_note(srcsn));
+    srcsn->magic = 0xBABEA1EE;
+    free_sourcenotetbl(srcsn->sources);
+    for (i = 0; i < srcsn->n_arches; i++) {
+	free_packages(srcsn->pkgs[i]);
+	/* block_free(srcsn->archname[i]); */
+    } 
+    block_free(srcsn->pkgs, sizeof(dpkg_packages*) * srcsn->n_arches);
+    block_free(srcsn->archname, sizeof(char*) * srcsn->n_arches);
+    free_source_note_listlist(srcsn->undo);
+    block_free(srcsn, sizeof(dpkg_sources_note));
+}
+
+static void new_op(dpkg_sources_note *srcsn) {
+    assert(is_sources_note(srcsn));
+    insert_source_note_listlist(&srcsn->undo, NULL);
+}
+static void save_source_note(dpkg_sources_note *srcsn, dpkg_source_note *srcn) {
+    source_note_list **where;
+    assert(is_sources_note(srcsn));
+    assert(srcsn->undo != NULL);
+
+    for (where = &srcsn->undo->value; 
+	 *where != NULL;
+	 where = &(*where)->next) 
+    {
+        if ((*where)->value->source == srcn->source) 
+	    return; 	/* already saved */
+    }
+
+    insert_source_note_list(where, copy_source_note(srcn));
+}
+static void save_empty_source_note(dpkg_sources_note *srcsn, dpkg_source *src) {
+    dpkg_source_note *srcn;
+    source_note_list **where;
+
+    for (where = &srcsn->undo->value; 
+	 *where != NULL; 
+	 where = &(*where)->next)
+    {
+        if ((*where)->value->source == src) 
+	    return; 	/* already saved */
+    }
+
+    srcn = block_malloc(sizeof(dpkg_source_note));
+    if (srcn == NULL) die("save_empty_source_note alloc:");
+    assert(is_sources_note(srcsn));
+
+    srcn->source = src;
+    srcn->n_arches = 0;
+    srcn->binaries = NULL;
+
+    insert_source_note_list(where, srcn);
+}
+
+typedef enum { DO_ARCHALL = 0, SKIP_ARCHALL = 1 } do_this;
+static void remove_binaries_by_arch(dpkg_sources_note *srcsn,
+				    dpkg_source_note *srcn, int archnum,
+				    do_this arch_all) 
+{
+    packagelist *p;
+    packagelist *leftovers = NULL, **addto = &leftovers;
+    assert(is_sources_note(srcsn));
+
+    assert(arch_all == SKIP_ARCHALL || NULL == lookup_sourcenotetbl(srcsn->sources,srcn->source->package));
+	/* if we're removing the entire binary, we should already have
+	 * removed the source. if we're removing just the binaries on this
+	 * arch (not arch:all) then we may be keeping the source
+	 *
+	 * really a logical XOR, I think. we don't rely on this assertion
+	 * here 
+	 */
+
+    for (p = srcn->binaries[archnum]; p != NULL; p = p->next) {
+	dpkg_collected_package *cpkg;
+	if (arch_all == SKIP_ARCHALL && p->value->arch_all) {
+		insert_packagelist(addto, p->value);
+		addto = &(*addto)->next;
+		continue;
+	}
+	cpkg = lookup_packagetbl(srcsn->pkgs[archnum]->packages,
+				 p->value->package);
+	remove_package(srcsn->pkgs[archnum], cpkg);
+    }
+    free_packagelist(srcn->binaries[archnum]);
+    srcn->binaries[archnum] = leftovers;
+}
+
+typedef enum { NOTUNDOABLE = 0, UNDOABLE = 1 } undoable;
+static void add_binaries_by_arch(dpkg_sources_note *srcsn,
+				 dpkg_source_note *srcn, dpkg_source *src,
+				 int archnum, undoable undoop, do_this arch_all) 
+{
+    ownedpackagelist *p;
+    const char *archname = srcsn->archname[archnum];
+    int origarchnum = -1;
+    int i;
+
+    assert(is_sources_note(srcsn));
+    assert(srcn == lookup_sourcenotetbl(srcsn->sources,srcn->source->package));
+    for (i = 0; i < src->owner->n_arches; i++) {
+	if (strcmp(archname, src->owner->archname[i]) == 0) {
+	    origarchnum = i;
+	    break;
+	}
+    }
+    if (origarchnum == -1) return; /* nothing to add, no biggie */
+
+    for (p = src->packages[origarchnum]; p != NULL; p = p->next) {
+	dpkg_collected_package *cpkg;
+
+	if (arch_all == SKIP_ARCHALL && p->value->arch_all) continue;
+
+	if ((cpkg = lookup_packagetbl(srcsn->pkgs[archnum]->packages, 
+				      p->value->package)))
+        {
+	    dpkg_source_note *srcnB;
+	    packagelist **p;
+
+	    if (!undoop) {
+		printf("conflict w/o undo: binary %s, owned by %s, replaced by %s\n", cpkg->pkg->package, cpkg->pkg->source, src->package);
+                fflush(stdout);
+            }
+	    srcnB = lookup_sourcenotetbl(srcsn->sources, cpkg->pkg->source);
+            assert(srcnB != NULL);
+                
+            for (p = &srcnB->binaries[archnum]; *p != NULL; p = &(*p)->next) {
+                if ((*p)->value == cpkg->pkg) break;
+            }
+            assert(*p != NULL); /* binary should be from source */
+
+	    assert(undoop);
+            save_source_note(srcsn, srcnB);
+	    remove_package(srcsn->pkgs[archnum], cpkg);
+            remove_packagelist(p);
+	}
+
+	add_package(srcsn->pkgs[archnum], p->value);
+	insert_packagelist(&srcn->binaries[archnum], p->value);
+    }
+}    
+
+void upgrade_source(dpkg_sources_note *srcsn, dpkg_source *src) {
+    dpkg_source_note *srcn;
+    int i;
+   
+    new_op(srcsn);
+ 
+    assert(is_sources_note(srcsn));
+    /* first, find the old source, if it exists */
+    srcn = remove_sourcenotetbl(srcsn->sources, src->package);
+    if (srcn != NULL) {
+        save_source_note(srcsn, srcn);
+	for (i = 0; i < srcn->n_arches; i++) {
+	    remove_binaries_by_arch(srcsn, srcn, i, DO_ARCHALL);
+	}
+	free_source_note(srcn);
+    } else {
+        save_empty_source_note(srcsn, src);
+    }
+    
+    /* then add the new one */    
+    srcn = new_source_note(src, srcsn->n_arches);   
+    add_sourcenotetbl(srcsn->sources, src->package, srcn);
+    for (i = 0; i < srcsn->n_arches; i++) {
+	add_binaries_by_arch(srcsn, srcn, src, i, UNDOABLE, DO_ARCHALL);
+    }
+    assert(is_sources_note(srcsn));
+}
+
+void upgrade_arch(dpkg_sources_note *srcsn, dpkg_source *src, char *arch) {
+    dpkg_source_note *srcn;
+    int archnum = -1;
+    int i;
+
+    assert(is_sources_note(srcsn));
+    /* first, find the old source */
+    srcn = lookup_sourcenotetbl(srcsn->sources, src->package);
+
+    assert(srcn != NULL);
+    new_op(srcsn);
+    save_source_note(srcsn, srcn);
+
+    /* then lookup the archnum */
+    for (i = 0; i < srcsn->n_arches; i++) {
+	if (strcmp(arch, srcsn->archname[i]) == 0) {
+	    archnum = i;
+	    break;
+	}
+    }
+    if (archnum == -1) die("upgrade_arch: unknown arch");
+    
+    /* then remove the old stuff and add the new */    
+    remove_binaries_by_arch(srcsn, srcn, archnum, SKIP_ARCHALL);
+    add_binaries_by_arch(srcsn, srcn, src, archnum, UNDOABLE, SKIP_ARCHALL);
+    assert(is_sources_note(srcsn));
+}
+
+void remove_source(dpkg_sources_note *srcsn, char *name) {
+    dpkg_source_note *srcn;
+    int i;
+
+    assert(is_sources_note(srcsn));
+    srcn = remove_sourcenotetbl(srcsn->sources, name);
+    assert(srcn != NULL);
+
+    new_op(srcsn);
+    save_source_note(srcsn, srcn);
+    for (i = 0; i < srcn->n_arches; i++) {
+        remove_binaries_by_arch(srcsn, srcn, i, DO_ARCHALL);
+    }
+    free_source_note(srcn);
+    assert(is_sources_note(srcsn));
+}
+
+int can_undo(dpkg_sources_note *srcsn) {
+    assert(is_sources_note(srcsn));
+    return srcsn->undo != NULL;
+}
+
+void undo_change(dpkg_sources_note *srcsn) {
+    dpkg_source_note *srcnO, *srcnC; /* old, current */
+    source_note_list *srcnl;
+    int i;
+
+    assert(is_sources_note(srcsn));
+    assert(can_undo(srcsn));
+
+    srcnl = remove_source_note_listlist(&srcsn->undo);
+
+    while(srcnl) {
+        srcnO = remove_source_note_list(&srcnl);
+        assert(srcnO != NULL); /* can_undo() implies this is true... */
+	
+        srcnC = remove_sourcenotetbl(srcsn->sources, srcnO->source->package);
+        if (srcnC != NULL) {
+	    for (i = 0; i < srcnC->n_arches; i++) {
+	        remove_binaries_by_arch(srcsn, srcnC, i, DO_ARCHALL);
+	    }
+	    free_source_note(srcnC);
+            assert(!lookup_sourcenotetbl(srcsn->sources, 
+					 srcnO->source->package));
+        }
+
+        if (srcnO->binaries == NULL) {
+            /* no original source */
+            assert(srcnC != NULL); /* some sort of no-op? freaky. */
+	    free_source_note(srcnO);
+        } else {
+            packagelist *p;
+            /* original source */
+            add_sourcenotetbl(srcsn->sources, srcnO->source->package, srcnO);
+            for (i = 0; i < srcsn->n_arches; i++) {
+                for (p = srcnO->binaries[i]; p != NULL; p = p->next) {
+		    add_package(srcsn->pkgs[i], p->value);
+                }
+            }
+        }
+    }
+}
+
+LIST_IMPL(source_note_list, dpkg_source_note *, free_source_note, 
+          block_malloc, block_free);
+LIST_IMPL(source_note_listlist, source_note_list *, free_source_note_list, 
+          block_malloc, block_free);
+
+void commit_changes(dpkg_sources_note *srcsn) {
+    assert(is_sources_note(srcsn));
+    free_source_note_listlist(srcsn->undo);
+    srcsn->undo = NULL;
+}
+
+dpkg_source_note *copy_source_note(dpkg_source_note *srcn) {
+    dpkg_source_note *srcn2;
+    packagelist *src, **dest;
+    int i;
+
+    assert(srcn->binaries != NULL);
+
+    srcn2 = block_malloc(sizeof(dpkg_source_note));
+    if (srcn2 == NULL) die("copy_source_note alloc:");
+
+    srcn2->source = srcn->source;
+    srcn2->n_arches = srcn->n_arches;
+    srcn2->binaries = block_malloc(sizeof(packagenamelist*) * srcn2->n_arches);
+    if (srcn2->binaries == NULL) die("copy_source_note alloc:");
+
+    for (i = 0; i < srcn2->n_arches; i++) {
+        dest = &(srcn2->binaries[i]);
+        *dest = NULL;
+        for (src = srcn->binaries[i]; src; src = src->next) {
+            insert_packagelist(dest, src->value);
+            dest = &((*dest)->next);
+        }
+    }
+
+    return srcn2;
+}
+
+void write_notes(char *dir, dpkg_sources_note *srcsn) {
+    FILE *src;
+    FILE *archfile[100];
+    char buf[1000];
+    int i;
+    sourcenotetbl_iter srciter;
+
+    assert(is_sources_note(srcsn));
+    snprintf(buf, 1000, "%s/Sources", dir);
+    src = fopen(buf, "w");
+    for (i = 0; i < srcsn->n_arches; i++) {
+	snprintf(buf, 1000, "%s/Packages_%s", dir, srcsn->archname[i]);
+	archfile[i] = fopen(buf, "w");
+    }
+
+    for (srciter = first_sourcenotetbl(srcsn->sources);
+	 !done_sourcenotetbl(srciter);
+	 srciter = next_sourcenotetbl(srciter))
+    {
+	packagelist *p;
+	int i;
+
+	if (!srciter.v->source->fake)
+		write_paragraph(src, srciter.v->source->details);
+	
+	for (i = 0; i < srcsn->n_arches; i++) {
+	    for (p = srciter.v->binaries[i]; p != NULL; p = p->next) {
+		write_paragraph(archfile[i], p->value->details);
+	    }
+	}
+    }
+    
+    fclose(src);
+    for (i = 0; i < srcsn->n_arches; i++) {
+	fclose(archfile[i]);
+    }
+}
+

=== added file 'update_out/dpkg.h'
--- update_out/dpkg.h	1970-01-01 00:00:00 +0000
+++ update_out/dpkg.h	2020-03-12 14:58:23 +0000
@@ -0,0 +1,207 @@
+#ifndef DPKG_H
+#define DPKG_H
+
+#include "templates.h"
+#include "memory.h"
+
+#include <stdio.h>
+
+/**************************************************************************
+ * Coping with an rfc822-esque field 
+ */
+
+typedef struct dpkg_entry dpkg_entry;
+struct dpkg_entry {
+    char *name;
+    char *value;
+};
+
+typedef struct dpkg_paragraph dpkg_paragraph;
+struct dpkg_paragraph {
+    int n_entries;
+    int n_allocated;
+    dpkg_entry *entry;
+};
+
+/**************************************************************************
+ * Coping with a package (or many pkgs) as an abstract entity 
+ */
+
+typedef enum {dr_NOOP,dr_LT,dr_LTEQ,dr_EQ,dr_GTEQ,dr_GT} dependency_relation;
+extern char *dependency_relation_sym[];
+
+typedef struct dependency dependency;
+struct dependency {
+    char *package;
+    dependency_relation op;
+    char *version;
+};
+
+LIST(deplist, dependency*);
+LIST(deplistlist, deplist*);
+
+LIST(packagenamelist, char*);
+LIST(ownedpackagenamelist, char*);
+
+typedef struct dpkg_package dpkg_package;
+
+struct dpkg_package {
+    char *package;
+    char *version;
+    
+    char *source;
+    char *source_ver;
+    
+    int priority;
+
+    int arch_all;
+    
+    deplistlist          *depends[4];
+    deplist              *conflicts;    
+    ownedpackagenamelist *provides;
+	
+    dpkg_paragraph *details;
+};
+
+LIST(packagelist, dpkg_package *);
+LIST(ownedpackagelist, dpkg_package *);
+
+typedef struct satisfieddep satisfieddep;
+
+struct satisfieddep {
+    /* dependency *dep; */
+    deplist *depl;
+    packagelist *pkgs;
+};
+
+LIST(satisfieddeplist, satisfieddep *);
+
+/**************************************************************************
+ * Coping with a source package (and collections thereof) as an abstract 
+ * entity, owning a bunch of binary packages 
+ */
+
+typedef struct dpkg_source dpkg_source;
+struct dpkg_source {
+    char *package;
+    char *version;
+
+    int fake;
+
+    struct dpkg_sources *owner;
+    ownedpackagelist **packages; /* one for each architecture */
+
+    dpkg_paragraph *details;
+};
+
+HASH(sourcetbl,char *,dpkg_source *);
+
+typedef struct dpkg_sources dpkg_sources;
+struct dpkg_sources {
+    int n_arches;
+    char **archname;
+    sourcetbl *sources;
+    ownedpackagelist **unclaimedpackages; /* one for each arch */
+};
+
+/**************************************************************************
+ */
+
+typedef struct dpkg_collected_package dpkg_collected_package;
+struct dpkg_collected_package {
+    dpkg_package *pkg;
+
+    int installed, conflicted;
+
+    enum { UNKNOWN, YES } installable;
+    packagenamelist *mayaffect;
+
+    /* on update, the installability_checked of each /mayaffect/ed package
+     * is cleared, and the mayaffect list is cleared.
+     * 
+     * note that installable = NO couldn't be maintained over adding a package
+     * to testing. installable = YES can be, thanks to the mayaffect list
+     * (once a package is removed, everything it mayaffect must be set back
+     * to unknown, but everything else is okay)
+     */
+};
+
+LIST(collpackagelist, dpkg_collected_package *);
+
+/**************************************************************************
+ */
+
+typedef struct dpkg_provision dpkg_provision;
+struct dpkg_provision {
+    char *version;
+    dpkg_collected_package *pkg;
+};
+
+LIST(virtualpkg, dpkg_provision);
+
+HASH(virtualpkgtbl,char *,virtualpkg *);
+HASH(packagetbl,char *,dpkg_collected_package *);
+
+typedef struct dpkg_packages dpkg_packages;
+struct dpkg_packages {
+    char *arch;
+    packagetbl *packages;
+    virtualpkgtbl *virtualpkgs;
+};
+
+typedef struct dpkg_source_note dpkg_source_note;
+struct dpkg_source_note {
+    dpkg_source *source;   /* unowned */
+    int n_arches;
+    packagelist **binaries; /* one for each arch */
+};
+HASH(sourcenotetbl, char *, dpkg_source_note *);
+
+LIST(source_note_list, dpkg_source_note *);
+	/* contains a copy of the previous source_note */
+LIST(source_note_listlist, source_note_list *);
+	/* contains a copy of all the source_notes modified by the last op */
+
+typedef struct dpkg_sources_note dpkg_sources_note;
+struct dpkg_sources_note {
+    unsigned long magic;
+    sourcenotetbl *sources;
+    int n_arches;
+    dpkg_packages **pkgs;
+    char **archname;
+
+    source_note_listlist *undo;
+};
+
+void free_packages(dpkg_packages *pkgs);
+void free_sources(dpkg_sources *s);
+
+dpkg_packages *get_architecture(dpkg_sources *srcs, char *arch);
+
+/* parsing things */
+int checkinstallable(dpkg_packages *pkgs, collpackagelist *instoneof);
+int checkinstallable2(dpkg_packages *pkgs, char *pkgname);
+satisfieddeplist *checkunsatisfiabledeps(dpkg_packages *pkgs, 
+					    deplistlist *deps);
+
+dpkg_sources *read_directory(char *dir, int n_arches, char *archname[]);
+void write_directory(char *dir, dpkg_sources *srcs);
+
+void free_source(dpkg_source *s);
+
+/* adding and deleting and stuff */
+dpkg_sources_note *new_sources_note(int n_arches, char **archname);
+void remove_source(dpkg_sources_note *srcsn, char *name);
+void upgrade_source(dpkg_sources_note *srcsn, dpkg_source *src);
+void upgrade_arch(dpkg_sources_note *srcsn, dpkg_source *src, char *arch);
+void write_notes(char *dir, dpkg_sources_note *srcsn);
+void free_sources_note(dpkg_sources_note *srcsn);
+void free_source_note(dpkg_source_note *srcn);
+void undo_change(dpkg_sources_note *srcsn);
+int can_undo(dpkg_sources_note *srcsn);
+void commit_changes(dpkg_sources_note *srcsn);
+
+int versioncmp(char *left, char *right);
+int cmpversions(char *left, int op, char *right);
+
+#endif

=== added file 'update_out/freelist.c'
--- update_out/freelist.c	1970-01-01 00:00:00 +0000
+++ update_out/freelist.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,188 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include "templates.h"
+
+typedef unsigned long ul;
+
+#define SIZE          (sizeof(ul) * 8)
+#define ROUND_DOWN(x) ((x) & ~(SIZE-1))
+#define ROUND_UP(x)   ROUND_DOWN((x) + (SIZE-1))
+#define NEXT_UP(x)    ROUND_DOWN((x) + SIZE)
+#define NEXT_DOWN(x)  ROUND_DOWN((x) - 1)
+
+#define SETBIT(s,p) \
+		assert( (bits[(s)/SIZE] & (p)) == (setp ? 0 : (p)) ); \
+		if (setp) bits[(s)/SIZE] |= (p); \
+		else      bits[(s)/SIZE] &= ~(p)
+
+#define GETBIT(s) (bits[ROUND_DOWN(s)/SIZE] & (1ul << (NEXT_UP(s) - s - 1)))
+
+size_t count_free_bits_back(ul *bits, size_t s) {
+	size_t cnt = 0;
+	ul w = ROUND_DOWN(s) / SIZE;
+	size_t add = s % SIZE;
+	ul off = (~0ul) << (SIZE - add);
+	ul H, d;
+
+	while ((bits[w] & off) == 0) {
+		cnt += add;
+		add = SIZE;
+		off = ~0ul;
+		if (w == 0)
+			return cnt;
+		w--;
+	}
+
+	H = add;
+	add = 0;
+	while ((d = (H - add) / 2) > 0) {
+		ul offM = (off >> d) & off;
+		if (bits[w] & offM) {
+			off = offM;
+			H = H - d;
+		} else {
+			add = H - d;
+		}
+	}
+	cnt += add;
+	return cnt;
+}
+
+size_t count_free_bits_after(ul *bits, size_t s, size_t end) {
+	size_t cnt = 0;
+	ul w = ROUND_DOWN(s) / SIZE;
+	size_t add = SIZE - s % SIZE;
+	ul off = (~0ul) >> (SIZE - add);
+	ul H, d;
+
+	end /= SIZE;
+
+	while ((bits[w] & off) == 0) {
+		cnt += add;
+		add = SIZE;
+		off = ~0ul;
+		w++;
+		if (w == end)
+			return cnt;
+	}
+
+	H = add;
+	add = 0;
+	while ((d = (H - add) / 2) > 0) {
+		ul offM = off << d;
+		if (bits[w] & offM) {
+			off = offM;
+			H = H - d;
+		} else {
+			add = H - d;
+		}
+	}
+	cnt += add;
+	return cnt;
+}
+
+void find_long_freebits(ul *bits, size_t s, ul *start, size_t *size) {
+        ul clen = 0;
+        ul bstart = 0, blen = 0;
+        ul i, k;
+
+        for (i = 0; i < s; i++) {
+                if (bits[i] == 0) {
+			clen++;
+		} else {
+			if (clen > blen) {
+				bstart = i - clen;
+				blen = clen;
+			}
+			clen = 0;
+		}
+	}
+
+	if (blen == 0) return;
+
+	bstart *= SIZE; blen *= SIZE;
+	k = count_free_bits_back(bits, bstart);
+	bstart -= k; blen += k;
+
+	blen += count_free_bits_after(bits, bstart + blen, s*SIZE);
+
+	*start = bstart; *size = blen;
+}
+
+void mark_bits(ul *bits, ul s, size_t size, int setp) {
+	ul e = s+size;
+
+	ul rds = ROUND_DOWN(s);
+	ul nus = rds + SIZE;
+	ul rue = ROUND_UP(e);
+
+	ul patl = (~0UL) >> (s % SIZE);
+	ul patr = (~0UL) << (rue - e);
+
+	assert(size > 0);
+
+	/* bits[s1..e1] get touched, but bits[s1], bits[e1] only partially
+	 *
+	 * if s1 == e1, then bits[s1] get touched from [s%SIZE, e%SIZE)
+	 * else
+	 *     bits[s1] gets touched from [s%SIZE, SIZE)
+	 *     bits[s2..e1) get reset completely
+	 *     bits[e1] gets touched from [0, e%SIZE)
+	 */
+
+	if (nus >= e) {
+		/* ROUND_DOWN(s) <= s < e <= NEXT_UP(s) */
+		SETBIT(rds, patl & patr);
+	} else {
+		/* ROUND_DOWN(s) <= s < NEXT_UP(s) <= NEXT_DOWN(e) < e */
+		ul rde = ROUND_DOWN(e);
+
+		SETBIT(rds, patl);
+		SETBIT(rde, patr);
+		while (nus < rde) {	
+			SETBIT(nus, ~0UL);
+			nus += SIZE;
+		}
+	}
+}
+
+void print_bits(ul *bits, ul s) {
+	ul i;
+	putchar(' ');
+	for (i = 0; i < s * SIZE; i++) {
+		putchar( GETBIT(i) ? '1' : '0' );
+	}
+}
+	
+#ifdef TESTBIN
+
+#define X 2
+int main(void) {
+	ul memory[X];
+	ul l, r;
+	ul k = 5;
+
+	memset(memory, 0, sizeof(memory));
+	for (l = 0; l < X*SIZE; l += k) {
+		for (r = 1; l+(r*r) < X*SIZE; r++) {
+
+			printf("%lu %lu (%lu %lu", l, r*r,
+			    (unsigned long) count_free_bits_back(memory, X*SIZE), (unsigned long) X*SIZE);
+			mark_bits(memory, l, r*r, 1);
+			printf("; %lu %lu %lu; %lu %lu %lu;): ", 
+			    (unsigned long) count_free_bits_back(memory, X*SIZE) + l + r*r, 
+			    (unsigned long) count_free_bits_after(memory, l + r*r, X*SIZE) + l + r*r,
+			    (unsigned long) X*SIZE,
+			    (unsigned long) count_free_bits_back(memory, l), 
+			    (unsigned long) count_free_bits_after(memory, 0, X*SIZE), 
+			    l);
+			print_bits(memory, X);
+			printf("\n");
+
+			mark_bits(memory, l, r*r, 0);
+		}
+	}
+
+	return 0;
+}
+#endif

=== added file 'update_out/freelist.h'
--- update_out/freelist.h	1970-01-01 00:00:00 +0000
+++ update_out/freelist.h	2020-03-12 14:58:23 +0000
@@ -0,0 +1,14 @@
+
+#ifndef FREELIST_H
+#define FREELIST_H
+
+#include <stdlib.h>
+
+typedef unsigned long flb_t;
+
+void mark_bits(flb_t *bits, flb_t s, size_t size, int setp);
+size_t count_free_bits_back(flb_t *bits, size_t s);
+size_t count_free_bits_after(flb_t *bits, size_t s, size_t end);
+void find_long_freebits(flb_t *bits, flb_t s, flb_t *start, size_t *size);
+
+#endif

=== added file 'update_out/index.html'
--- update_out/index.html	1970-01-01 00:00:00 +0000
+++ update_out/index.html	2020-03-12 14:58:23 +0000
@@ -0,0 +1,18 @@
+<a href="README">README</a><br>
+<a href="Makefile">Makefile</a><br>
+<a href="assert.c">assert.c</a><br>
+<a href="britney-py.c">britney-py.c</a><br>
+<a href="checklib.c">checklib.c</a><br>
+<a href="dpkg.c">dpkg.c</a><br>
+<a href="freelist.c">freelist.c</a><br>
+<a href="memory.c">memory.c</a><br>
+<a href="memory2.c">memory2.c</a><br>
+<a href="memory3.c">memory3.c</a><br>
+<a href="dpkg-lib.cpp">dpkg-lib.cpp</a><br>
+<a href="dpkg.h">dpkg.h</a><br>
+<a href="freelist.h">freelist.h</a><br>
+<a href="memory.h">memory.h</a><br>
+<a href="templates.h">templates.h</a><br>
+<a href="check_out.py">check_out.py</a><br>
+<a href="check_uptodate.py">check_uptodate.py</a><br>
+<a href="update_out.py">update_out.py</a><br>

=== added file 'update_out/memory.c'
--- update_out/memory.c	1970-01-01 00:00:00 +0000
+++ update_out/memory.c	2020-03-12 14:58:23 +0000
@@ -0,0 +1,389 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "memory.h"
+#include "templates.h"
+#include "freelist.h"
+
+/**** THEORY
+ * 
+
+So, we have blocks with a freelist
+
+        XXX............XXXXXXX..XXXXX.....XXXXXX......
+        
+Within a block, we work with segments. A segment is...
+
+	   ^..........|
+
+Every now and then we make sure we've got a decent sized segment.
+
+We have multiple blocks. They're kept ordered by the size of their
+current segment.
+
+ **********************************************/
+
+#define ALIGN 4
+