Merge lp:~spiv/bzr/chk-canonicalizer-613698 into lp:bzr

[Andrew Bennetts]

Add hidden option 'bzr reconcile --canonicalize-chks' to repair repos affected by bug 522637.

This adds 'bzr reconcile --canonicalize-chks' as a hidden option to repair repositories affected by bug 522637. The basic approach is like John's bzr-check-chk plugin: regenerate the CHK map from scratch and verify that the key is the same.

Because so few people will ever run this code (not many repositories seem to have been adversely affected by bug 522637), in principle I'd have been happy to have this as a plugin where its rough edges wouldn't grate. But it would have required a bit more duplication of code than I'd like (although now it's done a bit less that I was expecting, I think). So I wrote this as part of the reconcile infrastructure in the core tool, but left it as a hidden option.

I didn't integrate it into the regular reconcile code path because a) it's 2x slower than regular reconcile already without doing the other things reconcile does, and b) it's going to be so rarely used.

The use of NoDupeAddLinesDecorator is not strictly necessary, but it avoids exploding a 5MB repo of lp:do into over 200MB. That is fixed by repacking, but that sort of waste might have made this repair tool impossible to run on a larger repository.

This takes about 3 minutes to fix the broken lp:do repository on my laptop.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> Andrew Bennetts has proposed merging lp:~spiv/bzr/chk-canonicalizer-613698 into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
> Related bugs:
> #613698 need reconcile fixer for bug 522637 "missing referenced chk root keys"
> https://bugs.launchpad.net/bugs/613698
>
>
> This adds 'bzr reconcile --canonicalize-chks' as a hidden option to repair repositories affected by bug 522637. The basic approach is like John's bzr-check-chk plugin: regenerate the CHK map from scratch and verify that the key is the same.
>
> Because so few people will ever run this code (not many repositories seem to have been adversely affected by bug 522637), in principle I'd have been happy to have this as a plugin where its rough edges wouldn't grate. But it would have required a bit more duplication of code than I'd like (although now it's done a bit less that I was expecting, I think). So I wrote this as part of the reconcile infrastructure in the core tool, but left it as a hidden option.
>
> I didn't integrate it into the regular reconcile code path because a) it's 2x slower than regular reconcile already without doing the other things reconcile does, and b) it's going to be so rarely used.
>
> The use of NoDupeAddLinesDecorator is not strictly necessary, but it avoids exploding a 5MB repo of lp:do into over 200MB. That is fixed by repacking, but that sort of waste might have made this repair tool impossible to run on a larger repository.
>
> This takes about 3 minutes to fix the broken lp:do repository on my laptop.
>

- - repo_reconciler = self.repo.reconcile(thorough=True)
+ if self.canonicalize_chks:
+ repo_reconciler = self.repo.reconcile_canonicalize_chks()
+ else:
+ repo_reconciler = self.repo.reconcile(thorough=True)

^- Why use a new function rather than passing the parameter through?

I know there is a design principle that says don't add a parameter that
is static for callers (always True, always False for a given caller).
This isn't quite the same, since a user is selecting it.

Anyway, it *felt* to me more like a parameter to .reconcile()
(especially since it is a parameter up the stack).

The other thing I'm not sure about, it looks like you added:
+ @needs_write_lock
+ def reconcile_canonicalize_chks(self):

only to one repository type, which means you'd get an attribute error
running reconcile --canonicalize-chks on another repo type. But maybe I
misread that. (Better to just raise a BzrCommandError, I guess)

+class NoDupeAddLinesDecorator(object):

^- Is this actually necessary. I would have sworn I had a patch earlier
that did something exactly like this, because I saw conversions (I
think) waste tons of space that disappeared at repack. Then again, maybe
that was in InterDifferingSerializer or something along those lines.

Anyway, I think it might be reasonable to just push this as the common
case, rather than needing a wrapper. Certainly there you might be able
to do it better than a get_parent_map() call.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with...

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> Andrew Bennetts has proposed merging lp:~spiv/bzr/chk-canonicalizer-613698 into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
> Related bugs:
> #613698 need reconcile fixer for bug 522637 "missing referenced chk root keys"
> https://bugs.launchpad.net/bugs/613698
>
>
> This adds 'bzr reconcile --canonicalize-chks' as a hidden option to repair repositories affected by bug 522637. The basic approach is like John's bzr-check-chk plugin: regenerate the CHK map from scratch and verify that the key is the same.
>
> Because so few people will ever run this code (not many repositories seem to have been adversely affected by bug 522637), in principle I'd have been happy to have this as a plugin where its rough edges wouldn't grate. But it would have required a bit more duplication of code than I'd like (although now it's done a bit less that I was expecting, I think). So I wrote this as part of the reconcile infrastructure in the core tool, but left it as a hidden option.
>
> I didn't integrate it into the regular reconcile code path because a) it's 2x slower than regular reconcile already without doing the other things reconcile does, and b) it's going to be so rarely used.
>
> The use of NoDupeAddLinesDecorator is not strictly necessary, but it avoids exploding a 5MB repo of lp:do into over 200MB. That is fixed by repacking, but that sort of waste might have made this repair tool impossible to run on a larger repository.
>
> This takes about 3 minutes to fix the broken lp:do repository on my laptop.
>

btw, if we wanted to make this faster...

I think we could rework it so that the initial inventory is produced
from scratch, and then all others just use apply_inventory_delta.
(Possibly special cased if it actually finds an incorrect inventory.)

The total shape of the inventory is correct (otherwise all of this
wouldn't work anyway), and the deltas are correct. As such, it would
require a lot less regeneration of pages that are already correct.

That would make it possible to run it all reconcile runs, rather than
requiring a flag.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxjILcACgkQJdeBCYSNAAOj6gCeJ2HU6OJMgTAnsBvq1mIKkHCX
0rUAn2FouKNap31WoEEy+6P+ssXRJmac
=9oPA
-----END PGP SIGNATURE-----

[Andrew Bennetts (spiv)]

John A Meinel wrote:
[...]
> - - repo_reconciler = self.repo.reconcile(thorough=True)
> + if self.canonicalize_chks:
> + repo_reconciler = self.repo.reconcile_canonicalize_chks()
> + else:
> + repo_reconciler = self.repo.reconcile(thorough=True)
>
> ^- Why use a new function rather than passing the parameter through?
>
> I know there is a design principle that says don't add a parameter that
> is static for callers (always True, always False for a given caller).
> This isn't quite the same, since a user is selecting it.

The reason was that I didn't want to add it to all implementations of
reconcile, which felt a bit wrong considering it only applies to one
repository type.

> Anyway, it *felt* to me more like a parameter to .reconcile()
> (especially since it is a parameter up the stack).
>
> The other thing I'm not sure about, it looks like you added:
> + @needs_write_lock
> + def reconcile_canonicalize_chks(self):
>
> only to one repository type, which means you'd get an attribute error
> running reconcile --canonicalize-chks on another repo type. But maybe I
> misread that. (Better to just raise a BzrCommandError, I guess)

Yeah, I'll make cmd_reconcile check the format and raise that.

> +class NoDupeAddLinesDecorator(object):
>
> ^- Is this actually necessary. I would have sworn I had a patch earlier
> that did something exactly like this, because I saw conversions (I
> think) waste tons of space that disappeared at repack. Then again, maybe
> that was in InterDifferingSerializer or something along those lines.

Yep, it's necessary. 200MB vs. 5MB!

> Anyway, I think it might be reasonable to just push this as the common
> case, rather than needing a wrapper. Certainly there you might be able
> to do it better than a get_parent_map() call.

Yeah, quite possibly. This implementation does have the cost of
querying the index for 1 key every add_lines, which would be pretty bad
for performance where latency is a concern. But I'm also a bit
reluctant to make add_lines take any more parameters...

-Andrew.

[Andrew Bennetts (spiv)]

John A Meinel wrote:
[...]
> btw, if we wanted to make this faster...
>
> I think we could rework it so that the initial inventory is produced
> from scratch, and then all others just use apply_inventory_delta.
> (Possibly special cased if it actually finds an incorrect inventory.)

That's a good point (although it relies on apply_inventory_delta being
as reliable as CHKMap.from_dict, which was the bug in the first place,
but that seems reasonable).

I'd prefer to stop spending time on this, though :) The number of
affected repositories, as judged from the bug report, seems to be pretty
low, which makes sense given the fairly particular conditions needed to
trigger it (deleting < 5 entries from an inventory that is just over the
size that splits it from one CHK node to many). So for now I think it's
good enough to get this relatively crude repair tool in the hands of
the few users that need it. We can always come back and improve it
later if there's a reason to do that.

-Andrew.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> John A Meinel wrote:
> [...]
>> btw, if we wanted to make this faster...
>>
>> I think we could rework it so that the initial inventory is produced
>> from scratch, and then all others just use apply_inventory_delta.
>> (Possibly special cased if it actually finds an incorrect inventory.)
>
> That's a good point (although it relies on apply_inventory_delta being
> as reliable as CHKMap.from_dict, which was the bug in the first place,
> but that seems reasonable).
>
> I'd prefer to stop spending time on this, though :) The number of
> affected repositories, as judged from the bug report, seems to be pretty
> low, which makes sense given the fairly particular conditions needed to
> trigger it (deleting < 5 entries from an inventory that is just over the
> size that splits it from one CHK node to many). So for now I think it's
> good enough to get this relatively crude repair tool in the hands of
> the few users that need it. We can always come back and improve it
> later if there's a reason to do that.
>
> -Andrew.
>

Actually, it is

if delete_Count > _INTERESTING ...:

so it is deleting *more* that 5 entries from an inventory that causes it
to cross the boundary.

Which just means anything that deletes a lot of entries, enough to cross
the threshold.

That said, I don't know that you need to spend a lot more time on this.
Ideally we would also have 'bzr check' notice it, though.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxjU3QACgkQJdeBCYSNAAOlTQCeNxSBEeol3WJPCNwoJ09tbWfF
yIUAoLfOBGoRGzbIM0Cvy50a+DEKSt+g
=D6lS
-----END PGP SIGNATURE-----

[lp:~spiv/bzr/chk-canonicalizer-613698]

5381. By Andrew Bennetts on 2010-08-12

Don't traceback if a repository doesn't support reconcile_canonicalize_chks.

[Andrew Bennetts (spiv)]

John wrote:
> That said, I don't know that you need to spend a lot more time on this.

I'm going to go ahead and merge this then. I hope I'm not taking too many liberties by doing so, but I'm confident we can incrementally improve this after landing if we need to, and it'll be of more benefit landed than not.

> Ideally we would also have 'bzr check' notice it, though.

I agree. At the moment I'm not planning to spend time on that.

[Andrew Bennetts (spiv)]

sent to pqm by email

=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py	2010-08-07 20:31:22 +0000
+++ bzrlib/builtins.py	2010-08-12 04:26:01 +0000
@@ -1592,11 +1592,17 @@
 
     _see_also = ['check']
     takes_args = ['branch?']
+    takes_options = [
+        Option('canonicalize-chks',
+               help='Make sure CHKs are in canonical form (repairs '
+                    'bug 522637).',
+               hidden=True),
+        ]
 
-    def run(self, branch="."):
+    def run(self, branch=".", canonicalize_chks=False):
         from bzrlib.reconcile import reconcile
         dir = bzrdir.BzrDir.open(branch)
-        reconcile(dir)
+        reconcile(dir, canonicalize_chks=canonicalize_chks)
 
 
 class cmd_revision_history(Command):

=== modified file 'bzrlib/reconcile.py'
--- bzrlib/reconcile.py	2010-05-20 18:23:17 +0000
+++ bzrlib/reconcile.py	2010-08-12 04:26:01 +0000
@@ -36,7 +36,7 @@
 from bzrlib.versionedfile import AdapterFactory, FulltextContentFactory
 
 
-def reconcile(dir, other=None):
+def reconcile(dir, canonicalize_chks=False):
     """Reconcile the data in dir.
 
     Currently this is limited to a inventory 'reweave'.
@@ -46,18 +46,19 @@
     Directly using Reconciler is recommended for library users that
     desire fine grained control or analysis of the found issues.
 
-    :param other: another bzrdir to reconcile against.
+    :param canonicalize_chks: Make sure CHKs are in canonical form.
     """
-    reconciler = Reconciler(dir, other=other)
+    reconciler = Reconciler(dir, canonicalize_chks=canonicalize_chks)
     reconciler.reconcile()
 
 
 class Reconciler(object):
     """Reconcilers are used to reconcile existing data."""
 
-    def __init__(self, dir, other=None):
+    def __init__(self, dir, other=None, canonicalize_chks=False):
         """Create a Reconciler."""
         self.bzrdir = dir
+        self.canonicalize_chks = canonicalize_chks
 
     def reconcile(self):
         """Perform reconciliation.
@@ -98,7 +99,15 @@
         ui.ui_factory.note('Reconciling repository %s' %
             self.repo.user_url)
         self.pb.update("Reconciling repository", 0, 1)
-        repo_reconciler = self.repo.reconcile(thorough=True)
+        if self.canonicalize_chks:
+            try:
+                self.repo.reconcile_canonicalize_chks
+            except AttributeError:
+                raise errors.BzrError(
+                    "%s cannot canonicalize CHKs." % (self.repo,))
+            repo_reconciler = self.repo.reconcile_canonicalize_chks()
+        else:
+            repo_reconciler = self.repo.reconcile(thorough=True)
         self.inconsistent_parents = repo_reconciler.inconsistent_parents
         self.garbage_inventories = repo_reconciler.garbage_inventories
         if repo_reconciler.aborted:
@@ -496,6 +505,12 @@
     #  - unlock the names list
     # https://bugs.launchpad.net/bzr/+bug/154173
 
+    def __init__(self, repo, other=None, thorough=False,
+            canonicalize_chks=False):
+        super(PackReconciler, self).__init__(repo, other=other,
+            thorough=thorough)
+        self.canonicalize_chks = canonicalize_chks
+
     def _reconcile_steps(self):
         """Perform the steps to reconcile this repository."""
         if not self.thorough:
@@ -509,8 +524,12 @@
         total_inventories = len(list(
             collection.inventory_index.combined_index.iter_all_entries()))
         if len(all_revisions):
-            new_pack =  self.repo._reconcile_pack(collection, packs,
-                ".reconcile", all_revisions, self.pb)
+            if self.canonicalize_chks:
+                reconcile_meth = self.repo._canonicalize_chks_pack
+            else:
+                reconcile_meth = self.repo._reconcile_pack
+            new_pack = reconcile_meth(collection, packs, ".reconcile",
+                all_revisions, self.pb)
             if new_pack is not None:
                 self._discard_and_save(packs)
         else:

=== modified file 'bzrlib/repofmt/groupcompress_repo.py'
--- bzrlib/repofmt/groupcompress_repo.py	2010-05-14 13:25:05 +0000
+++ bzrlib/repofmt/groupcompress_repo.py	2010-08-12 04:26:01 +0000
@@ -32,11 +32,13 @@
     revision as _mod_revision,
     trace,
     ui,
+    versionedfile,
     )
 from bzrlib.btree_index import (
     BTreeGraphIndex,
     BTreeBuilder,
     )
+from bzrlib.decorators import needs_write_lock
 from bzrlib.groupcompress import (
     _GCGraphIndex,
     GroupCompressVersionedFiles,
@@ -425,8 +427,11 @@
         self._copy_stream(source_vf, target_vf, inventory_keys,
                           'inventories', self._get_filtered_inv_stream, 2)
 
+    def _get_chk_vfs_for_copy(self):
+        return self._build_vfs('chk', False, False)
+
     def _copy_chk_texts(self):
-        source_vf, target_vf = self._build_vfs('chk', False, False)
+        source_vf, target_vf = self._get_chk_vfs_for_copy()
         # TODO: This is technically spurious... if it is a performance issue,
         #       remove it
         total_keys = source_vf.keys()
@@ -578,6 +583,111 @@
         return new_pack.data_inserted() and self._data_changed
 
 
+class GCCHKCanonicalizingPacker(GCCHKPacker):
+    """A packer that ensures inventories have canonical-form CHK maps.
+    
+    Ideally this would be part of reconcile, but it's very slow and rarely
+    needed.  (It repairs repositories affected by
+    https://bugs.launchpad.net/bzr/+bug/522637).
+    """
+
+    def __init__(self, *args, **kwargs):
+        super(GCCHKCanonicalizingPacker, self).__init__(*args, **kwargs)
+        self._data_changed = False
+    
+    def _exhaust_stream(self, source_vf, keys, message, vf_to_stream, pb_offset):
+        """Create and exhaust a stream, but don't insert it.
+        
+        This is useful to get the side-effects of generating a stream.
+        """
+        self.pb.update('scanning %s' % (message,), pb_offset)
+        child_pb = ui.ui_factory.nested_progress_bar()
+        try:
+            list(vf_to_stream(source_vf, keys, message, child_pb))
+        finally:
+            child_pb.finished()
+
+    def _copy_inventory_texts(self):
+        source_vf, target_vf = self._build_vfs('inventory', True, True)
+        source_chk_vf, target_chk_vf = self._get_chk_vfs_for_copy()
+        inventory_keys = source_vf.keys()
+        # First, copy the existing CHKs on the assumption that most of them
+        # will be correct.  This will save us from having to reinsert (and
+        # recompress) these records later at the cost of perhaps preserving a
+        # few unused CHKs. 
+        # (Iterate but don't insert _get_filtered_inv_stream to populate the
+        # variables needed by GCCHKPacker._copy_chk_texts.)
+        self._exhaust_stream(source_vf, inventory_keys, 'inventories',
+                self._get_filtered_inv_stream, 2)
+        GCCHKPacker._copy_chk_texts(self)
+        # Now copy and fix the inventories, and any regenerated CHKs.
+        def chk_canonicalizing_inv_stream(source_vf, keys, message, pb=None):
+            return self._get_filtered_canonicalizing_inv_stream(
+                source_vf, keys, message, pb, source_chk_vf, target_chk_vf)
+        self._copy_stream(source_vf, target_vf, inventory_keys,
+                          'inventories', chk_canonicalizing_inv_stream, 4)
+
+    def _copy_chk_texts(self):
+        # No-op; in this class this happens during _copy_inventory_texts.
+        pass
+
+    def _get_filtered_canonicalizing_inv_stream(self, source_vf, keys, message,
+            pb=None, source_chk_vf=None, target_chk_vf=None):
+        """Filter the texts of inventories, regenerating CHKs to make sure they
+        are canonical.
+        """
+        total_keys = len(keys)
+        target_chk_vf = versionedfile.NoDupeAddLinesDecorator(target_chk_vf)
+        def _filtered_inv_stream():
+            stream = source_vf.get_record_stream(keys, 'groupcompress', True)
+            search_key_name = None
+            for idx, record in enumerate(stream):
+                # Inventories should always be with revisions; assume success.
+                bytes = record.get_bytes_as('fulltext')
+                chk_inv = inventory.CHKInventory.deserialise(
+                    source_chk_vf, bytes, record.key)
+                if pb is not None:
+                    pb.update('inv', idx, total_keys)
+                chk_inv.id_to_entry._ensure_root()
+                if search_key_name is None:
+                    # Find the name corresponding to the search_key_func
+                    search_key_reg = chk_map.search_key_registry
+                    for search_key_name, func in search_key_reg.iteritems():
+                        if func == chk_inv.id_to_entry._search_key_func:
+                            break
+                canonical_inv = inventory.CHKInventory.from_inventory(
+                    target_chk_vf, chk_inv,
+                    maximum_size=chk_inv.id_to_entry._root_node._maximum_size,
+                    search_key_name=search_key_name)
+                if chk_inv.id_to_entry.key() != canonical_inv.id_to_entry.key():
+                    trace.mutter(
+                        'Non-canonical CHK map for id_to_entry of inv: %s '
+                        '(root is %s, should be %s)' % (chk_inv.revision_id,
+                        chk_inv.id_to_entry.key()[0],
+                        canonical_inv.id_to_entry.key()[0]))
+                    self._data_changed = True
+                p_id_map = chk_inv.parent_id_basename_to_file_id
+                p_id_map._ensure_root()
+                canon_p_id_map = canonical_inv.parent_id_basename_to_file_id
+                if p_id_map.key() != canon_p_id_map.key():
+                    trace.mutter(
+                        'Non-canonical CHK map for parent_id_to_basename of '
+                        'inv: %s (root is %s, should be %s)'
+                        % (chk_inv.revision_id, p_id_map.key()[0],
+                           canon_p_id_map.key()[0]))
+                    self._data_changed = True
+                yield versionedfile.ChunkedContentFactory(record.key,
+                        record.parents, record.sha1,
+                        canonical_inv.to_lines())
+            # We have finished processing all of the inventory records, we
+            # don't need these sets anymore
+        return _filtered_inv_stream()
+
+    def _use_pack(self, new_pack):
+        """Override _use_pack to check for reconcile having changed content."""
+        return new_pack.data_inserted() and self._data_changed
+
+
 class GCRepositoryPackCollection(RepositoryPackCollection):
 
     pack_factory = GCPack
@@ -999,10 +1109,24 @@
         finally:
             pb.finished()
 
+    @needs_write_lock
+    def reconcile_canonicalize_chks(self):
+        """Reconcile this repository to make sure all CHKs are in canonical
+        form.
+        """
+        from bzrlib.reconcile import PackReconciler
+        reconciler = PackReconciler(self, thorough=True, canonicalize_chks=True)
+        reconciler.reconcile()
+        return reconciler
+
     def _reconcile_pack(self, collection, packs, extension, revs, pb):
         packer = GCCHKReconcilePacker(collection, packs, extension)
         return packer.pack(pb)
 
+    def _canonicalize_chks_pack(self, collection, packs, extension, revs, pb):
+        packer = GCCHKCanonicalizingPacker(collection, packs, extension, revs)
+        return packer.pack(pb)
+
     def _get_source(self, to_format):
         """Return a source for streaming from this repository."""
         if self._format._serializer == to_format._serializer:

=== modified file 'bzrlib/versionedfile.py'
--- bzrlib/versionedfile.py	2010-08-11 01:27:46 +0000
+++ bzrlib/versionedfile.py	2010-08-12 04:26:01 +0000
@@ -1725,6 +1725,46 @@
                 yield (l, key)
 
 
+class NoDupeAddLinesDecorator(object):
+    """Decorator for a VersionedFiles that skips doing an add_lines if the key
+    is already present.
+    """
+
+    def __init__(self, store):
+        self._store = store
+
+    def add_lines(self, key, parents, lines, parent_texts=None,
+            left_matching_blocks=None, nostore_sha=None, random_id=False,
+            check_content=True):
+        """See VersionedFiles.add_lines.
+        
+        This implementation may return None as the third element of the return
+        value when the original store wouldn't.
+        """
+        if nostore_sha:
+            raise NotImplementedError(
+                "NoDupeAddLinesDecorator.add_lines does not implement the "
+                "nostore_sha behaviour.")
+        if key[-1] is None:
+            sha1 = osutils.sha_strings(lines)
+            key = ("sha1:" + sha1,)
+        else:
+            sha1 = None
+        if key in self._store.get_parent_map([key]):
+            # This key has already been inserted, so don't do it again.
+            if sha1 is None:
+                sha1 = osutils.sha_strings(lines)
+            return sha1, sum(map(len, lines)), None
+        return self._store.add_lines(key, parents, lines,
+                parent_texts=parent_texts,
+                left_matching_blocks=left_matching_blocks,
+                nostore_sha=nostore_sha, random_id=random_id,
+                check_content=check_content)
+
+    def __getattr__(self, name):
+        return getattr(self._store, name)
+
+
 def network_bytes_to_kind_and_offset(network_bytes):
     """Strip of a record kind from the front of network_bytes.