Merge lp:~nknotts/dccl/encode-decode-iterators into lp:~dccl-dev/dccl/3.0

[Nathan Knotts]

I added an InputIterator overload to codec::decode
I also added a char* overload to codec::encode

For our application, our stream data is stored in a boost::circular_buffer. We did not want to incur the overhead of a copy/string allocation. I added the InputIterator overload to codec::decode to reduce/eliminate copying/allocation.

On the codec::encode side, I added a char* overload to reduce/eliminate allocations.

All unit tests that previously passed still pass.

However, some tests are failing (they were failing before this change)

The following tests FAILED:
  12 - dccl_test_ccl (OTHER_FAULT)
  13 - dccl_test_arithmetic (Failed)

[toby schneider (tes)]

This makes sense to me.

Minor style request: could you move the implementation of these methods to the bottom of codec.h after the declaration of dccl::Codec? (I want to keep the declaration of dccl::Codec clean):

1. template<typename CharIterator> unsigned id(CharIterator begin, CharIterator end)
2. template <typename CharIterator> void decode(CharIterator begin, CharIterator end, google::protobuf::Message* msg, bool header_only = false)

Otherwise should be fine. I'll give it one more detailed look over after you make that change and then I'll merge it in.

Also, dccl_test_ccl and dccl_test_arithmetic run fine on my machine for both lp:dccl/3.0 and this branch. I don't know if they are properly handling the .dylib extension on OSX (I think this is what you're using?). If you could open a separate bug report on those with the command line output when you run them, that'd be great.

Thanks-
-Toby

[lp:~nknotts/dccl/encode-decode-iterators]

315. By Nathan Knotts on 2015-01-20

src/codec.h: moved template implementations to bottom of file

[Nathan Knotts (nknotts)]

I am currently building on OSX. Based on the PPAs, I'm assuming you're building on Ubuntu/Debian, is that correct? I will try and track down why the tests are failing on my system. Any solution will be committed in a separate merge request.

[toby schneider (tes)]

Yes, I use Debian 6, 7 and Ubuntu 12.04/14.04 regularly. Ubuntu 14.04 is what I just tested your branch on.

[Nathan Knotts (nknotts)]

I did not realize those particular tests were using shared libraries. After setting my DYLD_LIBRARY_PATH, the tests now succeed.

[lp:~nknotts/dccl/encode-decode-iterators]

316. By Nathan Knotts on 2015-01-22

bitset.h: 0 initialize to_byte_string output buffer

[toby schneider (tes)]

Appears to be a clean implementation of char iterator overloads for Codec::encode, decode and related helper functions.

review: Approve

=== modified file 'src/binary.h'
--- src/binary.h	2014-10-08 18:19:16 +0000
+++ src/binary.h	2015-01-22 21:49:07 +0000
@@ -85,27 +85,30 @@
         return out;
     }
 
-    /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `in` is written to index 0 and 1 (first byte) of `out`
+    /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `begin` is written to index 0 and 1 (first byte) of `out`
     ///
-    /// \param in byte string to encode (e.g. "TOM")
+    /// \param begin iterator to first byte of string to encode (e.g. "TOM")
+    /// \param end iterator pointing to the past-the-end character of the string.
     /// \param out pointer to string to store result (e.g. "544f4d")
     /// \param upper_case set true to use upper case for the alphabet characters (i.e. A,B,C,D,E,F), otherwise lowercase is used (a,b,c,d,e,f).
-    inline void hex_encode(const std::string& in, std::string* out, bool upper_case = false)
+    template <typename CharIterator>
+    inline void hex_encode(CharIterator begin, CharIterator end, std::string* out, bool upper_case = false)
     {
         static const short char0_9_to_number = 48;
         static const short charA_F_to_number = 55; 
         static const short chara_f_to_number = 87; 
 
-        int in_size = in.size();
-        int out_size = in_size << 1;
-    
+        size_t in_size = std::distance(begin, end);
+        size_t out_size = in_size << 1;
+
+        out->clear();
         out->resize(out_size);
-        for(int i = 0, n = in_size;
-            i < n;
-            ++i)
+
+        size_t i = 0;
+        for(CharIterator it = begin; it != end; ++it)
         {
-            short msn = (in[i] >> 4) & 0x0f;
-            short lsn = in[i] & 0x0f;
+            short msn = (*it >> 4) & 0x0f;
+            short lsn = *it & 0x0f;
 
             if(msn >= 0 && msn <= 9)
                 (*out)[2*i] = msn + char0_9_to_number;
@@ -117,9 +120,29 @@
             else if(lsn >= 10 && lsn <= 15)
                 (*out)[2*i+1] = lsn + (upper_case ? charA_F_to_number : chara_f_to_number);
 
+            i++;
         }
     }
 
+    template <typename CharIterator>
+    inline std::string hex_encode(CharIterator begin, CharIterator end)
+    {
+        std::string out;
+        hex_encode(begin, end, &out);
+        return out;
+
+    }
+
+    /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `in` is written to index 0 and 1 (first byte) of `out`
+    ///
+    /// \param in byte string to encode (e.g. "TOM")
+    /// \param out pointer to string to store result (e.g. "544f4d")
+    /// \param upper_case set true to use upper case for the alphabet characters (i.e. A,B,C,D,E,F), otherwise lowercase is used (a,b,c,d,e,f).
+    inline void hex_encode(const std::string& in, std::string* out, bool upper_case = false)
+    {
+        hex_encode(in.begin(), in.end(), out, upper_case);
+    }
+
     inline std::string hex_encode(const std::string& in)
     {
         std::string out;

=== modified file 'src/bitset.h'
--- src/bitset.h	2015-01-14 01:25:28 +0000
+++ src/bitset.h	2015-01-22 21:49:07 +0000
@@ -28,6 +28,7 @@
 #include <algorithm>
 #include <limits>
 #include <string>
+#include <cassert>
 
 #include "exception.h"
 
@@ -319,15 +320,44 @@
             return s;
         }
 
+        /// \brief Generate a byte string representation of the Bitset, where each character represents 8 bits of the Bitset. The string is used as a byte container, and is not intended to be printed.
+        /// \param buf An output string containing the value of the Bitset, with the least signficant byte in string[0] and the most significant byte in string[size()-1]
+        /// \param max_len Maximum length of buf
+        /// \return number of bytes written to buf
+        size_t to_byte_string(char* buf, size_t max_len)
+        {
+            // number of bytes needed is ceil(size() / 8)
+            size_t len = this->size()/8 + (this->size()%8 ? 1 : 0);
+
+            assert( max_len >= len );
+
+            // initialize buffer to all zeroes
+            std::memset(buf, 0, len);
+
+            for(size_type i = 0, n = this->size(); i < n; ++i)
+                buf[i/8] |= static_cast<char>((*this)[i] << (i%8));
+
+            return len;
+        }
+
         /// \brief Sets the value of the Bitset to the contents of a byte string, where each character represents 8 bits of the Bitset.
         ///
         /// \param s A string container the values where the least signficant byte in string[0] and the most significant byte in string[size()-1]
         void from_byte_string(const std::string& s)
         {
-            this->resize(s.size() * 8);
+            from_byte_stream(s.begin(), s.end());
+        }
+
+        /// \brief Sets the value of the Bitset to the contents of a byte string, where each character represents 8 bits of the Bitset.
+        /// A string container the values where the least signficant byte in string[0] and the most significant byte in string[size()-1]
+        /// \param begin Iterator pointing to the begining of the input buffer
+        /// \param end Iterator pointing to the end of the input bufer
+        template<typename CharIterator>
+        void from_byte_stream(CharIterator begin, CharIterator end)
+        {
+            this->resize(std::distance(begin, end) * 8);
             int i = 0;
-            for(std::string::const_iterator it = s.begin(), n = s.end();
-                it != n; ++it)
+            for(CharIterator it = begin; it != end; ++it)
             {
                 for(size_type j = 0; j < 8; ++j)
                     (*this)[i*8+j] = (*it) & (1 << j);

=== modified file 'src/codec.cpp'
--- src/codec.cpp	2014-10-05 18:36:46 +0000
+++ src/codec.cpp	2015-01-22 21:49:07 +0000
@@ -150,8 +150,7 @@
     }
 }
 
-
-void dccl::Codec::encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only /* = false */)
+void dccl::Codec::encode_internal(const google::protobuf::Message& msg, bool header_only, Bitset& head_bits, Bitset& body_bits)
 {
     const Descriptor* desc = msg.GetDescriptor();
 
@@ -159,6 +158,9 @@
 
     try
     {
+        size_t head_byte_size = 0;
+        size_t body_byte_size = 0;
+
         if(!msg.IsInitialized() && !header_only)
             throw(Exception("Message is not properly initialized. All `required` fields must be set."));
         
@@ -173,26 +175,15 @@
         if(codec)
         {
             //fixed header
-            Bitset head_bits;
             id_codec()->field_encode(&head_bits, id(desc), 0);
             
             internal::MessageStack msg_stack;
             msg_stack.push(msg.GetDescriptor());
             codec->base_encode(&head_bits, msg, HEAD);
 
-            std::string body_bytes;
-            
             // given header of not even byte size (e.g. 01011), make even byte size (e.g. 00001011)
-            unsigned head_byte_size = ceil_bits2bytes(head_bits.size());
-            unsigned head_bits_diff = head_byte_size * BITS_IN_BYTE - (head_bits.size());
-            head_bits.resize(head_bits.size() + head_bits_diff);
-            
-            std::string head_bytes = head_bits.to_byte_string();
-        
-            dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_bytes.size() << "(" << head_bits.size() << ")" << std::endl;
-            dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl;
-            dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl;        
-
+            head_byte_size = ceil_bits2bytes(head_bits.size());
+            head_bits.resize(head_byte_size * BITS_IN_BYTE);
 
             if(header_only)
             {
@@ -200,23 +191,8 @@
             }
             else
             {
-                Bitset body_bits;
                 codec->base_encode(&body_bits, msg, BODY);
-                body_bytes = body_bits.to_byte_string();
-                dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl;
-                dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
-                dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " <<  body_bytes.size() << "(" << body_bits.size() << ")" <<  std::endl;
-
-                if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc)))
-                    encrypt(&body_bytes, head_bytes);
-
-                dlog.is(DEBUG3, ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
             }
-            
-            dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl;
-
-            *bytes += head_bytes + body_bytes;
-
         }
         else
         {
@@ -235,27 +211,83 @@
     }
 }
 
-
+size_t dccl::Codec::encode(char* bytes, size_t max_len, const google::protobuf::Message& msg, bool header_only /* = false */)
+{
+    const Descriptor* desc = msg.GetDescriptor();
+    Bitset head_bits;
+    Bitset body_bits;
+    encode_internal(msg, header_only, head_bits, body_bits);
+
+    size_t head_byte_size = ceil_bits2bytes(head_bits.size());
+    assert(max_len >= head_byte_size);
+    head_bits.to_byte_string(bytes, head_byte_size);
+
+    dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_byte_size << "(" << head_bits.size() << ")" << std::endl;
+    dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl;
+    dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(bytes, bytes+head_byte_size) << std::endl;
+
+    size_t body_byte_size = 0;
+    if (!header_only)
+    {
+        body_byte_size = ceil_bits2bytes(body_bits.size());
+        assert(max_len >= head_byte_size + body_byte_size);
+        body_bits.to_byte_string(bytes+head_byte_size, max_len-head_byte_size);
+
+        dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl;
+        dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(bytes+head_byte_size, bytes+head_byte_size+body_byte_size) << std::endl;
+        dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " <<  body_byte_size << "(" << body_bits.size() << ")" <<  std::endl;
+
+        if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc))) {
+            std::string head_bytes(bytes, bytes+head_byte_size);
+            std::string body_bytes(bytes+head_byte_size, bytes+head_byte_size+body_byte_size);
+            encrypt(&body_bytes, head_bytes);
+            std::memcpy(bytes+head_byte_size, body_bytes.data(), body_bytes.size());
+        }
+
+        dlog.is(logger::DEBUG3, logger::ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(bytes+head_byte_size, bytes+head_byte_size+body_byte_size) << std::endl;
+    }
+
+    dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl;
+
+    return head_byte_size + body_byte_size;
+}
+
+
+void dccl::Codec::encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only /* = false */)
+{
+    const Descriptor* desc = msg.GetDescriptor();
+    Bitset head_bits;
+    Bitset body_bits;
+    encode_internal(msg, header_only, head_bits, body_bits);
+
+    std::string head_bytes = head_bits.to_byte_string();
+
+    dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_bytes.size() << "(" << head_bits.size() << ")" << std::endl;
+    dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl;
+    dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl;
+
+    std::string body_bytes;
+    if (!header_only)
+    {
+        body_bytes = body_bits.to_byte_string();
+
+        dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl;
+        dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
+        dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " <<  body_bytes.size() << "(" << body_bits.size() << ")" <<  std::endl;
+
+        if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc)))
+            encrypt(&body_bytes, head_bytes);
+
+        dlog.is(logger::DEBUG3, logger::ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
+    }
+
+    dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl;
+    *bytes += head_bytes + body_bytes;
+}
 
 unsigned dccl::Codec::id(const std::string& bytes)
 {
-    unsigned id_min_size = 0, id_max_size = 0;
-    id_codec()->field_min_size(&id_min_size, 0);
-    id_codec()->field_max_size(&id_max_size, 0);
-    
-    if(bytes.length() < (id_min_size / BITS_IN_BYTE))
-        throw(Exception("Bytes passed (hex: " + hex_encode(bytes) + ") is too small to be a valid DCCL message"));
-        
-    Bitset fixed_header_bits;
-    fixed_header_bits.from_byte_string(bytes.substr(0, (size_t)std::ceil(double(id_max_size) / BITS_IN_BYTE)));
-
-    Bitset these_bits(&fixed_header_bits);
-    these_bits.get_more_bits(id_min_size);
-
-    boost::any return_value;
-    id_codec()->field_decode(&these_bits, &return_value, 0);
-    
-    return boost::any_cast<uint32>(return_value);
+    return id(bytes.begin(), bytes.end());
 }
 
 
@@ -268,96 +300,7 @@
 
 void dccl::Codec::decode(const std::string& bytes, google::protobuf::Message* msg, bool header_only /* = false */)
 {
-    try
-    {
-        unsigned this_id = id(bytes);
-
-        dlog.is(DEBUG1, DECODE) && dlog  << "Began decoding message of id: " << this_id << std::endl;
-        
-        if(!id2desc_.count(this_id))
-            throw(Exception("Message id " + boost::lexical_cast<std::string>(this_id) + " has not been validated. Call validate() before decoding this type."));
-
-        const Descriptor* desc = msg->GetDescriptor();
-        
-        dlog.is(DEBUG1, DECODE) && dlog  << "Type name: " << desc->full_name() << std::endl;
-        
-        boost::shared_ptr<FieldCodecBase> codec = FieldCodecManager::find(desc);
-        boost::shared_ptr<internal::FromProtoCppTypeBase> helper = internal::TypeHelper::find(desc);
-        
-        if(codec)
-        {
-            unsigned head_size_bits;
-            unsigned body_size_bits;
-            codec->base_max_size(&head_size_bits, desc, HEAD);
-            codec->base_max_size(&body_size_bits, desc, BODY);
-            unsigned id_size = 0;
-            id_codec()->field_size(&id_size, this_id, 0);            
-            head_size_bits += id_size;
-        
-            unsigned head_size_bytes = ceil_bits2bytes(head_size_bits);
-            unsigned body_size_bytes = ceil_bits2bytes(body_size_bits);
-    
-            dlog.is(DEBUG2, DECODE) && dlog  << "Head bytes (bits): " << head_size_bytes << "(" << head_size_bits
-                                    << "), max body bytes (bits): " << body_size_bytes << "(" << body_size_bits << ")" <<  std::endl;
-
-            std::string head_bytes = bytes.substr(0, head_size_bytes);
-            dlog.is(DEBUG3, DECODE) && dlog  << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl;
-            
-            Bitset head_bits;
-            head_bits.from_byte_string(head_bytes);    
-            dlog.is(DEBUG3, DECODE) && dlog  << "Unencrypted Head (bin): " << head_bits << std::endl;
-
-            // shift off ID bits
-            head_bits >>= id_size;
-
-            dlog.is(DEBUG3, DECODE) && dlog  << "Unencrypted Head after ID bits removal (bin): " << head_bits << std::endl;
-
-            internal::MessageStack msg_stack;
-            msg_stack.push(msg->GetDescriptor());
-            
-            codec->base_decode(&head_bits, msg, HEAD);
-            dlog.is(DEBUG2, DECODE) && dlog  << "after header decode, message is: " << *msg << std::endl;
-
-
-            if(header_only)
-            {
-                dlog.is(DEBUG2, DECODE) && dlog  << "as requested, skipping decrypting and decoding body." << std::endl;
-            }
-            else
-            {
-                std::string body_bytes = bytes.substr(head_size_bytes);
-                dlog.is(DEBUG3, DECODE) && dlog  << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
-                
-                if(!crypto_key_.empty() && !skip_crypto_ids_.count(this_id))
-                    decrypt(&body_bytes, head_bytes);
-                
-                dlog.is(DEBUG3, DECODE) && dlog  << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
-                
-                Bitset body_bits;
-                body_bits.from_byte_string(body_bytes);
-                dlog.is(DEBUG3, DECODE) && dlog  << "Unencrypted Body (bin): " << body_bits << std::endl;
-                
-                codec->base_decode(&body_bits, msg, BODY);
-                dlog.is(DEBUG2, DECODE) && dlog  << "after header & body decode, message is: " << *msg << std::endl;
-            }
-        }
-        else
-        {
-            throw(Exception("Failed to find (dccl.msg).codec `" + desc->options().GetExtension(dccl::msg).codec() + "`"));
-        }
-
-        dlog.is(DEBUG1, DECODE) && dlog  << "Successfully decoded message of type: " << desc->full_name() << std::endl;
-    }
-    catch(std::exception& e)
-    {
-        std::stringstream ss;
-        
-        ss << "Message " << hex_encode(bytes) <<  " failed to decode. Reason: " << e.what() << std::endl;
-
-        dlog.is(DEBUG1, DECODE) && dlog << ss.str() << std::endl;  
-        throw(Exception(ss.str()));
-    }    
-
+    decode(bytes.begin(), bytes.end(), msg, header_only);
 }
 
 // makes sure we can actual encode / decode a message of this descriptor given the loaded FieldCodecs

=== modified file 'src/codec.h'
--- src/codec.h	2014-10-08 18:19:16 +0000
+++ src/codec.h	2015-01-22 21:49:07 +0000
@@ -178,6 +178,10 @@
         unsigned id(const std::string& bytes);
 
         /// \brief Provides the DCCL ID given a DCCL type.
+        template<typename CharIterator>
+        unsigned id(CharIterator begin, CharIterator end);
+
+        /// \brief Provides the DCCL ID given a DCCL type.
         unsigned id(const google::protobuf::Descriptor* desc) const {
             return desc->options().GetExtension(dccl::msg).id();
         }            
@@ -201,6 +205,26 @@
         /// \throw Exception if message cannot be encoded.
         void encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only = false);
             
+        /// \brief Encodes a DCCL message
+        ///
+        /// \param bytes Output buffer to store encoded msg
+        /// \param max_len Maximum size of output buffer
+        /// \param msg Message to encode (must already have been validated)
+        /// \param header_only If true, only decode the header (do not try to decrypt (if applicable) and decode the message body)
+        /// \throw Exception if message cannot be encoded.
+        /// \return size of encoded message
+        size_t encode(char* bytes, size_t max_len, const google::protobuf::Message& msg, bool header_only = false);
+
+        /// \brief Decode a DCCL message when the type is known at compile time.
+        ///
+        /// \param begin Iterator to the first byte of encoded message to decode (must already have been validated)
+        /// \param end Iterator pointing to the past-the-end character of the message.
+        /// \param msg Pointer to any Google Protobuf Message generated by protoc (i.e. subclass of google::protobuf::Message). The decoded message will be written here.
+        /// \param header_only If true, only decode the header (do not try to decrypt (if applicable) and decode the message body)
+        /// \throw Exception if message cannot be decoded.
+        template <typename CharIterator>
+        void decode(CharIterator begin, CharIterator end, google::protobuf::Message* msg, bool header_only = false);
+
         /// \brief Decode a DCCL message when the type is known at compile time.
         ///
         /// \param bytes encoded message to decode (must already have been validated)
@@ -266,6 +290,8 @@
         Codec(const Codec&);
         Codec& operator= (const Codec&);
 
+        void encode_internal(const google::protobuf::Message& msg, bool header_only, Bitset& header_bits, Bitset& body_bits);
+
         void encrypt(std::string* s, const std::string& nonce);
         void decrypt(std::string* s, const std::string& nonce);
 
@@ -326,4 +352,127 @@
     return msg;
 }
 
+template<typename CharIterator>
+unsigned dccl::Codec::id(CharIterator begin, CharIterator end)
+{
+    unsigned id_min_size = 0, id_max_size = 0;
+    id_codec()->field_min_size(&id_min_size, 0);
+    id_codec()->field_max_size(&id_max_size, 0);
+
+    if(std::distance(begin, end) < (id_min_size / BITS_IN_BYTE))
+        throw(Exception("Bytes passed (hex: " + hex_encode(begin, end) + ") is too small to be a valid DCCL message"));
+
+    Bitset fixed_header_bits;
+    fixed_header_bits.from_byte_stream(begin, begin+(size_t)std::ceil(double(id_max_size) / BITS_IN_BYTE));
+
+    Bitset these_bits(&fixed_header_bits);
+    these_bits.get_more_bits(id_min_size);
+
+    boost::any return_value;
+    id_codec()->field_decode(&these_bits, &return_value, 0);
+
+    return boost::any_cast<uint32>(return_value);
+}
+
+template <typename CharIterator>
+void dccl::Codec::decode(CharIterator begin, CharIterator end, google::protobuf::Message* msg, bool header_only /*= false*/)
+{
+    try
+    {
+        unsigned this_id = id(begin, end);
+
+        dlog.is(logger::DEBUG1, logger::DECODE) && dlog  << "Began decoding message of id: " << this_id << std::endl;
+
+        if(!id2desc_.count(this_id))
+            throw(Exception("Message id " + boost::lexical_cast<std::string>(this_id) + " has not been validated. Call validate() before decoding this type."));
+
+        const google::protobuf::Descriptor* desc = msg->GetDescriptor();
+
+        dlog.is(logger::DEBUG1, logger::DECODE) && dlog  << "Type name: " << desc->full_name() << std::endl;
+
+        boost::shared_ptr<FieldCodecBase> codec = FieldCodecManager::find(desc);
+        boost::shared_ptr<internal::FromProtoCppTypeBase> helper = internal::TypeHelper::find(desc);
+
+        if(codec)
+        {
+            unsigned head_size_bits;
+            unsigned body_size_bits;
+            codec->base_max_size(&head_size_bits, desc, HEAD);
+            codec->base_max_size(&body_size_bits, desc, BODY);
+            unsigned id_size = 0;
+            id_codec()->field_size(&id_size, this_id, 0);
+            head_size_bits += id_size;
+
+            unsigned head_size_bytes = ceil_bits2bytes(head_size_bits);
+            unsigned body_size_bytes = ceil_bits2bytes(body_size_bits);
+
+            dlog.is(logger::DEBUG2, logger::DECODE) && dlog  << "Head bytes (bits): " << head_size_bytes << "(" << head_size_bits
+                                    << "), max body bytes (bits): " << body_size_bytes << "(" << body_size_bits << ")" <<  std::endl;
+
+            CharIterator head_bytes_end = begin + head_size_bytes;
+            dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Head (hex): " << hex_encode(begin, head_bytes_end) << std::endl;
+
+            Bitset head_bits;
+            head_bits.from_byte_stream(begin, head_bytes_end);
+            dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Head (bin): " << head_bits << std::endl;
+
+            // shift off ID bits
+            head_bits >>= id_size;
+
+            dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Head after ID bits removal (bin): " << head_bits << std::endl;
+
+            internal::MessageStack msg_stack;
+            msg_stack.push(msg->GetDescriptor());
+
+            codec->base_decode(&head_bits, msg, HEAD);
+            dlog.is(logger::DEBUG2, logger::DECODE) && dlog  << "after header decode, message is: " << *msg << std::endl;
+
+
+            if(header_only)
+            {
+                dlog.is(logger::DEBUG2, logger::DECODE) && dlog  << "as requested, skipping decrypting and decoding body." << std::endl;
+            }
+            else
+            {
+                dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Encrypted Body (hex): " << hex_encode(head_bytes_end, end) << std::endl;
+
+                Bitset body_bits;
+                if(!crypto_key_.empty() && !skip_crypto_ids_.count(this_id))
+                {
+                    std::string head_bytes(begin, head_bytes_end);
+                    std::string body_bytes(head_bytes_end, end);
+                    decrypt(&body_bytes, head_bytes);
+                    dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl;
+                    body_bits.from_byte_stream(body_bytes.begin(), body_bytes.end());
+                }
+                else
+                {
+                    dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Body (hex): " << hex_encode(head_bytes_end, end) << std::endl;
+                    body_bits.from_byte_stream(head_bytes_end, end);
+                }
+
+                dlog.is(logger::DEBUG3, logger::DECODE) && dlog  << "Unencrypted Body (bin): " << body_bits << std::endl;
+
+                codec->base_decode(&body_bits, msg, BODY);
+                dlog.is(logger::DEBUG2, logger::DECODE) && dlog  << "after header & body decode, message is: " << *msg << std::endl;
+            }
+        }
+        else
+        {
+            throw(Exception("Failed to find (dccl.msg).codec `" + desc->options().GetExtension(dccl::msg).codec() + "`"));
+        }
+
+        dlog.is(logger::DEBUG1, logger::DECODE) && dlog  << "Successfully decoded message of type: " << desc->full_name() << std::endl;
+    }
+    catch(std::exception& e)
+    {
+        std::stringstream ss;
+
+        ss << "Message " << hex_encode(begin, end) <<  " failed to decode. Reason: " << e.what() << std::endl;
+
+        dlog.is(logger::DEBUG1, logger::DECODE) && dlog << ss.str() << std::endl;
+        throw(Exception(ss.str()));
+    }
+}
+
 #endif