Merge lp:~nknotts/dccl/encode-decode-iterators into lp:~dccl-dev/dccl/3.0
- encode-decode-iterators
- Merge into 3.0
Status: | Merged |
---|---|
Merged at revision: | 314 |
Proposed branch: | lp:~nknotts/dccl/encode-decode-iterators |
Merge into: | lp:~dccl-dev/dccl/3.0 |
Diff against target: |
591 lines (+297/-152) 4 files modified
src/binary.h (+34/-11) src/bitset.h (+33/-3) src/codec.cpp (+81/-138) src/codec.h (+149/-0) |
To merge this branch: | bzr merge lp:~nknotts/dccl/encode-decode-iterators |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
toby schneider | Approve | ||
Review via email:
|
Commit message
Description of the change
I added an InputIterator overload to codec::decode
I also added a char* overload to codec::encode
For our application, our stream data is stored in a boost::
On the codec::encode side, I added a char* overload to reduce/eliminate allocations.
All unit tests that previously passed still pass.
However, some tests are failing (they were failing before this change)
The following tests FAILED:
12 - dccl_test_ccl (OTHER_FAULT)
13 - dccl_test_
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
toby schneider (tes) wrote : | # |
- 315. By Nathan Knotts
-
src/codec.h: moved template implementations to bottom of file
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nathan Knotts (nknotts) wrote : | # |
I am currently building on OSX. Based on the PPAs, I'm assuming you're building on Ubuntu/Debian, is that correct? I will try and track down why the tests are failing on my system. Any solution will be committed in a separate merge request.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
toby schneider (tes) wrote : | # |
Yes, I use Debian 6, 7 and Ubuntu 12.04/14.04 regularly. Ubuntu 14.04 is what I just tested your branch on.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nathan Knotts (nknotts) wrote : | # |
I did not realize those particular tests were using shared libraries. After setting my DYLD_LIBRARY_PATH, the tests now succeed.
- 316. By Nathan Knotts
-
bitset.h: 0 initialize to_byte_string output buffer
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
toby schneider (tes) wrote : | # |
Appears to be a clean implementation of char iterator overloads for Codec::encode, decode and related helper functions.
Preview Diff
1 | === modified file 'src/binary.h' | |||
2 | --- src/binary.h 2014-10-08 18:19:16 +0000 | |||
3 | +++ src/binary.h 2015-01-22 21:49:07 +0000 | |||
4 | @@ -85,27 +85,30 @@ | |||
5 | 85 | return out; | 85 | return out; |
6 | 86 | } | 86 | } |
7 | 87 | 87 | ||
9 | 88 | /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `in` is written to index 0 and 1 (first byte) of `out` | 88 | /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `begin` is written to index 0 and 1 (first byte) of `out` |
10 | 89 | /// | 89 | /// |
12 | 90 | /// \param in byte string to encode (e.g. "TOM") | 90 | /// \param begin iterator to first byte of string to encode (e.g. "TOM") |
13 | 91 | /// \param end iterator pointing to the past-the-end character of the string. | ||
14 | 91 | /// \param out pointer to string to store result (e.g. "544f4d") | 92 | /// \param out pointer to string to store result (e.g. "544f4d") |
15 | 92 | /// \param upper_case set true to use upper case for the alphabet characters (i.e. A,B,C,D,E,F), otherwise lowercase is used (a,b,c,d,e,f). | 93 | /// \param upper_case set true to use upper case for the alphabet characters (i.e. A,B,C,D,E,F), otherwise lowercase is used (a,b,c,d,e,f). |
17 | 93 | inline void hex_encode(const std::string& in, std::string* out, bool upper_case = false) | 94 | template <typename CharIterator> |
18 | 95 | inline void hex_encode(CharIterator begin, CharIterator end, std::string* out, bool upper_case = false) | ||
19 | 94 | { | 96 | { |
20 | 95 | static const short char0_9_to_number = 48; | 97 | static const short char0_9_to_number = 48; |
21 | 96 | static const short charA_F_to_number = 55; | 98 | static const short charA_F_to_number = 55; |
22 | 97 | static const short chara_f_to_number = 87; | 99 | static const short chara_f_to_number = 87; |
23 | 98 | 100 | ||
27 | 99 | int in_size = in.size(); | 101 | size_t in_size = std::distance(begin, end); |
28 | 100 | int out_size = in_size << 1; | 102 | size_t out_size = in_size << 1; |
29 | 101 | 103 | ||
30 | 104 | out->clear(); | ||
31 | 102 | out->resize(out_size); | 105 | out->resize(out_size); |
35 | 103 | for(int i = 0, n = in_size; | 106 | |
36 | 104 | i < n; | 107 | size_t i = 0; |
37 | 105 | ++i) | 108 | for(CharIterator it = begin; it != end; ++it) |
38 | 106 | { | 109 | { |
41 | 107 | short msn = (in[i] >> 4) & 0x0f; | 110 | short msn = (*it >> 4) & 0x0f; |
42 | 108 | short lsn = in[i] & 0x0f; | 111 | short lsn = *it & 0x0f; |
43 | 109 | 112 | ||
44 | 110 | if(msn >= 0 && msn <= 9) | 113 | if(msn >= 0 && msn <= 9) |
45 | 111 | (*out)[2*i] = msn + char0_9_to_number; | 114 | (*out)[2*i] = msn + char0_9_to_number; |
46 | @@ -117,9 +120,29 @@ | |||
47 | 117 | else if(lsn >= 10 && lsn <= 15) | 120 | else if(lsn >= 10 && lsn <= 15) |
48 | 118 | (*out)[2*i+1] = lsn + (upper_case ? charA_F_to_number : chara_f_to_number); | 121 | (*out)[2*i+1] = lsn + (upper_case ? charA_F_to_number : chara_f_to_number); |
49 | 119 | 122 | ||
50 | 123 | i++; | ||
51 | 120 | } | 124 | } |
52 | 121 | } | 125 | } |
53 | 122 | 126 | ||
54 | 127 | template <typename CharIterator> | ||
55 | 128 | inline std::string hex_encode(CharIterator begin, CharIterator end) | ||
56 | 129 | { | ||
57 | 130 | std::string out; | ||
58 | 131 | hex_encode(begin, end, &out); | ||
59 | 132 | return out; | ||
60 | 133 | |||
61 | 134 | } | ||
62 | 135 | |||
63 | 136 | /// \brief Encodes a (little-endian) hexadecimal string from a byte string. Index 0 of `in` is written to index 0 and 1 (first byte) of `out` | ||
64 | 137 | /// | ||
65 | 138 | /// \param in byte string to encode (e.g. "TOM") | ||
66 | 139 | /// \param out pointer to string to store result (e.g. "544f4d") | ||
67 | 140 | /// \param upper_case set true to use upper case for the alphabet characters (i.e. A,B,C,D,E,F), otherwise lowercase is used (a,b,c,d,e,f). | ||
68 | 141 | inline void hex_encode(const std::string& in, std::string* out, bool upper_case = false) | ||
69 | 142 | { | ||
70 | 143 | hex_encode(in.begin(), in.end(), out, upper_case); | ||
71 | 144 | } | ||
72 | 145 | |||
73 | 123 | inline std::string hex_encode(const std::string& in) | 146 | inline std::string hex_encode(const std::string& in) |
74 | 124 | { | 147 | { |
75 | 125 | std::string out; | 148 | std::string out; |
76 | 126 | 149 | ||
77 | === modified file 'src/bitset.h' | |||
78 | --- src/bitset.h 2015-01-14 01:25:28 +0000 | |||
79 | +++ src/bitset.h 2015-01-22 21:49:07 +0000 | |||
80 | @@ -28,6 +28,7 @@ | |||
81 | 28 | #include <algorithm> | 28 | #include <algorithm> |
82 | 29 | #include <limits> | 29 | #include <limits> |
83 | 30 | #include <string> | 30 | #include <string> |
84 | 31 | #include <cassert> | ||
85 | 31 | 32 | ||
86 | 32 | #include "exception.h" | 33 | #include "exception.h" |
87 | 33 | 34 | ||
88 | @@ -319,15 +320,44 @@ | |||
89 | 319 | return s; | 320 | return s; |
90 | 320 | } | 321 | } |
91 | 321 | 322 | ||
92 | 323 | /// \brief Generate a byte string representation of the Bitset, where each character represents 8 bits of the Bitset. The string is used as a byte container, and is not intended to be printed. | ||
93 | 324 | /// \param buf An output string containing the value of the Bitset, with the least signficant byte in string[0] and the most significant byte in string[size()-1] | ||
94 | 325 | /// \param max_len Maximum length of buf | ||
95 | 326 | /// \return number of bytes written to buf | ||
96 | 327 | size_t to_byte_string(char* buf, size_t max_len) | ||
97 | 328 | { | ||
98 | 329 | // number of bytes needed is ceil(size() / 8) | ||
99 | 330 | size_t len = this->size()/8 + (this->size()%8 ? 1 : 0); | ||
100 | 331 | |||
101 | 332 | assert( max_len >= len ); | ||
102 | 333 | |||
103 | 334 | // initialize buffer to all zeroes | ||
104 | 335 | std::memset(buf, 0, len); | ||
105 | 336 | |||
106 | 337 | for(size_type i = 0, n = this->size(); i < n; ++i) | ||
107 | 338 | buf[i/8] |= static_cast<char>((*this)[i] << (i%8)); | ||
108 | 339 | |||
109 | 340 | return len; | ||
110 | 341 | } | ||
111 | 342 | |||
112 | 322 | /// \brief Sets the value of the Bitset to the contents of a byte string, where each character represents 8 bits of the Bitset. | 343 | /// \brief Sets the value of the Bitset to the contents of a byte string, where each character represents 8 bits of the Bitset. |
113 | 323 | /// | 344 | /// |
114 | 324 | /// \param s A string container the values where the least signficant byte in string[0] and the most significant byte in string[size()-1] | 345 | /// \param s A string container the values where the least signficant byte in string[0] and the most significant byte in string[size()-1] |
115 | 325 | void from_byte_string(const std::string& s) | 346 | void from_byte_string(const std::string& s) |
116 | 326 | { | 347 | { |
118 | 327 | this->resize(s.size() * 8); | 348 | from_byte_stream(s.begin(), s.end()); |
119 | 349 | } | ||
120 | 350 | |||
121 | 351 | /// \brief Sets the value of the Bitset to the contents of a byte string, where each character represents 8 bits of the Bitset. | ||
122 | 352 | /// A string container the values where the least signficant byte in string[0] and the most significant byte in string[size()-1] | ||
123 | 353 | /// \param begin Iterator pointing to the begining of the input buffer | ||
124 | 354 | /// \param end Iterator pointing to the end of the input bufer | ||
125 | 355 | template<typename CharIterator> | ||
126 | 356 | void from_byte_stream(CharIterator begin, CharIterator end) | ||
127 | 357 | { | ||
128 | 358 | this->resize(std::distance(begin, end) * 8); | ||
129 | 328 | int i = 0; | 359 | int i = 0; |
132 | 329 | for(std::string::const_iterator it = s.begin(), n = s.end(); | 360 | for(CharIterator it = begin; it != end; ++it) |
131 | 330 | it != n; ++it) | ||
133 | 331 | { | 361 | { |
134 | 332 | for(size_type j = 0; j < 8; ++j) | 362 | for(size_type j = 0; j < 8; ++j) |
135 | 333 | (*this)[i*8+j] = (*it) & (1 << j); | 363 | (*this)[i*8+j] = (*it) & (1 << j); |
136 | 334 | 364 | ||
137 | === modified file 'src/codec.cpp' | |||
138 | --- src/codec.cpp 2014-10-05 18:36:46 +0000 | |||
139 | +++ src/codec.cpp 2015-01-22 21:49:07 +0000 | |||
140 | @@ -150,8 +150,7 @@ | |||
141 | 150 | } | 150 | } |
142 | 151 | } | 151 | } |
143 | 152 | 152 | ||
146 | 153 | 153 | void dccl::Codec::encode_internal(const google::protobuf::Message& msg, bool header_only, Bitset& head_bits, Bitset& body_bits) | |
145 | 154 | void dccl::Codec::encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only /* = false */) | ||
147 | 155 | { | 154 | { |
148 | 156 | const Descriptor* desc = msg.GetDescriptor(); | 155 | const Descriptor* desc = msg.GetDescriptor(); |
149 | 157 | 156 | ||
150 | @@ -159,6 +158,9 @@ | |||
151 | 159 | 158 | ||
152 | 160 | try | 159 | try |
153 | 161 | { | 160 | { |
154 | 161 | size_t head_byte_size = 0; | ||
155 | 162 | size_t body_byte_size = 0; | ||
156 | 163 | |||
157 | 162 | if(!msg.IsInitialized() && !header_only) | 164 | if(!msg.IsInitialized() && !header_only) |
158 | 163 | throw(Exception("Message is not properly initialized. All `required` fields must be set.")); | 165 | throw(Exception("Message is not properly initialized. All `required` fields must be set.")); |
159 | 164 | 166 | ||
160 | @@ -173,26 +175,15 @@ | |||
161 | 173 | if(codec) | 175 | if(codec) |
162 | 174 | { | 176 | { |
163 | 175 | //fixed header | 177 | //fixed header |
164 | 176 | Bitset head_bits; | ||
165 | 177 | id_codec()->field_encode(&head_bits, id(desc), 0); | 178 | id_codec()->field_encode(&head_bits, id(desc), 0); |
166 | 178 | 179 | ||
167 | 179 | internal::MessageStack msg_stack; | 180 | internal::MessageStack msg_stack; |
168 | 180 | msg_stack.push(msg.GetDescriptor()); | 181 | msg_stack.push(msg.GetDescriptor()); |
169 | 181 | codec->base_encode(&head_bits, msg, HEAD); | 182 | codec->base_encode(&head_bits, msg, HEAD); |
170 | 182 | 183 | ||
171 | 183 | std::string body_bytes; | ||
172 | 184 | |||
173 | 185 | // given header of not even byte size (e.g. 01011), make even byte size (e.g. 00001011) | 184 | // given header of not even byte size (e.g. 01011), make even byte size (e.g. 00001011) |
184 | 186 | unsigned head_byte_size = ceil_bits2bytes(head_bits.size()); | 185 | head_byte_size = ceil_bits2bytes(head_bits.size()); |
185 | 187 | unsigned head_bits_diff = head_byte_size * BITS_IN_BYTE - (head_bits.size()); | 186 | head_bits.resize(head_byte_size * BITS_IN_BYTE); |
176 | 188 | head_bits.resize(head_bits.size() + head_bits_diff); | ||
177 | 189 | |||
178 | 190 | std::string head_bytes = head_bits.to_byte_string(); | ||
179 | 191 | |||
180 | 192 | dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_bytes.size() << "(" << head_bits.size() << ")" << std::endl; | ||
181 | 193 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl; | ||
182 | 194 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl; | ||
183 | 195 | |||
186 | 196 | 187 | ||
187 | 197 | if(header_only) | 188 | if(header_only) |
188 | 198 | { | 189 | { |
189 | @@ -200,23 +191,8 @@ | |||
190 | 200 | } | 191 | } |
191 | 201 | else | 192 | else |
192 | 202 | { | 193 | { |
193 | 203 | Bitset body_bits; | ||
194 | 204 | codec->base_encode(&body_bits, msg, BODY); | 194 | codec->base_encode(&body_bits, msg, BODY); |
195 | 205 | body_bytes = body_bits.to_byte_string(); | ||
196 | 206 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl; | ||
197 | 207 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
198 | 208 | dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " << body_bytes.size() << "(" << body_bits.size() << ")" << std::endl; | ||
199 | 209 | |||
200 | 210 | if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc))) | ||
201 | 211 | encrypt(&body_bytes, head_bytes); | ||
202 | 212 | |||
203 | 213 | dlog.is(DEBUG3, ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
204 | 214 | } | 195 | } |
205 | 215 | |||
206 | 216 | dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl; | ||
207 | 217 | |||
208 | 218 | *bytes += head_bytes + body_bytes; | ||
209 | 219 | |||
210 | 220 | } | 196 | } |
211 | 221 | else | 197 | else |
212 | 222 | { | 198 | { |
213 | @@ -235,27 +211,83 @@ | |||
214 | 235 | } | 211 | } |
215 | 236 | } | 212 | } |
216 | 237 | 213 | ||
218 | 238 | 214 | size_t dccl::Codec::encode(char* bytes, size_t max_len, const google::protobuf::Message& msg, bool header_only /* = false */) | |
219 | 215 | { | ||
220 | 216 | const Descriptor* desc = msg.GetDescriptor(); | ||
221 | 217 | Bitset head_bits; | ||
222 | 218 | Bitset body_bits; | ||
223 | 219 | encode_internal(msg, header_only, head_bits, body_bits); | ||
224 | 220 | |||
225 | 221 | size_t head_byte_size = ceil_bits2bytes(head_bits.size()); | ||
226 | 222 | assert(max_len >= head_byte_size); | ||
227 | 223 | head_bits.to_byte_string(bytes, head_byte_size); | ||
228 | 224 | |||
229 | 225 | dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_byte_size << "(" << head_bits.size() << ")" << std::endl; | ||
230 | 226 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl; | ||
231 | 227 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(bytes, bytes+head_byte_size) << std::endl; | ||
232 | 228 | |||
233 | 229 | size_t body_byte_size = 0; | ||
234 | 230 | if (!header_only) | ||
235 | 231 | { | ||
236 | 232 | body_byte_size = ceil_bits2bytes(body_bits.size()); | ||
237 | 233 | assert(max_len >= head_byte_size + body_byte_size); | ||
238 | 234 | body_bits.to_byte_string(bytes+head_byte_size, max_len-head_byte_size); | ||
239 | 235 | |||
240 | 236 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl; | ||
241 | 237 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(bytes+head_byte_size, bytes+head_byte_size+body_byte_size) << std::endl; | ||
242 | 238 | dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " << body_byte_size << "(" << body_bits.size() << ")" << std::endl; | ||
243 | 239 | |||
244 | 240 | if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc))) { | ||
245 | 241 | std::string head_bytes(bytes, bytes+head_byte_size); | ||
246 | 242 | std::string body_bytes(bytes+head_byte_size, bytes+head_byte_size+body_byte_size); | ||
247 | 243 | encrypt(&body_bytes, head_bytes); | ||
248 | 244 | std::memcpy(bytes+head_byte_size, body_bytes.data(), body_bytes.size()); | ||
249 | 245 | } | ||
250 | 246 | |||
251 | 247 | dlog.is(logger::DEBUG3, logger::ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(bytes+head_byte_size, bytes+head_byte_size+body_byte_size) << std::endl; | ||
252 | 248 | } | ||
253 | 249 | |||
254 | 250 | dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl; | ||
255 | 251 | |||
256 | 252 | return head_byte_size + body_byte_size; | ||
257 | 253 | } | ||
258 | 254 | |||
259 | 255 | |||
260 | 256 | void dccl::Codec::encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only /* = false */) | ||
261 | 257 | { | ||
262 | 258 | const Descriptor* desc = msg.GetDescriptor(); | ||
263 | 259 | Bitset head_bits; | ||
264 | 260 | Bitset body_bits; | ||
265 | 261 | encode_internal(msg, header_only, head_bits, body_bits); | ||
266 | 262 | |||
267 | 263 | std::string head_bytes = head_bits.to_byte_string(); | ||
268 | 264 | |||
269 | 265 | dlog.is(DEBUG2, ENCODE) && dlog << "Head bytes (bits): " << head_bytes.size() << "(" << head_bits.size() << ")" << std::endl; | ||
270 | 266 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl; | ||
271 | 267 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl; | ||
272 | 268 | |||
273 | 269 | std::string body_bytes; | ||
274 | 270 | if (!header_only) | ||
275 | 271 | { | ||
276 | 272 | body_bytes = body_bits.to_byte_string(); | ||
277 | 273 | |||
278 | 274 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl; | ||
279 | 275 | dlog.is(DEBUG3, ENCODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
280 | 276 | dlog.is(DEBUG2, ENCODE) && dlog << "Body bytes (bits): " << body_bytes.size() << "(" << body_bits.size() << ")" << std::endl; | ||
281 | 277 | |||
282 | 278 | if(!crypto_key_.empty() && !skip_crypto_ids_.count(id(desc))) | ||
283 | 279 | encrypt(&body_bytes, head_bytes); | ||
284 | 280 | |||
285 | 281 | dlog.is(logger::DEBUG3, logger::ENCODE) && dlog << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
286 | 282 | } | ||
287 | 283 | |||
288 | 284 | dlog.is(DEBUG1, ENCODE) && dlog << "Successfully encoded message of type: " << desc->full_name() << std::endl; | ||
289 | 285 | *bytes += head_bytes + body_bytes; | ||
290 | 286 | } | ||
291 | 239 | 287 | ||
292 | 240 | unsigned dccl::Codec::id(const std::string& bytes) | 288 | unsigned dccl::Codec::id(const std::string& bytes) |
293 | 241 | { | 289 | { |
311 | 242 | unsigned id_min_size = 0, id_max_size = 0; | 290 | return id(bytes.begin(), bytes.end()); |
295 | 243 | id_codec()->field_min_size(&id_min_size, 0); | ||
296 | 244 | id_codec()->field_max_size(&id_max_size, 0); | ||
297 | 245 | |||
298 | 246 | if(bytes.length() < (id_min_size / BITS_IN_BYTE)) | ||
299 | 247 | throw(Exception("Bytes passed (hex: " + hex_encode(bytes) + ") is too small to be a valid DCCL message")); | ||
300 | 248 | |||
301 | 249 | Bitset fixed_header_bits; | ||
302 | 250 | fixed_header_bits.from_byte_string(bytes.substr(0, (size_t)std::ceil(double(id_max_size) / BITS_IN_BYTE))); | ||
303 | 251 | |||
304 | 252 | Bitset these_bits(&fixed_header_bits); | ||
305 | 253 | these_bits.get_more_bits(id_min_size); | ||
306 | 254 | |||
307 | 255 | boost::any return_value; | ||
308 | 256 | id_codec()->field_decode(&these_bits, &return_value, 0); | ||
309 | 257 | |||
310 | 258 | return boost::any_cast<uint32>(return_value); | ||
312 | 259 | } | 291 | } |
313 | 260 | 292 | ||
314 | 261 | 293 | ||
315 | @@ -268,96 +300,7 @@ | |||
316 | 268 | 300 | ||
317 | 269 | void dccl::Codec::decode(const std::string& bytes, google::protobuf::Message* msg, bool header_only /* = false */) | 301 | void dccl::Codec::decode(const std::string& bytes, google::protobuf::Message* msg, bool header_only /* = false */) |
318 | 270 | { | 302 | { |
409 | 271 | try | 303 | decode(bytes.begin(), bytes.end(), msg, header_only); |
320 | 272 | { | ||
321 | 273 | unsigned this_id = id(bytes); | ||
322 | 274 | |||
323 | 275 | dlog.is(DEBUG1, DECODE) && dlog << "Began decoding message of id: " << this_id << std::endl; | ||
324 | 276 | |||
325 | 277 | if(!id2desc_.count(this_id)) | ||
326 | 278 | throw(Exception("Message id " + boost::lexical_cast<std::string>(this_id) + " has not been validated. Call validate() before decoding this type.")); | ||
327 | 279 | |||
328 | 280 | const Descriptor* desc = msg->GetDescriptor(); | ||
329 | 281 | |||
330 | 282 | dlog.is(DEBUG1, DECODE) && dlog << "Type name: " << desc->full_name() << std::endl; | ||
331 | 283 | |||
332 | 284 | boost::shared_ptr<FieldCodecBase> codec = FieldCodecManager::find(desc); | ||
333 | 285 | boost::shared_ptr<internal::FromProtoCppTypeBase> helper = internal::TypeHelper::find(desc); | ||
334 | 286 | |||
335 | 287 | if(codec) | ||
336 | 288 | { | ||
337 | 289 | unsigned head_size_bits; | ||
338 | 290 | unsigned body_size_bits; | ||
339 | 291 | codec->base_max_size(&head_size_bits, desc, HEAD); | ||
340 | 292 | codec->base_max_size(&body_size_bits, desc, BODY); | ||
341 | 293 | unsigned id_size = 0; | ||
342 | 294 | id_codec()->field_size(&id_size, this_id, 0); | ||
343 | 295 | head_size_bits += id_size; | ||
344 | 296 | |||
345 | 297 | unsigned head_size_bytes = ceil_bits2bytes(head_size_bits); | ||
346 | 298 | unsigned body_size_bytes = ceil_bits2bytes(body_size_bits); | ||
347 | 299 | |||
348 | 300 | dlog.is(DEBUG2, DECODE) && dlog << "Head bytes (bits): " << head_size_bytes << "(" << head_size_bits | ||
349 | 301 | << "), max body bytes (bits): " << body_size_bytes << "(" << body_size_bits << ")" << std::endl; | ||
350 | 302 | |||
351 | 303 | std::string head_bytes = bytes.substr(0, head_size_bytes); | ||
352 | 304 | dlog.is(DEBUG3, DECODE) && dlog << "Unencrypted Head (hex): " << hex_encode(head_bytes) << std::endl; | ||
353 | 305 | |||
354 | 306 | Bitset head_bits; | ||
355 | 307 | head_bits.from_byte_string(head_bytes); | ||
356 | 308 | dlog.is(DEBUG3, DECODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl; | ||
357 | 309 | |||
358 | 310 | // shift off ID bits | ||
359 | 311 | head_bits >>= id_size; | ||
360 | 312 | |||
361 | 313 | dlog.is(DEBUG3, DECODE) && dlog << "Unencrypted Head after ID bits removal (bin): " << head_bits << std::endl; | ||
362 | 314 | |||
363 | 315 | internal::MessageStack msg_stack; | ||
364 | 316 | msg_stack.push(msg->GetDescriptor()); | ||
365 | 317 | |||
366 | 318 | codec->base_decode(&head_bits, msg, HEAD); | ||
367 | 319 | dlog.is(DEBUG2, DECODE) && dlog << "after header decode, message is: " << *msg << std::endl; | ||
368 | 320 | |||
369 | 321 | |||
370 | 322 | if(header_only) | ||
371 | 323 | { | ||
372 | 324 | dlog.is(DEBUG2, DECODE) && dlog << "as requested, skipping decrypting and decoding body." << std::endl; | ||
373 | 325 | } | ||
374 | 326 | else | ||
375 | 327 | { | ||
376 | 328 | std::string body_bytes = bytes.substr(head_size_bytes); | ||
377 | 329 | dlog.is(DEBUG3, DECODE) && dlog << "Encrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
378 | 330 | |||
379 | 331 | if(!crypto_key_.empty() && !skip_crypto_ids_.count(this_id)) | ||
380 | 332 | decrypt(&body_bytes, head_bytes); | ||
381 | 333 | |||
382 | 334 | dlog.is(DEBUG3, DECODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
383 | 335 | |||
384 | 336 | Bitset body_bits; | ||
385 | 337 | body_bits.from_byte_string(body_bytes); | ||
386 | 338 | dlog.is(DEBUG3, DECODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl; | ||
387 | 339 | |||
388 | 340 | codec->base_decode(&body_bits, msg, BODY); | ||
389 | 341 | dlog.is(DEBUG2, DECODE) && dlog << "after header & body decode, message is: " << *msg << std::endl; | ||
390 | 342 | } | ||
391 | 343 | } | ||
392 | 344 | else | ||
393 | 345 | { | ||
394 | 346 | throw(Exception("Failed to find (dccl.msg).codec `" + desc->options().GetExtension(dccl::msg).codec() + "`")); | ||
395 | 347 | } | ||
396 | 348 | |||
397 | 349 | dlog.is(DEBUG1, DECODE) && dlog << "Successfully decoded message of type: " << desc->full_name() << std::endl; | ||
398 | 350 | } | ||
399 | 351 | catch(std::exception& e) | ||
400 | 352 | { | ||
401 | 353 | std::stringstream ss; | ||
402 | 354 | |||
403 | 355 | ss << "Message " << hex_encode(bytes) << " failed to decode. Reason: " << e.what() << std::endl; | ||
404 | 356 | |||
405 | 357 | dlog.is(DEBUG1, DECODE) && dlog << ss.str() << std::endl; | ||
406 | 358 | throw(Exception(ss.str())); | ||
407 | 359 | } | ||
408 | 360 | |||
410 | 361 | } | 304 | } |
411 | 362 | 305 | ||
412 | 363 | // makes sure we can actual encode / decode a message of this descriptor given the loaded FieldCodecs | 306 | // makes sure we can actual encode / decode a message of this descriptor given the loaded FieldCodecs |
413 | 364 | 307 | ||
414 | === modified file 'src/codec.h' | |||
415 | --- src/codec.h 2014-10-08 18:19:16 +0000 | |||
416 | +++ src/codec.h 2015-01-22 21:49:07 +0000 | |||
417 | @@ -178,6 +178,10 @@ | |||
418 | 178 | unsigned id(const std::string& bytes); | 178 | unsigned id(const std::string& bytes); |
419 | 179 | 179 | ||
420 | 180 | /// \brief Provides the DCCL ID given a DCCL type. | 180 | /// \brief Provides the DCCL ID given a DCCL type. |
421 | 181 | template<typename CharIterator> | ||
422 | 182 | unsigned id(CharIterator begin, CharIterator end); | ||
423 | 183 | |||
424 | 184 | /// \brief Provides the DCCL ID given a DCCL type. | ||
425 | 181 | unsigned id(const google::protobuf::Descriptor* desc) const { | 185 | unsigned id(const google::protobuf::Descriptor* desc) const { |
426 | 182 | return desc->options().GetExtension(dccl::msg).id(); | 186 | return desc->options().GetExtension(dccl::msg).id(); |
427 | 183 | } | 187 | } |
428 | @@ -201,6 +205,26 @@ | |||
429 | 201 | /// \throw Exception if message cannot be encoded. | 205 | /// \throw Exception if message cannot be encoded. |
430 | 202 | void encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only = false); | 206 | void encode(std::string* bytes, const google::protobuf::Message& msg, bool header_only = false); |
431 | 203 | 207 | ||
432 | 208 | /// \brief Encodes a DCCL message | ||
433 | 209 | /// | ||
434 | 210 | /// \param bytes Output buffer to store encoded msg | ||
435 | 211 | /// \param max_len Maximum size of output buffer | ||
436 | 212 | /// \param msg Message to encode (must already have been validated) | ||
437 | 213 | /// \param header_only If true, only decode the header (do not try to decrypt (if applicable) and decode the message body) | ||
438 | 214 | /// \throw Exception if message cannot be encoded. | ||
439 | 215 | /// \return size of encoded message | ||
440 | 216 | size_t encode(char* bytes, size_t max_len, const google::protobuf::Message& msg, bool header_only = false); | ||
441 | 217 | |||
442 | 218 | /// \brief Decode a DCCL message when the type is known at compile time. | ||
443 | 219 | /// | ||
444 | 220 | /// \param begin Iterator to the first byte of encoded message to decode (must already have been validated) | ||
445 | 221 | /// \param end Iterator pointing to the past-the-end character of the message. | ||
446 | 222 | /// \param msg Pointer to any Google Protobuf Message generated by protoc (i.e. subclass of google::protobuf::Message). The decoded message will be written here. | ||
447 | 223 | /// \param header_only If true, only decode the header (do not try to decrypt (if applicable) and decode the message body) | ||
448 | 224 | /// \throw Exception if message cannot be decoded. | ||
449 | 225 | template <typename CharIterator> | ||
450 | 226 | void decode(CharIterator begin, CharIterator end, google::protobuf::Message* msg, bool header_only = false); | ||
451 | 227 | |||
452 | 204 | /// \brief Decode a DCCL message when the type is known at compile time. | 228 | /// \brief Decode a DCCL message when the type is known at compile time. |
453 | 205 | /// | 229 | /// |
454 | 206 | /// \param bytes encoded message to decode (must already have been validated) | 230 | /// \param bytes encoded message to decode (must already have been validated) |
455 | @@ -266,6 +290,8 @@ | |||
456 | 266 | Codec(const Codec&); | 290 | Codec(const Codec&); |
457 | 267 | Codec& operator= (const Codec&); | 291 | Codec& operator= (const Codec&); |
458 | 268 | 292 | ||
459 | 293 | void encode_internal(const google::protobuf::Message& msg, bool header_only, Bitset& header_bits, Bitset& body_bits); | ||
460 | 294 | |||
461 | 269 | void encrypt(std::string* s, const std::string& nonce); | 295 | void encrypt(std::string* s, const std::string& nonce); |
462 | 270 | void decrypt(std::string* s, const std::string& nonce); | 296 | void decrypt(std::string* s, const std::string& nonce); |
463 | 271 | 297 | ||
464 | @@ -326,4 +352,127 @@ | |||
465 | 326 | return msg; | 352 | return msg; |
466 | 327 | } | 353 | } |
467 | 328 | 354 | ||
468 | 355 | template<typename CharIterator> | ||
469 | 356 | unsigned dccl::Codec::id(CharIterator begin, CharIterator end) | ||
470 | 357 | { | ||
471 | 358 | unsigned id_min_size = 0, id_max_size = 0; | ||
472 | 359 | id_codec()->field_min_size(&id_min_size, 0); | ||
473 | 360 | id_codec()->field_max_size(&id_max_size, 0); | ||
474 | 361 | |||
475 | 362 | if(std::distance(begin, end) < (id_min_size / BITS_IN_BYTE)) | ||
476 | 363 | throw(Exception("Bytes passed (hex: " + hex_encode(begin, end) + ") is too small to be a valid DCCL message")); | ||
477 | 364 | |||
478 | 365 | Bitset fixed_header_bits; | ||
479 | 366 | fixed_header_bits.from_byte_stream(begin, begin+(size_t)std::ceil(double(id_max_size) / BITS_IN_BYTE)); | ||
480 | 367 | |||
481 | 368 | Bitset these_bits(&fixed_header_bits); | ||
482 | 369 | these_bits.get_more_bits(id_min_size); | ||
483 | 370 | |||
484 | 371 | boost::any return_value; | ||
485 | 372 | id_codec()->field_decode(&these_bits, &return_value, 0); | ||
486 | 373 | |||
487 | 374 | return boost::any_cast<uint32>(return_value); | ||
488 | 375 | } | ||
489 | 376 | |||
490 | 377 | template <typename CharIterator> | ||
491 | 378 | void dccl::Codec::decode(CharIterator begin, CharIterator end, google::protobuf::Message* msg, bool header_only /*= false*/) | ||
492 | 379 | { | ||
493 | 380 | try | ||
494 | 381 | { | ||
495 | 382 | unsigned this_id = id(begin, end); | ||
496 | 383 | |||
497 | 384 | dlog.is(logger::DEBUG1, logger::DECODE) && dlog << "Began decoding message of id: " << this_id << std::endl; | ||
498 | 385 | |||
499 | 386 | if(!id2desc_.count(this_id)) | ||
500 | 387 | throw(Exception("Message id " + boost::lexical_cast<std::string>(this_id) + " has not been validated. Call validate() before decoding this type.")); | ||
501 | 388 | |||
502 | 389 | const google::protobuf::Descriptor* desc = msg->GetDescriptor(); | ||
503 | 390 | |||
504 | 391 | dlog.is(logger::DEBUG1, logger::DECODE) && dlog << "Type name: " << desc->full_name() << std::endl; | ||
505 | 392 | |||
506 | 393 | boost::shared_ptr<FieldCodecBase> codec = FieldCodecManager::find(desc); | ||
507 | 394 | boost::shared_ptr<internal::FromProtoCppTypeBase> helper = internal::TypeHelper::find(desc); | ||
508 | 395 | |||
509 | 396 | if(codec) | ||
510 | 397 | { | ||
511 | 398 | unsigned head_size_bits; | ||
512 | 399 | unsigned body_size_bits; | ||
513 | 400 | codec->base_max_size(&head_size_bits, desc, HEAD); | ||
514 | 401 | codec->base_max_size(&body_size_bits, desc, BODY); | ||
515 | 402 | unsigned id_size = 0; | ||
516 | 403 | id_codec()->field_size(&id_size, this_id, 0); | ||
517 | 404 | head_size_bits += id_size; | ||
518 | 405 | |||
519 | 406 | unsigned head_size_bytes = ceil_bits2bytes(head_size_bits); | ||
520 | 407 | unsigned body_size_bytes = ceil_bits2bytes(body_size_bits); | ||
521 | 408 | |||
522 | 409 | dlog.is(logger::DEBUG2, logger::DECODE) && dlog << "Head bytes (bits): " << head_size_bytes << "(" << head_size_bits | ||
523 | 410 | << "), max body bytes (bits): " << body_size_bytes << "(" << body_size_bits << ")" << std::endl; | ||
524 | 411 | |||
525 | 412 | CharIterator head_bytes_end = begin + head_size_bytes; | ||
526 | 413 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Head (hex): " << hex_encode(begin, head_bytes_end) << std::endl; | ||
527 | 414 | |||
528 | 415 | Bitset head_bits; | ||
529 | 416 | head_bits.from_byte_stream(begin, head_bytes_end); | ||
530 | 417 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Head (bin): " << head_bits << std::endl; | ||
531 | 418 | |||
532 | 419 | // shift off ID bits | ||
533 | 420 | head_bits >>= id_size; | ||
534 | 421 | |||
535 | 422 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Head after ID bits removal (bin): " << head_bits << std::endl; | ||
536 | 423 | |||
537 | 424 | internal::MessageStack msg_stack; | ||
538 | 425 | msg_stack.push(msg->GetDescriptor()); | ||
539 | 426 | |||
540 | 427 | codec->base_decode(&head_bits, msg, HEAD); | ||
541 | 428 | dlog.is(logger::DEBUG2, logger::DECODE) && dlog << "after header decode, message is: " << *msg << std::endl; | ||
542 | 429 | |||
543 | 430 | |||
544 | 431 | if(header_only) | ||
545 | 432 | { | ||
546 | 433 | dlog.is(logger::DEBUG2, logger::DECODE) && dlog << "as requested, skipping decrypting and decoding body." << std::endl; | ||
547 | 434 | } | ||
548 | 435 | else | ||
549 | 436 | { | ||
550 | 437 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Encrypted Body (hex): " << hex_encode(head_bytes_end, end) << std::endl; | ||
551 | 438 | |||
552 | 439 | Bitset body_bits; | ||
553 | 440 | if(!crypto_key_.empty() && !skip_crypto_ids_.count(this_id)) | ||
554 | 441 | { | ||
555 | 442 | std::string head_bytes(begin, head_bytes_end); | ||
556 | 443 | std::string body_bytes(head_bytes_end, end); | ||
557 | 444 | decrypt(&body_bytes, head_bytes); | ||
558 | 445 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Body (hex): " << hex_encode(body_bytes) << std::endl; | ||
559 | 446 | body_bits.from_byte_stream(body_bytes.begin(), body_bytes.end()); | ||
560 | 447 | } | ||
561 | 448 | else | ||
562 | 449 | { | ||
563 | 450 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Body (hex): " << hex_encode(head_bytes_end, end) << std::endl; | ||
564 | 451 | body_bits.from_byte_stream(head_bytes_end, end); | ||
565 | 452 | } | ||
566 | 453 | |||
567 | 454 | dlog.is(logger::DEBUG3, logger::DECODE) && dlog << "Unencrypted Body (bin): " << body_bits << std::endl; | ||
568 | 455 | |||
569 | 456 | codec->base_decode(&body_bits, msg, BODY); | ||
570 | 457 | dlog.is(logger::DEBUG2, logger::DECODE) && dlog << "after header & body decode, message is: " << *msg << std::endl; | ||
571 | 458 | } | ||
572 | 459 | } | ||
573 | 460 | else | ||
574 | 461 | { | ||
575 | 462 | throw(Exception("Failed to find (dccl.msg).codec `" + desc->options().GetExtension(dccl::msg).codec() + "`")); | ||
576 | 463 | } | ||
577 | 464 | |||
578 | 465 | dlog.is(logger::DEBUG1, logger::DECODE) && dlog << "Successfully decoded message of type: " << desc->full_name() << std::endl; | ||
579 | 466 | } | ||
580 | 467 | catch(std::exception& e) | ||
581 | 468 | { | ||
582 | 469 | std::stringstream ss; | ||
583 | 470 | |||
584 | 471 | ss << "Message " << hex_encode(begin, end) << " failed to decode. Reason: " << e.what() << std::endl; | ||
585 | 472 | |||
586 | 473 | dlog.is(logger::DEBUG1, logger::DECODE) && dlog << ss.str() << std::endl; | ||
587 | 474 | throw(Exception(ss.str())); | ||
588 | 475 | } | ||
589 | 476 | } | ||
590 | 477 | |||
591 | 329 | #endif | 478 | #endif |
This makes sense to me.
Minor style request: could you move the implementation of these methods to the bottom of codec.h after the declaration of dccl::Codec? (I want to keep the declaration of dccl::Codec clean):
1. template<typename CharIterator> unsigned id(CharIterator begin, CharIterator end) :protobuf: :Message* msg, bool header_only = false)
2. template <typename CharIterator> void decode(CharIterator begin, CharIterator end, google:
Otherwise should be fine. I'll give it one more detailed look over after you make that change and then I'll merge it in.
Also, dccl_test_ccl and dccl_test_ arithmetic run fine on my machine for both lp:dccl/3.0 and this branch. I don't know if they are properly handling the .dylib extension on OSX (I think this is what you're using?). If you could open a separate bug report on those with the command line output when you run them, that'd be great.
Thanks-
-Toby