Merge lp:~jimbaker/pyjuju/format-2-raw into lp:pyjuju

Proposed by Jim Baker on 2012-09-01
Status: Merged
Approved by: Kapil Thangavelu on 2012-09-04
Approved revision: 576
Merged at revision: 576
Proposed branch: lp:~jimbaker/pyjuju/format-2-raw
Merge into: lp:pyjuju
Diff against target: 661 lines (+284/-147)
8 files modified
juju/charm/config.py (+1/-1)
juju/control/tests/test_config_set.py (+1/-22)
juju/hooks/cli.py (+17/-2)
juju/hooks/protocol.py (+0/-1)
juju/hooks/tests/test_cli.py (+7/-24)
juju/hooks/tests/test_invoker.py (+230/-40)
juju/lib/format.py (+17/-24)
juju/lib/tests/test_format.py (+11/-33)
To merge this branch: bzr merge lp:~jimbaker/pyjuju/format-2-raw
Reviewer Review Type Date Requested Status
Juju Engineering 2012-09-01 Pending
Review via email: mp+122380@code.launchpad.net

Description of the change

Modify format 2 so that it supports raw strings

This branch modifies the current format 2 support as follows:

config-get/set:
 - type int: Reads "1"; Writes "1" (no quotes)
 - type float: Reads "1" or "1.0"; Writes "1.0" (no quotes)
 - type boolean: Reads lower(v) in "true" or "false"; Writes "true" or "false"
 - type string: Reads raw data; Writes raw data

relation-get/set
 - String: Reads raw data; Writes raw data

relation-get -
 - YAML output with string keys and raw data values (no nesting!)

 - JSON output with string keys and raw data values (no nesting!) -
   however, high byte strings that are not legal Unicode are base64
   encoded. This should be the same as seen in golang and seems to be
   a fairly common standard

I have tested this with format 1 and format 2 charms interoperating,
as well as various scenarios of raw string input/ouput to ensure that
no extraneous bytes are added, or bytes are lost, and have added
appropriate unit tests. In particular, this branch ensures the support
of null bytes in raw strings. Note that bash can be tricky to use with
null bytes, but this is not a limitation of Juju itself.

https://codereview.appspot.com/6490069/

To post a comment you must log in.
Jim Baker (jimbaker) wrote :

Reviewers: mp+122380_code.launchpad.net,

Message:
Please take a look.

Description:
Modify format 2 so that it supports raw strings

This branch modifies the current format 2 support as follows:

config-get/set:
  - type int: Reads "1"; Writes "1" (no quotes)
  - type float: Reads "1" or "1.0"; Writes "1.0" (no quotes)
  - type boolean: Reads lower(v) in "true" or "false"; Writes "true" or
"false"
  - type string: Reads raw data; Writes raw data

relation-get/set
  - String: Reads raw data; Writes raw data

relation-get -
  - YAML output with string keys and raw data values (no nesting!)

  - JSON output with string keys and raw data values (no nesting!) -
    however, high byte strings that are not legal Unicode are base64
    encoded. This should be the same as seen in golang and seems to be
    a fairly common standard

I have tested this with format 1 and format 2 charms interoperating,
as well as various scenarios of raw string input/ouput to ensure that
no extraneous bytes are added, or bytes are lost, and have added
appropriate unit tests. In particular, this branch ensures the support
of null bytes in raw strings. Note that bash can be tricky to use with
null bytes, but this is not a limitation of Juju itself.

https://code.launchpad.net/~jimbaker/juju/format-2-raw/+merge/122380

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/6490069/

Affected files:
   A [revision details]
   M juju/charm/config.py
   M juju/control/tests/test_config_set.py
   M juju/hooks/cli.py
   M juju/hooks/protocol.py
   M juju/hooks/tests/test_cli.py
   M juju/hooks/tests/test_invoker.py
   M juju/lib/format.py
   M juju/lib/tests/test_format.py

Kapil Thangavelu (hazmat) wrote :

looks great, +1

https://codereview.appspot.com/6490069/diff/1/juju/hooks/cli.py
File juju/hooks/cli.py (right):

https://codereview.appspot.com/6490069/diff/1/juju/hooks/cli.py#newcode233
juju/hooks/cli.py:233: # encoded; workaround by firt testing whether it
can be
s/firt/first

https://codereview.appspot.com/6490069/

Clint Byrum (clint-fewbar) wrote :

Can we please document this in lp:juju/docs before merging it?

Kapil Thangavelu (hazmat) wrote :

Thats an independent task and branch that should follow. At the moment no
one is using format-2, and we should have the right implementation in place
b4 documenting usage.

On Tue, Sep 4, 2012 at 7:02 PM, Clint Byrum <email address hidden> wrote:

> Can we please document this in lp:juju/docs before merging it?
> --
> https://code.launchpad.net/~jimbaker/juju/format-2-raw/+merge/122380
> You are subscribed to branch lp:juju.
>

Clint Byrum (clint-fewbar) wrote :

Excerpts from Kapil Thangavelu's message of 2012-09-06 18:01:26 UTC:
> Thats an independent task and branch that should follow. At the moment no
> one is using format-2, and we should have the right implementation in place
> b4 documenting usage.

I see it a bit different, in that I'd like to see a specification
somewhere before the implementation is merged (not worked on.. just
before it is merged).

Either way, documentation of the format needs to be done before we cut
a 0.6 release.

Kapil Thangavelu (hazmat) wrote :

On Thu, Sep 6, 2012 at 5:31 PM, Clint Byrum <email address hidden> wrote:

> Excerpts from Kapil Thangavelu's message of 2012-09-06 18:01:26 UTC:
> > Thats an independent task and branch that should follow. At the moment no
> > one is using format-2, and we should have the right implementation in
> place
> > b4 documenting usage.
>
> I see it a bit different, in that I'd like to see a specification
> somewhere before the implementation is merged (not worked on.. just
> before it is merged).
>
>
This has been the subject of at least three threads on list, with the most
recent providing an exact specification of the delta.

> Either way, documentation of the format needs to be done before we cut
> a 0.6 release.
>

Agreed.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'juju/charm/config.py'
--- juju/charm/config.py 2012-07-02 03:14:51 +0000
+++ juju/charm/config.py 2012-09-01 05:47:20 +0000
@@ -144,7 +144,7 @@
144 # output format, False144 # output format, False
145 raise ServiceConfigValueError(145 raise ServiceConfigValueError(
146 "Invalid value for %s: %s" % (146 "Invalid value for %s: %s" % (
147 name, YAMLFormat().format(value)))147 name, YAMLFormat().format_raw(value)))
148 return value148 return value
149149
150 def get_defaults(self):150 def get_defaults(self):
151151
=== modified file 'juju/control/tests/test_config_set.py'
--- juju/control/tests/test_config_set.py 2012-07-02 23:55:27 +0000
+++ juju/control/tests/test_config_set.py 2012-09-01 05:47:20 +0000
@@ -261,7 +261,7 @@
261 self.setup_exit(0)261 self.setup_exit(0)
262 self.mocker.replay()262 self.mocker.replay()
263 main(["set", "mysql-format-v2",263 main(["set", "mysql-format-v2",
264 "monkey-madness='barrels of monkeys'"])264 "monkey-madness=barrels of monkeys"])
265 yield finished265 yield finished
266 self.assertEqual(266 self.assertEqual(
267 self.output.getvalue(),267 self.output.getvalue(),
@@ -290,24 +290,3 @@
290 state,290 state,
291 {"awesome": False, "monkey-madness": 0.5,291 {"awesome": False, "monkey-madness": 0.5,
292 "query-cache-size": -1, "tuning-level": "safest"})292 "query-cache-size": -1, "tuning-level": "safest"})
293
294 @inlineCallbacks
295 def test_invalid_string_option_format_v2(self):
296 """Verify that config settings reject invalid string"""
297 self.service_state = yield self.add_service_from_charm(
298 "mysql-format-v2")
299 finished = self.setup_cli_reactor()
300 self.setup_exit(0)
301 self.mocker.replay()
302 # YAML does a fair amount of coercion... note that it needs to
303 # be quoted if it looks like a boolean/int/string
304 main(["set", "mysql-format-v2", "tuning-level=FALSE"])
305 yield finished
306 self.assertEqual(
307 self.output.getvalue(),
308 "Invalid value for tuning-level: false\n")
309 state = yield self.service_state.get_config()
310 self.assertEqual(
311 state,
312 {"awesome": False, "monkey-madness": 0.5,
313 "query-cache-size": -1, "tuning-level": "safest"})
314293
=== modified file 'juju/hooks/cli.py'
--- juju/hooks/cli.py 2012-06-14 14:33:22 +0000
+++ juju/hooks/cli.py 2012-09-01 05:47:20 +0000
@@ -1,4 +1,6 @@
1import argparse1import argparse
2import base64
3import copy
2import json4import json
3import logging5import logging
4import os6import os
@@ -223,12 +225,25 @@
223 stream.flush()225 stream.flush()
224226
225 def format_json(self, result, stream):227 def format_json(self, result, stream):
226 print >>stream, json.dumps(result)228 encoded = copy.copy(result)
229 if isinstance(result, dict):
230 for k, v in result.iteritems():
231 # Workaround the fact that JSON does not work with str
232 # values that have high bytes and are not actually UTF-8
233 # encoded; workaround by firt testing whether it can be
234 # decoded as UTF-8, and if not, wrapping as Base64
235 # encoded.
236 if isinstance(v, str):
237 try:
238 v.decode("utf8")
239 except UnicodeDecodeError:
240 encoded[k] = base64.b64encode(v)
241 json.dump(encoded, stream)
227242
228 def format_smart(self, result, stream):243 def format_smart(self, result, stream):
229 if result is not None:244 if result is not None:
230 charm_formatter = get_charm_formatter_from_env()245 charm_formatter = get_charm_formatter_from_env()
231 print >>stream, charm_formatter.format(result)246 stream.write(charm_formatter.format_raw(result))
232247
233248
234def parse_log_level(level):249def parse_log_level(level):
235250
=== modified file 'juju/hooks/protocol.py'
--- juju/hooks/protocol.py 2012-07-03 00:39:39 +0000
+++ juju/hooks/protocol.py 2012-09-01 05:47:20 +0000
@@ -335,7 +335,6 @@
335 # status reporting335 # status reporting
336 yield container.open_port(port, proto)336 yield container.open_port(port, proto)
337337
338
339 yield service_unit_state.open_port(port, proto)338 yield service_unit_state.open_port(port, proto)
340 yield self.factory.log(logging.DEBUG, "opened %s/%s" % (port, proto))339 yield self.factory.log(logging.DEBUG, "opened %s/%s" % (port, proto))
341 defer.returnValue({})340 defer.returnValue({})
342341
=== modified file 'juju/hooks/tests/test_cli.py'
--- juju/hooks/tests/test_cli.py 2012-06-15 01:16:13 +0000
+++ juju/hooks/tests/test_cli.py 2012-09-01 05:47:20 +0000
@@ -8,7 +8,6 @@
8from contextlib import closing8from contextlib import closing
99
10from twisted.internet.defer import inlineCallbacks, returnValue10from twisted.internet.defer import inlineCallbacks, returnValue
11import yaml
1211
13from juju.hooks.cli import (12from juju.hooks.cli import (
14 CommandLineClient, parse_log_level, parse_port_protocol)13 CommandLineClient, parse_log_level, parse_port_protocol)
@@ -423,39 +422,23 @@
423 with closing(StringIO.StringIO()) as output:422 with closing(StringIO.StringIO()) as output:
424 cli.format_smart(sample, output)423 cli.format_smart(sample, output)
425 self.assertEqual(output.getvalue(), formatted)424 self.assertEqual(output.getvalue(), formatted)
426 self.assertEqual(sample, yaml.safe_load(output.getvalue()))
427425
428 def test_format_smart_v2(self):426 def test_format_smart_v2(self):
429 """Verifies smart format v2 writes correct YAML"""427 """Verifies smart format v2 writes raw strings properly"""
430 self.change_environment(_JUJU_CHARM_FORMAT="2")428 self.change_environment(_JUJU_CHARM_FORMAT="2")
431429
432 # For each case, verify actual output serialization along with430 # For each case, verify actual output serialization along with
433 # roundtripping through YAML431 # roundtripping through YAML
434 self.assert_smart_output(None, "") # No newline in output for None432 self.assert_smart_output(None, "") # No newline in output for None
435 self.assert_smart_output("", "''\n")433 self.assert_smart_output("", "")
436 self.assert_smart_output("A string", "A string\n")434 self.assert_smart_output("A string", "A string")
437 # Note: YAML uses b64 encoding for byte strings tagged by !!binary435 self.assert_smart_output(
438 self.assert_smart_output(436 "High bytes: \xCA\xFE", "High bytes: \xca\xfe")
439 "High bytes: \xCA\xFE",437 self.assert_smart_output("中文", "\xe4\xb8\xad\xe6\x96\x87")
440 "!!binary |\n SGlnaCBieXRlczogyv4=\n")
441 self.assert_smart_output(u"", "''\n")
442 self.assert_smart_output(
443 u"A unicode string (but really ascii)",
444 "A unicode string (but really ascii)\n")
445 # Any non-ascii Unicode will use UTF-8 encoding
446 self.assert_smart_output(u"中文", "\xe4\xb8\xad\xe6\x96\x87\n")
447 self.assert_smart_output({}, "{}\n")
448 self.assert_smart_output(438 self.assert_smart_output(
449 {u"public-address": u"ec2-1-2-3-4.compute-1.amazonaws.com",439 {u"public-address": u"ec2-1-2-3-4.compute-1.amazonaws.com",
450 u"foo": u"bar",440 u"foo": u"bar",
451 u"configured": True},441 u"configured": True},
452 ("configured: true\n"442 ("configured: true\n"
453 "foo: bar\n"443 "foo: bar\n"
454 "public-address: ec2-1-2-3-4.compute-1.amazonaws.com\n"))444 "public-address: ec2-1-2-3-4.compute-1.amazonaws.com"))
455 self.assert_smart_output(False, "false\n")
456 self.assert_smart_output(True, "true\n")
457 self.assert_smart_output(0.0, "0.0\n")
458 self.assert_smart_output(3.14159, "3.14159\n")
459 self.assert_smart_output(6.02214178e23, "6.02214178e+23\n")
460 self.assert_smart_output(0, "0\n")
461 self.assert_smart_output(42, "42\n")
462445
=== modified file 'juju/hooks/tests/test_invoker.py'
--- juju/hooks/tests/test_invoker.py 2012-07-03 07:30:53 +0000
+++ juju/hooks/tests/test_invoker.py 2012-09-01 05:47:20 +0000
@@ -1,6 +1,7 @@
1# -*- encoding: utf-8 -*-1# -*- encoding: utf-8 -*-
22
3from StringIO import StringIO3from StringIO import StringIO
4import base64
4import json5import json
5import logging6import logging
6import os7import os
@@ -192,7 +193,7 @@
192193
193 def create_hook(self, hook, arguments):194 def create_hook(self, hook, arguments):
194 bin_path = self.get_cli_hook(hook)195 bin_path = self.get_cli_hook(hook)
195 fn = self.makeFile("#!/bin/sh\n'%s' %s" % (bin_path, arguments))196 fn = self.makeFile("#!/bin/bash\n'%s' %s" % (bin_path, arguments))
196 # make the hook executable197 # make the hook executable
197 os.chmod(fn, stat.S_IEXEC | stat.S_IREAD)198 os.chmod(fn, stat.S_IEXEC | stat.S_IREAD)
198 return fn199 return fn
@@ -1516,7 +1517,6 @@
1516 # we don't see units in the other container1517 # we don't see units in the other container
1517 self.assertNotIn("mysql/0", self.log.getvalue())1518 self.assertNotIn("mysql/0", self.log.getvalue())
15181519
1519
1520 @defer.inlineCallbacks1520 @defer.inlineCallbacks
1521 def test_open_and_close_ports(self):1521 def test_open_and_close_ports(self):
1522 """Verify that port hook commands run and changes are immediate."""1522 """Verify that port hook commands run and changes are immediate."""
@@ -1547,15 +1547,14 @@
1547 [{"port": 80, "proto": "tcp"},1547 [{"port": 80, "proto": "tcp"},
1548 {"port": 53, "proto": "udp"}])1548 {"port": 53, "proto": "udp"}])
15491549
1550
1551 result = yield exe(self.create_hook("close-port", "80/tcp"))1550 result = yield exe(self.create_hook("close-port", "80/tcp"))
1552 self.assertEqual(result, 0)1551 self.assertEqual(result, 0)
1553 self.assertEqual(1552 self.assertEqual(
1554 (yield unit_state.get_open_ports()),1553 (yield unit_state.get_open_ports()),
1555 [{"port": 53, "proto": "udp"} ,])1554 [{"port": 53, "proto": "udp"}])
1556 self.assertEqual(1555 self.assertEqual(
1557 (yield container_state.get_open_ports()),1556 (yield container_state.get_open_ports()),
1558 [{"port": 53, "proto": "udp"},])1557 [{"port": 53, "proto": "udp"}])
15591558
1560 yield exe.ended1559 yield exe.ended
1561 self.assertLogLines(1560 self.assertLogLines(
@@ -1658,6 +1657,16 @@
1658 "mysql", charm_name="mysql-format-v2")1657 "mysql", charm_name="mysql-format-v2")
1659 yield super(TestCharmFormatV2, self)._default_relations()1658 yield super(TestCharmFormatV2, self)._default_relations()
16601659
1660 def make_zipped_file(self):
1661 data_file = self.makeFile()
1662 with open(data_file, "wb") as f:
1663 # gzipped of 'abc' - however gzip will also includes the
1664 # source file name, so easiest to keep it stable here as
1665 # standard data
1666 f.write("\x1f\x8b\x08\x08\xbb\x8bAP\x02\xfftmpr"
1667 "fyP0e\x00KLJ\x06\x00\xc2A$5\x03\x00\x00\x00")
1668 return data_file
1669
1661 @defer.inlineCallbacks1670 @defer.inlineCallbacks
1662 def test_environment(self):1671 def test_environment(self):
1663 """Ensure that an explicit setting of format: 2 works properly"""1672 """Ensure that an explicit setting of format: 2 works properly"""
@@ -1667,7 +1676,7 @@
1667 self.assertEqual(env["_JUJU_CHARM_FORMAT"], "2")1676 self.assertEqual(env["_JUJU_CHARM_FORMAT"], "2")
16681677
1669 @defer.inlineCallbacks1678 @defer.inlineCallbacks
1670 def test_output(self):1679 def test_smart_output(self):
1671 """Verify roundtripping"""1680 """Verify roundtripping"""
1672 hook_debug_log = capture_separate_log("hook", level=logging.DEBUG)1681 hook_debug_log = capture_separate_log("hook", level=logging.DEBUG)
1673 hook_log = capture_separate_log("hook", level=logging.INFO)1682 hook_log = capture_separate_log("hook", level=logging.INFO)
@@ -1675,54 +1684,235 @@
1675 "database:42", "add", "mysql/0", self.relation,1684 "database:42", "add", "mysql/0", self.relation,
1676 client_id="client_id")1685 client_id="client_id")
16771686
1678 # Byte strings are also supported by staying completely in1687 # Test the support of raw strings, both from a file and from
1679 # YAML, so test that corner case. This also means users need1688 # command line. Unicode can also be used - this is just
1680 # to present valid YAML for any input:1689 # rendered as UTF-8 in the shell; the source here is also
1681 data_file = self.makeFile(1690 # UTF-8 - note it is not a Unicode string, it's a bytestring.
1682 yaml.safe_dump("But when I do drink, I prefer \xCA\xFE"))1691 data_file = self.make_zipped_file()
1683 set_hook = self.create_hook(1692 set_hook = self.create_hook(
1684 "relation-set",1693 "relation-set",
1685 "b=true i=42 f=1.23 s=ascii u=中文 d=@%s" % data_file)1694 "b=true f=1.23 i=42 s=ascii u=中文 d=@%s "
1695 "r=\"$(echo -en 'But when I do drink, I prefer \\xCA\\xFE')\"" % (
1696 data_file))
1686 yield exe(set_hook)1697 yield exe(set_hook)
1698
1687 result = yield exe(self.create_hook("relation-get", "- mysql/0"))1699 result = yield exe(self.create_hook("relation-get", "- mysql/0"))
1688 self.assertEqual(result, 0)1700 self.assertEqual(result, 0)
16891701
1690 # YAML guarantees that the keys will be sorted1702 # relation-get - uses YAML to dump keys. YAML guarantees that
1691 # lexicographically; note that we output UTF-8 for Unicode1703 # the keys will be sorted lexicographically; note that we
1692 # when dumping YAML, so our source text (with this test file1704 # output UTF-8 for Unicode when dumping YAML, so our source
1693 # in UTF-8 itself) matches the output text, as seen in the1705 # text (with this test file in UTF-8 itself) matches the
1694 # characters for "zhongwen" (Chinese language).1706 # output text, as seen in the characters for "zhongwen"
1707 # (Chinese language).
1695 self.assertEqual(1708 self.assertEqual(
1696 hook_log.getvalue(),1709 hook_log.getvalue(),
1697 "b: true\n"1710 "b: 'true'\n"
1698 "d: !!binary |\n QnV0IHdoZW4gSSBkbyBkcmluaywgSSBwcmVmZXIgyv4=\n"1711 "d: !!binary |\n H4sICLuLQVAC/3RtcHJmeVAwZQBLTEoGAMJBJDUDAAAA\n"
1699 "f: 1.23\n"1712 "f: '1.23'\n"
1700 "i: 42\n"1713 "i: '42'\n"
1701 "private-address: mysql-0.example.com\n"1714 "private-address: mysql-0.example.com\n"
1715 "r: !!binary |\n QnV0IHdoZW4gSSBkbyBkcmluaywgSSBwcmVmZXIgyv4=\n"
1702 "s: ascii\n"1716 "s: ascii\n"
1703 "u: 中文\n\n")1717 "u: 中文\n")
17041718
1705 # Log lines are not simply converted into Unicode, as in v1 format1719 # Note: backslashes are necessarily doubled here; r"XYZ"
1720 # strings don't help with hexescapes
1706 self.assertLogLines(1721 self.assertLogLines(
1707 hook_debug_log.getvalue(),1722 hook_debug_log.getvalue(),
1708 ["Flushed values for hook %r on 'database:42'" % (1723 ["Flushed values for hook %r on 'database:42'" % (
1709 os.path.basename(set_hook),),1724 os.path.basename(set_hook),),
1710 " Setting changed: 'b'=True (was unset)",1725 " Setting changed: 'b'='true' (was unset)",
1711 " Setting changed: 'd'='But when I do drink, "1726 " Setting changed: 'd'='\\x1f\\x8b\\x08\\x08\\xbb\\x8bAP\\x02"
1727 "\\xfftmprfyP0e\\x00KLJ\\x06\\x00\\xc2A$5\\x03"
1728 "\\x00\\x00\\x00' (was unset)",
1729 " Setting changed: 'f'='1.23' (was unset)",
1730 " Setting changed: 'i'='42' (was unset)",
1731 " Setting changed: 'r'='But when I do drink, "
1712 "I prefer \\xca\\xfe' (was unset)",1732 "I prefer \\xca\\xfe' (was unset)",
1713 " Setting changed: 'f'=1.23 (was unset)",
1714 " Setting changed: 'i'=42 (was unset)",
1715 " Setting changed: 's'='ascii' (was unset)",1733 " Setting changed: 's'='ascii' (was unset)",
1716 " Setting changed: 'u'=u'\\u4e2d\\u6587' (was unset)"])1734 " Setting changed: 'u'=u'\\u4e2d\\u6587' (was unset)"
17171735 ])
1718 # Also ensure that invalid YAML is rejected; unlike earlier,1736
1719 # this was not encoded with yaml.safe_dump1737 @defer.inlineCallbacks
1720 data_file = self.makeFile(1738 def test_exact_roundtrip_binary_data(self):
1721 "But when I do drink, I prefer \xCA\xFE")1739 """Verify that binary data, including \x00, is roundtripped exactly"""
1722 hook = self.create_hook("relation-set", "d=@%s" % data_file)1740 hook_log = capture_separate_log("hook", level=logging.INFO)
1723 e = yield self.assertFailure(exe(hook), errors.CharmInvocationError)1741 exe = yield self.ua.get_invoker(
1724 self.assertEqual(str(e), "Error processing %r: exit code 1." % hook)1742 "database:42", "add", "mysql/0", self.relation,
1725 self.assertIn(1743 client_id="client_id")
1726 "yaml.reader.ReaderError: \'utf8\' codec can\'t decode byte #xca: "1744 data_file = self.make_zipped_file()
1727 "invalid continuation byte\n in \"<string>\", position 30",1745
1728 hook_log.getvalue())1746 # relation-set can only read null bytes from a file; bash
1747 # would otherwise silently drop
1748 set_hook = self.create_hook("relation-set", "zipped=@%s" % (
1749 data_file))
1750 result = yield exe(set_hook)
1751 self.assertEqual(result, 0)
1752
1753 # Abuse the create_hook method a little bit by adding a pipe
1754 get_hook = self.create_hook("relation-get", "zipped mysql/0 | zcat")
1755 result = yield exe(get_hook)
1756 self.assertEqual(result, 0)
1757
1758 # Using the hook log for this verification does generate one
1759 # extra \n (seen elsewhere in our tests), but this is just
1760 # test noise: we are guaranteed roundtrip fidelity by using
1761 # the picky tool that is zcat - no extraneous data accepted.
1762 self.assertEqual(hook_log.getvalue(), "abc\n")
1763
1764 @defer.inlineCallbacks
1765 def test_json_output(self):
1766 """Verify roundtripping"""
1767 hook_log = capture_separate_log("hook", level=logging.INFO)
1768 exe = yield self.ua.get_invoker(
1769 "database:42", "add", "mysql/0", self.relation,
1770 client_id="client_id")
1771
1772 # Test the support of raw strings, both from a file and from
1773 # command line. In addition, test Unicode indirectly by using
1774 # UTF-8. Because the source of this file is marked as UTF-8,
1775 # we can embed such characters directly in bytestrings, not
1776 # just Unicode strings. This also works within the context of
1777 # the shell.
1778 raw = "But when I do drink, I prefer \xca\xfe"
1779 data_file = self.makeFile(raw)
1780 set_hook = self.create_hook(
1781 "relation-set",
1782 "b=true f=1.23 i=42 s=ascii u=中文 d=@%s "
1783 "r=\"$(echo -en 'But when I do drink, I prefer \\xCA\\xFE')\"" % (
1784 data_file,))
1785 yield exe(set_hook)
1786
1787 result = yield exe(self.create_hook(
1788 "relation-get", "--format=json - mysql/0"))
1789 self.assertEqual(result, 0)
1790
1791 # YAML serialization internally has converted (transparently)
1792 # UTF-8 to Unicode, which can be rendered by JSON. However the
1793 # "cafe" bytestring is invalid JSON, so verify that it's been
1794 # Base64 encoded.
1795 encoded = base64.b64encode(raw)
1796 self.assertEqual(
1797 hook_log.getvalue(),
1798 '{"b": "true", '
1799 '"d": "%s", '
1800 '"f": "1.23", '
1801 '"i": "42", '
1802 '"private-address": "mysql-0.example.com", '
1803 '"s": "ascii", '
1804 '"r": "%s", '
1805 '"u": "\\u4e2d\\u6587"}\n' % (encoded, encoded))
1806
1807 @defer.inlineCallbacks
1808 def common_relation_set(self):
1809 hook_log = capture_separate_log("hook", level=logging.INFO)
1810 exe = yield self.ua.get_invoker(
1811 "database:42", "add", "mysql/0",
1812 self.relation, client_id="client_id")
1813 raw = "But when I do drink, I prefer \xCA\xFE"
1814 data_file = self.makeFile(raw)
1815 set_hook = self.create_hook(
1816 "relation-set",
1817 "s='some text' u=中文 d=@%s "
1818 "r=\"$(echo -en 'But when I do drink, I prefer \\xCA\\xFE')\"" % (
1819 data_file))
1820 result = yield exe(set_hook)
1821 self.assertEqual(result, 0)
1822 defer.returnValue((exe, hook_log))
1823
1824 @defer.inlineCallbacks
1825 def test_relation_get_ascii(self):
1826 """Verify that ascii data is roundtripped"""
1827 exe, hook_log = yield self.common_relation_set()
1828 result = yield exe(self.create_hook("relation-get", "s mysql/0"))
1829 self.assertEqual(result, 0)
1830 self.assertEqual(hook_log.getvalue(), "some text\n")
1831
1832 @defer.inlineCallbacks
1833 def test_relation_get_raw(self):
1834 """Verify that raw data is roundtripped"""
1835 exe, hook_log = yield self.common_relation_set()
1836 result = yield exe(self.create_hook("relation-get", "r mysql/0"))
1837 self.assertEqual(result, 0)
1838 self.assertEqual(
1839 hook_log.getvalue(), "But when I do drink, I prefer \xca\xfe\n")
1840
1841 @defer.inlineCallbacks
1842 def test_relation_get_unicode(self):
1843 """Verify Unicode is roundtripped (via UTF-8) through the shell"""
1844 exe, hook_log = yield self.common_relation_set()
1845
1846 result = yield exe(self.create_hook("relation-get", "u mysql/0"))
1847 self.assertEqual(result, 0)
1848 self.assertEqual(hook_log.getvalue(), "中文\n")
1849
1850 @defer.inlineCallbacks
1851 def setup_config(self):
1852 hook_log = self.capture_logging("hook")
1853 exe = yield self.ua.get_invoker(
1854 "db:42", "add", "mysql/0", self.relation, client_id="client_id")
1855 context = yield self.ua.get_context("client_id")
1856 config = yield context.get_config()
1857 with open(self.make_zipped_file(), "rb") as f:
1858 data = f.read()
1859 config.update({
1860 "b": True,
1861 "f": 1.23,
1862 "i": 42,
1863 "s": "some text",
1864 # uses UTF-8 encoding in this test script
1865 "u": "中文",
1866 # use high byte and null byte characters
1867 "r": data
1868 })
1869 yield config.write()
1870 defer.returnValue((exe, hook_log))
1871
1872 @defer.inlineCallbacks
1873 def test_config_get_boolean(self):
1874 """Validate that config-get returns lowercase names of booleans."""
1875 exe, hook_log = yield self.setup_config()
1876 result = yield exe(self.create_hook("config-get", "b"))
1877 self.assertEqual(result, 0)
1878 self.assertEqual(hook_log.getvalue(), "true\n")
1879
1880 @defer.inlineCallbacks
1881 def test_config_get_float(self):
1882 """Validate that config-get returns floats without quotes."""
1883 exe, hook_log = yield self.setup_config()
1884 result = yield exe(self.create_hook("config-get", "f"))
1885 self.assertEqual(result, 0)
1886 self.assertEqual(hook_log.getvalue(), "1.23\n")
1887
1888 @defer.inlineCallbacks
1889 def test_config_get_int(self):
1890 """Validate that config-get returns ints without quotes."""
1891 exe, hook_log = yield self.setup_config()
1892 result = yield exe(self.create_hook("config-get", "i"))
1893 self.assertEqual(result, 0)
1894 self.assertEqual(hook_log.getvalue(), "42\n")
1895
1896 @defer.inlineCallbacks
1897 def test_config_get_ascii(self):
1898 """Validate that config-get returns ascii strings."""
1899 exe, hook_log = yield self.setup_config()
1900 result = yield exe(self.create_hook("config-get", "s"))
1901 self.assertEqual(result, 0)
1902 self.assertEqual(hook_log.getvalue(), "some text\n")
1903
1904 @defer.inlineCallbacks
1905 def test_config_get_raw(self):
1906 """Validate config-get can work with high and null bytes."""
1907 exe, hook_log = yield self.setup_config()
1908 result = yield exe(self.create_hook("config-get", "r | zcat"))
1909 self.assertEqual(result, 0)
1910 self.assertEqual(hook_log.getvalue(), "abc\n")
1911
1912 @defer.inlineCallbacks
1913 def test_config_get_unicode(self):
1914 """Validate that config-get returns raw strings containing UTF-8."""
1915 exe, hook_log = yield self.setup_config()
1916 result = yield exe(self.create_hook("config-get", "u"))
1917 self.assertEqual(result, 0)
1918 self.assertEqual(hook_log.getvalue(), "中文\n")
17291919
=== modified file 'juju/lib/format.py'
--- juju/lib/format.py 2012-06-22 19:08:23 +0000
+++ juju/lib/format.py 2012-09-01 05:47:20 +0000
@@ -42,6 +42,14 @@
4242
43 return data43 return data
4444
45 def _parse_value(self, key, value):
46 """Interprets value as a str"""
47 return value
48
49 def should_delete(self, value):
50 """Whether `value` implies corresponding key should be deleted"""
51 return not value.strip()
52
4553
46class PythonFormat(BaseFormat):54class PythonFormat(BaseFormat):
47 """Supports backwards compatibility through str and JSON encoding."""55 """Supports backwards compatibility through str and JSON encoding."""
@@ -52,9 +60,9 @@
52 """Formats `data` using Python str encoding"""60 """Formats `data` using Python str encoding"""
53 return str(data)61 return str(data)
5462
55 def _parse_value(self, key, value):63 def format_raw(self, data):
56 """Interprets value as a str"""64 """Add extra \n seen in Python format, so not truly raw"""
57 return value65 return self.format(data) + "\n"
5866
59 # For the old format: 1, using JSON serialization introduces some67 # For the old format: 1, using JSON serialization introduces some
60 # subtle issues around Unicode conversion that then later results68 # subtle issues around Unicode conversion that then later results
@@ -69,13 +77,6 @@
69 """Loads data, but also converts str to Unicode"""77 """Loads data, but also converts str to Unicode"""
70 return json.loads(data)78 return json.loads(data)
7179
72 def should_delete(self, value):
73 """Whether `value` implies corresponding key should be deleted"""
74 # In format: 1, all values are strings, but possibly with
75 # spaces. The strip reduces strings consisting only of spaces,
76 # or otherwise empty, to an empty string.
77 return not value.strip()
78
7980
80class YAMLFormat(BaseFormat):81class YAMLFormat(BaseFormat):
81 """New format that uses YAML, so no unexpected encoding issues"""82 """New format that uses YAML, so no unexpected encoding issues"""
@@ -98,13 +99,12 @@
98 # Also remove any extra \n, will still be valid yaml99 # Also remove any extra \n, will still be valid yaml
99 return serialized.rstrip("\n")100 return serialized.rstrip("\n")
100101
101 def _parse_value(self, key, value):102 def format_raw(self, data):
102 # Ensure roundtripping to/from yaml if format: 2; in103 """Formats `data` as a raw string if str, otherwise as YAML"""
103 # particular this ensures that true/false work as desired104 if isinstance(data, str):
104 try:105 return data
105 return yaml.safe_load(value)106 else:
106 except Exception:107 return self.format(data)
107 raise JujuError("Invalid YAML value (argument:%s)" % key)
108108
109 # Use the same format for dump109 # Use the same format for dump
110 dump = format110 dump = format
@@ -113,13 +113,6 @@
113 """Loads data safely, ensuring no Python specific type info leaks"""113 """Loads data safely, ensuring no Python specific type info leaks"""
114 return yaml.safe_load(data)114 return yaml.safe_load(data)
115115
116 def should_delete(self, value):
117 """Whether `value` implies corresponding key should be deleted"""
118 # In format: 2, values were already parsed by yaml.safe_load;
119 # in particular, a string consisting only of whitespace is
120 # parsed as None.
121 return value is None
122
123116
124def is_valid_charm_format(charm_format):117def is_valid_charm_format(charm_format):
125 """True if `charm_format` is a valid format"""118 """True if `charm_format` is a valid format"""
126119
=== modified file 'juju/lib/tests/test_format.py'
--- juju/lib/tests/test_format.py 2012-06-22 19:08:23 +0000
+++ juju/lib/tests/test_format.py 2012-09-01 05:47:20 +0000
@@ -223,7 +223,7 @@
223 def assert_parse(self, data):223 def assert_parse(self, data):
224 """Verify input parses as expected, including from a data file"""224 """Verify input parses as expected, including from a data file"""
225 formatter = YAMLFormat()225 formatter = YAMLFormat()
226 formatted = formatter.format(data)226 formatted = formatter.format_raw(data)
227 data_file = self.makeFile(formatted)227 data_file = self.makeFile(formatted)
228 kvs = ["formatted=%s" % formatted,228 kvs = ["formatted=%s" % formatted,
229 "file=@%s" % data_file]229 "file=@%s" % data_file]
@@ -234,27 +234,9 @@
234 def test_parse_keyvalue_pairs(self):234 def test_parse_keyvalue_pairs(self):
235 """Verify key value pairs parse for a wide range of YAML inputs."""235 """Verify key value pairs parse for a wide range of YAML inputs."""
236 formatter = YAMLFormat()236 formatter = YAMLFormat()
237 self.assert_parse(None)
238 self.assert_parse("")237 self.assert_parse("")
239 self.assert_parse("A string")238 self.assert_parse("A string")
240 self.assert_parse("High bytes: \xCA\xFE")239 self.assert_parse("High bytes: \xCA\xFE")
241 self.assert_parse(u"")
242 self.assert_parse(u"A unicode string (but really ascii)")
243 self.assert_parse(u"中文")
244 self.assert_parse({})
245 self.assert_parse(
246 {u"public-address": u"ec2-1-2-3-4.compute-1.amazonaws.com",
247 u"foo": u"bar",
248 u"configured": True})
249 self.assert_parse([])
250 self.assert_parse(["abc", "xyz", 42, True])
251 self.assert_parse(False)
252 self.assert_parse(True)
253 self.assert_parse(0.0)
254 self.assert_parse(3.14159)
255 self.assert_parse(6.02214178e23)
256 self.assert_parse(0)
257 self.assert_parse(42)
258240
259 # Raises an error if no such file241 # Raises an error if no such file
260 e = self.assertRaises(242 e = self.assertRaises(
@@ -271,14 +253,6 @@
271 self.assertEquals(253 self.assertEquals(
272 str(e), "Expected `option=value`. Found `foobar`")254 str(e), "Expected `option=value`. Found `foobar`")
273255
274 # Raises an error if the value is invalid YAML
275 e = self.assertRaises(
276 JujuError,
277 formatter.parse_keyvalue_pairs, ["content=\xCA\FE"])
278 self.assertEquals(
279 str(e),
280 "Invalid YAML value (argument:content)")
281
282 def assert_dump_load(self, data, expected):256 def assert_dump_load(self, data, expected):
283 """Asserts expected formatting and roundtrip through dump/load"""257 """Asserts expected formatting and roundtrip through dump/load"""
284 formatter = YAMLFormat()258 formatter = YAMLFormat()
@@ -320,11 +294,15 @@
320 self.assert_dump_load(42, "data: 42")294 self.assert_dump_load(42, "data: 42")
321295
322 def test_should_delete(self):296 def test_should_delete(self):
323 """Verify only `None` values (as YAML loaded) indicate deletion"""297 """Verify empty or whitespace only strings indicate deletion"""
324 formatter = YAMLFormat()298 formatter = PythonFormat()
325 self.assertFalse(formatter.should_delete("0"))299 self.assertFalse(formatter.should_delete("0"))
326 self.assertFalse(formatter.should_delete("something"))300 self.assertFalse(formatter.should_delete("something"))
327 self.assertFalse(formatter.should_delete(""))301 self.assertTrue(formatter.should_delete(""))
328 self.assertFalse(formatter.should_delete(" "))302 self.assertTrue(formatter.should_delete(" "))
329 self.assertFalse(formatter.should_delete(0))303
330 self.assertTrue(formatter.should_delete(None))304 # Verify that format: 1 can only work with str values
305 e = self.assertRaises(AttributeError, formatter.should_delete, 42)
306 self.assertEqual(str(e), "'int' object has no attribute 'strip'")
307 e = self.assertRaises(AttributeError, formatter.should_delete, None)
308 self.assertEqual(str(e), "'NoneType' object has no attribute 'strip'")

Subscribers

People subscribed via source and target branches

to status/vote changes: