juju-core

Merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:~go-bot/juju-core/trunk

  1. juju-metadata-generate-tools
  2. Merge into trunk
Proposed by Andrew Wilkins
Status: Merged
Approved by: Andrew Wilkins
Approved revision: no longer in the source branch.
Merged at revision: 1741
Proposed branch: lp:~axwalk/juju-core/juju-metadata-generate-tools
Merge into: lp:~go-bot/juju-core/trunk
Diff against target: 1260 lines (+943/-115)
17 files modified
cmd/plugins/juju-metadata/metadata.go (+1/-0)
cmd/plugins/juju-metadata/metadataplugin_test.go (+1/-0)
cmd/plugins/juju-metadata/toolsmetadata.go (+182/-0)
cmd/plugins/juju-metadata/toolsmetadata_test.go (+259/-0)
environs/filestorage/filestorage.go (+83/-0)
environs/localstorage/storage.go (+6/-0)
environs/localstorage/storage_test.go (+9/-0)
environs/simplestreams/json.go (+55/-0)
environs/simplestreams/json_test.go (+34/-0)
environs/simplestreams/simplestreams.go (+20/-34)
environs/simplestreams/simplestreams_test.go (+1/-0)
environs/sync/sync.go (+2/-73)
environs/tools/boilerplate.go (+1/-1)
environs/tools/marshal.go (+86/-0)
environs/tools/marshal_test.go (+150/-0)
environs/tools/simplestreams.go (+12/-7)
environs/tools/simplestreams_test.go (+41/-0)
To merge this branch: bzr merge lp:~axwalk/juju-core/juju-metadata-generate-tools
Reviewer Review Type Date Requested Status
Juju Engineering Pending
Review via email: mp+183070@code.launchpad.net

Commit message

juju-metadata: add generate-tools subcommand

This is a new subcommand for the juju-metadata plugin,
which can be used to generate simplestreams metadata
(index and products) for an existing set of tools.

The command will list the tools in the public bucket
or a local directory, and generate the index and
tools metadata in the same location.

https://codereview.appspot.com/13417043/

Description of the change

juju-metadata: add generate-tools subcommand

This is a new subcommand for the juju-metadata plugin,
which can be used to generate simplestreams metadata
(index and products) for an existing set of tools.

The command will list the tools in the public bucket
or a local directory, and generate the index and
tools metadata in the same location.

https://codereview.appspot.com/13417043/

To post a comment you must log in.
Revision history for this message
Andrew Wilkins (axwalk) wrote :

Reviewers: mp+183070_code.launchpad.net,

Message:
Please take a look.

Description:
juju-metadata: add generate-tools subcommand

This is a new subcommand for the juju-metadata plugin,
which can be used to generate simplestreams metadata
(index and products) for an existing set of tools.

The command will list the tools in the public bucket
or a local directory, and generate the index and
tools metadata in the same location.

https://code.launchpad.net/~axwalk/juju-core/juju-metadata-generate-tools/+merge/183070

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/13417043/

Affected files:
   A [revision details]
   M cmd/plugins/juju-metadata/metadata.go
   M cmd/plugins/juju-metadata/metadataplugin_test.go
   A cmd/plugins/juju-metadata/toolsmetadata.go
   A cmd/plugins/juju-metadata/toolsmetadata_test.go
   M environs/emptystorage.go
   M environs/emptystorage_test.go
   M environs/localstorage/storage.go
   M environs/localstorage/storage_test.go
   M environs/simplestreams/simplestreams.go
   M environs/simplestreams/simplestreams_test.go
   M environs/tools/boilerplate.go
   M environs/tools/simplestreams.go

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Please take a look.

https://codereview.appspot.com/13417043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Please take a look.

https://codereview.appspot.com/13417043/

Revision history for this message
Ian Booth (wallyworld) wrote :
Download full text (3.9 KiB)

Coming along nicely. But not quite there. See various comments.

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go
File cmd/plugins/juju-metadata/toolsmetadata.go (right):

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode30
cmd/plugins/juju-metadata/toolsmetadata.go:30: const pathPrefix =
"tools/"
I'd make this "releases"

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode55
cmd/plugins/juju-metadata/toolsmetadata.go:55: env, err :=
environs.NewFromName(c.EnvName)
There's little point using an env to read the tools - the tools are
readily accessible directly from "https://juju-dist.s3.amazonaws.com/"
(see synctools). Ideally, this tool would look also at the specified
directory (via -d) and read the tools tarballs from there (under
releases) and generate the metadata. Or even better, take a URL like
file:// as an arg (but dir will be fine for the first iteration of
this).

The only reason an env would be used is so that tools metadata could be
written to the env's private bucket. However, what will ultimately
happen is synctools will evolve to not just copy the tools tarballs like
it does now, but also write the metadata using the new
MarshallToolsMetadata method in tools. This command will be used for dev
oriented things like easily adding new tool tarballs into the mix so
they can used by juju.

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode74
cmd/plugins/juju-metadata/toolsmetadata.go:74: // it's from the public
or private storage at this point.
urlPath should have "releases" prepended.
If the base url is http://juju.canonical.com/tools, then tools tarballs
will be at
releases/foo.tar.gz etc
and simplestreams metadata will be at
streams/v1/index.json etc

https://codereview.appspot.com/13417043/diff/7001/environs/emptystorage.go
File environs/emptystorage.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/emptystorage.go#newcode17
environs/emptystorage.go:17: var EmptyStorage Storage = struct {
Do we still need this given the implementation has changed?

https://codereview.appspot.com/13417043/diff/7001/environs/localstorage/storage.go
File environs/localstorage/storage.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/localstorage/storage.go#newcode71
environs/localstorage/storage.go:71: }
What's this used for?

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams.go
File environs/simplestreams/simplestreams.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams.go#newcode179
environs/simplestreams/simplestreams.go:179: func (c *ItemCollection)
UnmarshalJSON(b []byte) error {
Please move this and associated code/tests to json.go

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams_test.go
File environs/simplestreams/simplestreams_test.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams_test.go#newcode439
environs/simplestreams/simplestreams_...

Read more...

Revision history for this message
Andrew Wilkins (axwalk) wrote :
Download full text (4.1 KiB)

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go
File cmd/plugins/juju-metadata/toolsmetadata.go (right):

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode30
cmd/plugins/juju-metadata/toolsmetadata.go:30: const pathPrefix =
"tools/"
On 2013/08/30 06:40:42, wallyworld wrote:
> I'd make this "releases"

Done.

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode55
cmd/plugins/juju-metadata/toolsmetadata.go:55: env, err :=
environs.NewFromName(c.EnvName)
On 2013/08/30 06:40:42, wallyworld wrote:
> There's little point using an env to read the tools - the tools are
 readily
> accessible directly from "https://juju-dist.s3.amazonaws.com/" (see
 synctools).

Gotcha. Sorry, I missed the bit in sync-tools that did the magic. I'll
move the fileStorageReader out of there so I can reuse it.

https://codereview.appspot.com/13417043/diff/7001/cmd/plugins/juju-metadata/toolsmetadata.go#newcode74
cmd/plugins/juju-metadata/toolsmetadata.go:74: // it's from the public
or private storage at this point.
On 2013/08/30 06:40:42, wallyworld wrote:
> urlPath should have "releases" prepended.
> If the base url is http://juju.canonical.com/tools, then tools
 tarballs will be
> at
> releases/foo.tar.gz etc
> and simplestreams metadata will be at
> streams/v1/index.json etc

Done.

https://codereview.appspot.com/13417043/diff/7001/environs/emptystorage.go
File environs/emptystorage.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/emptystorage.go#newcode17
environs/emptystorage.go:17: var EmptyStorage Storage = struct {
On 2013/08/30 06:40:42, wallyworld wrote:
> Do we still need this given the implementation has changed?

No, I'll back it out.

https://codereview.appspot.com/13417043/diff/7001/environs/localstorage/storage.go
File environs/localstorage/storage.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/localstorage/storage.go#newcode71
environs/localstorage/storage.go:71: }
On 2013/08/30 06:40:42, wallyworld wrote:
> What's this used for?

Drive-by fix. List shouldn't error if there are no matching files.

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams.go
File environs/simplestreams/simplestreams.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams.go#newcode179
environs/simplestreams/simplestreams.go:179: func (c *ItemCollection)
UnmarshalJSON(b []byte) error {
On 2013/08/30 06:40:42, wallyworld wrote:
> Please move this and associated code/tests to json.go

Done.

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams_test.go
File environs/simplestreams/simplestreams_test.go (right):

https://codereview.appspot.com/13417043/diff/7001/environs/simplestreams/simplestreams_test.go#newcode439
environs/simplestreams/simplestreams_test.go:439: "c": 123
On 2013/08/30 06:40:42, wallyworld wrote:
> Could we please make this a big number to test the issue is solved

This test isn't verifying that problem. It's verifying that we get
default JSON unmarshalling behaviour ...

Read more...

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Please take a look.

https://codereview.appspot.com/13417043/

Revision history for this message
Ian Booth (wallyworld) wrote :

LGTM with the issues highlighted addressed

https://codereview.appspot.com/13417043/diff/18001/environs/localstorage/storage.go
File environs/localstorage/storage.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/localstorage/storage.go#newcode71
environs/localstorage/storage.go:71: }
I'd prefer returning an empty array here ie []string{}
Returning nil, nil seems wrong

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json.go
File environs/simplestreams/json.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json.go#newcode20
environs/simplestreams/json.go:20: type CloudMetadata struct {
Sorry if I was unclear, the structs which model the simplestreams data
stay where they were; it's only the json marshall/unmarshall code which
goes in here

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go
File environs/simplestreams/json_test.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go#newcode22
environs/simplestreams/json_test.go:22: "c": 123
Please make this a big number so that we can ensure the fix actually
works. I know it's tested elsewhere too, but this code is where the
implementation is

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go#newcode28
environs/simplestreams/json_test.go:28: "c": float64(123),
why float when we want an int?

https://codereview.appspot.com/13417043/diff/18001/environs/tools/marshal.go
File environs/tools/marshal.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/tools/marshal.go#newcode23
environs/tools/marshal.go:23: func MarshalToolsMetadataJSON(metadata
[]*ToolsMetadata) (index, products []byte, err error) {
Where are the tests for the methods in this file?

https://codereview.appspot.com/13417043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2013/09/02 01:44:13, wallyworld wrote:
> LGTM with the issues highlighted addressed

https://codereview.appspot.com/13417043/diff/18001/environs/localstorage/storage.go
> File environs/localstorage/storage.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/localstorage/storage.go#newcode71
> environs/localstorage/storage.go:71: }
> I'd prefer returning an empty array here ie []string{}
> Returning nil, nil seems wrong

Done.

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json.go
> File environs/simplestreams/json.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json.go#newcode20
> environs/simplestreams/json.go:20: type CloudMetadata struct {
> Sorry if I was unclear, the structs which model the simplestreams data
 stay
> where they were; it's only the json marshall/unmarshall code which
 goes in here

Fixed. I've left "type itemCollection" (the ItemCollection clone), and
ItemCollection unmarshalling methods in json.go.

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go
> File environs/simplestreams/json_test.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go#newcode22
> environs/simplestreams/json_test.go:22: "c": 123
> Please make this a big number so that we can ensure the fix actually
 works. I
> know it's tested elsewhere too, but this code is where the
 implementation is

As discussed on IRC, this is not testing the fix for numbers
unmarshalling to int64. This is testing what happens if someone attempts
to un/marshal an ItemCollection directly, not through
ParseCloudMetadata.

https://codereview.appspot.com/13417043/diff/18001/environs/simplestreams/json_test.go#newcode28
> environs/simplestreams/json_test.go:28: "c": float64(123),
> why float when we want an int?

As above. Default encoding/json behaviour for parsing a number into an
interface{} is to unmarshal as float64.

https://codereview.appspot.com/13417043/diff/18001/environs/tools/marshal.go
> File environs/tools/marshal.go (right):

https://codereview.appspot.com/13417043/diff/18001/environs/tools/marshal.go#newcode23
> environs/tools/marshal.go:23: func MarshalToolsMetadataJSON(metadata
> []*ToolsMetadata) (index, products []byte, err error) {
> Where are the tests for the methods in this file?

On their way :)
There's tests in cmd/juju-metadata, just haven't migrated yet. Will do
before landing.

https://codereview.appspot.com/13417043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Please take a look.

https://codereview.appspot.com/13417043/

Revision history for this message
Go Bot (go-bot) wrote :

Attempt to merge into lp:juju-core failed due to conflicts:

text conflict in environs/localstorage/storage.go

Revision history for this message
Go Bot (go-bot) wrote :

The attempt to merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:juju-core failed. Below is the output from the failed tests.

# launchpad.net/juju-core/cmd/plugins/juju-metadata
cmd/plugins/juju-metadata/toolsmetadata.go:70: not enough arguments in call to "launchpad.net/juju-core/environs/tools".ReadList
cmd/plugins/juju-metadata/toolsmetadata.go:73: not enough arguments in call to "launchpad.net/juju-core/environs/tools".ReadList

Revision history for this message
Go Bot (go-bot) wrote :
Download full text (25.5 KiB)

The attempt to merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core/agent 0.721s
ok launchpad.net/juju-core/agent/tools 0.307s
ok launchpad.net/juju-core/bzr 6.797s
ok launchpad.net/juju-core/cert 1.882s
ok launchpad.net/juju-core/charm 0.574s
? launchpad.net/juju-core/charm/hooks [no test files]
ok launchpad.net/juju-core/cloudinit 0.020s
ok launchpad.net/juju-core/cmd 0.220s
? launchpad.net/juju-core/cmd/builddb [no test files]
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/juju 115.012s

----------------------------------------------------------------------
FAIL: machine_test.go:377: MachineSuite.TestManageStateServesAPI

[LOG] 82.52202 INFO juju environs/testing: uploading FAKE tools 1.15.0-precise-amd64
[LOG] 82.53892 INFO juju.environs.boostrap bootstrapping environment "dummyenv"
[LOG] 82.53895 INFO juju.environs.tools reading tools with major.minor version 1.15
[LOG] 82.53897 INFO juju.environs.tools filtering tools by version: 1.15.0
[LOG] 82.53898 INFO juju.environs.tools filtering tools by series: precise
[LOG] 82.53899 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 82.53900 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 82.53903 DEBUG juju.environs.tools found 1.15.0-precise-amd64
[LOG] 82.53907 INFO juju environs/dummy: would pick tools from 1.15.0-precise-amd64
[LOG] 82.57437 INFO juju.state opening state; mongo addresses: ["localhost:45918"]; entity ""
[LOG] 82.59163 INFO juju.state connection established
[LOG] 82.61661 INFO juju.state initializing environment
[LOG] 82.63706 INFO juju state/api: listening on "127.0.0.1:58455"
[LOG] 82.65638 INFO juju.state opening state; mongo addresses: ["localhost:45918"]; entity ""
[LOG] 82.66323 INFO juju.state connection established
[LOG] 82.67894 INFO juju juju: authorization error while connecting to state server; retrying
[LOG] 82.67899 INFO juju.state opening state; mongo addresses: ["localhost:45918"]; entity ""
[LOG] 82.68542 INFO juju.state connection established
[LOG] 82.71861 INFO juju state/api: dialing "wss://127.0.0.1:58455/"
[LOG] 82.72164 INFO juju state/api: connection established
[LOG] 82.72176 DEBUG juju rpc/jsoncodec: <- {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"user-admin","Password":"dummy-secret","Nonce":""}}
[LOG] 82.72211 DEBUG juju rpc/jsoncodec: -> {"RequestId":1,"Response":{}}
[LOG] 82.72240 INFO juju.container.lxc lxcObjectFactory replaced with mock lxc factory
[LOG] 82.72344 DEBUG juju rpc/jsoncodec: <- {"RequestId":2,"Type":"Pinger","Request":"Ping","Params":{}}
[LOG] 82.72351 DEBUG juju rpc/jsoncodec: -> {"RequestId":2,"Response":{}}
[LOG] 82.79369 INFO juju machine agent machine-0 start
[LOG] 82.80775 INFO juju Starting StateWorker for machine-0
[LOG] 82.80780 INFO juju worker: start "state"
[LOG] 82.80783 INFO juju.state opening state; mongo addresses: ["localhost:45918"]; entity "machine-0"
[LOG] 82.80811 INFO juju worker: start "api"
[LOG] 82.80832 INFO juju state/api: dialing "wss://localhost:46426/"
[...

Revision history for this message
Go Bot (go-bot) wrote :
Download full text (7.0 KiB)

The attempt to merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core/agent 0.685s
ok launchpad.net/juju-core/agent/tools 0.239s
ok launchpad.net/juju-core/bzr 6.874s
ok launchpad.net/juju-core/cert 1.697s
ok launchpad.net/juju-core/charm 0.995s
? launchpad.net/juju-core/charm/hooks [no test files]
ok launchpad.net/juju-core/cloudinit 0.019s
ok launchpad.net/juju-core/cmd 0.206s
? launchpad.net/juju-core/cmd/builddb [no test files]
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/juju 116.277s
ok launchpad.net/juju-core/cmd/jujud 37.436s

----------------------------------------------------------------------
FAIL: toolsmetadata_test.go:202: ToolsMetadataSuite.TestPatchLevels

clearing private storage
removing files: []
clearing public storage
removing files: []
toolsmetadata_test.go:233:
    c.Assert(metadata[0], gc.DeepEquals, &tools.ToolsMetadata{
        Release: "precise",
        Version: currentVersion.String(),
        Arch: "amd64",
        Size: 85,
        Path: fmt.Sprintf("releases/juju-%s-precise-amd64.tgz", currentVersion),
        FileType: "tar.gz",
        SHA256: "6bd6e4ff34f88ac91f3a8ce975e7cdff30f1d57545a396f02f7c5858b0733951",
    })
... obtained *tools.ToolsMetadata = &tools.ToolsMetadata{Release:"precise", Version:"1.15.0", Arch:"amd64", Size:85, Path:"releases/juju-1.15.0-precise-amd64.tgz", FileType:"tar.gz", SHA256:"00c804a07ef599a29fd0087afe46d89ca937dfec3ade4ca669e5f177bb9541a3"}
... expected *tools.ToolsMetadata = &tools.ToolsMetadata{Release:"precise", Version:"1.15.0", Arch:"amd64", Size:85, Path:"releases/juju-1.15.0-precise-amd64.tgz", FileType:"tar.gz", SHA256:"6bd6e4ff34f88ac91f3a8ce975e7cdff30f1d57545a396f02f7c5858b0733951"}

OOPS: 31 passed, 1 FAILED
--- FAIL: Test (1.29 seconds)
FAIL
FAIL launchpad.net/juju-core/cmd/plugins/juju-metadata 1.569s
ok launchpad.net/juju-core/constraints 0.027s
ok launchpad.net/juju-core/container/lxc 0.344s
? launchpad.net/juju-core/container/lxc/mock [no test files]
ok launchpad.net/juju-core/downloader 5.272s
ok launchpad.net/juju-core/environs 1.159s
ok launchpad.net/juju-core/environs/bootstrap 6.441s
ok launchpad.net/juju-core/environs/cloudinit 0.459s
ok launchpad.net/juju-core/environs/config 0.762s
? launchpad.net/juju-core/environs/filestorage [no test files]
ok launchpad.net/juju-core/environs/imagemetadata 0.304s
ok launchpad.net/juju-core/environs/instances 0.286s
ok launchpad.net/juju-core/environs/jujutest 0.232s
ok launchpad.net/juju-core/environs/localstorage 0.235s
ok launchpad.net/juju-core/environs/manual 1.044s
ok launchpad.net/juju-core/environs/simplestreams 0.287s
? launchpad.net/juju-core/environs/simplestreams/testing [no test files]
ok launchpad.net/juju-core/environs/sync 0.274s
ok launchpad.net/juju-core/environs/testing 0.010s
ok launchpad.net/juju-core/environs/tools 24.015s
? launchpad.net/juju-core/errors [no test files]
ok launchpad.net/juju-core/instance 0.018s
ok ...

Read more...

Revision history for this message
John A Meinel (jameinel) wrote :

https://codereview.appspot.com/13417043/diff/27001/cmd/plugins/juju-metadata/toolsmetadata_test.go
File cmd/plugins/juju-metadata/toolsmetadata_test.go (right):

https://codereview.appspot.com/13417043/diff/27001/cmd/plugins/juju-metadata/toolsmetadata_test.go#newcode232
cmd/plugins/juju-metadata/toolsmetadata_test.go:232: SHA256:
"9268ba87201b1514171cc09334db6f680e1e013f5ae584f1b43252c743eea841",
How did you come up with these SHA sums? It certainly seems very
site-specific because the actual SHA will end up depending on the
details of the compression (by default gzip compression inserts date
stamps into the stream, so you won't ever be able to predict the
compression of a newly generated file).

If we can validate them through some other path, I would love to see it,
but if we have to, you can do:

    SHA256: metadata[0].SHA256

Can we grab the actual .tgz before it is uploaded and check its sha that
way?

https://codereview.appspot.com/13417043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

https://codereview.appspot.com/13417043/diff/27001/cmd/plugins/juju-metadata/toolsmetadata_test.go
File cmd/plugins/juju-metadata/toolsmetadata_test.go (right):

https://codereview.appspot.com/13417043/diff/27001/cmd/plugins/juju-metadata/toolsmetadata_test.go#newcode232
cmd/plugins/juju-metadata/toolsmetadata_test.go:232: SHA256:
"9268ba87201b1514171cc09334db6f680e1e013f5ae584f1b43252c743eea841",
On 2013/09/02 05:48:30, jameinel wrote:
> How did you come up with these SHA sums? It certainly seems very
 site-specific
> because the actual SHA will end up depending on the details of the
 compression
> (by default gzip compression inserts date stamps into the stream, so
 you won't
> ever be able to predict the compression of a newly generated file).

I've changed this a bit already, so the contents of the files are not
tgz, but a well defined content string. So the SHA1 sum is just a sum of
that content.

> If we can validate them through some other path, I would love to see
it, but if
> we have to, you can do:

> SHA256: metadata[0].SHA256

What would be the point? I'm checking the value of metadata[N] :)

> Can we grab the actual .tgz before it is uploaded and check its sha
that way?

Yes, that's probably more sensible. I'll do that.

https://codereview.appspot.com/13417043/

Revision history for this message
Go Bot (go-bot) wrote :
Download full text (15.1 KiB)

The attempt to merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core/agent 0.693s
ok launchpad.net/juju-core/agent/tools 0.234s
ok launchpad.net/juju-core/bzr 6.646s
ok launchpad.net/juju-core/cert 2.389s
ok launchpad.net/juju-core/charm 0.513s
? launchpad.net/juju-core/charm/hooks [no test files]
ok launchpad.net/juju-core/cloudinit 0.025s
ok launchpad.net/juju-core/cmd 0.228s
? launchpad.net/juju-core/cmd/builddb [no test files]
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/juju 118.184s

----------------------------------------------------------------------
FAIL: machine_test.go:377: MachineSuite.TestManageStateServesAPI

[LOG] 3.36609 INFO juju environs/testing: uploading FAKE tools 1.15.0-precise-amd64
[LOG] 3.38755 INFO juju.environs.boostrap bootstrapping environment "dummyenv"
[LOG] 3.38758 INFO juju.environs.tools reading tools with major.minor version 1.15
[LOG] 3.38760 INFO juju.environs.tools filtering tools by version: 1.15.0
[LOG] 3.38761 INFO juju.environs.tools filtering tools by series: precise
[LOG] 3.38762 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 3.38764 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 3.38771 DEBUG juju.environs.tools found 1.15.0-precise-amd64
[LOG] 3.38775 INFO juju environs/dummy: would pick tools from 1.15.0-precise-amd64
[LOG] 3.41266 INFO juju.state opening state; mongo addresses: ["localhost:48899"]; entity ""
[LOG] 3.43032 INFO juju.state connection established
[LOG] 3.46754 INFO juju.state initializing environment
[LOG] 3.50535 INFO juju state/api: listening on "127.0.0.1:47605"
[LOG] 3.52603 INFO juju.state opening state; mongo addresses: ["localhost:48899"]; entity ""
[LOG] 3.54228 INFO juju.state connection established
[LOG] 3.54291 INFO juju juju: authorization error while connecting to state server; retrying
[LOG] 3.54298 INFO juju.state opening state; mongo addresses: ["localhost:48899"]; entity ""
[LOG] 3.54845 INFO juju.state connection established
[LOG] 3.57492 INFO juju state/api: dialing "wss://127.0.0.1:47605/"
[LOG] 3.57794 INFO juju state/api: connection established
[LOG] 3.57806 DEBUG juju rpc/jsoncodec: <- {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"user-admin","Password":"dummy-secret","Nonce":""}}
[LOG] 3.57838 DEBUG juju rpc/jsoncodec: -> {"RequestId":1,"Response":{}}
[LOG] 3.57864 INFO juju.container.lxc lxcObjectFactory replaced with mock lxc factory
[LOG] 3.57970 DEBUG juju rpc/jsoncodec: <- {"RequestId":2,"Type":"Pinger","Request":"Ping","Params":{}}
[LOG] 3.57976 DEBUG juju rpc/jsoncodec: -> {"RequestId":2,"Response":{}}
[LOG] 3.65935 INFO juju machine agent machine-0 start
[LOG] 3.67458 INFO juju Starting StateWorker for machine-0
[LOG] 3.67464 INFO juju worker: start "state"
[LOG] 3.67466 INFO juju.state opening state; mongo addresses: ["localhost:48899"]; entity "machine-0"
[LOG] 3.67490 INFO juju worker: start "api"
[LOG] 3.67511 INFO juju state/api: dialing "wss://localhost:35271/"
[LOG] 3.67574 ERROR juju state/a...

Revision history for this message
Go Bot (go-bot) wrote :
Download full text (14.3 KiB)

The attempt to merge lp:~axwalk/juju-core/juju-metadata-generate-tools into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core/agent 0.686s
ok launchpad.net/juju-core/agent/tools 0.297s
ok launchpad.net/juju-core/bzr 7.482s
ok launchpad.net/juju-core/cert 2.013s
ok launchpad.net/juju-core/charm 0.538s
? launchpad.net/juju-core/charm/hooks [no test files]
ok launchpad.net/juju-core/cloudinit 0.020s
ok launchpad.net/juju-core/cmd 0.219s
? launchpad.net/juju-core/cmd/builddb [no test files]
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/juju 119.881s

----------------------------------------------------------------------
FAIL: machine_test.go:377: MachineSuite.TestManageStateServesAPI

[LOG] 2.87215 INFO juju environs/testing: uploading FAKE tools 1.15.0-precise-amd64
[LOG] 2.88895 INFO juju.environs.boostrap bootstrapping environment "dummyenv"
[LOG] 2.88898 INFO juju.environs.tools reading tools with major.minor version 1.15
[LOG] 2.88901 INFO juju.environs.tools filtering tools by version: 1.15.0
[LOG] 2.88901 INFO juju.environs.tools filtering tools by series: precise
[LOG] 2.88902 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 2.88904 DEBUG juju.environs.tools reading v1.15 tools
[LOG] 2.88907 DEBUG juju.environs.tools found 1.15.0-precise-amd64
[LOG] 2.88911 INFO juju environs/dummy: would pick tools from 1.15.0-precise-amd64
[LOG] 2.91988 INFO juju.state opening state; mongo addresses: ["localhost:60164"]; entity ""
[LOG] 2.93285 INFO juju.state connection established
[LOG] 2.96211 INFO juju.state initializing environment
[LOG] 2.99495 INFO juju state/api: listening on "127.0.0.1:57889"
[LOG] 3.01132 INFO juju.state opening state; mongo addresses: ["localhost:60164"]; entity ""
[LOG] 3.03637 INFO juju.state connection established
[LOG] 3.03833 INFO juju juju: authorization error while connecting to state server; retrying
[LOG] 3.03845 INFO juju.state opening state; mongo addresses: ["localhost:60164"]; entity ""
[LOG] 3.04553 INFO juju.state connection established
[LOG] 3.07147 INFO juju state/api: dialing "wss://127.0.0.1:57889/"
[LOG] 3.07452 INFO juju state/api: connection established
[LOG] 3.07463 DEBUG juju rpc/jsoncodec: <- {"RequestId":1,"Type":"Admin","Request":"Login","Params":{"AuthTag":"user-admin","Password":"dummy-secret","Nonce":""}}
[LOG] 3.07497 DEBUG juju rpc/jsoncodec: -> {"RequestId":1,"Response":{}}
[LOG] 3.07523 INFO juju.container.lxc lxcObjectFactory replaced with mock lxc factory
[LOG] 3.07630 DEBUG juju rpc/jsoncodec: <- {"RequestId":2,"Type":"Pinger","Request":"Ping","Params":{}}
[LOG] 3.07637 DEBUG juju rpc/jsoncodec: -> {"RequestId":2,"Response":{}}
[LOG] 3.14233 INFO juju machine agent machine-0 start
[LOG] 3.15640 INFO juju Starting StateWorker for machine-0
[LOG] 3.15647 INFO juju worker: start "state"
[LOG] 3.15650 INFO juju.state opening state; mongo addresses: ["localhost:60164"]; entity "machine-0"
[LOG] 3.15680 INFO juju worker: start "api"
[LOG] 3.15707 INFO juju state/api: dialing "wss://localhost:49733/"
[LOG] 3.15769 ERROR juju state/a...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'cmd/plugins/juju-metadata/metadata.go'
2--- cmd/plugins/juju-metadata/metadata.go 2013-08-19 23:32:17 +0000
3+++ cmd/plugins/juju-metadata/metadata.go 2013-09-02 06:06:26 +0000
4@@ -34,6 +34,7 @@
5
6 metadatacmd.Register(&ValidateImageMetadataCommand{})
7 metadatacmd.Register(&ImageMetadataCommand{})
8+ metadatacmd.Register(&ToolsMetadataCommand{})
9
10 metadatacmd.Register(&ValidateToolsMetadataCommand{})
11
12
13=== modified file 'cmd/plugins/juju-metadata/metadataplugin_test.go'
14--- cmd/plugins/juju-metadata/metadataplugin_test.go 2013-08-13 04:55:12 +0000
15+++ cmd/plugins/juju-metadata/metadataplugin_test.go 2013-09-02 06:06:26 +0000
16@@ -30,6 +30,7 @@
17
18 var metadataCommandNames = []string{
19 "generate-image",
20+ "generate-tools",
21 "help",
22 "validate-images",
23 "validate-tools",
24
25=== added file 'cmd/plugins/juju-metadata/toolsmetadata.go'
26--- cmd/plugins/juju-metadata/toolsmetadata.go 1970-01-01 00:00:00 +0000
27+++ cmd/plugins/juju-metadata/toolsmetadata.go 2013-09-02 06:06:26 +0000
28@@ -0,0 +1,182 @@
29+// Copyright 2013 Canonical Ltd.
30+// Licensed under the AGPLv3, see LICENCE file for details.
31+
32+package main
33+
34+import (
35+ "crypto/sha256"
36+ "fmt"
37+ "hash"
38+ "io"
39+ "io/ioutil"
40+ "net/http"
41+ "os"
42+ "path/filepath"
43+ "time"
44+
45+ "launchpad.net/gnuflag"
46+
47+ "launchpad.net/juju-core/cmd"
48+ "launchpad.net/juju-core/environs"
49+ "launchpad.net/juju-core/environs/config"
50+ "launchpad.net/juju-core/environs/localstorage"
51+ "launchpad.net/juju-core/environs/simplestreams"
52+ "launchpad.net/juju-core/environs/sync"
53+ "launchpad.net/juju-core/environs/tools"
54+ "launchpad.net/juju-core/provider/ec2"
55+ "launchpad.net/juju-core/utils"
56+ "launchpad.net/juju-core/version"
57+)
58+
59+var DefaultToolsLocation = sync.DefaultToolsLocation
60+
61+// ToolsMetadataCommand is used to generate simplestreams metadata for
62+// juju tools.
63+type ToolsMetadataCommand struct {
64+ cmd.EnvCommandBase
65+ fetch bool
66+ metadataDir string
67+
68+ // noS3 is used in testing to disable the use of S3 public storage
69+ // as a backup.
70+ noS3 bool
71+}
72+
73+func (c *ToolsMetadataCommand) Info() *cmd.Info {
74+ return &cmd.Info{
75+ Name: "generate-tools",
76+ Purpose: "generate simplestreams tools metadata",
77+ }
78+}
79+
80+func (c *ToolsMetadataCommand) SetFlags(f *gnuflag.FlagSet) {
81+ c.EnvCommandBase.SetFlags(f)
82+ f.BoolVar(&c.fetch, "fetch", true, "fetch tools and compute content size and hash")
83+ f.StringVar(&c.metadataDir, "d", "", "local directory in which to store metadata")
84+}
85+
86+func (c *ToolsMetadataCommand) Run(context *cmd.Context) error {
87+ if c.metadataDir == "" {
88+ c.metadataDir = config.JujuHome()
89+ }
90+ c.metadataDir = utils.NormalizePath(c.metadataDir)
91+
92+ // Create a StorageReader that will get a tools list from the local disk.
93+ // Since ReadList expects tools to be immediately under "tools/", and we
94+ // want them to be in tools/releases, we have to wrap the storage.
95+ listener, err := localstorage.Serve("127.0.0.1:0", c.metadataDir)
96+ if err != nil {
97+ return err
98+ }
99+ defer listener.Close()
100+ var sourceStorage environs.StorageReader = localstorage.Client(listener.Addr().String())
101+ sourceStorage = prefixedToolsStorage{sourceStorage}
102+
103+ fmt.Fprintln(context.Stdout, "Finding tools...")
104+ const minorVersion = -1
105+ toolsList, err := tools.ReadList(sourceStorage, version.Current.Major, minorVersion)
106+ if err == tools.ErrNoTools && !c.noS3 {
107+ sourceStorage = ec2.NewHTTPStorageReader(sync.DefaultToolsLocation)
108+ toolsList, err = tools.ReadList(sourceStorage, version.Current.Major, minorVersion)
109+ }
110+ if err != nil {
111+ return err
112+ }
113+
114+ metadata := make([]*tools.ToolsMetadata, len(toolsList))
115+ for i, t := range toolsList {
116+ var size int64
117+ var sha256hex string
118+ if c.fetch {
119+ fmt.Fprintln(context.Stdout, "Fetching tools to generate hash:", t.URL)
120+ var sha256hash hash.Hash
121+ size, sha256hash, err = fetchToolsHash(t.URL)
122+ if err != nil {
123+ return err
124+ }
125+ sha256hex = fmt.Sprintf("%x", sha256hash.Sum(nil))
126+ }
127+
128+ path := fmt.Sprintf("releases/juju-%s-%s-%s.tgz", t.Version.Number, t.Version.Series, t.Version.Arch)
129+ metadata[i] = &tools.ToolsMetadata{
130+ Release: t.Version.Series,
131+ Version: t.Version.Number.String(),
132+ Arch: t.Version.Arch,
133+ Path: path,
134+ FileType: "tar.gz",
135+ Size: size,
136+ SHA256: sha256hex,
137+ }
138+ }
139+
140+ index, products, err := tools.MarshalToolsMetadataJSON(metadata, time.Now())
141+ if err != nil {
142+ return err
143+ }
144+ objects := []struct {
145+ path string
146+ data []byte
147+ }{
148+ {simplestreams.DefaultIndexPath + simplestreams.UnsignedSuffix, index},
149+ {tools.ProductMetadataPath, products},
150+ }
151+ for _, object := range objects {
152+ path := filepath.Join(c.metadataDir, "tools", object.path)
153+ fmt.Fprintf(context.Stdout, "Writing %s\n", path)
154+ if err = writeFile(path, object.data); err != nil {
155+ return err
156+ }
157+ }
158+ return nil
159+}
160+
161+func writeFile(path string, data []byte) error {
162+ dir := filepath.Dir(path)
163+ if err := os.MkdirAll(dir, 0755); err != nil && !os.IsExist(err) {
164+ return err
165+ }
166+ return ioutil.WriteFile(path, data, 0644)
167+}
168+
169+// fetchToolsHash fetches the file at the specified URL,
170+// and calculates its size in bytes and computes a SHA256
171+// hash of its contents.
172+func fetchToolsHash(url string) (size int64, sha256hash hash.Hash, err error) {
173+ resp, err := http.Get(url)
174+ if err != nil {
175+ return 0, nil, err
176+ }
177+ sha256hash = sha256.New()
178+ size, err = io.Copy(sha256hash, resp.Body)
179+ resp.Body.Close()
180+ return size, sha256hash, err
181+}
182+
183+const fromprefix = "tools/"
184+const toprefix = "tools/releases/"
185+
186+type prefixedToolsStorage struct {
187+ environs.StorageReader
188+}
189+
190+func (s prefixedToolsStorage) translate(name string) string {
191+ return toprefix + name[len(fromprefix):]
192+}
193+
194+func (s prefixedToolsStorage) Get(name string) (io.ReadCloser, error) {
195+ return s.StorageReader.Get(s.translate(name))
196+}
197+
198+func (s prefixedToolsStorage) List(prefix string) ([]string, error) {
199+ names, err := s.StorageReader.List(s.translate(prefix))
200+ if err == nil {
201+ for i, name := range names {
202+ names[i] = fromprefix + name[len(toprefix):]
203+ }
204+ }
205+ return names, err
206+}
207+
208+func (s prefixedToolsStorage) URL(name string) (string, error) {
209+ return s.StorageReader.URL(s.translate(name))
210+}
211
212=== added file 'cmd/plugins/juju-metadata/toolsmetadata_test.go'
213--- cmd/plugins/juju-metadata/toolsmetadata_test.go 1970-01-01 00:00:00 +0000
214+++ cmd/plugins/juju-metadata/toolsmetadata_test.go 2013-09-02 06:06:26 +0000
215@@ -0,0 +1,259 @@
216+// Copyright 2013 Canonical Ltd.
217+// Licensed under the AGPLv3, see LICENCE file for details.
218+
219+package main
220+
221+import (
222+ "bytes"
223+ "crypto/sha256"
224+ "fmt"
225+ "io"
226+ "io/ioutil"
227+ "net/http"
228+ "os"
229+ "path"
230+ "path/filepath"
231+ "regexp"
232+ "sort"
233+ "strings"
234+
235+ gc "launchpad.net/gocheck"
236+
237+ "launchpad.net/juju-core/cmd"
238+ "launchpad.net/juju-core/environs"
239+ "launchpad.net/juju-core/environs/config"
240+ "launchpad.net/juju-core/environs/simplestreams"
241+ envtesting "launchpad.net/juju-core/environs/testing"
242+ "launchpad.net/juju-core/environs/tools"
243+ _ "launchpad.net/juju-core/provider/dummy"
244+ coretesting "launchpad.net/juju-core/testing"
245+ "launchpad.net/juju-core/utils/set"
246+ "launchpad.net/juju-core/version"
247+)
248+
249+type ToolsMetadataSuite struct {
250+ home *coretesting.FakeHome
251+ env environs.Environ
252+}
253+
254+var _ = gc.Suite(&ToolsMetadataSuite{})
255+
256+func (s *ToolsMetadataSuite) SetUpTest(c *gc.C) {
257+ s.home = coretesting.MakeSampleHome(c)
258+ env, err := environs.PrepareFromName("erewhemos")
259+ c.Assert(err, gc.IsNil)
260+ s.env = env
261+ envtesting.RemoveAllTools(c, s.env)
262+}
263+
264+func (s *ToolsMetadataSuite) TearDownTest(c *gc.C) {
265+ s.home.Restore()
266+}
267+
268+func (s *ToolsMetadataSuite) parseMetadata(c *gc.C, metadataDir string) []*tools.ToolsMetadata {
269+ params := simplestreams.ValueParams{
270+ DataType: tools.ContentDownload,
271+ ValueTemplate: tools.ToolsMetadata{},
272+ }
273+
274+ baseURL := "file://" + metadataDir + "/tools"
275+ transport := &http.Transport{}
276+ transport.RegisterProtocol("file", http.NewFileTransport(http.Dir("/")))
277+ old := simplestreams.SetHttpClient(&http.Client{Transport: transport})
278+ defer simplestreams.SetHttpClient(old)
279+
280+ const requireSigned = false
281+ indexPath := simplestreams.DefaultIndexPath + simplestreams.UnsignedSuffix
282+ indexRef, err := simplestreams.GetIndexWithFormat(baseURL, indexPath, "index:1.0", requireSigned, params)
283+ c.Assert(err, gc.IsNil)
284+ c.Assert(indexRef.Indexes, gc.HasLen, 1)
285+
286+ toolsIndexMetadata := indexRef.Indexes["com.ubuntu.juju:released:tools"]
287+ c.Assert(toolsIndexMetadata, gc.NotNil)
288+
289+ data, err := ioutil.ReadFile(filepath.Join(metadataDir, "tools", toolsIndexMetadata.ProductsFilePath))
290+ c.Assert(err, gc.IsNil)
291+
292+ url := path.Join(baseURL, toolsIndexMetadata.ProductsFilePath)
293+ c.Assert(err, gc.IsNil)
294+ cloudMetadata, err := simplestreams.ParseCloudMetadata(data, "products:1.0", url, tools.ToolsMetadata{})
295+ c.Assert(err, gc.IsNil)
296+
297+ toolsMetadataMap := make(map[string]*tools.ToolsMetadata)
298+ var expectedProductIds set.Strings
299+ var toolsVersions set.Strings
300+ for _, mc := range cloudMetadata.Products {
301+ for _, items := range mc.Items {
302+ for key, item := range items.Items {
303+ toolsMetadata := item.(*tools.ToolsMetadata)
304+ toolsMetadataMap[key] = toolsMetadata
305+ toolsVersions.Add(key)
306+ productId := fmt.Sprintf("com.ubuntu.juju:%s:%s", toolsMetadata.Version, toolsMetadata.Arch)
307+ expectedProductIds.Add(productId)
308+ }
309+ }
310+ }
311+
312+ // Make sure index's product IDs are all represented in the products metadata.
313+ sort.Strings(toolsIndexMetadata.ProductIds)
314+ c.Assert(toolsIndexMetadata.ProductIds, gc.DeepEquals, expectedProductIds.SortedValues())
315+
316+ toolsMetadata := make([]*tools.ToolsMetadata, len(toolsMetadataMap))
317+ for i, key := range toolsVersions.SortedValues() {
318+ toolsMetadata[i] = toolsMetadataMap[key]
319+ }
320+ return toolsMetadata
321+}
322+
323+func (s *ToolsMetadataSuite) makeTools(c *gc.C, metadataDir string, versionStrings []string) {
324+ toolsDir := filepath.Join(metadataDir, "tools", "releases")
325+ c.Assert(os.MkdirAll(toolsDir, 0755), gc.IsNil)
326+ for _, versionString := range versionStrings {
327+ binary := version.MustParseBinary(versionString)
328+ path := filepath.Join(toolsDir, fmt.Sprintf("juju-%s.tgz", binary))
329+ err := ioutil.WriteFile(path, []byte(binary.String()), 0644)
330+ c.Assert(err, gc.IsNil)
331+ }
332+}
333+
334+var currentVersionStrings = []string{
335+ // only these ones will make it into the JSON files.
336+ version.CurrentNumber().String() + "-quantal-amd64",
337+ version.CurrentNumber().String() + "-quantal-arm",
338+ version.CurrentNumber().String() + "-quantal-i386",
339+}
340+
341+var versionStrings = append([]string{
342+ "1.12.0-precise-amd64",
343+ "1.12.0-precise-i386",
344+ "1.12.0-raring-amd64",
345+ "1.12.0-raring-i386",
346+ "1.13.0-precise-amd64",
347+}, currentVersionStrings...)
348+
349+var expectedOutputCommon = makeExpectedOutputCommon()
350+
351+func makeExpectedOutputCommon() string {
352+ expected := `Finding tools\.\.\.
353+Fetching tools to generate hash: http://.*/tools/releases/juju-1\.12\.0-precise-amd64\.tgz
354+Fetching tools to generate hash: http://.*/tools/releases/juju-1\.12\.0-precise-i386\.tgz
355+Fetching tools to generate hash: http://.*/tools/releases/juju-1\.12\.0-raring-amd64\.tgz
356+Fetching tools to generate hash: http://.*/tools/releases/juju-1\.12\.0-raring-i386\.tgz
357+Fetching tools to generate hash: http://.*/tools/releases/juju-1\.13\.0-precise-amd64\.tgz
358+`
359+ f := "Fetching tools to generate hash: http://.*/tools/releases/juju-%s\\.tgz\n"
360+ for _, v := range currentVersionStrings {
361+ expected += fmt.Sprintf(f, regexp.QuoteMeta(v))
362+ }
363+ return strings.TrimSpace(expected)
364+}
365+
366+var expectedOutputDirectory = expectedOutputCommon + `
367+Writing %s/tools/streams/v1/index\.json
368+Writing %s/tools/streams/v1/com\.ubuntu\.juju:released:tools\.json
369+`
370+
371+func (s *ToolsMetadataSuite) TestGenerateDefaultDirectory(c *gc.C) {
372+ metadataDir := config.JujuHome() // default metadata dir
373+ s.makeTools(c, metadataDir, versionStrings)
374+ ctx := coretesting.Context(c)
375+ code := cmd.Main(&ToolsMetadataCommand{noS3: true}, ctx, nil)
376+ c.Assert(code, gc.Equals, 0)
377+ output := ctx.Stdout.(*bytes.Buffer).String()
378+ expected := fmt.Sprintf(expectedOutputDirectory, metadataDir, metadataDir)
379+ c.Assert(output, gc.Matches, expected)
380+ metadata := s.parseMetadata(c, metadataDir)
381+ c.Assert(metadata, gc.HasLen, len(versionStrings))
382+ obtainedVersionStrings := make([]string, len(versionStrings))
383+ for i, metadata := range metadata {
384+ s := fmt.Sprintf("%s-%s-%s", metadata.Version, metadata.Release, metadata.Arch)
385+ obtainedVersionStrings[i] = s
386+ }
387+ c.Assert(obtainedVersionStrings, gc.DeepEquals, versionStrings)
388+}
389+
390+func (s *ToolsMetadataSuite) TestGenerateDirectory(c *gc.C) {
391+ metadataDir := c.MkDir()
392+ s.makeTools(c, metadataDir, versionStrings)
393+ ctx := coretesting.Context(c)
394+ code := cmd.Main(&ToolsMetadataCommand{noS3: true}, ctx, []string{"-d", metadataDir})
395+ c.Assert(code, gc.Equals, 0)
396+ output := ctx.Stdout.(*bytes.Buffer).String()
397+ expected := fmt.Sprintf(expectedOutputDirectory, metadataDir, metadataDir)
398+ c.Assert(output, gc.Matches, expected)
399+ metadata := s.parseMetadata(c, metadataDir)
400+ c.Assert(metadata, gc.HasLen, len(versionStrings))
401+ obtainedVersionStrings := make([]string, len(versionStrings))
402+ for i, metadata := range metadata {
403+ s := fmt.Sprintf("%s-%s-%s", metadata.Version, metadata.Release, metadata.Arch)
404+ obtainedVersionStrings[i] = s
405+ }
406+ c.Assert(obtainedVersionStrings, gc.DeepEquals, versionStrings)
407+}
408+
409+func (s *ToolsMetadataSuite) TestNoTools(c *gc.C) {
410+ ctx := coretesting.Context(c)
411+ code := cmd.Main(&ToolsMetadataCommand{noS3: true}, ctx, nil)
412+ c.Assert(code, gc.Equals, 1)
413+ stdout := ctx.Stdout.(*bytes.Buffer).String()
414+ c.Assert(stdout, gc.Matches, "Finding tools\\.\\.\\.\n")
415+ stderr := ctx.Stderr.(*bytes.Buffer).String()
416+ c.Assert(stderr, gc.Matches, "error: no tools available\n")
417+}
418+
419+func (s *ToolsMetadataSuite) TestPatchLevels(c *gc.C) {
420+ currentVersion := version.CurrentNumber()
421+ currentVersion.Build = 0
422+ versionStrings := []string{
423+ currentVersion.String() + "-precise-amd64",
424+ currentVersion.String() + ".1-precise-amd64",
425+ }
426+ metadataDir := config.JujuHome() // default metadata dir
427+ s.makeTools(c, metadataDir, versionStrings)
428+ ctx := coretesting.Context(c)
429+ code := cmd.Main(&ToolsMetadataCommand{noS3: true}, ctx, nil)
430+ c.Assert(code, gc.Equals, 0)
431+ output := ctx.Stdout.(*bytes.Buffer).String()
432+ expectedOutput := fmt.Sprintf(`
433+Finding tools\.\.\.
434+Fetching tools to generate hash: http://.*/tools/releases/juju-%s\.tgz
435+Fetching tools to generate hash: http://.*/tools/releases/juju-%s\.tgz
436+Writing %s/tools/streams/v1/index\.json
437+Writing %s/tools/streams/v1/com\.ubuntu\.juju:released:tools\.json
438+`[1:], regexp.QuoteMeta(versionStrings[0]), regexp.QuoteMeta(versionStrings[1]), metadataDir, metadataDir)
439+ c.Assert(output, gc.Matches, expectedOutput)
440+ metadata := s.parseMetadata(c, metadataDir)
441+ c.Assert(metadata, gc.HasLen, 2)
442+
443+ filename := fmt.Sprintf("juju-%s-precise-amd64.tgz", currentVersion)
444+ c.Assert(metadata[0], gc.DeepEquals, &tools.ToolsMetadata{
445+ Release: "precise",
446+ Version: currentVersion.String(),
447+ Arch: "amd64",
448+ Size: 20,
449+ Path: "releases/" + filename,
450+ FileType: "tar.gz",
451+ SHA256: sha256sum(c, filepath.Join(metadataDir, "tools", "releases", filename)),
452+ })
453+
454+ filename = fmt.Sprintf("juju-%s.1-precise-amd64.tgz", currentVersion)
455+ c.Assert(metadata[1], gc.DeepEquals, &tools.ToolsMetadata{
456+ Release: "precise",
457+ Version: currentVersion.String() + ".1",
458+ Arch: "amd64",
459+ Size: 22,
460+ Path: "releases/" + filename,
461+ FileType: "tar.gz",
462+ SHA256: sha256sum(c, filepath.Join(metadataDir, "tools", "releases", filename)),
463+ })
464+}
465+
466+func sha256sum(c *gc.C, path string) string {
467+ f, err := os.Open(path)
468+ c.Assert(err, gc.IsNil)
469+ defer f.Close()
470+ hash := sha256.New()
471+ _, err = io.Copy(hash, f)
472+ c.Assert(err, gc.IsNil)
473+ return fmt.Sprintf("%x", hash.Sum(nil))
474+}
475
476=== added directory 'environs/filestorage'
477=== added file 'environs/filestorage/filestorage.go'
478--- environs/filestorage/filestorage.go 1970-01-01 00:00:00 +0000
479+++ environs/filestorage/filestorage.go 2013-09-02 06:06:26 +0000
480@@ -0,0 +1,83 @@
481+// Copyright 2012, 2013 Canonical Ltd.
482+// Licensed under the AGPLv3, see LICENCE file for details.
483+
484+package filestorage
485+
486+import (
487+ "fmt"
488+ "io"
489+ "os"
490+ "path"
491+ "path/filepath"
492+ "sort"
493+
494+ "launchpad.net/juju-core/environs"
495+ "launchpad.net/juju-core/utils"
496+)
497+
498+// fileStorageReader implements StorageReader backed
499+// by the local filesystem.
500+type fileStorageReader struct {
501+ path string
502+}
503+
504+// newFileStorageReader returns a new storage reader for
505+// a directory inside the local file system.
506+func NewFileStorageReader(path string) (environs.StorageReader, error) {
507+ p := filepath.Clean(path)
508+ fi, err := os.Stat(p)
509+ if err != nil {
510+ return nil, err
511+ }
512+ if !fi.Mode().IsDir() {
513+ return nil, fmt.Errorf("specified source path is not a directory: %s", path)
514+ }
515+ return &fileStorageReader{p}, nil
516+}
517+
518+// Get implements environs.StorageReader.Get.
519+func (f *fileStorageReader) Get(name string) (io.ReadCloser, error) {
520+ filename, err := f.URL(name)
521+ if err != nil {
522+ return nil, err
523+ }
524+ file, err := os.Open(filename)
525+ if err != nil {
526+ return nil, err
527+ }
528+ return file, nil
529+}
530+
531+// List implements environs.StorageReader.List.
532+func (f *fileStorageReader) List(prefix string) ([]string, error) {
533+ // Add one for the missing path separator.
534+ pathlen := len(f.path) + 1
535+ pattern := filepath.Join(f.path, prefix+"*")
536+ matches, err := filepath.Glob(pattern)
537+ if err != nil {
538+ return nil, err
539+ }
540+ list := []string{}
541+ for _, match := range matches {
542+ fi, err := os.Stat(match)
543+ if err != nil {
544+ return nil, err
545+ }
546+ if !fi.Mode().IsDir() {
547+ filename := match[pathlen:]
548+ list = append(list, filename)
549+ }
550+ }
551+ sort.Strings(list)
552+ return list, nil
553+}
554+
555+// URL implements environs.StorageReader.URL.
556+func (f *fileStorageReader) URL(name string) (string, error) {
557+ return path.Join(f.path, name), nil
558+}
559+
560+// ConsistencyStrategy implements environs.StorageReader.ConsistencyStrategy.
561+func (f *fileStorageReader) ConsistencyStrategy() utils.AttemptStrategy {
562+ return utils.AttemptStrategy{}
563+}
564
565=== modified file 'environs/localstorage/storage.go'
566--- environs/localstorage/storage.go 2013-08-30 16:31:45 +0000
567+++ environs/localstorage/storage.go 2013-09-02 06:06:26 +0000
568@@ -63,6 +63,12 @@
569 return nil, err
570 }
571 if resp.StatusCode != http.StatusOK {
572+ // If the path is not found, it's not an error
573+ // because it's only created when the first
574+ // file is put.
575+ if resp.StatusCode == http.StatusNotFound {
576+ return []string{}, nil
577+ }
578 return nil, fmt.Errorf("%s", resp.Status)
579 }
580 defer resp.Body.Close()
581
582=== modified file 'environs/localstorage/storage_test.go'
583--- environs/localstorage/storage_test.go 2013-08-19 11:19:07 +0000
584+++ environs/localstorage/storage_test.go 2013-09-02 06:06:26 +0000
585@@ -22,6 +22,15 @@
586
587 var _ = gc.Suite(&storageSuite{})
588
589+func (s *storageSuite) TestList(c *gc.C) {
590+ listener, _, _ := startServer(c)
591+ defer listener.Close()
592+ storage := localstorage.Client(listener.Addr().String())
593+ names, err := storage.List("a/b/c")
594+ c.Assert(err, gc.IsNil)
595+ c.Assert(names, gc.HasLen, 0)
596+}
597+
598 // TestPersistence tests the adding, reading, listing and removing
599 // of files from the local storage.
600 func (s *storageSuite) TestPersistence(c *gc.C) {
601
602=== added file 'environs/simplestreams/json.go'
603--- environs/simplestreams/json.go 1970-01-01 00:00:00 +0000
604+++ environs/simplestreams/json.go 2013-09-02 06:06:26 +0000
605@@ -0,0 +1,55 @@
606+// Copyright 2013 Canonical Ltd.
607+// Licensed under the AGPLv3, see LICENCE file for details.
608+
609+package simplestreams
610+
611+import (
612+ "encoding/json"
613+ "reflect"
614+)
615+
616+// itemCollection is a clone of ItemCollection, but
617+// does not implement json.Unmarshaler.
618+type itemCollection struct {
619+ Items map[string]*json.RawMessage `json:"items"`
620+ Arch string `json:"arch,omitempty"`
621+ Version string `json:"version,omitempty"`
622+ RegionName string `json:"region,omitempty"`
623+ Endpoint string `json:"endpoint,omitempty"`
624+}
625+
626+// ItemsCollection.UnmarshalJSON unmarshals the ItemCollection,
627+// storing the raw bytes for each item. These can later be
628+// unmarshalled again into product-specific types.
629+func (c *ItemCollection) UnmarshalJSON(b []byte) error {
630+ var raw itemCollection
631+ if err := json.Unmarshal(b, &raw); err != nil {
632+ return err
633+ }
634+ c.rawItems = raw.Items
635+ c.Items = make(map[string]interface{}, len(raw.Items))
636+ c.Arch = raw.Arch
637+ c.Version = raw.Version
638+ c.RegionName = raw.RegionName
639+ c.Endpoint = raw.Endpoint
640+ for key, rawv := range raw.Items {
641+ var v interface{}
642+ if err := json.Unmarshal([]byte(*rawv), &v); err != nil {
643+ return err
644+ }
645+ c.Items[key] = v
646+ }
647+ return nil
648+}
649+
650+func (c *ItemCollection) construct(itemType reflect.Type) error {
651+ for key, rawv := range c.rawItems {
652+ itemValuePtr := reflect.New(itemType)
653+ err := json.Unmarshal([]byte(*rawv), itemValuePtr.Interface())
654+ if err != nil {
655+ return err
656+ }
657+ c.Items[key] = itemValuePtr.Interface()
658+ }
659+ return nil
660+}
661
662=== added file 'environs/simplestreams/json_test.go'
663--- environs/simplestreams/json_test.go 1970-01-01 00:00:00 +0000
664+++ environs/simplestreams/json_test.go 2013-09-02 06:06:26 +0000
665@@ -0,0 +1,34 @@
666+package simplestreams_test
667+
668+import (
669+ "encoding/json"
670+
671+ gc "launchpad.net/gocheck"
672+
673+ "launchpad.net/juju-core/environs/simplestreams"
674+)
675+
676+type jsonSuite struct{}
677+
678+func (s *jsonSuite) TestItemCollectionMarshalling(c *gc.C) {
679+ // Ensure that unmarshalling a simplestreams.ItemCollection
680+ // directly (not through ParseCloudMetadata) doesn't
681+ // cause any surprises.
682+ var m simplestreams.ItemCollection
683+ m.Items = make(map[string]interface{})
684+ err := json.Unmarshal([]byte(`{
685+ "items": {
686+ "a": "b",
687+ "c": 123
688+ }
689+ }`), &m)
690+ c.Assert(err, gc.IsNil)
691+ c.Assert(m.Items, gc.DeepEquals, map[string]interface{}{
692+ "a": "b",
693+ "c": float64(123),
694+ })
695+ // Ensure marshalling works as expected, too.
696+ b, err := json.Marshal(&m)
697+ c.Assert(err, gc.IsNil)
698+ c.Assert(string(b), gc.Equals, `{"items":{"a":"b","c":123}}`)
699+}
700
701=== modified file 'environs/simplestreams/simplestreams.go'
702--- environs/simplestreams/simplestreams.go 2013-08-26 12:23:09 +0000
703+++ environs/simplestreams/simplestreams.go 2013-09-02 06:06:26 +0000
704@@ -134,32 +134,33 @@
705 type attributeValues map[string]string
706 type aliasesByAttribute map[string]attributeValues
707
708-// Exported for testing
709 type CloudMetadata struct {
710 Products map[string]MetadataCatalog `json:"products"`
711- Aliases map[string]aliasesByAttribute `json:"_aliases"`
712+ Aliases map[string]aliasesByAttribute `json:"_aliases,omitempty"`
713 Updated string `json:"updated"`
714 Format string `json:"format"`
715 }
716
717-type itemsByVersion map[string]*ItemCollection
718-
719 type MetadataCatalog struct {
720- Series string `json:"release"`
721- Version string `json:"version"`
722- Arch string `json:"arch"`
723- RegionName string `json:"region"`
724- Endpoint string `json:"endpoint"`
725- Items itemsByVersion `json:"versions"`
726+ Series string `json:"release,omitempty"`
727+ Version string `json:"version,omitempty"`
728+ Arch string `json:"arch,omitempty"`
729+ RegionName string `json:"region,omitempty"`
730+ Endpoint string `json:"endpoint,omitempty"`
731+
732+ // Items is a mapping from version to an ItemCollection,
733+ // where the version is the date the items were produced,
734+ // in the format YYYYMMDD.
735+ Items map[string]*ItemCollection `json:"versions"`
736 }
737
738-// Exported for testing
739 type ItemCollection struct {
740+ rawItems map[string]*json.RawMessage
741 Items map[string]interface{} `json:"items"`
742- Arch string `json:"arch"`
743- Version string `json:"version"`
744- RegionName string `json:"region"`
745- Endpoint string `json:"endpoint"`
746+ Arch string `json:"arch,omitempty"`
747+ Version string `json:"version,omitempty"`
748+ RegionName string `json:"region,omitempty"`
749+ Endpoint string `json:"endpoint,omitempty"`
750 }
751
752 // These structs define the model used for metadata indices.
753@@ -181,8 +182,8 @@
754 Updated string `json:"updated"`
755 Format string `json:"format"`
756 DataType string `json:"datatype"`
757- CloudName string `json:"cloudname"`
758- Clouds []CloudSpec `json:"clouds"`
759+ CloudName string `json:"cloudname,omitempty"`
760+ Clouds []CloudSpec `json:"clouds,omitempty"`
761 ProductsFilePath string `json:"path"`
762 ProductIds []string `json:"products"`
763 }
764@@ -706,29 +707,14 @@
765 func (metadata *CloudMetadata) construct(valueType reflect.Type) error {
766 for _, metadataCatalog := range metadata.Products {
767 for _, ItemCollection := range metadataCatalog.Items {
768- for i, item := range ItemCollection.Items {
769- val, err := structFromMap(valueType, item.(map[string]interface{}))
770- if err != nil {
771- return err
772- }
773- ItemCollection.Items[i] = val
774+ if err := ItemCollection.construct(valueType); err != nil {
775+ return err
776 }
777 }
778 }
779 return nil
780 }
781
782-// structFromMap marshalls a mapf of values into a metadata struct.
783-func structFromMap(valueType reflect.Type, attr map[string]interface{}) (interface{}, error) {
784- data, err := json.Marshal(attr)
785- if err != nil {
786- return nil, err
787- }
788- val := reflect.New(valueType).Interface()
789- err = json.Unmarshal([]byte(data), &val)
790- return val, err
791-}
792-
793 type structTags map[reflect.Type]map[string]int
794
795 var tagsForType structTags = make(structTags)
796
797=== modified file 'environs/simplestreams/simplestreams_test.go'
798--- environs/simplestreams/simplestreams_test.go 2013-08-16 05:39:37 +0000
799+++ environs/simplestreams/simplestreams_test.go 2013-09-02 06:06:26 +0000
800@@ -17,6 +17,7 @@
801 func Test(t *testing.T) {
802 registerSimpleStreamsTests()
803 gc.Suite(&signingSuite{})
804+ gc.Suite(&jsonSuite{})
805 gc.TestingT(t)
806 }
807
808
809=== modified file 'environs/sync/sync.go'
810--- environs/sync/sync.go 2013-08-28 00:25:31 +0000
811+++ environs/sync/sync.go 2013-09-02 06:06:26 +0000
812@@ -7,19 +7,15 @@
813 "bytes"
814 "fmt"
815 "io"
816- "os"
817- "path"
818- "path/filepath"
819- "sort"
820 "time"
821
822 "launchpad.net/loggo"
823
824 "launchpad.net/juju-core/environs"
825+ "launchpad.net/juju-core/environs/filestorage"
826 envtools "launchpad.net/juju-core/environs/tools"
827 "launchpad.net/juju-core/provider/ec2"
828 coretools "launchpad.net/juju-core/tools"
829- "launchpad.net/juju-core/utils"
830 "launchpad.net/juju-core/version"
831 )
832
833@@ -123,7 +119,7 @@
834 if ctx.Source == "" {
835 return ec2.NewHTTPStorageReader(DefaultToolsLocation), nil
836 }
837- return newFileStorageReader(ctx.Source)
838+ return filestorage.NewFileStorageReader(ctx.Source)
839 }
840
841 // copyTools copies a set of tools from the source to the target.
842@@ -165,73 +161,6 @@
843 return nil
844 }
845
846-// fileStorageReader implements StorageReader backed
847-// by the local filesystem.
848-type fileStorageReader struct {
849- path string
850-}
851-
852-// newFileStorageReader returns a new storage reader for
853-// a directory inside the local file system.
854-func newFileStorageReader(path string) (environs.StorageReader, error) {
855- p := filepath.Clean(path)
856- fi, err := os.Stat(p)
857- if err != nil {
858- return nil, err
859- }
860- if !fi.Mode().IsDir() {
861- return nil, fmt.Errorf("specified source path is not a directory: %s", path)
862- }
863- return &fileStorageReader{p}, nil
864-}
865-
866-// Get implements environs.StorageReader.Get.
867-func (f *fileStorageReader) Get(name string) (io.ReadCloser, error) {
868- filename, err := f.URL(name)
869- if err != nil {
870- return nil, err
871- }
872- file, err := os.Open(filename)
873- if err != nil {
874- return nil, err
875- }
876- return file, nil
877-}
878-
879-// List implements environs.StorageReader.List.
880-func (f *fileStorageReader) List(prefix string) ([]string, error) {
881- // Add one for the missing path separator.
882- pathlen := len(f.path) + 1
883- pattern := filepath.Join(f.path, prefix+"*")
884- matches, err := filepath.Glob(pattern)
885- if err != nil {
886- return nil, err
887- }
888- list := []string{}
889- for _, match := range matches {
890- fi, err := os.Stat(match)
891- if err != nil {
892- return nil, err
893- }
894- if !fi.Mode().IsDir() {
895- filename := match[pathlen:]
896- list = append(list, filename)
897- }
898- }
899- sort.Strings(list)
900- return list, nil
901-}
902-
903-// URL implements environs.StorageReader.URL.
904-func (f *fileStorageReader) URL(name string) (string, error) {
905- return path.Join(f.path, name), nil
906-}
907-
908-// ConsistencyStrategy implements environs.StorageReader.ConsistencyStrategy.
909-func (f *fileStorageReader) ConsistencyStrategy() utils.AttemptStrategy {
910- return utils.AttemptStrategy{}
911-}
912-
913 // NewSyncLogWriter creates a loggo writer for registration
914 // by the callers of Sync. This way the logged output can also
915 // be displayed otherwise, e.g. on the screen.
916
917=== modified file 'environs/tools/boilerplate.go'
918--- environs/tools/boilerplate.go 2013-08-21 02:01:09 +0000
919+++ environs/tools/boilerplate.go 2013-09-02 06:06:26 +0000
920@@ -74,7 +74,7 @@
921
922 type toolsMetadataParams struct {
923 ToolsBinaryPath string
924- ToolsBinarySize float64
925+ ToolsBinarySize int64
926 ToolsBinarySHA256 string
927 Region string
928 URL string
929
930=== added file 'environs/tools/marshal.go'
931--- environs/tools/marshal.go 1970-01-01 00:00:00 +0000
932+++ environs/tools/marshal.go 2013-09-02 06:06:26 +0000
933@@ -0,0 +1,86 @@
934+// Copyright 2013 Canonical Ltd.
935+// Licensed under the AGPLv3, see LICENCE file for details.
936+
937+// The tools package supports locating, parsing, and filtering Ubuntu tools metadata in simplestreams format.
938+// See http://launchpad.net/simplestreams and in particular the doc/README file in that project for more information
939+// about the file formats.
940+package tools
941+
942+import (
943+ "encoding/json"
944+ "fmt"
945+ "time"
946+
947+ "launchpad.net/juju-core/environs/simplestreams"
948+ "launchpad.net/juju-core/utils/set"
949+)
950+
951+const (
952+ ProductMetadataPath = "streams/v1/com.ubuntu.juju:released:tools.json"
953+)
954+
955+// MarshalToolsMetadataJSON marshals tools metadata to index and products JSON.
956+//
957+// updated is the time at which the JSON file was updated.
958+func MarshalToolsMetadataJSON(metadata []*ToolsMetadata, updated time.Time) (index, products []byte, err error) {
959+ if index, err = MarshalToolsMetadataIndexJSON(metadata, updated); err != nil {
960+ return nil, nil, err
961+ }
962+ if products, err = MarshalToolsMetadataProductsJSON(metadata, updated); err != nil {
963+ return nil, nil, err
964+ }
965+ return index, products, err
966+}
967+
968+// MarshalToolsMetadataIndexJSON marshals tools metadata to index JSON.
969+//
970+// updated is the time at which the JSON file was updated.
971+func MarshalToolsMetadataIndexJSON(metadata []*ToolsMetadata, updated time.Time) (out []byte, err error) {
972+ productIds := make([]string, len(metadata))
973+ for i, t := range metadata {
974+ productIds[i] = t.productId()
975+ }
976+ var indices simplestreams.Indices
977+ indices.Updated = updated.Format(time.RFC1123Z)
978+ indices.Format = "index:1.0"
979+ indices.Indexes = map[string]*simplestreams.IndexMetadata{
980+ "com.ubuntu.juju:released:tools": &simplestreams.IndexMetadata{
981+ Updated: indices.Updated,
982+ Format: "products:1.0",
983+ DataType: "content-download",
984+ ProductsFilePath: ProductMetadataPath,
985+ ProductIds: set.NewStrings(productIds...).SortedValues(),
986+ },
987+ }
988+ return json.MarshalIndent(&indices, "", " ")
989+}
990+
991+// MarshalToolsMetadataProductsJSON marshals tools metadata to products JSON.
992+//
993+// updated is the time at which the JSON file was updated.
994+func MarshalToolsMetadataProductsJSON(metadata []*ToolsMetadata, updated time.Time) (out []byte, err error) {
995+ var cloud simplestreams.CloudMetadata
996+ cloud.Updated = updated.Format(time.RFC1123Z)
997+ cloud.Format = "products:1.0"
998+ cloud.Products = make(map[string]simplestreams.MetadataCatalog)
999+ itemsversion := updated.Format("20060201") // YYYYMMDD
1000+ for _, t := range metadata {
1001+ id := t.productId()
1002+ itemid := fmt.Sprintf("%s-%s-%s", t.Version, t.Release, t.Arch)
1003+ if catalog, ok := cloud.Products[id]; ok {
1004+ catalog.Items[itemsversion].Items[itemid] = t
1005+ } else {
1006+ catalog = simplestreams.MetadataCatalog{
1007+ Arch: t.Arch,
1008+ Version: t.Version,
1009+ Items: map[string]*simplestreams.ItemCollection{
1010+ itemsversion: &simplestreams.ItemCollection{
1011+ Items: map[string]interface{}{itemid: t},
1012+ },
1013+ },
1014+ }
1015+ cloud.Products[id] = catalog
1016+ }
1017+ }
1018+ return json.MarshalIndent(&cloud, "", " ")
1019+}
1020
1021=== added file 'environs/tools/marshal_test.go'
1022--- environs/tools/marshal_test.go 1970-01-01 00:00:00 +0000
1023+++ environs/tools/marshal_test.go 2013-09-02 06:06:26 +0000
1024@@ -0,0 +1,150 @@
1025+// Copyright 2013 Canonical Ltd.
1026+// Licensed under the AGPLv3, see LICENCE file for details.
1027+
1028+package tools_test
1029+
1030+import (
1031+ "time"
1032+
1033+ gc "launchpad.net/gocheck"
1034+
1035+ "launchpad.net/juju-core/environs/tools"
1036+ jc "launchpad.net/juju-core/testing/checkers"
1037+)
1038+
1039+var _ = gc.Suite(&marshalSuite{})
1040+
1041+type marshalSuite struct{}
1042+
1043+func (s *marshalSuite) TestLargeNumber(c *gc.C) {
1044+ metadata := []*tools.ToolsMetadata{
1045+ &tools.ToolsMetadata{
1046+ Release: "saucy",
1047+ Version: "1.2.3.4",
1048+ Arch: "arm",
1049+ Size: 9223372036854775807,
1050+ Path: "/somewhere/over/the/rainbow.tar.gz",
1051+ FileType: "tar.gz",
1052+ },
1053+ }
1054+ _, products, err := tools.MarshalToolsMetadataJSON(metadata, time.Now())
1055+ c.Assert(err, gc.IsNil)
1056+ c.Assert(string(products), jc.Contains, `"size": 9223372036854775807`)
1057+}
1058+
1059+var expectedIndex = `{
1060+ "index": {
1061+ "com.ubuntu.juju:released:tools": {
1062+ "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
1063+ "format": "products:1.0",
1064+ "datatype": "content-download",
1065+ "path": "streams/v1/com.ubuntu.juju:released:tools.json",
1066+ "products": [
1067+ "com.ubuntu.juju:1.2.3.4:arm",
1068+ "com.ubuntu.juju:4.3.2.1:amd64"
1069+ ]
1070+ }
1071+ },
1072+ "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
1073+ "format": "index:1.0"
1074+}`
1075+
1076+var expectedProducts = `{
1077+ "products": {
1078+ "com.ubuntu.juju:1.2.3.4:arm": {
1079+ "version": "1.2.3.4",
1080+ "arch": "arm",
1081+ "versions": {
1082+ "19700101": {
1083+ "items": {
1084+ "1.2.3.4-precise-arm": {
1085+ "release": "precise",
1086+ "version": "1.2.3.4",
1087+ "arch": "arm",
1088+ "size": 42,
1089+ "path": "toenlightenment.tar.gz",
1090+ "ftype": "tar.gz",
1091+ "sha256": ""
1092+ },
1093+ "1.2.3.4-saucy-arm": {
1094+ "release": "saucy",
1095+ "version": "1.2.3.4",
1096+ "arch": "arm",
1097+ "size": 9223372036854775807,
1098+ "path": "/somewhere/over/the/rainbow.tar.gz",
1099+ "ftype": "tar.gz",
1100+ "sha256": ""
1101+ }
1102+ }
1103+ }
1104+ }
1105+ },
1106+ "com.ubuntu.juju:4.3.2.1:amd64": {
1107+ "version": "4.3.2.1",
1108+ "arch": "amd64",
1109+ "versions": {
1110+ "19700101": {
1111+ "items": {
1112+ "4.3.2.1-precise-amd64": {
1113+ "release": "precise",
1114+ "version": "4.3.2.1",
1115+ "arch": "amd64",
1116+ "size": 0,
1117+ "path": "whatever.tar.gz",
1118+ "ftype": "tar.gz",
1119+ "sha256": "afb14e65c794464e378def12cbad6a96f9186d69"
1120+ }
1121+ }
1122+ }
1123+ }
1124+ }
1125+ },
1126+ "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
1127+ "format": "products:1.0"
1128+}`
1129+
1130+var toolMetadataForTesting = []*tools.ToolsMetadata{
1131+ &tools.ToolsMetadata{
1132+ Release: "saucy",
1133+ Version: "1.2.3.4",
1134+ Arch: "arm",
1135+ Size: 9223372036854775807,
1136+ Path: "/somewhere/over/the/rainbow.tar.gz",
1137+ FileType: "tar.gz",
1138+ },
1139+ &tools.ToolsMetadata{
1140+ Release: "precise",
1141+ Version: "1.2.3.4",
1142+ Arch: "arm",
1143+ Size: 42,
1144+ Path: "toenlightenment.tar.gz",
1145+ FileType: "tar.gz",
1146+ },
1147+ &tools.ToolsMetadata{
1148+ Release: "precise",
1149+ Version: "4.3.2.1",
1150+ Arch: "amd64",
1151+ Path: "whatever.tar.gz",
1152+ FileType: "tar.gz",
1153+ SHA256: "afb14e65c794464e378def12cbad6a96f9186d69",
1154+ },
1155+}
1156+
1157+func (s *marshalSuite) TestMarshalIndex(c *gc.C) {
1158+ index, err := tools.MarshalToolsMetadataIndexJSON(toolMetadataForTesting, time.Unix(0, 0).UTC())
1159+ c.Assert(err, gc.IsNil)
1160+ c.Assert(string(index), gc.Equals, expectedIndex)
1161+}
1162+
1163+func (s *marshalSuite) TestMarshalProducts(c *gc.C) {
1164+ products, err := tools.MarshalToolsMetadataProductsJSON(toolMetadataForTesting, time.Unix(0, 0).UTC())
1165+ c.Assert(err, gc.IsNil)
1166+ c.Assert(string(products), gc.Equals, expectedProducts)
1167+}
1168+
1169+func (s *marshalSuite) TestMarshal(c *gc.C) {
1170+ index, products, err := tools.MarshalToolsMetadataJSON(toolMetadataForTesting, time.Unix(0, 0).UTC())
1171+ c.Assert(err, gc.IsNil)
1172+ c.Assert(string(index), gc.Equals, expectedIndex)
1173+ c.Assert(string(products), gc.Equals, expectedProducts)
1174+}
1175
1176=== modified file 'environs/tools/simplestreams.go'
1177--- environs/tools/simplestreams.go 2013-08-21 02:01:09 +0000
1178+++ environs/tools/simplestreams.go 2013-09-02 06:06:26 +0000
1179@@ -8,6 +8,7 @@
1180
1181 import (
1182 "fmt"
1183+
1184 "launchpad.net/juju-core/environs/simplestreams"
1185 )
1186
1187@@ -44,13 +45,17 @@
1188
1189 // ToolsMetadata holds information about a particular tools tarball.
1190 type ToolsMetadata struct {
1191- Release string `json:"release"`
1192- Version string `json:"version"`
1193- Arch string `json:"arch"`
1194- Size float64 `json:"size"`
1195- Path string `json:"path"`
1196- FileType string `json:"ftype"`
1197- SHA256 string `json:"sha256"`
1198+ Release string `json:"release"`
1199+ Version string `json:"version"`
1200+ Arch string `json:"arch"`
1201+ Size int64 `json:"size"`
1202+ Path string `json:"path"`
1203+ FileType string `json:"ftype"`
1204+ SHA256 string `json:"sha256"`
1205+}
1206+
1207+func (t *ToolsMetadata) productId() string {
1208+ return fmt.Sprintf("com.ubuntu.juju:%s:%s", t.Version, t.Arch)
1209 }
1210
1211 // Fetch returns a list of tools for the specified cloud matching the constraint.
1212
1213=== modified file 'environs/tools/simplestreams_test.go'
1214--- environs/tools/simplestreams_test.go 2013-08-21 01:58:26 +0000
1215+++ environs/tools/simplestreams_test.go 2013-09-02 06:06:26 +0000
1216@@ -233,3 +233,44 @@
1217 c.Assert(err, gc.IsNil)
1218 c.Assert(ids, gc.DeepEquals, []string{"com.ubuntu.juju:1.10.1:amd64"})
1219 }
1220+
1221+func (s *productSpecSuite) TestLargeNumber(c *gc.C) {
1222+ json := `{
1223+ "updated": "Fri, 30 Aug 2013 16:12:58 +0800",
1224+ "format": "products:1.0",
1225+ "products": {
1226+ "com.ubuntu.juju:1.10.0:amd64": {
1227+ "version": "1.10.0",
1228+ "arch": "amd64",
1229+ "versions": {
1230+ "20133008": {
1231+ "items": {
1232+ "1.10.0-precise-amd64": {
1233+ "release": "precise",
1234+ "version": "1.10.0",
1235+ "arch": "amd64",
1236+ "size": 9223372036854775807,
1237+ "path": "releases/juju-1.10.0-precise-amd64.tgz",
1238+ "ftype": "tar.gz",
1239+ "sha256": ""
1240+ }
1241+ }
1242+ }
1243+ }
1244+ }
1245+ }
1246+ }`
1247+ cloudMetadata, err := simplestreams.ParseCloudMetadata([]byte(json), "products:1.0", "", tools.ToolsMetadata{})
1248+ c.Assert(err, gc.IsNil)
1249+ c.Assert(cloudMetadata.Products, gc.HasLen, 1)
1250+ product := cloudMetadata.Products["com.ubuntu.juju:1.10.0:amd64"]
1251+ c.Assert(product, gc.NotNil)
1252+ c.Assert(product.Items, gc.HasLen, 1)
1253+ version := product.Items["20133008"]
1254+ c.Assert(version, gc.NotNil)
1255+ c.Assert(version.Items, gc.HasLen, 1)
1256+ item := version.Items["1.10.0-precise-amd64"]
1257+ c.Assert(item, gc.NotNil)
1258+ c.Assert(item, gc.FitsTypeOf, &tools.ToolsMetadata{})
1259+ c.Assert(item.(*tools.ToolsMetadata).Size, gc.Equals, int64(9223372036854775807))
1260+}

Subscribers

People subscribed via source and target branches

to status/vote changes: