juju-core

Merge lp:~wallyworld/juju-core/marshal-image-metadata into lp:~go-bot/juju-core/trunk

  1. marshal-image-metadata
  2. Merge into trunk
Proposed by Ian Booth
Status: Merged
Approved by: Ian Booth
Approved revision: no longer in the source branch.
Merged at revision: 1993
Proposed branch: lp:~wallyworld/juju-core/marshal-image-metadata
Merge into: lp:~go-bot/juju-core/trunk
Prerequisite: lp:~wallyworld/juju-core/improve-image-metadata-command
Diff against target: 677 lines (+406/-137)
12 files modified
cmd/plugins/juju-metadata/imagemetadata.go (+6/-1)
cmd/plugins/juju-metadata/imagemetadata_test.go (+1/-1)
cmd/plugins/juju-metadata/validateimagemetadata_test.go (+6/-1)
environs/imagemetadata/generate.go (+37/-124)
environs/imagemetadata/generate_test.go (+40/-0)
environs/imagemetadata/marshal.go (+97/-0)
environs/imagemetadata/marshal_test.go (+127/-0)
environs/imagemetadata/simplestreams.go (+15/-6)
environs/imagemetadata/testing/testing.go (+70/-0)
environs/imagemetadata/validation_test.go (+4/-1)
environs/simplestreams/simplestreams.go (+2/-2)
environs/tools/marshal.go (+1/-1)
To merge this branch: bzr merge lp:~wallyworld/juju-core/marshal-image-metadata
Reviewer Review Type Date Requested Status
Juju Engineering Pending
Review via email: mp+190542@code.launchpad.net

Commit message

Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://codereview.appspot.com/14540055/

Description of the change

Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://codereview.appspot.com/14540055/

To post a comment you must log in.
Revision history for this message
Ian Booth (wallyworld) wrote :

Reviewers: mp+190542_code.launchpad.net,

Message:
Please take a look.

Description:
Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://code.launchpad.net/~wallyworld/juju-core/marshal-image-metadata/+merge/190542

Requires:
https://code.launchpad.net/~wallyworld/juju-core/improve-image-metadata-command/+merge/190517

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/14540055/

Affected files (+398, -135 lines):
   A [revision details]
   M cmd/plugins/juju-metadata/imagemetadata.go
   M environs/imagemetadata/generate.go
   A environs/imagemetadata/generate_test.go
   A environs/imagemetadata/marshal.go
   A environs/imagemetadata/marshal_test.go
   M environs/imagemetadata/simplestreams.go
   A environs/imagemetadata/testing/testing.go
   M environs/imagemetadata/validation_test.go
   M environs/simplestreams/simplestreams.go
   M environs/tools/marshal.go

Revision history for this message
Ian Booth (wallyworld) wrote :

Please take a look.

https://codereview.appspot.com/14540055/

Revision history for this message
Tim Penhey (thumper) wrote :

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go
File environs/imagemetadata/generate.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go#newcode15
environs/imagemetadata/generate.go:15: metadataInfo, err :=
generateMetadata(series, im, cloudSpec)
This has me queezy.

Surely WriteMetadata would take the metadataInfo as an input parameter?

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go
File environs/imagemetadata/generate_test.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode17
environs/imagemetadata/generate_test.go:17: type generateSuite struct{}
testbase.LoggingSuite

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode34
environs/imagemetadata/generate_test.go:34: c.Assert(len(metadata),
gc.Equals, 1)
gc.HasLen

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go
File environs/imagemetadata/simplestreams.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go#newcode121
environs/imagemetadata/simplestreams.go:121: Release string
`json:"-"`
What does json:"-" do?

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go
File environs/tools/marshal.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go#newcode69
environs/tools/marshal.go:69: itemsversion := updated.Format("20060102")
// YYYYMMDD
Some of these changes are the same as the branch I just reviewed.

https://codereview.appspot.com/14540055/

Revision history for this message
Ian Booth (wallyworld) wrote :

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go
File environs/imagemetadata/generate.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go#newcode15
environs/imagemetadata/generate.go:15: metadataInfo, err :=
generateMetadata(series, im, cloudSpec)
On 2013/10/16 01:26:11, thumper wrote:
> This has me queezy.

> Surely WriteMetadata would take the metadataInfo as an input
parameter?

The sematics of the write function is to take a ImageMetadata struct and
write out the simplestreams metadata. In order to write the metadata, it
needs to be generated from the struct. The method could be called
WriteAndGenerate I guess. But the generation and writing to file is an
atomic operation.

A subsequent branch renames the method anyway and refactors the content
to align with what Andrew did for tools metadata.

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go
File environs/imagemetadata/generate_test.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode17
environs/imagemetadata/generate_test.go:17: type generateSuite struct{}
On 2013/10/16 01:26:11, thumper wrote:
> testbase.LoggingSuite

Will fix

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode34
environs/imagemetadata/generate_test.go:34: c.Assert(len(metadata),
gc.Equals, 1)
On 2013/10/16 01:26:11, thumper wrote:
> gc.HasLen

Will fix

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go
File environs/imagemetadata/simplestreams.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go#newcode121
environs/imagemetadata/simplestreams.go:121: Release string
`json:"-"`
On 2013/10/16 01:26:11, thumper wrote:
> What does json:"-" do?

Tells json not to marshall that attribute. It's there to provide info
for other mthods in the struct to do their thing, but we don't want it
serialised to json.

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go
File environs/tools/marshal.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go#newcode69
environs/tools/marshal.go:69: itemsversion := updated.Format("20060102")
// YYYYMMDD
On 2013/10/16 01:26:11, thumper wrote:
> Some of these changes are the same as the branch I just reviewed.

I fixed this in a downstream branch - I didn't pump the changes back up
the pipe sorry.

https://codereview.appspot.com/14540055/

Revision history for this message
Ian Booth (wallyworld) wrote :

Please take a look.

https://codereview.appspot.com/14540055/

Revision history for this message
Tim Penhey (thumper) wrote :

LGTM

https://codereview.appspot.com/14540055/

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'cmd/plugins/juju-metadata/imagemetadata.go'
--- cmd/plugins/juju-metadata/imagemetadata.go 2013-10-16 02:56:51 +0000
+++ cmd/plugins/juju-metadata/imagemetadata.go 2013-10-16 04:12:27 +0000
@@ -14,6 +14,7 @@
14 "launchpad.net/juju-core/environs"14 "launchpad.net/juju-core/environs"
15 "launchpad.net/juju-core/environs/config"15 "launchpad.net/juju-core/environs/config"
16 "launchpad.net/juju-core/environs/configstore"16 "launchpad.net/juju-core/environs/configstore"
17 "launchpad.net/juju-core/environs/filestorage"
17 "launchpad.net/juju-core/environs/imagemetadata"18 "launchpad.net/juju-core/environs/imagemetadata"
18 "launchpad.net/juju-core/environs/simplestreams"19 "launchpad.net/juju-core/environs/simplestreams"
19)20)
@@ -152,7 +153,11 @@
152 Region: c.Region,153 Region: c.Region,
153 Endpoint: c.Endpoint,154 Endpoint: c.Endpoint,
154 }155 }
155 _, err := imagemetadata.GenerateMetadata(c.Series, &im, &cloudSpec, c.Dir)156 targetStorage, err := filestorage.NewFileStorageWriter(c.Dir, filestorage.UseDefaultTmpDir)
157 if err != nil {
158 return err
159 }
160 err = imagemetadata.WriteMetadata(c.Series, &im, &cloudSpec, targetStorage)
156 if err != nil {161 if err != nil {
157 return fmt.Errorf("image metadata files could not be created: %v", err)162 return fmt.Errorf("image metadata files could not be created: %v", err)
158 }163 }
159164
=== modified file 'cmd/plugins/juju-metadata/imagemetadata_test.go'
--- cmd/plugins/juju-metadata/imagemetadata_test.go 2013-10-16 02:56:51 +0000
+++ cmd/plugins/juju-metadata/imagemetadata_test.go 2013-10-16 04:12:27 +0000
@@ -108,7 +108,7 @@
108108
109const (109const (
110 defaultIndexFileName = "index.json"110 defaultIndexFileName = "index.json"
111 defaultImageFileName = "imagemetadata.json"111 defaultImageFileName = "com.ubuntu.cloud:released:imagemetadata.json"
112)112)
113113
114func (s *ImageMetadataSuite) TestImageMetadataFilesNoEnv(c *gc.C) {114func (s *ImageMetadataSuite) TestImageMetadataFilesNoEnv(c *gc.C) {
115115
=== modified file 'cmd/plugins/juju-metadata/validateimagemetadata_test.go'
--- cmd/plugins/juju-metadata/validateimagemetadata_test.go 2013-10-16 03:38:32 +0000
+++ cmd/plugins/juju-metadata/validateimagemetadata_test.go 2013-10-16 04:12:27 +0000
@@ -11,6 +11,7 @@
11 gc "launchpad.net/gocheck"11 gc "launchpad.net/gocheck"
1212
13 "launchpad.net/juju-core/cmd"13 "launchpad.net/juju-core/cmd"
14 "launchpad.net/juju-core/environs/filestorage"
14 "launchpad.net/juju-core/environs/imagemetadata"15 "launchpad.net/juju-core/environs/imagemetadata"
15 "launchpad.net/juju-core/environs/simplestreams"16 "launchpad.net/juju-core/environs/simplestreams"
16 coretesting "launchpad.net/juju-core/testing"17 coretesting "launchpad.net/juju-core/testing"
@@ -73,7 +74,11 @@
73 Region: region,74 Region: region,
74 Endpoint: endpoint,75 Endpoint: endpoint,
75 }76 }
76 _, err := imagemetadata.GenerateMetadata(series, &im, &cloudSpec, s.metadataDir)77 targetStorage, err := filestorage.NewFileStorageWriter(s.metadataDir, filestorage.UseDefaultTmpDir)
78 if err != nil {
79 return err
80 }
81 err = imagemetadata.WriteMetadata(series, &im, &cloudSpec, targetStorage)
77 if err != nil {82 if err != nil {
78 return err83 return err
79 }84 }
8085
=== modified file 'environs/imagemetadata/generate.go'
--- environs/imagemetadata/generate.go 2013-10-11 01:20:34 +0000
+++ environs/imagemetadata/generate.go 2013-10-16 04:12:27 +0000
@@ -5,135 +5,48 @@
55
6import (6import (
7 "bytes"7 "bytes"
8 "fmt"
9 "io/ioutil"
10 "os"
11 "path/filepath"
12 "text/template"
13 "time"8 "time"
149
15 "launchpad.net/juju-core/environs/simplestreams"10 "launchpad.net/juju-core/environs/simplestreams"
16)11 "launchpad.net/juju-core/environs/storage"
1712)
18const (13
19 defaultIndexFileName = "index.json"14func WriteMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec, metadataStore storage.Storage) error {
20 defaultImageFileName = "imagemetadata.json"15 metadataInfo, err := generateMetadata(series, im, cloudSpec)
21)16 if err != nil {
22
23// GenerateMetadata generates some basic simplestreams metadata using the specified cloud and image details.
24func GenerateMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec, dest string) ([]string, error) {
25 indexFileName := defaultIndexFileName
26 imageFileName := defaultImageFileName
27 now := time.Now()
28 imparams := imageMetadataParams{
29 Id: im.Id,
30 Arch: im.Arch,
31 Region: cloudSpec.Region,
32 URL: cloudSpec.Endpoint,
33 Path: "streams/v1",
34 ImageFileName: imageFileName,
35 Updated: now.Format(time.RFC1123Z),
36 VersionKey: now.Format("20060102"),
37 }
38
39 var err error
40 imparams.Version, err = simplestreams.SeriesVersion(series)
41 if err != nil {
42 return nil, fmt.Errorf("invalid series %q", series)
43 }
44
45 streamsPath := filepath.Join(dest, "streams", "v1")
46 if err = os.MkdirAll(streamsPath, 0755); err != nil {
47 return nil, err
48 }
49 indexFileName = filepath.Join(streamsPath, indexFileName)
50 imageFileName = filepath.Join(streamsPath, imageFileName)
51 err = writeJsonFile(imparams, indexFileName, indexBoilerplate)
52 if err != nil {
53 return nil, err
54 }
55 err = writeJsonFile(imparams, imageFileName, productBoilerplate)
56 if err != nil {
57 return nil, err
58 }
59 return []string{indexFileName, imageFileName}, nil
60}
61
62type imageMetadataParams struct {
63 Region string
64 URL string
65 Updated string
66 Arch string
67 Path string
68 Series string
69 Version string
70 VersionKey string
71 Id string
72 ImageFileName string
73}
74
75func writeJsonFile(imparams imageMetadataParams, filename, boilerplate string) error {
76 t := template.Must(template.New("").Parse(boilerplate))
77 var metadata bytes.Buffer
78 if err := t.Execute(&metadata, imparams); err != nil {
79 panic(fmt.Errorf("cannot generate %s metdata: %v", filename, err))
80 }
81 data := metadata.Bytes()
82 if err := ioutil.WriteFile(filename, data, 0666); err != nil {
83 return err17 return err
84 }18 }
19 for _, md := range metadataInfo {
20 err = metadataStore.Put(md.Path, bytes.NewReader(md.Data), int64(len(md.Data)))
21 if err != nil {
22 return err
23 }
24 }
85 return nil25 return nil
86}26}
8727
88var indexBoilerplate = `28type MetadataFile struct {
89{29 Path string
90 "index": {30 Data []byte
91 "com.ubuntu.cloud:custom": {31}
92 "updated": "{{.Updated}}",32
93 "clouds": [33// generateMetadata generates some basic simplestreams metadata using the specified cloud and image details.
94 {34func generateMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec) ([]MetadataFile, error) {
95 "region": "{{.Region}}",35 metadata := &ImageMetadata{
96 "endpoint": "{{.URL}}"36 Id: im.Id,
97 }37 Arch: im.Arch,
98 ],38 Release: series,
99 "cloudname": "custom",39 RegionName: cloudSpec.Region,
100 "datatype": "image-ids",40 Endpoint: cloudSpec.Endpoint,
101 "format": "products:1.0",41 }
102 "products": [42
103 "com.ubuntu.cloud:server:{{.Version}}:{{.Arch}}"43 index, products, err := MarshalImageMetadataJSON([]*ImageMetadata{metadata}, cloudSpec, time.Now())
104 ],44 if err != nil {
105 "path": "{{.Path}}/{{.ImageFileName}}"45 return nil, err
106 }46 }
107 },47 objects := []MetadataFile{
108 "updated": "{{.Updated}}",48 {simplestreams.UnsignedIndex, index},
109 "format": "index:1.0"49 {ProductMetadataPath, products},
110}50 }
111`51 return objects, nil
11252}
113var productBoilerplate = `
114{
115 "content_id": "com.ubuntu.cloud:custom",
116 "format": "products:1.0",
117 "updated": "{{.Updated}}",
118 "datatype": "image-ids",
119 "products": {
120 "com.ubuntu.cloud:server:{{.Version}}:{{.Arch}}": {
121 "release": "{{.Series}}",
122 "version": "{{.Version}}",
123 "arch": "{{.Arch}}",
124 "versions": {
125 "{{.VersionKey}}": {
126 "items": {
127 "{{.Id}}": {
128 "region": "{{.Region}}",
129 "id": "{{.Id}}"
130 }
131 },
132 "pubname": "ubuntu-{{.Series}}-{{.Version}}-{{.Arch}}-server-{{.VersionKey}}",
133 "label": "custom"
134 }
135 }
136 }
137 }
138}
139`
14053
=== added file 'environs/imagemetadata/generate_test.go'
--- environs/imagemetadata/generate_test.go 1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/generate_test.go 2013-10-16 04:12:27 +0000
@@ -0,0 +1,40 @@
1// Copyright 2013 Canonical Ltd.
2// Licensed under the AGPLv3, see LICENCE file for details.
3
4package imagemetadata_test
5
6import (
7 gc "launchpad.net/gocheck"
8
9 "launchpad.net/juju-core/environs/filestorage"
10 "launchpad.net/juju-core/environs/imagemetadata"
11 "launchpad.net/juju-core/environs/imagemetadata/testing"
12 "launchpad.net/juju-core/environs/simplestreams"
13 "launchpad.net/juju-core/testing/testbase"
14)
15
16var _ = gc.Suite(&generateSuite{})
17
18type generateSuite struct {
19 testbase.LoggingSuite
20}
21
22func (s *generateSuite) TestWriteMetadata(c *gc.C) {
23 im := &imagemetadata.ImageMetadata{
24 Id: "1234",
25 Arch: "amd64",
26 }
27 cloudSpec := &simplestreams.CloudSpec{
28 Region: "region",
29 Endpoint: "endpoint",
30 }
31 dir := c.MkDir()
32 targetStorage, err := filestorage.NewFileStorageWriter(dir, filestorage.UseDefaultTmpDir)
33 c.Assert(err, gc.IsNil)
34 err = imagemetadata.WriteMetadata("raring", im, cloudSpec, targetStorage)
35 c.Assert(err, gc.IsNil)
36 metadata := testing.ParseMetadata(c, dir)
37 c.Assert(metadata, gc.HasLen, 1)
38 im.RegionName = cloudSpec.Region
39 c.Assert(im, gc.DeepEquals, metadata[0])
40}
041
=== added file 'environs/imagemetadata/marshal.go'
--- environs/imagemetadata/marshal.go 1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/marshal.go 2013-10-16 04:12:27 +0000
@@ -0,0 +1,97 @@
1// Copyright 2013 Canonical Ltd.
2// Licensed under the AGPLv3, see LICENCE file for details.
3
4package imagemetadata
5
6import (
7 "encoding/json"
8 "time"
9
10 "launchpad.net/juju-core/environs/simplestreams"
11 "launchpad.net/juju-core/utils/set"
12)
13
14const (
15 ProductMetadataPath = "streams/v1/com.ubuntu.cloud:released:imagemetadata.json"
16)
17
18// MarshalImageMetadataJSON marshals image metadata to index and products JSON.
19//
20// updated is the time at which the JSON file was updated.
21func MarshalImageMetadataJSON(metadata []*ImageMetadata, cloudSpec *simplestreams.CloudSpec, updated time.Time) (index, products []byte, err error) {
22 if index, err = MarshalImageMetadataIndexJSON(metadata, cloudSpec, updated); err != nil {
23 return nil, nil, err
24 }
25 if products, err = MarshalImageMetadataProductsJSON(metadata, updated); err != nil {
26 return nil, nil, err
27 }
28 return index, products, err
29}
30
31// MarshalImageMetadataIndexJSON marshals image metadata to index JSON.
32//
33// updated is the time at which the JSON file was updated.
34func MarshalImageMetadataIndexJSON(metadata []*ImageMetadata, cloudSpec *simplestreams.CloudSpec, updated time.Time) (out []byte, err error) {
35 productIds := make([]string, len(metadata))
36 for i, t := range metadata {
37 productIds[i], err = t.productId()
38 if err != nil {
39 return nil, err
40 }
41 }
42 var indices simplestreams.Indices
43 indices.Updated = updated.Format(time.RFC1123Z)
44 indices.Format = "index:1.0"
45 indices.Indexes = map[string]*simplestreams.IndexMetadata{
46 "com.ubuntu.cloud:custom": &simplestreams.IndexMetadata{
47 CloudName: "custom",
48 Updated: indices.Updated,
49 Format: "products:1.0",
50 DataType: "image-ids",
51 ProductsFilePath: ProductMetadataPath,
52 ProductIds: set.NewStrings(productIds...).SortedValues(),
53 Clouds: []simplestreams.CloudSpec{*cloudSpec},
54 },
55 }
56 return json.MarshalIndent(&indices, "", " ")
57}
58
59// MarshalImageMetadataProductsJSON marshals image metadata to products JSON.
60//
61// updated is the time at which the JSON file was updated.
62func MarshalImageMetadataProductsJSON(metadata []*ImageMetadata, updated time.Time) (out []byte, err error) {
63 var cloud simplestreams.CloudMetadata
64 cloud.Updated = updated.Format(time.RFC1123Z)
65 cloud.Format = "products:1.0"
66 cloud.Products = make(map[string]simplestreams.MetadataCatalog)
67 itemsversion := updated.Format("20060201") // YYYYMMDD
68 for _, t := range metadata {
69 id, err := t.productId()
70 if err != nil {
71 return nil, err
72 }
73 version, err := simplestreams.SeriesVersion(t.Release)
74 if err != nil {
75 return nil, err
76 }
77 toWrite := &ImageMetadata{
78 Id: t.Id,
79 }
80 if catalog, ok := cloud.Products[id]; ok {
81 catalog.Items[itemsversion].Items[t.Id] = t
82 } else {
83 catalog = simplestreams.MetadataCatalog{
84 Arch: t.Arch,
85 RegionName: t.RegionName,
86 Version: version,
87 Items: map[string]*simplestreams.ItemCollection{
88 itemsversion: &simplestreams.ItemCollection{
89 Items: map[string]interface{}{t.Id: toWrite},
90 },
91 },
92 }
93 cloud.Products[id] = catalog
94 }
95 }
96 return json.MarshalIndent(&cloud, "", " ")
97}
098
=== added file 'environs/imagemetadata/marshal_test.go'
--- environs/imagemetadata/marshal_test.go 1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/marshal_test.go 2013-10-16 04:12:27 +0000
@@ -0,0 +1,127 @@
1// Copyright 2013 Canonical Ltd.
2// Licensed under the AGPLv3, see LICENCE file for details.
3
4package imagemetadata_test
5
6import (
7 "time"
8
9 gc "launchpad.net/gocheck"
10
11 "launchpad.net/juju-core/environs/imagemetadata"
12 "launchpad.net/juju-core/environs/simplestreams"
13)
14
15var _ = gc.Suite(&marshalSuite{})
16
17type marshalSuite struct{}
18
19var expectedIndex = `{
20 "index": {
21 "com.ubuntu.cloud:custom": {
22 "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
23 "format": "products:1.0",
24 "datatype": "image-ids",
25 "cloudname": "custom",
26 "clouds": [
27 {
28 "region": "region",
29 "endpoint": "endpoint"
30 }
31 ],
32 "path": "streams/v1/com.ubuntu.cloud:released:imagemetadata.json",
33 "products": [
34 "com.ubuntu.cloud:server:12.04:amd64",
35 "com.ubuntu.cloud:server:12.04:arm",
36 "com.ubuntu.cloud:server:13.10:arm"
37 ]
38 }
39 },
40 "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
41 "format": "index:1.0"
42}`
43
44var expectedProducts = `{
45 "products": {
46 "com.ubuntu.cloud:server:12.04:amd64": {
47 "version": "12.04",
48 "arch": "amd64",
49 "versions": {
50 "19700101": {
51 "items": {
52 "abcd": {
53 "id": "abcd"
54 }
55 }
56 }
57 }
58 },
59 "com.ubuntu.cloud:server:12.04:arm": {
60 "version": "12.04",
61 "arch": "arm",
62 "versions": {
63 "19700101": {
64 "items": {
65 "5678": {
66 "id": "5678"
67 }
68 }
69 }
70 }
71 },
72 "com.ubuntu.cloud:server:13.10:arm": {
73 "version": "13.10",
74 "arch": "arm",
75 "versions": {
76 "19700101": {
77 "items": {
78 "1234": {
79 "id": "1234"
80 }
81 }
82 }
83 }
84 }
85 },
86 "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
87 "format": "products:1.0"
88}`
89
90var imageMetadataForTesting = []*imagemetadata.ImageMetadata{
91 &imagemetadata.ImageMetadata{
92 Id: "1234",
93 Release: "saucy",
94 Arch: "arm",
95 },
96 &imagemetadata.ImageMetadata{
97 Id: "5678",
98 Release: "precise",
99 Arch: "arm",
100 },
101 &imagemetadata.ImageMetadata{
102 Id: "abcd",
103 Release: "precise",
104 Arch: "amd64",
105 },
106}
107
108func (s *marshalSuite) TestMarshalIndex(c *gc.C) {
109 cloudSpec := &simplestreams.CloudSpec{Region: "region", Endpoint: "endpoint"}
110 index, err := imagemetadata.MarshalImageMetadataIndexJSON(imageMetadataForTesting, cloudSpec, time.Unix(0, 0).UTC())
111 c.Assert(err, gc.IsNil)
112 c.Assert(string(index), gc.Equals, expectedIndex)
113}
114
115func (s *marshalSuite) TestMarshalProducts(c *gc.C) {
116 products, err := imagemetadata.MarshalImageMetadataProductsJSON(imageMetadataForTesting, time.Unix(0, 0).UTC())
117 c.Assert(err, gc.IsNil)
118 c.Assert(string(products), gc.Equals, expectedProducts)
119}
120
121func (s *marshalSuite) TestMarshal(c *gc.C) {
122 cloudSpec := &simplestreams.CloudSpec{Region: "region", Endpoint: "endpoint"}
123 index, products, err := imagemetadata.MarshalImageMetadataJSON(imageMetadataForTesting, cloudSpec, time.Unix(0, 0).UTC())
124 c.Assert(err, gc.IsNil)
125 c.Assert(string(index), gc.Equals, expectedIndex)
126 c.Assert(string(products), gc.Equals, expectedProducts)
127}
0128
=== modified file 'environs/imagemetadata/simplestreams.go'
--- environs/imagemetadata/simplestreams.go 2013-10-10 11:40:54 +0000
+++ environs/imagemetadata/simplestreams.go 2013-10-16 04:12:27 +0000
@@ -115,12 +115,21 @@
115// ImageMetadata holds information about a particular cloud image.115// ImageMetadata holds information about a particular cloud image.
116type ImageMetadata struct {116type ImageMetadata struct {
117 Id string `json:"id"`117 Id string `json:"id"`
118 Storage string `json:"root_store"`118 Storage string `json:"root_store,omitempty"`
119 VType string `json:"virt"`119 VType string `json:"virt,omitempty"`
120 Arch string `json:"arch"`120 Arch string `json:"arch,omitempty"`
121 RegionAlias string `json:"crsn"`121 Release string `json:"-"`
122 RegionName string `json:"region"`122 RegionAlias string `json:"crsn,omitempty"`
123 Endpoint string `json:"endpoint"`123 RegionName string `json:"region,omitempty"`
124 Endpoint string `json:"endpoint,omitempty"`
125}
126
127func (t *ImageMetadata) productId() (string, error) {
128 seriesVersion, err := simplestreams.SeriesVersion(t.Release)
129 if err != nil {
130 return "", err
131 }
132 return fmt.Sprintf("com.ubuntu.cloud:server:%s:%s", seriesVersion, t.Arch), nil
124}133}
125134
126// Fetch returns a list of images for the specified cloud matching the constraint.135// Fetch returns a list of images for the specified cloud matching the constraint.
127136
=== added directory 'environs/imagemetadata/testing'
=== added file 'environs/imagemetadata/testing/testing.go'
--- environs/imagemetadata/testing/testing.go 1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/testing/testing.go 2013-10-16 04:12:27 +0000
@@ -0,0 +1,70 @@
1// Copyright 2013 Canonical Ltd.
2// Licensed under the AGPLv3, see LICENCE file for details.
3
4package testing
5
6import (
7 "fmt"
8 "io/ioutil"
9 "path/filepath"
10 "sort"
11
12 gc "launchpad.net/gocheck"
13
14 "launchpad.net/juju-core/environs/imagemetadata"
15 "launchpad.net/juju-core/environs/simplestreams"
16 "launchpad.net/juju-core/utils/set"
17)
18
19// ParseMetadata loads ImageMetadata from the specified directory.
20func ParseMetadata(c *gc.C, metadataDir string) []*imagemetadata.ImageMetadata {
21 params := simplestreams.ValueParams{
22 DataType: "image-ids",
23 ValueTemplate: imagemetadata.ImageMetadata{},
24 }
25
26 source := simplestreams.NewURLDataSource("file://"+metadataDir, simplestreams.VerifySSLHostnames)
27
28 const requireSigned = false
29 indexPath := simplestreams.UnsignedIndex
30 indexRef, err := simplestreams.GetIndexWithFormat(
31 source, indexPath, "index:1.0", requireSigned, simplestreams.CloudSpec{}, params)
32 c.Assert(err, gc.IsNil)
33 c.Assert(indexRef.Indexes, gc.HasLen, 1)
34
35 imageIndexMetadata := indexRef.Indexes["com.ubuntu.cloud:custom"]
36 c.Assert(imageIndexMetadata, gc.NotNil)
37
38 data, err := ioutil.ReadFile(filepath.Join(metadataDir, imageIndexMetadata.ProductsFilePath))
39 c.Assert(err, gc.IsNil)
40
41 url, err := source.URL(imageIndexMetadata.ProductsFilePath)
42 c.Assert(err, gc.IsNil)
43 cloudMetadata, err := simplestreams.ParseCloudMetadata(data, "products:1.0", url, imagemetadata.ImageMetadata{})
44 c.Assert(err, gc.IsNil)
45
46 imageMetadataMap := make(map[string]*imagemetadata.ImageMetadata)
47 var expectedProductIds set.Strings
48 var imageVersions set.Strings
49 for _, mc := range cloudMetadata.Products {
50 for _, items := range mc.Items {
51 for key, item := range items.Items {
52 imageMetadata := item.(*imagemetadata.ImageMetadata)
53 imageMetadataMap[key] = imageMetadata
54 imageVersions.Add(key)
55 productId := fmt.Sprintf("com.ubuntu.cloud:server:%s:%s", mc.Version, imageMetadata.Arch)
56 expectedProductIds.Add(productId)
57 }
58 }
59 }
60
61 // Make sure index's product IDs are all represented in the products metadata.
62 sort.Strings(imageIndexMetadata.ProductIds)
63 c.Assert(imageIndexMetadata.ProductIds, gc.DeepEquals, expectedProductIds.SortedValues())
64
65 imageMetadata := make([]*imagemetadata.ImageMetadata, len(imageMetadataMap))
66 for i, key := range imageVersions.SortedValues() {
67 imageMetadata[i] = imageMetadataMap[key]
68 }
69 return imageMetadata
70}
071
=== modified file 'environs/imagemetadata/validation_test.go'
--- environs/imagemetadata/validation_test.go 2013-10-11 04:09:04 +0000
+++ environs/imagemetadata/validation_test.go 2013-10-16 04:12:27 +0000
@@ -6,6 +6,7 @@
6import (6import (
7 gc "launchpad.net/gocheck"7 gc "launchpad.net/gocheck"
88
9 "launchpad.net/juju-core/environs/filestorage"
9 "launchpad.net/juju-core/environs/imagemetadata"10 "launchpad.net/juju-core/environs/imagemetadata"
10 "launchpad.net/juju-core/environs/simplestreams"11 "launchpad.net/juju-core/environs/simplestreams"
11 "launchpad.net/juju-core/testing/testbase"12 "launchpad.net/juju-core/testing/testbase"
@@ -27,7 +28,9 @@
27 Region: region,28 Region: region,
28 Endpoint: endpoint,29 Endpoint: endpoint,
29 }30 }
30 _, err := imagemetadata.GenerateMetadata(series, &im, &cloudSpec, s.metadataDir)31 targetStorage, err := filestorage.NewFileStorageWriter(s.metadataDir, filestorage.UseDefaultTmpDir)
32 c.Assert(err, gc.IsNil)
33 err = imagemetadata.WriteMetadata(series, &im, &cloudSpec, targetStorage)
31 if err != nil {34 if err != nil {
32 return err35 return err
33 }36 }
3437
=== modified file 'environs/simplestreams/simplestreams.go'
--- environs/simplestreams/simplestreams.go 2013-10-10 11:40:54 +0000
+++ environs/simplestreams/simplestreams.go 2013-10-16 04:12:27 +0000
@@ -32,8 +32,8 @@
3232
33// CloudSpec uniquely defines a specific cloud deployment.33// CloudSpec uniquely defines a specific cloud deployment.
34type CloudSpec struct {34type CloudSpec struct {
35 Region string35 Region string `json:"region"`
36 Endpoint string36 Endpoint string `json:"endpoint"`
37}37}
3838
39// HasRegion is implemented by instances which can provide a region to which they belong.39// HasRegion is implemented by instances which can provide a region to which they belong.
4040
=== modified file 'environs/tools/marshal.go'
--- environs/tools/marshal.go 2013-09-02 23:29:30 +0000
+++ environs/tools/marshal.go 2013-10-16 04:12:27 +0000
@@ -66,7 +66,7 @@
66 cloud.Updated = updated.Format(time.RFC1123Z)66 cloud.Updated = updated.Format(time.RFC1123Z)
67 cloud.Format = "products:1.0"67 cloud.Format = "products:1.0"
68 cloud.Products = make(map[string]simplestreams.MetadataCatalog)68 cloud.Products = make(map[string]simplestreams.MetadataCatalog)
69 itemsversion := updated.Format("20060201") // YYYYMMDD69 itemsversion := updated.Format("20060102") // YYYYMMDD
70 for _, t := range metadata {70 for _, t := range metadata {
71 id, err := t.productId()71 id, err := t.productId()
72 if err != nil {72 if err != nil {

Subscribers

People subscribed via source and target branches

to status/vote changes: