Merge lp:~wallyworld/juju-core/marshal-image-metadata into lp:~go-bot/juju-core/trunk

[Ian Booth]

Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://codereview.appspot.com/14540055/

Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://codereview.appspot.com/14540055/

[Ian Booth (wallyworld)]

Reviewers: mp+190542_code.launchpad.net,

Message:
Please take a look.

Description:
Write image metadata using json serialisation

This branch writes outs simplestreams image metadata using
json serialisation instead of a template based approach. It
is a pre-requisit to allow the next step of merging new image
metadata instead of overwriting. This will be used when setting
up private clouds.

https://code.launchpad.net/~wallyworld/juju-core/marshal-image-metadata/+merge/190542

Requires:
https://code.launchpad.net/~wallyworld/juju-core/improve-image-metadata-command/+merge/190517

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/14540055/

Affected files (+398, -135 lines):
   A [revision details]
   M cmd/plugins/juju-metadata/imagemetadata.go
   M environs/imagemetadata/generate.go
   A environs/imagemetadata/generate_test.go
   A environs/imagemetadata/marshal.go
   A environs/imagemetadata/marshal_test.go
   M environs/imagemetadata/simplestreams.go
   A environs/imagemetadata/testing/testing.go
   M environs/imagemetadata/validation_test.go
   M environs/simplestreams/simplestreams.go
   M environs/tools/marshal.go

[Ian Booth (wallyworld)]

Please take a look.

https://codereview.appspot.com/14540055/

[Tim Penhey (thumper)]

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go
File environs/imagemetadata/generate.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go#newcode15
environs/imagemetadata/generate.go:15: metadataInfo, err :=
generateMetadata(series, im, cloudSpec)
This has me queezy.

Surely WriteMetadata would take the metadataInfo as an input parameter?

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go
File environs/imagemetadata/generate_test.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode17
environs/imagemetadata/generate_test.go:17: type generateSuite struct{}
testbase.LoggingSuite

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode34
environs/imagemetadata/generate_test.go:34: c.Assert(len(metadata),
gc.Equals, 1)
gc.HasLen

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go
File environs/imagemetadata/simplestreams.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go#newcode121
environs/imagemetadata/simplestreams.go:121: Release string
`json:"-"`
What does json:"-" do?

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go
File environs/tools/marshal.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go#newcode69
environs/tools/marshal.go:69: itemsversion := updated.Format("20060102")
// YYYYMMDD
Some of these changes are the same as the branch I just reviewed.

https://codereview.appspot.com/14540055/

[Ian Booth (wallyworld)]

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go
File environs/imagemetadata/generate.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate.go#newcode15
environs/imagemetadata/generate.go:15: metadataInfo, err :=
generateMetadata(series, im, cloudSpec)
On 2013/10/16 01:26:11, thumper wrote:
> This has me queezy.

> Surely WriteMetadata would take the metadataInfo as an input
parameter?

The sematics of the write function is to take a ImageMetadata struct and
write out the simplestreams metadata. In order to write the metadata, it
needs to be generated from the struct. The method could be called
WriteAndGenerate I guess. But the generation and writing to file is an
atomic operation.

A subsequent branch renames the method anyway and refactors the content
to align with what Andrew did for tools metadata.

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go
File environs/imagemetadata/generate_test.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode17
environs/imagemetadata/generate_test.go:17: type generateSuite struct{}
On 2013/10/16 01:26:11, thumper wrote:
> testbase.LoggingSuite

Will fix

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/generate_test.go#newcode34
environs/imagemetadata/generate_test.go:34: c.Assert(len(metadata),
gc.Equals, 1)
On 2013/10/16 01:26:11, thumper wrote:
> gc.HasLen

Will fix

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go
File environs/imagemetadata/simplestreams.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/imagemetadata/simplestreams.go#newcode121
environs/imagemetadata/simplestreams.go:121: Release string
`json:"-"`
On 2013/10/16 01:26:11, thumper wrote:
> What does json:"-" do?

Tells json not to marshall that attribute. It's there to provide info
for other mthods in the struct to do their thing, but we don't want it
serialised to json.

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go
File environs/tools/marshal.go (right):

https://codereview.appspot.com/14540055/diff/4001/environs/tools/marshal.go#newcode69
environs/tools/marshal.go:69: itemsversion := updated.Format("20060102")
// YYYYMMDD
On 2013/10/16 01:26:11, thumper wrote:
> Some of these changes are the same as the branch I just reviewed.

I fixed this in a downstream branch - I didn't pump the changes back up
the pipe sorry.

https://codereview.appspot.com/14540055/

[Ian Booth (wallyworld)]

Please take a look.

https://codereview.appspot.com/14540055/

[Tim Penhey (thumper)]

LGTM

https://codereview.appspot.com/14540055/

=== modified file 'cmd/plugins/juju-metadata/imagemetadata.go'
--- cmd/plugins/juju-metadata/imagemetadata.go	2013-10-16 02:56:51 +0000
+++ cmd/plugins/juju-metadata/imagemetadata.go	2013-10-16 04:12:27 +0000
@@ -14,6 +14,7 @@
 	"launchpad.net/juju-core/environs"
 	"launchpad.net/juju-core/environs/config"
 	"launchpad.net/juju-core/environs/configstore"
+	"launchpad.net/juju-core/environs/filestorage"
 	"launchpad.net/juju-core/environs/imagemetadata"
 	"launchpad.net/juju-core/environs/simplestreams"
 )
@@ -152,7 +153,11 @@
 		Region:   c.Region,
 		Endpoint: c.Endpoint,
 	}
-	_, err := imagemetadata.GenerateMetadata(c.Series, &im, &cloudSpec, c.Dir)
+	targetStorage, err := filestorage.NewFileStorageWriter(c.Dir, filestorage.UseDefaultTmpDir)
+	if err != nil {
+		return err
+	}
+	err = imagemetadata.WriteMetadata(c.Series, &im, &cloudSpec, targetStorage)
 	if err != nil {
 		return fmt.Errorf("image metadata files could not be created: %v", err)
 	}

=== modified file 'cmd/plugins/juju-metadata/imagemetadata_test.go'
--- cmd/plugins/juju-metadata/imagemetadata_test.go	2013-10-16 02:56:51 +0000
+++ cmd/plugins/juju-metadata/imagemetadata_test.go	2013-10-16 04:12:27 +0000
@@ -108,7 +108,7 @@
 
 const (
 	defaultIndexFileName = "index.json"
-	defaultImageFileName = "imagemetadata.json"
+	defaultImageFileName = "com.ubuntu.cloud:released:imagemetadata.json"
 )
 
 func (s *ImageMetadataSuite) TestImageMetadataFilesNoEnv(c *gc.C) {

=== modified file 'cmd/plugins/juju-metadata/validateimagemetadata_test.go'
--- cmd/plugins/juju-metadata/validateimagemetadata_test.go	2013-10-16 03:38:32 +0000
+++ cmd/plugins/juju-metadata/validateimagemetadata_test.go	2013-10-16 04:12:27 +0000
@@ -11,6 +11,7 @@
 	gc "launchpad.net/gocheck"
 
 	"launchpad.net/juju-core/cmd"
+	"launchpad.net/juju-core/environs/filestorage"
 	"launchpad.net/juju-core/environs/imagemetadata"
 	"launchpad.net/juju-core/environs/simplestreams"
 	coretesting "launchpad.net/juju-core/testing"
@@ -73,7 +74,11 @@
 		Region:   region,
 		Endpoint: endpoint,
 	}
-	_, err := imagemetadata.GenerateMetadata(series, &im, &cloudSpec, s.metadataDir)
+	targetStorage, err := filestorage.NewFileStorageWriter(s.metadataDir, filestorage.UseDefaultTmpDir)
+	if err != nil {
+		return err
+	}
+	err = imagemetadata.WriteMetadata(series, &im, &cloudSpec, targetStorage)
 	if err != nil {
 		return err
 	}

=== modified file 'environs/imagemetadata/generate.go'
--- environs/imagemetadata/generate.go	2013-10-11 01:20:34 +0000
+++ environs/imagemetadata/generate.go	2013-10-16 04:12:27 +0000
@@ -5,135 +5,48 @@
 
 import (
 	"bytes"
-	"fmt"
-	"io/ioutil"
-	"os"
-	"path/filepath"
-	"text/template"
 	"time"
 
 	"launchpad.net/juju-core/environs/simplestreams"
-)
-
-const (
-	defaultIndexFileName = "index.json"
-	defaultImageFileName = "imagemetadata.json"
-)
-
-// GenerateMetadata generates some basic simplestreams metadata using the specified cloud and image details.
-func GenerateMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec, dest string) ([]string, error) {
-	indexFileName := defaultIndexFileName
-	imageFileName := defaultImageFileName
-	now := time.Now()
-	imparams := imageMetadataParams{
-		Id:            im.Id,
-		Arch:          im.Arch,
-		Region:        cloudSpec.Region,
-		URL:           cloudSpec.Endpoint,
-		Path:          "streams/v1",
-		ImageFileName: imageFileName,
-		Updated:       now.Format(time.RFC1123Z),
-		VersionKey:    now.Format("20060102"),
-	}
-
-	var err error
-	imparams.Version, err = simplestreams.SeriesVersion(series)
-	if err != nil {
-		return nil, fmt.Errorf("invalid series %q", series)
-	}
-
-	streamsPath := filepath.Join(dest, "streams", "v1")
-	if err = os.MkdirAll(streamsPath, 0755); err != nil {
-		return nil, err
-	}
-	indexFileName = filepath.Join(streamsPath, indexFileName)
-	imageFileName = filepath.Join(streamsPath, imageFileName)
-	err = writeJsonFile(imparams, indexFileName, indexBoilerplate)
-	if err != nil {
-		return nil, err
-	}
-	err = writeJsonFile(imparams, imageFileName, productBoilerplate)
-	if err != nil {
-		return nil, err
-	}
-	return []string{indexFileName, imageFileName}, nil
-}
-
-type imageMetadataParams struct {
-	Region        string
-	URL           string
-	Updated       string
-	Arch          string
-	Path          string
-	Series        string
-	Version       string
-	VersionKey    string
-	Id            string
-	ImageFileName string
-}
-
-func writeJsonFile(imparams imageMetadataParams, filename, boilerplate string) error {
-	t := template.Must(template.New("").Parse(boilerplate))
-	var metadata bytes.Buffer
-	if err := t.Execute(&metadata, imparams); err != nil {
-		panic(fmt.Errorf("cannot generate %s metdata: %v", filename, err))
-	}
-	data := metadata.Bytes()
-	if err := ioutil.WriteFile(filename, data, 0666); err != nil {
+	"launchpad.net/juju-core/environs/storage"
+)
+
+func WriteMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec, metadataStore storage.Storage) error {
+	metadataInfo, err := generateMetadata(series, im, cloudSpec)
+	if err != nil {
 		return err
 	}
+	for _, md := range metadataInfo {
+		err = metadataStore.Put(md.Path, bytes.NewReader(md.Data), int64(len(md.Data)))
+		if err != nil {
+			return err
+		}
+	}
 	return nil
 }
 
-var indexBoilerplate = `
-{
- "index": {
-   "com.ubuntu.cloud:custom": {
-     "updated": "{{.Updated}}",
-     "clouds": [
-       {
-         "region": "{{.Region}}",
-         "endpoint": "{{.URL}}"
-       }
-     ],
-     "cloudname": "custom",
-     "datatype": "image-ids",
-     "format": "products:1.0",
-     "products": [
-       "com.ubuntu.cloud:server:{{.Version}}:{{.Arch}}"
-     ],
-     "path": "{{.Path}}/{{.ImageFileName}}"
-   }
- },
- "updated": "{{.Updated}}",
- "format": "index:1.0"
-}
-`
-
-var productBoilerplate = `
-{
-  "content_id": "com.ubuntu.cloud:custom",
-  "format": "products:1.0",
-  "updated": "{{.Updated}}",
-  "datatype": "image-ids",
-  "products": {
-    "com.ubuntu.cloud:server:{{.Version}}:{{.Arch}}": {
-      "release": "{{.Series}}",
-      "version": "{{.Version}}",
-      "arch": "{{.Arch}}",
-      "versions": {
-        "{{.VersionKey}}": {
-          "items": {
-            "{{.Id}}": {
-              "region": "{{.Region}}",
-              "id": "{{.Id}}"
-            }
-          },
-          "pubname": "ubuntu-{{.Series}}-{{.Version}}-{{.Arch}}-server-{{.VersionKey}}",
-          "label": "custom"
-        }
-      }
-    }
-  }
-}
-`
+type MetadataFile struct {
+	Path string
+	Data []byte
+}
+
+// generateMetadata generates some basic simplestreams metadata using the specified cloud and image details.
+func generateMetadata(series string, im *ImageMetadata, cloudSpec *simplestreams.CloudSpec) ([]MetadataFile, error) {
+	metadata := &ImageMetadata{
+		Id:         im.Id,
+		Arch:       im.Arch,
+		Release:    series,
+		RegionName: cloudSpec.Region,
+		Endpoint:   cloudSpec.Endpoint,
+	}
+
+	index, products, err := MarshalImageMetadataJSON([]*ImageMetadata{metadata}, cloudSpec, time.Now())
+	if err != nil {
+		return nil, err
+	}
+	objects := []MetadataFile{
+		{simplestreams.UnsignedIndex, index},
+		{ProductMetadataPath, products},
+	}
+	return objects, nil
+}

=== added file 'environs/imagemetadata/generate_test.go'
--- environs/imagemetadata/generate_test.go	1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/generate_test.go	2013-10-16 04:12:27 +0000
@@ -0,0 +1,40 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package imagemetadata_test
+
+import (
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/environs/filestorage"
+	"launchpad.net/juju-core/environs/imagemetadata"
+	"launchpad.net/juju-core/environs/imagemetadata/testing"
+	"launchpad.net/juju-core/environs/simplestreams"
+	"launchpad.net/juju-core/testing/testbase"
+)
+
+var _ = gc.Suite(&generateSuite{})
+
+type generateSuite struct {
+	testbase.LoggingSuite
+}
+
+func (s *generateSuite) TestWriteMetadata(c *gc.C) {
+	im := &imagemetadata.ImageMetadata{
+		Id:   "1234",
+		Arch: "amd64",
+	}
+	cloudSpec := &simplestreams.CloudSpec{
+		Region:   "region",
+		Endpoint: "endpoint",
+	}
+	dir := c.MkDir()
+	targetStorage, err := filestorage.NewFileStorageWriter(dir, filestorage.UseDefaultTmpDir)
+	c.Assert(err, gc.IsNil)
+	err = imagemetadata.WriteMetadata("raring", im, cloudSpec, targetStorage)
+	c.Assert(err, gc.IsNil)
+	metadata := testing.ParseMetadata(c, dir)
+	c.Assert(metadata, gc.HasLen, 1)
+	im.RegionName = cloudSpec.Region
+	c.Assert(im, gc.DeepEquals, metadata[0])
+}

=== added file 'environs/imagemetadata/marshal.go'
--- environs/imagemetadata/marshal.go	1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/marshal.go	2013-10-16 04:12:27 +0000
@@ -0,0 +1,97 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package imagemetadata
+
+import (
+	"encoding/json"
+	"time"
+
+	"launchpad.net/juju-core/environs/simplestreams"
+	"launchpad.net/juju-core/utils/set"
+)
+
+const (
+	ProductMetadataPath = "streams/v1/com.ubuntu.cloud:released:imagemetadata.json"
+)
+
+// MarshalImageMetadataJSON marshals image metadata to index and products JSON.
+//
+// updated is the time at which the JSON file was updated.
+func MarshalImageMetadataJSON(metadata []*ImageMetadata, cloudSpec *simplestreams.CloudSpec, updated time.Time) (index, products []byte, err error) {
+	if index, err = MarshalImageMetadataIndexJSON(metadata, cloudSpec, updated); err != nil {
+		return nil, nil, err
+	}
+	if products, err = MarshalImageMetadataProductsJSON(metadata, updated); err != nil {
+		return nil, nil, err
+	}
+	return index, products, err
+}
+
+// MarshalImageMetadataIndexJSON marshals image metadata to index JSON.
+//
+// updated is the time at which the JSON file was updated.
+func MarshalImageMetadataIndexJSON(metadata []*ImageMetadata, cloudSpec *simplestreams.CloudSpec, updated time.Time) (out []byte, err error) {
+	productIds := make([]string, len(metadata))
+	for i, t := range metadata {
+		productIds[i], err = t.productId()
+		if err != nil {
+			return nil, err
+		}
+	}
+	var indices simplestreams.Indices
+	indices.Updated = updated.Format(time.RFC1123Z)
+	indices.Format = "index:1.0"
+	indices.Indexes = map[string]*simplestreams.IndexMetadata{
+		"com.ubuntu.cloud:custom": &simplestreams.IndexMetadata{
+			CloudName:        "custom",
+			Updated:          indices.Updated,
+			Format:           "products:1.0",
+			DataType:         "image-ids",
+			ProductsFilePath: ProductMetadataPath,
+			ProductIds:       set.NewStrings(productIds...).SortedValues(),
+			Clouds:           []simplestreams.CloudSpec{*cloudSpec},
+		},
+	}
+	return json.MarshalIndent(&indices, "", "    ")
+}
+
+// MarshalImageMetadataProductsJSON marshals image metadata to products JSON.
+//
+// updated is the time at which the JSON file was updated.
+func MarshalImageMetadataProductsJSON(metadata []*ImageMetadata, updated time.Time) (out []byte, err error) {
+	var cloud simplestreams.CloudMetadata
+	cloud.Updated = updated.Format(time.RFC1123Z)
+	cloud.Format = "products:1.0"
+	cloud.Products = make(map[string]simplestreams.MetadataCatalog)
+	itemsversion := updated.Format("20060201") // YYYYMMDD
+	for _, t := range metadata {
+		id, err := t.productId()
+		if err != nil {
+			return nil, err
+		}
+		version, err := simplestreams.SeriesVersion(t.Release)
+		if err != nil {
+			return nil, err
+		}
+		toWrite := &ImageMetadata{
+			Id: t.Id,
+		}
+		if catalog, ok := cloud.Products[id]; ok {
+			catalog.Items[itemsversion].Items[t.Id] = t
+		} else {
+			catalog = simplestreams.MetadataCatalog{
+				Arch:       t.Arch,
+				RegionName: t.RegionName,
+				Version:    version,
+				Items: map[string]*simplestreams.ItemCollection{
+					itemsversion: &simplestreams.ItemCollection{
+						Items: map[string]interface{}{t.Id: toWrite},
+					},
+				},
+			}
+			cloud.Products[id] = catalog
+		}
+	}
+	return json.MarshalIndent(&cloud, "", "    ")
+}

=== added file 'environs/imagemetadata/marshal_test.go'
--- environs/imagemetadata/marshal_test.go	1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/marshal_test.go	2013-10-16 04:12:27 +0000
@@ -0,0 +1,127 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package imagemetadata_test
+
+import (
+	"time"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/environs/imagemetadata"
+	"launchpad.net/juju-core/environs/simplestreams"
+)
+
+var _ = gc.Suite(&marshalSuite{})
+
+type marshalSuite struct{}
+
+var expectedIndex = `{
+    "index": {
+        "com.ubuntu.cloud:custom": {
+            "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
+            "format": "products:1.0",
+            "datatype": "image-ids",
+            "cloudname": "custom",
+            "clouds": [
+                {
+                    "region": "region",
+                    "endpoint": "endpoint"
+                }
+            ],
+            "path": "streams/v1/com.ubuntu.cloud:released:imagemetadata.json",
+            "products": [
+                "com.ubuntu.cloud:server:12.04:amd64",
+                "com.ubuntu.cloud:server:12.04:arm",
+                "com.ubuntu.cloud:server:13.10:arm"
+            ]
+        }
+    },
+    "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
+    "format": "index:1.0"
+}`
+
+var expectedProducts = `{
+    "products": {
+        "com.ubuntu.cloud:server:12.04:amd64": {
+            "version": "12.04",
+            "arch": "amd64",
+            "versions": {
+                "19700101": {
+                    "items": {
+                        "abcd": {
+                            "id": "abcd"
+                        }
+                    }
+                }
+            }
+        },
+        "com.ubuntu.cloud:server:12.04:arm": {
+            "version": "12.04",
+            "arch": "arm",
+            "versions": {
+                "19700101": {
+                    "items": {
+                        "5678": {
+                            "id": "5678"
+                        }
+                    }
+                }
+            }
+        },
+        "com.ubuntu.cloud:server:13.10:arm": {
+            "version": "13.10",
+            "arch": "arm",
+            "versions": {
+                "19700101": {
+                    "items": {
+                        "1234": {
+                            "id": "1234"
+                        }
+                    }
+                }
+            }
+        }
+    },
+    "updated": "Thu, 01 Jan 1970 00:00:00 +0000",
+    "format": "products:1.0"
+}`
+
+var imageMetadataForTesting = []*imagemetadata.ImageMetadata{
+	&imagemetadata.ImageMetadata{
+		Id:      "1234",
+		Release: "saucy",
+		Arch:    "arm",
+	},
+	&imagemetadata.ImageMetadata{
+		Id:      "5678",
+		Release: "precise",
+		Arch:    "arm",
+	},
+	&imagemetadata.ImageMetadata{
+		Id:      "abcd",
+		Release: "precise",
+		Arch:    "amd64",
+	},
+}
+
+func (s *marshalSuite) TestMarshalIndex(c *gc.C) {
+	cloudSpec := &simplestreams.CloudSpec{Region: "region", Endpoint: "endpoint"}
+	index, err := imagemetadata.MarshalImageMetadataIndexJSON(imageMetadataForTesting, cloudSpec, time.Unix(0, 0).UTC())
+	c.Assert(err, gc.IsNil)
+	c.Assert(string(index), gc.Equals, expectedIndex)
+}
+
+func (s *marshalSuite) TestMarshalProducts(c *gc.C) {
+	products, err := imagemetadata.MarshalImageMetadataProductsJSON(imageMetadataForTesting, time.Unix(0, 0).UTC())
+	c.Assert(err, gc.IsNil)
+	c.Assert(string(products), gc.Equals, expectedProducts)
+}
+
+func (s *marshalSuite) TestMarshal(c *gc.C) {
+	cloudSpec := &simplestreams.CloudSpec{Region: "region", Endpoint: "endpoint"}
+	index, products, err := imagemetadata.MarshalImageMetadataJSON(imageMetadataForTesting, cloudSpec, time.Unix(0, 0).UTC())
+	c.Assert(err, gc.IsNil)
+	c.Assert(string(index), gc.Equals, expectedIndex)
+	c.Assert(string(products), gc.Equals, expectedProducts)
+}

=== modified file 'environs/imagemetadata/simplestreams.go'
--- environs/imagemetadata/simplestreams.go	2013-10-10 11:40:54 +0000
+++ environs/imagemetadata/simplestreams.go	2013-10-16 04:12:27 +0000
@@ -115,12 +115,21 @@
 // ImageMetadata holds information about a particular cloud image.
 type ImageMetadata struct {
 	Id          string `json:"id"`
-	Storage     string `json:"root_store"`
-	VType       string `json:"virt"`
-	Arch        string `json:"arch"`
-	RegionAlias string `json:"crsn"`
-	RegionName  string `json:"region"`
-	Endpoint    string `json:"endpoint"`
+	Storage     string `json:"root_store,omitempty"`
+	VType       string `json:"virt,omitempty"`
+	Arch        string `json:"arch,omitempty"`
+	Release     string `json:"-"`
+	RegionAlias string `json:"crsn,omitempty"`
+	RegionName  string `json:"region,omitempty"`
+	Endpoint    string `json:"endpoint,omitempty"`
+}
+
+func (t *ImageMetadata) productId() (string, error) {
+	seriesVersion, err := simplestreams.SeriesVersion(t.Release)
+	if err != nil {
+		return "", err
+	}
+	return fmt.Sprintf("com.ubuntu.cloud:server:%s:%s", seriesVersion, t.Arch), nil
 }
 
 // Fetch returns a list of images for the specified cloud matching the constraint.

=== added directory 'environs/imagemetadata/testing'
=== added file 'environs/imagemetadata/testing/testing.go'
--- environs/imagemetadata/testing/testing.go	1970-01-01 00:00:00 +0000
+++ environs/imagemetadata/testing/testing.go	2013-10-16 04:12:27 +0000
@@ -0,0 +1,70 @@
+// Copyright 2013 Canonical Ltd.
+// Licensed under the AGPLv3, see LICENCE file for details.
+
+package testing
+
+import (
+	"fmt"
+	"io/ioutil"
+	"path/filepath"
+	"sort"
+
+	gc "launchpad.net/gocheck"
+
+	"launchpad.net/juju-core/environs/imagemetadata"
+	"launchpad.net/juju-core/environs/simplestreams"
+	"launchpad.net/juju-core/utils/set"
+)
+
+// ParseMetadata loads ImageMetadata from the specified directory.
+func ParseMetadata(c *gc.C, metadataDir string) []*imagemetadata.ImageMetadata {
+	params := simplestreams.ValueParams{
+		DataType:      "image-ids",
+		ValueTemplate: imagemetadata.ImageMetadata{},
+	}
+
+	source := simplestreams.NewURLDataSource("file://"+metadataDir, simplestreams.VerifySSLHostnames)
+
+	const requireSigned = false
+	indexPath := simplestreams.UnsignedIndex
+	indexRef, err := simplestreams.GetIndexWithFormat(
+		source, indexPath, "index:1.0", requireSigned, simplestreams.CloudSpec{}, params)
+	c.Assert(err, gc.IsNil)
+	c.Assert(indexRef.Indexes, gc.HasLen, 1)
+
+	imageIndexMetadata := indexRef.Indexes["com.ubuntu.cloud:custom"]
+	c.Assert(imageIndexMetadata, gc.NotNil)
+
+	data, err := ioutil.ReadFile(filepath.Join(metadataDir, imageIndexMetadata.ProductsFilePath))
+	c.Assert(err, gc.IsNil)
+
+	url, err := source.URL(imageIndexMetadata.ProductsFilePath)
+	c.Assert(err, gc.IsNil)
+	cloudMetadata, err := simplestreams.ParseCloudMetadata(data, "products:1.0", url, imagemetadata.ImageMetadata{})
+	c.Assert(err, gc.IsNil)
+
+	imageMetadataMap := make(map[string]*imagemetadata.ImageMetadata)
+	var expectedProductIds set.Strings
+	var imageVersions set.Strings
+	for _, mc := range cloudMetadata.Products {
+		for _, items := range mc.Items {
+			for key, item := range items.Items {
+				imageMetadata := item.(*imagemetadata.ImageMetadata)
+				imageMetadataMap[key] = imageMetadata
+				imageVersions.Add(key)
+				productId := fmt.Sprintf("com.ubuntu.cloud:server:%s:%s", mc.Version, imageMetadata.Arch)
+				expectedProductIds.Add(productId)
+			}
+		}
+	}
+
+	// Make sure index's product IDs are all represented in the products metadata.
+	sort.Strings(imageIndexMetadata.ProductIds)
+	c.Assert(imageIndexMetadata.ProductIds, gc.DeepEquals, expectedProductIds.SortedValues())
+
+	imageMetadata := make([]*imagemetadata.ImageMetadata, len(imageMetadataMap))
+	for i, key := range imageVersions.SortedValues() {
+		imageMetadata[i] = imageMetadataMap[key]
+	}
+	return imageMetadata
+}

=== modified file 'environs/imagemetadata/validation_test.go'
--- environs/imagemetadata/validation_test.go	2013-10-11 04:09:04 +0000
+++ environs/imagemetadata/validation_test.go	2013-10-16 04:12:27 +0000
@@ -6,6 +6,7 @@
 import (
 	gc "launchpad.net/gocheck"
 
+	"launchpad.net/juju-core/environs/filestorage"
 	"launchpad.net/juju-core/environs/imagemetadata"
 	"launchpad.net/juju-core/environs/simplestreams"
 	"launchpad.net/juju-core/testing/testbase"
@@ -27,7 +28,9 @@
 		Region:   region,
 		Endpoint: endpoint,
 	}
-	_, err := imagemetadata.GenerateMetadata(series, &im, &cloudSpec, s.metadataDir)
+	targetStorage, err := filestorage.NewFileStorageWriter(s.metadataDir, filestorage.UseDefaultTmpDir)
+	c.Assert(err, gc.IsNil)
+	err = imagemetadata.WriteMetadata(series, &im, &cloudSpec, targetStorage)
 	if err != nil {
 		return err
 	}

=== modified file 'environs/simplestreams/simplestreams.go'
--- environs/simplestreams/simplestreams.go	2013-10-10 11:40:54 +0000
+++ environs/simplestreams/simplestreams.go	2013-10-16 04:12:27 +0000
@@ -32,8 +32,8 @@
 
 // CloudSpec uniquely defines a specific cloud deployment.
 type CloudSpec struct {
-	Region   string
-	Endpoint string
+	Region   string `json:"region"`
+	Endpoint string `json:"endpoint"`
 }
 
 // HasRegion is implemented by instances which can provide a region to which they belong.

=== modified file 'environs/tools/marshal.go'
--- environs/tools/marshal.go	2013-09-02 23:29:30 +0000
+++ environs/tools/marshal.go	2013-10-16 04:12:27 +0000
@@ -66,7 +66,7 @@
 	cloud.Updated = updated.Format(time.RFC1123Z)
 	cloud.Format = "products:1.0"
 	cloud.Products = make(map[string]simplestreams.MetadataCatalog)
-	itemsversion := updated.Format("20060201") // YYYYMMDD
+	itemsversion := updated.Format("20060102") // YYYYMMDD
 	for _, t := range metadata {
 		id, err := t.productId()
 		if err != nil {