Merge lp:~gz/juju-core/1.16_ssl_verification_bootstrap_state_1268913 into lp:juju-core/1.16

Proposed by Martin Packman
Status: Merged
Approved by: Martin Packman
Approved revision: no longer in the source branch.
Merged at revision: 2002
Proposed branch: lp:~gz/juju-core/1.16_ssl_verification_bootstrap_state_1268913
Merge into: lp:juju-core/1.16
Diff against target: 183 lines (+63/-16)
5 files modified
cmd/jujud/bootstrap.go (+1/-1)
environs/httpstorage/storage.go (+4/-1)
provider/common/bootstrap_test.go (+1/-1)
provider/common/state.go (+11/-2)
provider/common/state_test.go (+46/-11)
To merge this branch: bzr merge lp:~gz/juju-core/1.16_ssl_verification_bootstrap_state_1268913
Reviewer Review Type Date Requested Status
Juju Engineering Pending
Review via email: mp+203037@code.launchpad.net

Commit message

Obey ssl-hostname-verification for provider-state

When reading the provider-state file from cloud storage on
bootstrap, skip validating the https cert if the config
ssl-hostname-verification is set to false.

https://codereview.appspot.com/56560043/

R=axwalk

Description of the change

Obey ssl-hostname-verification for provider-state

When reading the provider-state file from cloud storage on
bootstrap, skip validating the https cert if the config
ssl-hostname-verification is set to false.

Change simplified to not test the valid cert case, avoiding
the need for a usable testing https server.

https://codereview.appspot.com/56560043/

To post a comment you must log in.
Revision history for this message
Martin Packman (gz) wrote :

Reviewers: mp+203037_code.launchpad.net,

Message:
Please take a look.

Description:
Obey ssl-hostname-verification for provider-state

When reading the provider-state file from cloud storage on
bootstrap, skip validating the https cert if the config
ssl-hostname-verification is set to false.

In order to reuse the existing testing infrastructure for
localStorage, I had to add a new PromotableStorage
interface which does the (overly magic) lookup of the
https endpoint via http head and uses it for *all*
requests, not just the modification ones.

On trunk, I don't see why ClientTLS shouldn't be switched
to just use https all the time.

https://code.launchpad.net/~gz/juju-core/1.16_ssl_verification_bootstrap_state_1268913/+merge/203037

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/56560043/

Affected files (+169, -53 lines):
   A [revision details]
   M cmd/jujud/bootstrap.go
   M environs/httpstorage/backend_test.go
   M environs/httpstorage/storage.go
   M environs/testing/storage.go
   M provider/common/bootstrap_test.go
   M provider/common/state.go
   M provider/common/state_test.go

Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2014/01/27 18:11:55, gz wrote:
> Please take a look.

"On trunk, I don't see why ClientTLS shouldn't be switched
to just use https all the time."

I suppose we could just use HTTPS, but disable host key verification.
The certificate isn't known to clients other than management nodes/CLI.

https://codereview.appspot.com/56560043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

https://codereview.appspot.com/56560043/diff/20001/environs/httpstorage/storage.go
File environs/httpstorage/storage.go (right):

https://codereview.appspot.com/56560043/diff/20001/environs/httpstorage/storage.go#newcode26
environs/httpstorage/storage.go:26: type PromotableStorage interface {
I'm not keen on this. I think a better way to test, rather than further
exposing the guts of this package, would be to use a
net/http/httputil.ReverseProxy.

https://codereview.appspot.com/56560043/diff/20001/provider/common/state.go
File provider/common/state.go (right):

https://codereview.appspot.com/56560043/diff/20001/provider/common/state.go#newcode71
provider/common/state.go:71: func LoadStateFromURL(url string,
disableSSLHostnameVerification bool) (*BootstrapState, error) {
This is fine for now, but I wonder if we shouldn't have a method of
obtaining an http.Client for an environment. We now have HTTP proxy
settings and SSL hostname verification as options. We could centralise
all that and pass an http.Client in here.

https://codereview.appspot.com/56560043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2014/01/29 07:53:19, axw wrote:
> On 2014/01/27 18:11:55, gz wrote:
> > Please take a look.

> "On trunk, I don't see why ClientTLS shouldn't be switched
> to just use https all the time."

> I suppose we could just use HTTPS, but disable host key verification.
The
> certificate isn't known to clients other than management nodes/CLI.

William pointed out to me that actually, yes, we do have the (public) CA
cert. So we could indeed use HTTPS all the time. It would require us to
somehow get the cert into cloud-init so wget can use it when fetching
tools, and to use it when fetching from storage generally.

https://codereview.appspot.com/56560043/

Revision history for this message
Martin Packman (gz) wrote :

Please take a look.

https://codereview.appspot.com/56560043/diff/20001/environs/httpstorage/storage.go
File environs/httpstorage/storage.go (right):

https://codereview.appspot.com/56560043/diff/20001/environs/httpstorage/storage.go#newcode26
environs/httpstorage/storage.go:26: type PromotableStorage interface {
On 2014/01/29 08:12:59, axw wrote:
> I'm not keen on this. I think a better way to test, rather than
further exposing
> the guts of this package, would be to use a
net/http/httputil.ReverseProxy.

Okay, I'll revert most of this and write something independant.

https://codereview.appspot.com/56560043/diff/20001/provider/common/state.go
File provider/common/state.go (right):

https://codereview.appspot.com/56560043/diff/20001/provider/common/state.go#newcode71
provider/common/state.go:71: func LoadStateFromURL(url string,
disableSSLHostnameVerification bool) (*BootstrapState, error) {
On 2014/01/29 08:12:59, axw wrote:
> This is fine for now, but I wonder if we shouldn't have a method of
obtaining an
> http.Client for an environment. We now have HTTP proxy settings and
SSL hostname
> verification as options. We could centralise all that and pass an
http.Client in
> here.

Yes, on trunk there should be some refactoring like that. There are a
couple of places that want a http.Client obeying thig setting and we may
want more.

https://codereview.appspot.com/56560043/

Revision history for this message
Martin Packman (gz) wrote :
Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2014/01/30 13:57:41, gz wrote:
> Please take a look.

Thanks, much nicer I think. LGTM.

https://codereview.appspot.com/56560043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

Oops, forgot to publish this.

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go
File provider/common/state_test.go (right):

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go#newcode46
provider/common/state_test.go:46: // testingHTTPServer creates a tempdir
backed https server with invalid certs
s/invalid/self-signed/?

https://codereview.appspot.com/56560043/

Revision history for this message
Martin Packman (gz) wrote :

Please take a look.

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go
File provider/common/state_test.go (right):

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go#newcode46
provider/common/state_test.go:46: // testingHTTPServer creates a tempdir
backed https server with invalid certs
On 2014/01/31 01:57:27, axw wrote:
> s/invalid/self-signed/?

Good call. And really, it's just without certs right?

https://codereview.appspot.com/56560043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2014/01/31 12:33:34, gz wrote:
> Please take a look.

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go
> File provider/common/state_test.go (right):

https://codereview.appspot.com/56560043/diff/60001/provider/common/state_test.go#newcode46
> provider/common/state_test.go:46: // testingHTTPServer creates a
tempdir backed
> https server with invalid certs
> On 2014/01/31 01:57:27, axw wrote:
> > s/invalid/self-signed/?

> Good call. And really, it's just without certs right?

No, there's a certificate (there has to be), it's just self-signed. The
SSL certificate verification fails because the CA is unknown.

https://codereview.appspot.com/56560043/

Revision history for this message
Martin Packman (gz) wrote :

On 2014/01/31 12:52:59, axw wrote:

> No, there's a certificate (there has to be), it's just self-signed.
The SSL
> certificate verification fails because the CA is unknown.

Hm, but we're not supplying it, so presumably the httptest server stuff
has its own one?

https://codereview.appspot.com/56560043/

Revision history for this message
Andrew Wilkins (axwalk) wrote :

On 2014/02/01 14:54:54, gz wrote:
> On 2014/01/31 12:52:59, axw wrote:
> >
> > No, there's a certificate (there has to be), it's just self-signed.
The SSL
> > certificate verification fails because the CA is unknown.

> Hm, but we're not supplying it, so presumably the httptest server
stuff has its
> own one?

Yes:
http://golang.org/src/pkg/net/http/httptest/server.go?s=3153:3180#L107

https://codereview.appspot.com/56560043/

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'cmd/jujud/bootstrap.go'
--- cmd/jujud/bootstrap.go 2013-09-30 19:40:06 +0000
+++ cmd/jujud/bootstrap.go 2014-01-31 12:32:29 +0000
@@ -96,7 +96,7 @@
96 return fmt.Errorf("cannot read provider-state-url file: %v", err)96 return fmt.Errorf("cannot read provider-state-url file: %v", err)
97 }97 }
98 stateInfoURL := strings.Split(string(data), "\n")[0]98 stateInfoURL := strings.Split(string(data), "\n")[0]
99 bsState, err := common.LoadStateFromURL(stateInfoURL)99 bsState, err := common.LoadStateFromURL(stateInfoURL, !cfg.SSLHostnameVerification())
100 if err != nil {100 if err != nil {
101 return fmt.Errorf("cannot load state from URL %q (read from %q): %v", stateInfoURL, providerStateURLFile, err)101 return fmt.Errorf("cannot load state from URL %q (read from %q): %v", stateInfoURL, providerStateURLFile, err)
102 }102 }
103103
=== modified file 'environs/httpstorage/storage.go'
--- environs/httpstorage/storage.go 2013-10-02 05:46:56 +0000
+++ environs/httpstorage/storage.go 2014-01-31 12:32:29 +0000
@@ -64,12 +64,15 @@
64}64}
6565
66func (s *localStorage) getHTTPSBaseURL() (string, error) {66func (s *localStorage) getHTTPSBaseURL() (string, error) {
67 url, _ := s.URL("/") // never fails67 url, _ := s.URL("") // never fails
68 resp, err := s.client.Head(url)68 resp, err := s.client.Head(url)
69 if err != nil {69 if err != nil {
70 return "", err70 return "", err
71 }71 }
72 resp.Body.Close()72 resp.Body.Close()
73 if resp.StatusCode != http.StatusOK {
74 return "", fmt.Errorf("Could not access file storage: %v %s", url, resp.Status)
75 }
73 httpsURL, err := resp.Location()76 httpsURL, err := resp.Location()
74 if err != nil {77 if err != nil {
75 return "", err78 return "", err
7679
=== modified file 'provider/common/bootstrap_test.go'
--- provider/common/bootstrap_test.go 2013-10-02 00:29:29 +0000
+++ provider/common/bootstrap_test.go 2014-01-31 12:32:29 +0000
@@ -151,7 +151,7 @@
151 err := common.Bootstrap(env, constraints.Value{}, nil)151 err := common.Bootstrap(env, constraints.Value{}, nil)
152 c.Assert(err, gc.IsNil)152 c.Assert(err, gc.IsNil)
153153
154 savedState, err := common.LoadStateFromURL(checkURL)154 savedState, err := common.LoadStateFromURL(checkURL, false)
155 c.Assert(err, gc.IsNil)155 c.Assert(err, gc.IsNil)
156 c.Assert(savedState, gc.DeepEquals, &common.BootstrapState{156 c.Assert(savedState, gc.DeepEquals, &common.BootstrapState{
157 StateInstances: []instance.Id{instance.Id(checkInstanceId)},157 StateInstances: []instance.Id{instance.Id(checkInstanceId)},
158158
=== modified file 'provider/common/state.go'
--- provider/common/state.go 2013-10-01 12:03:33 +0000
+++ provider/common/state.go 2014-01-31 12:32:29 +0000
@@ -21,6 +21,7 @@
21 "launchpad.net/juju-core/log"21 "launchpad.net/juju-core/log"
22 "launchpad.net/juju-core/state"22 "launchpad.net/juju-core/state"
23 "launchpad.net/juju-core/state/api"23 "launchpad.net/juju-core/state/api"
24 "launchpad.net/juju-core/utils"
24)25)
2526
26// StateFile is the name of the file where the provider's state is stored.27// StateFile is the name of the file where the provider's state is stored.
@@ -67,11 +68,19 @@
67}68}
6869
69// LoadStateFromURL reads state from the given URL.70// LoadStateFromURL reads state from the given URL.
70func LoadStateFromURL(url string) (*BootstrapState, error) {71func LoadStateFromURL(url string, disableSSLHostnameVerification bool) (*BootstrapState, error) {
71 resp, err := http.Get(url)72 client := http.DefaultClient
73 if disableSSLHostnameVerification {
74 logger.Infof("hostname SSL verification disabled")
75 client = utils.GetNonValidatingHTTPClient()
76 }
77 resp, err := client.Get(url)
72 if err != nil {78 if err != nil {
73 return nil, err79 return nil, err
74 }80 }
81 if resp.StatusCode != http.StatusOK {
82 return nil, fmt.Errorf("could not load state from url: %v %s", url, resp.Status)
83 }
75 return loadState(resp.Body)84 return loadState(resp.Body)
76}85}
7786
7887
=== modified file 'provider/common/state_test.go'
--- provider/common/state_test.go 2013-10-02 10:39:12 +0000
+++ provider/common/state_test.go 2014-01-31 12:32:29 +0000
@@ -4,8 +4,10 @@
4package common_test4package common_test
55
6import (6import (
7 "bytes"
8 "io/ioutil"7 "io/ioutil"
8 "net/http"
9 "net/http/httptest"
10 "path/filepath"
911
10 gc "launchpad.net/gocheck"12 gc "launchpad.net/gocheck"
11 "launchpad.net/goyaml"13 "launchpad.net/goyaml"
@@ -30,10 +32,26 @@
30 AddCleanup(testbase.CleanupFunc)32 AddCleanup(testbase.CleanupFunc)
31}33}
3234
33func newStorage(suite cleaner, c *gc.C) storage.Storage {35func newStorageWithDataDir(suite cleaner, c *gc.C) (stor storage.Storage, dataDir string) {
34 closer, stor, _ := envtesting.CreateLocalTestStorage(c)36 closer, stor, dataDir := envtesting.CreateLocalTestStorage(c)
35 suite.AddCleanup(func(*gc.C) { closer.Close() })37 suite.AddCleanup(func(*gc.C) { closer.Close() })
36 return stor38 return
39}
40
41func newStorage(suite cleaner, c *gc.C) (stor storage.Storage) {
42 stor, _ = newStorageWithDataDir(suite, c)
43 return
44}
45
46// testingHTTPServer creates a tempdir backed https server without certs
47func testingHTTPSServer(suite cleaner, c *gc.C) (baseURL string, dataDir string) {
48 dataDir = c.MkDir()
49 mux := http.NewServeMux()
50 mux.Handle("/", http.FileServer(http.Dir(dataDir)))
51 server := httptest.NewTLSServer(mux)
52 suite.AddCleanup(func(*gc.C) { server.Close() })
53 baseURL = server.URL
54 return
37}55}
3856
39func (suite *StateSuite) TestCreateStateFileWritesEmptyStateFile(c *gc.C) {57func (suite *StateSuite) TestCreateStateFileWritesEmptyStateFile(c *gc.C) {
@@ -72,32 +90,49 @@
72 c.Check(content, gc.DeepEquals, marshaledState)90 c.Check(content, gc.DeepEquals, marshaledState)
73}91}
7492
75func (suite *StateSuite) setUpSavedState(c *gc.C, stor storage.Storage) common.BootstrapState {93func (suite *StateSuite) setUpSavedState(c *gc.C, dataDir string) common.BootstrapState {
76 arch := "amd64"94 arch := "amd64"
77 state := common.BootstrapState{95 state := common.BootstrapState{
78 StateInstances: []instance.Id{instance.Id("an-instance-id")},96 StateInstances: []instance.Id{instance.Id("an-instance-id")},
79 Characteristics: []instance.HardwareCharacteristics{{Arch: &arch}}}97 Characteristics: []instance.HardwareCharacteristics{{Arch: &arch}}}
80 content, err := goyaml.Marshal(state)98 content, err := goyaml.Marshal(state)
81 c.Assert(err, gc.IsNil)99 c.Assert(err, gc.IsNil)
82 err = stor.Put(common.StateFile, ioutil.NopCloser(bytes.NewReader(content)), int64(len(content)))100 err = ioutil.WriteFile(filepath.Join(dataDir, common.StateFile), []byte(content), 0644)
83 c.Assert(err, gc.IsNil)101 c.Assert(err, gc.IsNil)
84 return state102 return state
85}103}
86104
87func (suite *StateSuite) TestLoadStateReadsStateFile(c *gc.C) {105func (suite *StateSuite) TestLoadStateReadsStateFile(c *gc.C) {
88 storage := newStorage(suite, c)106 storage, dataDir := newStorageWithDataDir(suite, c)
89 state := suite.setUpSavedState(c, storage)107 state := suite.setUpSavedState(c, dataDir)
90 storedState, err := common.LoadState(storage)108 storedState, err := common.LoadState(storage)
91 c.Assert(err, gc.IsNil)109 c.Assert(err, gc.IsNil)
92 c.Check(*storedState, gc.DeepEquals, state)110 c.Check(*storedState, gc.DeepEquals, state)
93}111}
94112
95func (suite *StateSuite) TestLoadStateFromURLReadsStateFile(c *gc.C) {113func (suite *StateSuite) TestLoadStateFromURLReadsStateFile(c *gc.C) {
96 stor := newStorage(suite, c)114 stor, dataDir := newStorageWithDataDir(suite, c)
97 state := suite.setUpSavedState(c, stor)115 state := suite.setUpSavedState(c, dataDir)
98 url, err := stor.URL(common.StateFile)116 url, err := stor.URL(common.StateFile)
99 c.Assert(err, gc.IsNil)117 c.Assert(err, gc.IsNil)
100 storedState, err := common.LoadStateFromURL(url)118 storedState, err := common.LoadStateFromURL(url, false)
119 c.Assert(err, gc.IsNil)
120 c.Check(*storedState, gc.DeepEquals, state)
121}
122
123func (suite *StateSuite) TestLoadStateFromURLBadCert(c *gc.C) {
124 baseURL, _ := testingHTTPSServer(suite, c)
125 url := baseURL + "/" + common.StateFile
126 storedState, err := common.LoadStateFromURL(url, false)
127 c.Assert(err, gc.ErrorMatches, ".*/provider-state:.* certificate signed by unknown authority")
128 c.Assert(storedState, gc.IsNil)
129}
130
131func (suite *StateSuite) TestLoadStateFromURLBadCertPermitted(c *gc.C) {
132 baseURL, dataDir := testingHTTPSServer(suite, c)
133 state := suite.setUpSavedState(c, dataDir)
134 url := baseURL + "/" + common.StateFile
135 storedState, err := common.LoadStateFromURL(url, true)
101 c.Assert(err, gc.IsNil)136 c.Assert(err, gc.IsNil)
102 c.Check(*storedState, gc.DeepEquals, state)137 c.Check(*storedState, gc.DeepEquals, state)
103}138}

Subscribers

People subscribed via source and target branches

to all changes: