Merge lp:~jtv/juju-core/create-gwacl-sessions into lp:~go-bot/juju-core/trunk
- create-gwacl-sessions
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | John A Meinel |
Approved revision: | no longer in the source branch. |
Merged at revision: | 1313 |
Proposed branch: | lp:~jtv/juju-core/create-gwacl-sessions |
Merge into: | lp:~go-bot/juju-core/trunk |
Diff against target: |
212 lines (+164/-1) 2 files modified
environs/azure/environ.go (+34/-1) environs/azure/environ_test.go (+130/-0) |
To merge this branch: | bzr merge lp:~jtv/juju-core/create-gwacl-sessions |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Tim Penhey (community) | Approve | ||
Review via email: mp+170032@code.launchpad.net |
Commit message
Create gwacl session objects for management & API.
I commented out one of the new functions at the last moment (as well as its
test), even though the test passes. This was done because we're still working
out a detail in gwacl that may result in an incompatible change.
For some of the code in here, including Config(), the only interesting aspect
to test really is the locking. It's the one thing that can go wrong. None of
the other providers seem to test for this, and it's a bit of a losing game in
that whenever you forget to lock, you're likely to forget to test for locking
as well. But I wrote up a simple, reusable pattern so that at least we can
guard against regressions.
Description of the change
The pattern was discussed with Gavin, the gwacl work with Raphaël.
John A Meinel (jameinel) wrote : | # |
Данило Шеган (danilo) wrote : | # |
This generally LGTM. I have further suggestions on the lock testing
pattern, but it may end up being too much work that you can't commit to
right now.
Also, I am not sure of the value of having an unused method commented
out. I know that you plan to change the implementation of it RSN, but
why is it better to keep it commented out instead of having it in there.
If there's desire for it to not be accidentally used by anyone, I'd
remove it completely from code.
https:/
File environs/
https:/
environs/
guarantee that the lock is obeyed. The
If you are going to go to this lengths to test the locking (which I
applaud), you might as well go the full distance :)
How about you simply introduce a separate lock (or you might be able to
use select with channels) which ensures that goroutine 2 only runs after
goroutine 1 has started executing. If you make it into a generic
functionality that'd perhaps live in juju-core/testing/ package, it'd
suddenly become even more useful. If you feel that generalizing this
would take you too far away from your immediate goals (though I know you
love generalizing code :), feel free to ignore this advice.
Jeroen T. Vermeulen (jtv) wrote : | # |
John A Meinel writes:
> You could improve the pattern by ensuring goroutine 2 starts first.
Never say never with concurrency, because I could always be missing something, but I *wanted* goroutine 1 to start first. If goroutine 2 starts first, it is likely to complete right away before goroutine 1 starts (with current implementations if GOMAXPROCS=1) and then the test becomes a tautology.
> You could also encapsulate all of the logic in a helper with closures.
Ooo, thanks for taking this on! I'll see if I can work it into a next branch, if you don't mind, so as to keep branches short and focused.
Tim Penhey (thumper) wrote : | # |
On 2013/06/20 02:24:00, jtv.canonical wrote:
> I came up with a much simpler and more generic form for the helper:
> https:/
LGTM for what I could follow.
Tim Penhey (thumper) : | # |
John A Meinel (jameinel) wrote : | # |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 2013-06-20 3:01, Jeroen T. Vermeulen wrote:
> John A Meinel writes:
>
>> You could improve the pattern by ensuring goroutine 2 starts
>> first.
>
> Never say never with concurrency, because I could always be missing
> something, but I *wanted* goroutine 1 to start first. If goroutine
> 2 starts first, it is likely to complete right away before
> goroutine 1 starts (with current implementations if GOMAXPROCS=1)
> and then the test becomes a tautology.
>
I probably misspoke in my original email, but if you look at the code
I wrote, goroutine 2 waits for goroutine 1 to send on a channel, so
goroutine 2 (concurrent) will always know that goroutine 1 (base) has
started before it executes.
Now, the one change you might want to make, is to have 'startedChan'
be a buffered channel. That way baseAction is not blocked until
concurrentAction starts (but concurrentAction is blocked until
baseAction has at least started).
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://
iEYEARECAAYFAlH
q7AAn07kfXcomIH
=3dFc
-----END PGP SIGNATURE-----
Go Bot (go-bot) wrote : | # |
The attempt to merge lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core failed. Below is the output from the failed tests.
environs/
Jeroen T. Vermeulen (jtv) wrote : | # |
No idea why this didn't work; it passed local tests and I can paste that exact import string into a "go get -u" command and it works.
Retrying.
Go Bot (go-bot) wrote : | # |
The attempt to merge lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core failed. Below is the output from the failed tests.
environs/
Go Bot (go-bot) wrote : | # |
The attempt to merge lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core failed. Below is the output from the failed tests.
../gwacl/
Go Bot (go-bot) wrote : | # |
The attempt to merge lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core failed. Below is the output from the failed tests.
ok launchpad.
ok launchpad.
ok launchpad.
? launchpad.
ok launchpad.
ok launchpad.
? launchpad.
? launchpad.
? launchpad.
ok launchpad.
-------
FAIL: agent.go:0: MachineSuite.
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
/home/tarmac/
c.Fatal("Test left sockets in a dirty state")
... Error: Test left sockets in a dirty state
-------
PANIC: machine_
[LOG] 25.93298 INFO juju environs/testing: uploading FAKE tools 1.11.1-
[LOG] 25.93306 INFO juju environs: reading tools with major version 1
[LOG] 25.93308 DEBUG juju environs/tools: reading v1.* tools
[LOG] 25.93309 INFO juju environs: falling back to public bucket
[LOG] 25.93311 DEBUG juju environs/tools: reading v1.* tools
[LOG] 25.93316 DEBUG juju environs/tools: found 1.11.1-
[LOG] 25.93317 INFO juju environs: filtering tools by series: precise
[LOG] 25.93320 INFO juju environs: filtering tools by version: 1.11.1
[LOG] 25.93322 INFO juju environs/dummy: would pick tools from 1.11.1-
[LOG] 25.96834 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 25.97192 INFO juju state: connection established
[LOG] 26.03032 INFO juju state: initializing environment
[LOG] 26.05076 INFO juju state/api: listening on "localhost:0"
[LOG] 26.07939 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 26.08279 INFO juju state: connection established
[LOG] 26.08328 INFO juju juju: authorization error while connecting to state server; retrying
[LOG] 26.08334 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 2...
Go Bot (go-bot) wrote : | # |
The attempt to merge lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core failed. Below is the output from the failed tests.
ok launchpad.
ok launchpad.
ok launchpad.
? launchpad.
ok launchpad.
ok launchpad.
? launchpad.
? launchpad.
? launchpad.
ok launchpad.
-------
FAIL: agent.go:0: MachineSuite.
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
Waiting for sockets to die: 1 in use, 0 alive
/home/tarmac/
c.Fatal("Test left sockets in a dirty state")
... Error: Test left sockets in a dirty state
-------
PANIC: machine_
[LOG] 10.36633 INFO juju environs/testing: uploading FAKE tools 1.11.1-
[LOG] 10.36639 INFO juju environs: reading tools with major version 1
[LOG] 10.36641 DEBUG juju environs/tools: reading v1.* tools
[LOG] 10.36643 INFO juju environs: falling back to public bucket
[LOG] 10.36645 DEBUG juju environs/tools: reading v1.* tools
[LOG] 10.36649 DEBUG juju environs/tools: found 1.11.1-
[LOG] 10.36651 INFO juju environs: filtering tools by series: precise
[LOG] 10.36653 INFO juju environs: filtering tools by version: 1.11.1
[LOG] 10.36656 INFO juju environs/dummy: would pick tools from 1.11.1-
[LOG] 10.40127 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 10.40487 INFO juju state: connection established
[LOG] 10.47989 INFO juju state: initializing environment
[LOG] 10.50059 INFO juju state/api: listening on "localhost:0"
[LOG] 10.52953 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 10.53294 INFO juju state: connection established
[LOG] 10.53350 INFO juju juju: authorization error while connecting to state server; retrying
[LOG] 10.53356 INFO juju state: opening state; mongo addresses: ["localhost:
[LOG] 1...
Preview Diff
1 | === modified file 'environs/azure/environ.go' |
2 | --- environs/azure/environ.go 2013-06-18 07:46:08 +0000 |
3 | +++ environs/azure/environ.go 2013-06-18 10:04:29 +0000 |
4 | @@ -4,6 +4,7 @@ |
5 | package azure |
6 | |
7 | import ( |
8 | + "launchpad.net/gwacl" |
9 | "launchpad.net/juju-core/constraints" |
10 | "launchpad.net/juju-core/environs" |
11 | "launchpad.net/juju-core/environs/config" |
12 | @@ -64,7 +65,8 @@ |
13 | |
14 | // Config is specified in the Environ interface. |
15 | func (env *azureEnviron) Config() *config.Config { |
16 | - panic("unimplemented") |
17 | + snap := env.getSnapshot() |
18 | + return snap.ecfg.Config |
19 | } |
20 | |
21 | // SetConfig is specified in the Environ interface. |
22 | @@ -131,3 +133,34 @@ |
23 | func (env *azureEnviron) Provider() environs.EnvironProvider { |
24 | panic("unimplemented") |
25 | } |
26 | + |
27 | +// TODO: Temporarily deactivating this code. Passing certificate in-memory |
28 | +// may require gwacl change. |
29 | +/* |
30 | +// getManagementAPI obtains a context object for interfacing with Azure's |
31 | +// management API. |
32 | +// For now, each invocation just returns a separate object. This is probably |
33 | +// wasteful (each context gets its own SSL connection) and may need optimizing |
34 | +// later. |
35 | +func (env *azureEnviron) getManagementAPI() (*gwacl.ManagementAPI, error) { |
36 | + snap := env.getSnapshot() |
37 | + subscription := snap.ecfg.ManagementSubscriptionId() |
38 | + cert := snap.ecfg.ManagementCertificate() |
39 | + return gwacl.NewManagementAPI(subscription, cert) |
40 | +} |
41 | +*/ |
42 | + |
43 | +// getStorageContext obtains a context object for interfacing with Azure's |
44 | +// storage API. |
45 | +// For now, each invocation just returns a separate object. This is probably |
46 | +// wasteful (each context gets its own SSL connection) and may need optimizing |
47 | +// later. |
48 | +func (env *azureEnviron) getStorageContext() (*gwacl.StorageContext, error) { |
49 | + snap := env.getSnapshot() |
50 | + context := gwacl.StorageContext{ |
51 | + Account: snap.ecfg.StorageAccountName(), |
52 | + Key: snap.ecfg.StorageAccountKey(), |
53 | + } |
54 | + // There is currently no way for this to fail. |
55 | + return &context, nil |
56 | +} |
57 | |
58 | === modified file 'environs/azure/environ_test.go' |
59 | --- environs/azure/environ_test.go 2013-06-17 10:24:42 +0000 |
60 | +++ environs/azure/environ_test.go 2013-06-18 10:04:29 +0000 |
61 | @@ -5,6 +5,7 @@ |
62 | |
63 | import ( |
64 | . "launchpad.net/gocheck" |
65 | + "launchpad.net/juju-core/environs/config" |
66 | "sync" |
67 | ) |
68 | |
69 | @@ -14,6 +15,53 @@ |
70 | |
71 | var _ = Suite(new(EnvironSuite)) |
72 | |
73 | +func makeEnviron(c *C) *azureEnviron { |
74 | + attrs := makeAzureConfigMap(c) |
75 | + cfg, err := config.New(attrs) |
76 | + c.Assert(err, IsNil) |
77 | + ecfg, err := azureEnvironProvider{}.newConfig(cfg) |
78 | + c.Assert(err, IsNil) |
79 | + return &azureEnviron{ |
80 | + name: "env", |
81 | + ecfg: ecfg, |
82 | + } |
83 | +} |
84 | + |
85 | +// A note on locking tests. Proper locking is hard to test for. Tests here |
86 | +// use a fixed pattern to verify that a function obeys a particular lock: |
87 | +// |
88 | +// 1. Create a channel for the function's result. |
89 | +// 2. Grab the lock. |
90 | +// 3. Launch goroutine 1: invoke the function and report result to the channel. |
91 | +// 4. Launch goroutine 2: modify the object and then release the lock. |
92 | +// 5. Retrieve result from the channel. |
93 | +// 6. Test that the result reflects goroutine 2's modification. |
94 | +// 7. Test that the lock was released in the end. |
95 | +// |
96 | +// If the function obeys the lock, it can't complete until goroutine 2 has |
97 | +// completed. If it doesn't, it can. The pattern aims for this scenario: |
98 | +// |
99 | +// The mainline code blocks on the channel. |
100 | +// Goroutine 1 starts. It invokes the function you want to test. |
101 | +// The function tries to grab the lock, and blocks. |
102 | +// Goroutine 2 starts. It releases the lock and exits. |
103 | +// The function in goroutine 1 is now unblocked. |
104 | +// |
105 | +// It would be simpler to have just one goroutine (and skip the channel), and |
106 | +// release the lock inline. But then the ordering depends on a more |
107 | +// fundamental choice within the language implementation: it may choose to |
108 | +// start running a goroutine immediately at the "go" statement, or it may |
109 | +// continue executing the inline code and postpone execution of the goroutine |
110 | +// until the inline code blocks. |
111 | +// |
112 | +// The pattern is still not a full guarantee that the lock is obeyed. The |
113 | +// language implementation might choose to run the goroutines in LIFO order, |
114 | +// and then the locking would not be exercised. The lock would simply be |
115 | +// available by the time goroutine 1 ran, and the test would never fail unless |
116 | +// the function you're testing neglected to release the lock. But as long as |
117 | +// there is a reasonable chance of the first goroutine starting before the |
118 | +// second, there is a chance of exposing a function that disobeys the lock. |
119 | + |
120 | func (EnvironSuite) TestGetSnapshot(c *C) { |
121 | original := azureEnviron{name: "this-env", ecfg: new(azureEnvironConfig)} |
122 | snapshot := original.getSnapshot() |
123 | @@ -32,7 +80,89 @@ |
124 | c.Check(snapshot.Mutex, Equals, sync.Mutex{}) |
125 | } |
126 | |
127 | +func (EnvironSuite) TestGetSnapshotLocksEnviron(c *C) { |
128 | + // This tests follows the locking-test pattern. See comment above. |
129 | + // If you want to change how this works, you probably want to update |
130 | + // any other tests with the same pattern as well. |
131 | + original := azureEnviron{name: "old-name"} |
132 | + // 1. Result comes out of this channel. |
133 | + snaps := make(chan *azureEnviron) |
134 | + // 2. Stop a well-behaved getSnapshot from running (for now). |
135 | + original.Lock() |
136 | + // 3. Goroutine 1: ask for a snapshot. The point of the test is that |
137 | + // this blocks until we release our lock. |
138 | + go func() { |
139 | + snaps <- original.getSnapshot() |
140 | + }() |
141 | + // 4. Goroutine 2: release the lock. The getSnapshot call can't |
142 | + // complete until we've done this. |
143 | + go func() { |
144 | + original.name = "new-name" |
145 | + original.Unlock() |
146 | + }() |
147 | + // 5. Let the goroutines do their work. |
148 | + snapshot := <-snaps |
149 | + // 6. Test: the snapshot was made only after the lock was released. |
150 | + c.Check(snapshot.name, Equals, "new-name") |
151 | + // 7. Test: getSnapshot released the lock. |
152 | + c.Check(original.Mutex, Equals, sync.Mutex{}) |
153 | +} |
154 | + |
155 | func (EnvironSuite) TestName(c *C) { |
156 | env := azureEnviron{name: "foo"} |
157 | c.Check(env.Name(), Equals, env.name) |
158 | } |
159 | + |
160 | +func (EnvironSuite) TestConfigReturnsConfig(c *C) { |
161 | + cfg := new(config.Config) |
162 | + ecfg := azureEnvironConfig{Config: cfg} |
163 | + env := azureEnviron{ecfg: &ecfg} |
164 | + c.Check(env.Config(), Equals, cfg) |
165 | +} |
166 | + |
167 | +func (EnvironSuite) TestConfigLocksEnviron(c *C) { |
168 | + // This tests follows the locking-test pattern. See comment above. |
169 | + // If you want to change how this works, you probably want to update |
170 | + // any other tests with the same pattern as well. |
171 | + env := azureEnviron{name: "env", ecfg: new(azureEnvironConfig)} |
172 | + newConfig := new(config.Config) |
173 | + // 1. Create results channel. |
174 | + configs := make(chan *config.Config) |
175 | + // 2. Stop a well-behaved Config() from running, for now. |
176 | + env.Lock() |
177 | + // 3. Goroutine 1: call Config(). We want to test that this locks. |
178 | + go func() { |
179 | + configs <- env.Config() |
180 | + }() |
181 | + // 4. Goroutine 2: change the Environ object, and release the lock. |
182 | + go func() { |
183 | + env.ecfg = &azureEnvironConfig{Config: newConfig} |
184 | + env.Unlock() |
185 | + }() |
186 | + // 5. Let the goroutines do their work. |
187 | + config := <-configs |
188 | + // 6. Test that goroutine 2 completed before Config did. |
189 | + c.Check(config, Equals, newConfig) |
190 | + // 7. Test: Config() released the lock. |
191 | + c.Check(env.Mutex, Equals, sync.Mutex{}) |
192 | +} |
193 | + |
194 | +// TODO: Temporarily deactivating this code. Passing certificate in-memory |
195 | +// may require gwacl change. |
196 | +/* |
197 | +func (EnvironSuite) TestGetManagementAPI(c *C) { |
198 | + env := makeEnviron(c) |
199 | + management, err := env.getManagementAPI() |
200 | + c.Assert(err, IsNil) |
201 | + c.Check(management, NotNil) |
202 | +} |
203 | +*/ |
204 | + |
205 | +func (EnvironSuite) TestGetStorageContext(c *C) { |
206 | + env := makeEnviron(c) |
207 | + storage, err := env.getStorageContext() |
208 | + c.Assert(err, IsNil) |
209 | + c.Assert(storage, NotNil) |
210 | + c.Check(storage.Account, Equals, env.ecfg.StorageAccountName()) |
211 | + c.Check(storage.Key, Equals, env.ecfg.StorageAccountKey()) |
212 | +} |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 2013-06-18 14:05, Jeroen T. Vermeulen wrote: /code.launchpad .net/~jtv/ juju-core/ create- gwacl-sessions/ +merge/ 170032 /codereview. appspot. com/10367045/
> Jeroen T. Vermeulen has proposed merging
> lp:~jtv/juju-core/create-gwacl-sessions into lp:juju-core.
>
> Commit message: Create gwacl session objects for management & API.
>
> I commented out one of the new functions at the last moment (as
> well as its test), even though the test passes. This was done
> because we're still working out a detail in gwacl that may result
> in an incompatible change.
>
> For some of the code in here, including Config(), the only
> interesting aspect to test really is the locking. It's the one
> thing that can go wrong. None of the other providers seem to test
> for this, and it's a bit of a losing game in that whenever you
> forget to lock, you're likely to forget to test for locking as
> well. But I wrote up a simple, reusable pattern so that at least
> we can guard against regressions.
>
> Requested reviews: juju hackers (juju)
>
> For more details, see:
> https:/
>
>
>
The pattern was discussed with Gavin, the gwacl work with
> Raphaël.
>
> https:/
>
You could improve the pattern by ensuring goroutine 2 starts first.
You could also encapsulate all of the logic in a helper with closures.
So something like:
func CheckLocking(c *C, lock sync.Locker, baseAction func() ction()
interface{}, concurrentAction func()) interface{} {
resultChan := make(chan interface{})
startedChan := make(chan bool)
lock.Lock()
go func() {
startedChan <- true
resultChan <- baseAction()
}()
go func() {
//Make sure the baseAction has started
<-startedChan
concurrentA
lock.Unlock()
}()
return <-resultChan
}
then your test becomes:
func (EnvironSuite) TestGetSnapshot LocksEnviron( c *C) { getSnapshot( )
original. name = "new-name" original, base, concurrent) (azureSnapshot) snapshot. name, Equals, "new-name")
c.Check( original. Mutex, Equals, sync.Mutex{})
original := azureEnviron{name: "old-name"}
base := func () interface{} {
return original.
}
concurrent := func() {
}
result := CheckLocking(
snapshot := result.
c.Check(
}
You can make it shorter by inlining the functions: LocksEnviron( c *C) { original, getSnapshot( )} }).(azureSnapsh ot) snapshot. name, Equals, "new-name")
c.Check( original. Mutex, Equals, sync.Mutex{})
func (EnvironSuite) TestGetSnapshot
original := azureEnviron{name: "old-name"}
snapshot := CheckLocking(
func() interface{} {return original.
func() {original.name = "new-name"
c.Check(
}
Thoughts?
John
=:->
-----BEGIN PGP SIGNATURE----- www.enigmail. net/
BPO0ACgkQJdeBCY SNAAOCCgCeOJ3aF kYQmmLHBVfdG1ue hEQB RJCC/JZSDTjJ8aE mA
Version: GnuPG v1.4.13 (Cygwin)
Comment: Using GnuPG with Thunderbird - http://
iEYEARECAAYFAlH
JxAAnixDGf5LQ3u
=pE6P
-----END PGP SIGNATURE-----