Merge lp:~axwalk/juju-core/testing-mgo-addrinuse into lp:~go-bot/juju-core/trunk

Proposed by Andrew Wilkins
Status: Merged
Approved by: Andrew Wilkins
Approved revision: no longer in the source branch.
Merged at revision: 2729
Proposed branch: lp:~axwalk/juju-core/testing-mgo-addrinuse
Merge into: lp:~go-bot/juju-core/trunk
Diff against target: 86 lines (+38/-13)
1 file modified
testing/mgo.go (+38/-13)
To merge this branch: bzr merge lp:~axwalk/juju-core/testing-mgo-addrinuse
Reviewer Review Type Date Requested Status
Juju Engineering Pending
Review via email: mp+219456@code.launchpad.net

Commit message

testing: reattempt MgoInstance.run if addr in use

Mongod does not support listening on port 0 to
acquire an ephemeral port, so our tests must
choose a port which may then be allocated to
another process. This CL adds code to reattempt
the starting of mongod if the port is in use,
up to a maximum of 5 attempts.

https://codereview.appspot.com/94410046/

Description of the change

testing: reattempt MgoInstance.run if addr in use

Mongod does not support listening on port 0 to
acquire an ephemeral port, so our tests must
choose a port which may then be allocated to
another process. This CL adds code to reattempt
the starting of mongod if the port is in use,
up to a maximum of 5 attempts.

https://codereview.appspot.com/94410046/

To post a comment you must log in.
Revision history for this message
Andrew Wilkins (axwalk) wrote :
Download full text (3.3 KiB)

Reviewers: mp+219456_code.launchpad.net,

Message:
Please take a look.

Description:
testing: reattempt MgoInstance.run if addr in use

Mongod does not support listening on port 0 to
acquire an ephemeral port, so our tests must
choose a port which may then be allocated to
another process. This CL adds code to reattempt
the starting of mongod if the port is in use,
up to a maximum of 5 attempts.

https://code.launchpad.net/~axwalk/juju-core/testing-mgo-addrinuse/+merge/219456

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/94410046/

Affected files (+39, -13 lines):
   A [revision details]
   M testing/mgo.go

Index: [revision details]
=== added file '[revision details]'
--- [revision details] 2012-01-01 00:00:00 +0000
+++ [revision details] 2012-01-01 00:00:00 +0000
@@ -0,0 +1,2 @@
+Old revision: tarmac-20140513233806-k5sipwwqoqnt8gzr
+New revision: <email address hidden>

Index: testing/mgo.go
=== modified file 'testing/mgo.go'
--- testing/mgo.go 2014-05-12 07:59:38 +0000
+++ testing/mgo.go 2014-05-14 05:26:43 +0000
@@ -38,6 +38,11 @@
   waitingForConnectionsRe = regexp.MustCompile(".*initandlisten.*waiting
for connections.*")
  )

+const (
+ // maximum number of times to attempt starting mongod
+ maxStartMongodAttempts = 5
+)
+
  type MgoInstance struct {
   // addr holds the address of the MongoDB server
   addr string
@@ -107,18 +112,29 @@
   if err != nil {
    return fmt.Errorf("cannot write cert/key PEM: %v", err)
   }
- inst.port = FindTCPPort()
- inst.addr = fmt.Sprintf("localhost:%d", inst.port)
- inst.dir = dbdir
- inst.ssl = ssl
- if err := inst.run(); err != nil {
- inst.addr = ""
- inst.port = 0
- os.RemoveAll(inst.dir)
- inst.dir = ""
- logger.Warningf("failed to start mongo: %v", err)
- } else {
- logger.Debugf("started mongod pid %d in %s on port %d",
inst.server.Process.Pid, dbdir, inst.port)
+
+ // Attempt to start mongo up to maxStartMongodAttempts times,
+ // as the port we choose may be taken from us in the mean time.
+ for i := 0; i < maxStartMongodAttempts; i++ {
+ inst.port = FindTCPPort()
+ inst.addr = fmt.Sprintf("localhost:%d", inst.port)
+ inst.dir = dbdir
+ inst.ssl = ssl
+ err = inst.run()
+ switch err.(type) {
+ case addrAlreadyInUseError:
+ logger.Debugf("failed to start mongo: %v, trying another port", err)
+ continue
+ case nil:
+ logger.Debugf("started mongod pid %d in %s on port %d",
inst.server.Process.Pid, dbdir, inst.port)
+ default:
+ inst.addr = ""
+ inst.port = 0
+ os.RemoveAll(inst.dir)
+ inst.dir = ""
+ logger.Warningf("failed to start mongo: %v", err)
+ }
+ break
   }
   return err
  }
@@ -186,7 +202,11 @@
    if readUntilMatching(prefix, io.TeeReader(out, &buf),
waitingForConnectionsRe) {
     listening <- nil
    } else {
- listening <- fmt.Errorf("mongod failed to listen on port %v", mgoport)
+ err := fmt.Errorf("mongod failed to listen on port %v", mgoport)
+ if strings.Contains(buf.String(), "addr already in use") {
+ err = addrAlreadyInUseError{err}
+ }
+ listening <- err
    }
    // Capture the last 20 lines of output from mongod, to log
    /...

Read more...

Revision history for this message
Ian Booth (wallyworld) wrote :
Revision history for this message
Go Bot (go-bot) wrote :
Download full text (68.4 KiB)

The attempt to merge lp:~axwalk/juju-core/testing-mgo-addrinuse into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core 0.014s
ok launchpad.net/juju-core/agent 1.013s
ok launchpad.net/juju-core/agent/mongo 0.446s
ok launchpad.net/juju-core/agent/tools 0.187s
ok launchpad.net/juju-core/bzr 4.927s
ok launchpad.net/juju-core/cert 2.628s
ok launchpad.net/juju-core/charm 0.472s
? launchpad.net/juju-core/charm/hooks [no test files]
? launchpad.net/juju-core/charm/testing [no test files]
ok launchpad.net/juju-core/cloudinit 0.030s
ok launchpad.net/juju-core/cloudinit/sshinit 0.788s
ok launchpad.net/juju-core/cmd 0.201s
ok launchpad.net/juju-core/cmd/charm-admin 0.263s
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/envcmd 0.176s
ok launchpad.net/juju-core/cmd/juju 225.050s

----------------------------------------------------------------------
PANIC: agent.go:0: MachineSuite.SetUpTest

[LOG] 0:00.016 ERROR juju.testing mongodb has exited without being killed
[LOG] 0:00.016 ERROR juju.testing mongod: [35 lines omitted]
[LOG] 0:00.016 ERROR juju.testing mongod: Wed May 14 06:56:07.659 Got signal: 6 (Aborted).
[LOG] 0:00.016 ERROR juju.testing mongod: Wed May 14 06:56:07.662 Backtrace:
[LOG] 0:00.016 ERROR juju.testing mongod: 0xb6ca63 0x6e3c00 0x7f96102474a0 0x7f9610247425 0x7f961024ab8b 0x6e337f 0x7f9610b98846 0x7f9610b98873 0x7f9610b9896e 0xb31ac1 0xb5f7b8 0xb529f4 0xb57c4c 0x854553 0x7f96116e3ce9 0x7f9611f30e9a 0x7f96103053fd
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xb6ca63]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x350) [0x6e3c00]
[LOG] 0:00.016 ERROR juju.testing mongod: /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f96102474a0]
[LOG] 0:00.016 ERROR juju.testing mongod: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f9610247425]
[LOG] 0:00.016 ERROR juju.testing mongod: /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f961024ab8b]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo11myterminateEv+0x3f) [0x6e337f]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5846) [0x7f9610b98846]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb5873) [0x7f9610b98873]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb596e) [0x7f9610b9896e]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo9uassertedEiPKc+0x91) [0xb31ac1]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo10SSLManagerC2ERKNS_9SSLParamsE+0x338) [0xb5f7b8]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo8ListenerC1ERKSsS2_ib+0x114) [0xb529f4]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo13MiniWebServerC1ERKSsS2_i+0xc) [0xb57c4c]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/bin/mongod(_ZN5mongo15webServerThreadEPKNS_11AdminAccessE+0x53) [0x854553]
[LOG] 0:00.016 ERROR juju.testing mongod: /usr/lib/libboost_thread....

Revision history for this message
Andrew Wilkins (axwalk) wrote :

So, mongo is crashing now... I'm going to disable the builtin HTTP server, which is what appears to be causing problems. We don't use it at all, AFAIK. Likewise Unix sockets.

Revision history for this message
Andrew Wilkins (axwalk) wrote :
Revision history for this message
Go Bot (go-bot) wrote :
Download full text (11.3 KiB)

The attempt to merge lp:~axwalk/juju-core/testing-mgo-addrinuse into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core 0.016s
ok launchpad.net/juju-core/agent 0.836s
ok launchpad.net/juju-core/agent/mongo 0.483s
ok launchpad.net/juju-core/agent/tools 0.181s
ok launchpad.net/juju-core/bzr 5.139s
ok launchpad.net/juju-core/cert 2.305s
ok launchpad.net/juju-core/charm 0.409s
? launchpad.net/juju-core/charm/hooks [no test files]
? launchpad.net/juju-core/charm/testing [no test files]
ok launchpad.net/juju-core/cloudinit 0.030s
ok launchpad.net/juju-core/cloudinit/sshinit 0.742s
ok launchpad.net/juju-core/cmd 0.168s
ok launchpad.net/juju-core/cmd/charm-admin 0.305s
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/envcmd 0.165s
ok launchpad.net/juju-core/cmd/juju 222.221s
ok launchpad.net/juju-core/cmd/jujud 66.550s
ok launchpad.net/juju-core/cmd/plugins/juju-metadata 9.727s
? launchpad.net/juju-core/cmd/plugins/juju-restore [no test files]
ok launchpad.net/juju-core/cmd/plugins/local 0.207s
? launchpad.net/juju-core/cmd/plugins/local/juju-local [no test files]
ok launchpad.net/juju-core/constraints 0.026s
ok launchpad.net/juju-core/container 0.035s
ok launchpad.net/juju-core/container/factory 0.037s
ok launchpad.net/juju-core/container/kvm 0.192s
ok launchpad.net/juju-core/container/kvm/mock 0.046s
? launchpad.net/juju-core/container/kvm/testing [no test files]
ok launchpad.net/juju-core/container/lxc 4.356s
? launchpad.net/juju-core/container/lxc/mock [no test files]
? launchpad.net/juju-core/container/lxc/testing [no test files]
? launchpad.net/juju-core/container/testing [no test files]
ok launchpad.net/juju-core/downloader 5.222s
ok launchpad.net/juju-core/environs 2.348s
ok launchpad.net/juju-core/environs/bootstrap 11.631s
ok launchpad.net/juju-core/environs/cloudinit 0.447s
ok launchpad.net/juju-core/environs/config 1.581s
ok launchpad.net/juju-core/environs/configstore 0.028s
ok launchpad.net/juju-core/environs/filestorage 0.025s
ok launchpad.net/juju-core/environs/httpstorage 0.661s
ok launchpad.net/juju-core/environs/imagemetadata 0.443s
? launchpad.net/juju-core/environs/imagemetadata/testing [no test files]
ok launchpad.net/juju-core/environs/instances 0.046s
ok launchpad.net/juju-core/environs/jujutest 0.171s
ok launchpad.net/juju-core/environs/manual 10.022s
? launchpad.net/juju-core/environs/network [no test files]
ok launchpad.net/juju-core/environs/simplestreams 0.277s
? launchpad.net/juju-core/environs/simplestreams/testing [no test files]
ok launchpad.net/juju-core/environs/sshstorage 0.907s
ok launchpad.net/juju-core/environs/storage 0.804s
ok launchpad.net/juju-core/environs/sync 51.854s
ok launchpad.net/juju-core/environs/testing 0.138s
ok launchpad.net/juju-core/environs/tools 4.668s
? launchpad.net/juju-core/environs/tools/testing [no test files]
ok launchpad.net/juju-core/errors 0.016s
ok launchpad.net/juju-core/instance 0.017s
? launchpad.net/juju-core/instance/tes...

Revision history for this message
Go Bot (go-bot) wrote :
Download full text (11.8 KiB)

The attempt to merge lp:~axwalk/juju-core/testing-mgo-addrinuse into lp:juju-core failed. Below is the output from the failed tests.

ok launchpad.net/juju-core 0.015s
ok launchpad.net/juju-core/agent 0.848s
ok launchpad.net/juju-core/agent/mongo 0.459s
ok launchpad.net/juju-core/agent/tools 0.168s
ok launchpad.net/juju-core/bzr 5.249s
ok launchpad.net/juju-core/cert 2.298s
ok launchpad.net/juju-core/charm 0.441s
? launchpad.net/juju-core/charm/hooks [no test files]
? launchpad.net/juju-core/charm/testing [no test files]
ok launchpad.net/juju-core/cloudinit 0.029s
ok launchpad.net/juju-core/cloudinit/sshinit 0.775s
ok launchpad.net/juju-core/cmd 0.193s
ok launchpad.net/juju-core/cmd/charm-admin 0.252s
? launchpad.net/juju-core/cmd/charmd [no test files]
? launchpad.net/juju-core/cmd/charmload [no test files]
ok launchpad.net/juju-core/cmd/envcmd 0.146s
ok launchpad.net/juju-core/cmd/juju 220.073s
ok launchpad.net/juju-core/cmd/jujud 68.188s
ok launchpad.net/juju-core/cmd/plugins/juju-metadata 6.682s
? launchpad.net/juju-core/cmd/plugins/juju-restore [no test files]
ok launchpad.net/juju-core/cmd/plugins/local 0.179s
? launchpad.net/juju-core/cmd/plugins/local/juju-local [no test files]
ok launchpad.net/juju-core/constraints 0.038s
ok launchpad.net/juju-core/container 0.041s
ok launchpad.net/juju-core/container/factory 0.065s
ok launchpad.net/juju-core/container/kvm 0.149s
ok launchpad.net/juju-core/container/kvm/mock 0.034s
? launchpad.net/juju-core/container/kvm/testing [no test files]
ok launchpad.net/juju-core/container/lxc 4.252s
? launchpad.net/juju-core/container/lxc/mock [no test files]
? launchpad.net/juju-core/container/lxc/testing [no test files]
? launchpad.net/juju-core/container/testing [no test files]
ok launchpad.net/juju-core/downloader 5.259s
ok launchpad.net/juju-core/environs 2.380s
ok launchpad.net/juju-core/environs/bootstrap 11.944s
ok launchpad.net/juju-core/environs/cloudinit 0.438s
ok launchpad.net/juju-core/environs/config 1.622s
ok launchpad.net/juju-core/environs/configstore 0.028s
ok launchpad.net/juju-core/environs/filestorage 0.026s
ok launchpad.net/juju-core/environs/httpstorage 0.669s
ok launchpad.net/juju-core/environs/imagemetadata 0.509s
? launchpad.net/juju-core/environs/imagemetadata/testing [no test files]
ok launchpad.net/juju-core/environs/instances 0.043s
ok launchpad.net/juju-core/environs/jujutest 0.155s
ok launchpad.net/juju-core/environs/manual 9.530s
? launchpad.net/juju-core/environs/network [no test files]
ok launchpad.net/juju-core/environs/simplestreams 0.231s
? launchpad.net/juju-core/environs/simplestreams/testing [no test files]
ok launchpad.net/juju-core/environs/sshstorage 0.924s
ok launchpad.net/juju-core/environs/storage 0.855s
ok launchpad.net/juju-core/environs/sync 51.699s
ok launchpad.net/juju-core/environs/testing 0.140s
ok launchpad.net/juju-core/environs/tools 4.868s
? launchpad.net/juju-core/environs/tools/testing [no test files]
ok launchpad.net/juju-core/errors 0.020s
ok launchpad.net/juju-core/instance 0.020s
? launchpad.net/juju-core/instance/test...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'testing/mgo.go'
--- testing/mgo.go 2014-05-12 07:59:38 +0000
+++ testing/mgo.go 2014-05-14 07:43:49 +0000
@@ -38,6 +38,11 @@
38 waitingForConnectionsRe = regexp.MustCompile(".*initandlisten.*waiting for connections.*")38 waitingForConnectionsRe = regexp.MustCompile(".*initandlisten.*waiting for connections.*")
39)39)
4040
41const (
42 // maximum number of times to attempt starting mongod
43 maxStartMongodAttempts = 5
44)
45
41type MgoInstance struct {46type MgoInstance struct {
42 // addr holds the address of the MongoDB server47 // addr holds the address of the MongoDB server
43 addr string48 addr string
@@ -107,18 +112,29 @@
107 if err != nil {112 if err != nil {
108 return fmt.Errorf("cannot write cert/key PEM: %v", err)113 return fmt.Errorf("cannot write cert/key PEM: %v", err)
109 }114 }
110 inst.port = FindTCPPort()115
111 inst.addr = fmt.Sprintf("localhost:%d", inst.port)116 // Attempt to start mongo up to maxStartMongodAttempts times,
112 inst.dir = dbdir117 // as the port we choose may be taken from us in the mean time.
113 inst.ssl = ssl118 for i := 0; i < maxStartMongodAttempts; i++ {
114 if err := inst.run(); err != nil {119 inst.port = FindTCPPort()
115 inst.addr = ""120 inst.addr = fmt.Sprintf("localhost:%d", inst.port)
116 inst.port = 0121 inst.dir = dbdir
117 os.RemoveAll(inst.dir)122 inst.ssl = ssl
118 inst.dir = ""123 err = inst.run()
119 logger.Warningf("failed to start mongo: %v", err)124 switch err.(type) {
120 } else {125 case addrAlreadyInUseError:
121 logger.Debugf("started mongod pid %d in %s on port %d", inst.server.Process.Pid, dbdir, inst.port)126 logger.Debugf("failed to start mongo: %v, trying another port", err)
127 continue
128 case nil:
129 logger.Debugf("started mongod pid %d in %s on port %d", inst.server.Process.Pid, dbdir, inst.port)
130 default:
131 inst.addr = ""
132 inst.port = 0
133 os.RemoveAll(inst.dir)
134 inst.dir = ""
135 logger.Warningf("failed to start mongo: %v", err)
136 }
137 break
122 }138 }
123 return err139 return err
124}140}
@@ -139,6 +155,7 @@
139 "--noprealloc",155 "--noprealloc",
140 "--smallfiles",156 "--smallfiles",
141 "--nojournal",157 "--nojournal",
158 "--nohttpinterface",
142 "--nounixsocket",159 "--nounixsocket",
143 "--oplogSize", "10",160 "--oplogSize", "10",
144 "--keyFile", filepath.Join(inst.dir, "keyfile"),161 "--keyFile", filepath.Join(inst.dir, "keyfile"),
@@ -186,7 +203,11 @@
186 if readUntilMatching(prefix, io.TeeReader(out, &buf), waitingForConnectionsRe) {203 if readUntilMatching(prefix, io.TeeReader(out, &buf), waitingForConnectionsRe) {
187 listening <- nil204 listening <- nil
188 } else {205 } else {
189 listening <- fmt.Errorf("mongod failed to listen on port %v", mgoport)206 err := fmt.Errorf("mongod failed to listen on port %v", mgoport)
207 if strings.Contains(buf.String(), "addr already in use") {
208 err = addrAlreadyInUseError{err}
209 }
210 listening <- err
190 }211 }
191 // Capture the last 20 lines of output from mongod, to log212 // Capture the last 20 lines of output from mongod, to log
192 // in the event of unclean exit.213 // in the event of unclean exit.
@@ -528,3 +549,7 @@
528 l.Close()549 l.Close()
529 return l.Addr().(*net.TCPAddr).Port550 return l.Addr().(*net.TCPAddr).Port
530}551}
552
553type addrAlreadyInUseError struct {
554 error
555}

Subscribers

People subscribed via source and target branches

to status/vote changes: