juju-core

state/apiserver: multiple data races

Bug #1305386 reported by Dave Cheney
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Dave Cheney
juju-core 1.19.2
mgo
Invalid
Undecided
Unassigned

Bug Description

http://paste.ubuntu.com/7229156/

machine-0: WARNING: DATA RACE
machine-0: Write by goroutine 154:
machine-0: labix.org/v2/mgo.(*Query).Iter()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2253 +0x5b5
machine-0: labix.org/v2/mgo.(*Query).All()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2557 +0x46
machine-0: launchpad.net/juju-core/state.(*Unit).findCleanMachineQuery()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/unit.go:1142 +0x1ad
machine-0: launchpad.net/juju-core/state.(*Unit).assignToCleanMaybeEmptyMachine()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/unit.go:1231 +0x2cc
machine-0: launchpad.net/juju-core/state.(*Unit).AssignToCleanEmptyMachine()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/unit.go:1119 +0x5d
machine-0: launchpad.net/juju-core/state.(*State).AssignUnit()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/state.go:1375 +0x53a
machine-0: launchpad.net/juju-core/juju.AddUnits()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/juju/deploy.go:151 +0xfd8
machine-0: runtime.call1073741824()
machine-0: /home/ubuntu/go/src/pkg/runtime/asm_amd64.s:371 +0xaf
machine-0: launchpad.net/juju-core/state/apiserver/client.(*Client).ServiceDeploy()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/client/client.go:293 +0x52f
machine-0: launchpad.net/juju-core/state/apiserver/client.(*Client).ServiceDeployWithNetworks()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/client/client.go:301 +0x74
machine-0: runtime.call256()
machine-0: /home/ubuntu/go/src/pkg/runtime/asm_amd64.s:349 +0x45
machine-0: reflect.Value.Call()
machine-0: /home/ubuntu/go/src/pkg/reflect/value.go:411 +0xe4
machine-0: launchpad.net/juju-core/rpc/rpcreflect.func·005()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/rpcreflect/type.go:306 +0x204
machine-0: launchpad.net/juju-core/rpc/rpcreflect.MethodCaller.Call()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/rpcreflect/value.go:103 +0x18c
machine-0: launchpad.net/juju-core/rpc.(*Conn).runRequest()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:485 +0x149
machine-0: Previous write by goroutine 24:
machine-0: labix.org/v2/mgo.func·009()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:3168 +0x4e4
machine-0: labix.org/v2/mgo.(*mongoSocket).readLoop()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:538 +0xefd
machine-0: Goroutine 154 (running) created at:
machine-0: launchpad.net/juju-core/rpc.(*Conn).handleRequest()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:417 +0x7cf
machine-0: launchpad.net/juju-core/rpc.(*Conn).loop()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:349 +0x18d
machine-0: launchpad.net/juju-core/rpc.(*Conn).input()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:317 +0x42
machine-0: Goroutine 24 (running) created at:
machine-0: labix.org/v2/mgo.newSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:161 +0x3cf
machine-0: labix.org/v2/mgo.(*mongoServer).Connect()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:164 +0x592
machine-0: labix.org/v2/mgo.(*mongoServer).AcquireSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:119 +0x4d7
machine-0: labix.org/v2/mgo.(*mongoServer).pinger()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:275 +0x1fb
machine-0: labix.org/v2/mgo.newServer()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:79 +0x166
machine-0: labix.org/v2/mgo.(*mongoCluster).server()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:375 +0x12c
machine-0: labix.org/v2/mgo.func·004()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:435 +0x72f
machine-0: ==================
machine-0: ==================
machine-0: WARNING: DATA RACE
machine-0: Write by goroutine 154:
machine-0: labix.org/v2/mgo.(*Query).Iter()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2253 +0x5b5
machine-0: launchpad.net/juju-core/state.(*Unit).assignToCleanMaybeEmptyMachine()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/unit.go:1241 +0x3b9
machine-0: launchpad.net/juju-core/state.(*Unit).AssignToCleanEmptyMachine()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/unit.go:1119 +0x5d
machine-0: launchpad.net/juju-core/state.(*State).AssignUnit()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/state.go:1375 +0x53a
machine-0: launchpad.net/juju-core/juju.AddUnits()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/juju/deploy.go:151 +0xfd8
machine-0: runtime.call1073741824()
machine-0: /home/ubuntu/go/src/pkg/runtime/asm_amd64.s:371 +0xaf
machine-0: launchpad.net/juju-core/state/apiserver/client.(*Client).ServiceDeploy()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/client/client.go:293 +0x52f
machine-0: launchpad.net/juju-core/state/apiserver/client.(*Client).ServiceDeployWithNetworks()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/client/client.go:301 +0x74
machine-0: runtime.call256()
machine-0: /home/ubuntu/go/src/pkg/runtime/asm_amd64.s:349 +0x45
machine-0: reflect.Value.Call()
machine-0: /home/ubuntu/go/src/pkg/reflect/value.go:411 +0xe4
machine-0: launchpad.net/juju-core/rpc/rpcreflect.func·005()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/rpcreflect/type.go:306 +0x204
machine-0: launchpad.net/juju-core/rpc/rpcreflect.MethodCaller.Call()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/rpcreflect/value.go:103 +0x18c
machine-0: launchpad.net/juju-core/rpc.(*Conn).runRequest()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:485 +0x149
machine-0: Previous write by goroutine 24:
machine-0: labix.org/v2/mgo.func·009()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:3168 +0x4e4
machine-0: labix.org/v2/mgo.(*mongoSocket).readLoop()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:538 +0xefd
machine-0: Goroutine 154 (running) created at:
machine-0: launchpad.net/juju-core/rpc.(*Conn).handleRequest()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:417 +0x7cf
machine-0: launchpad.net/juju-core/rpc.(*Conn).loop()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:349 +0x18d
machine-0: launchpad.net/juju-core/rpc.(*Conn).input()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/rpc/server.go:317 +0x42
machine-0: Goroutine 24 (running) created at:
machine-0: labix.org/v2/mgo.newSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:161 +0x3cf
machine-0: labix.org/v2/mgo.(*mongoServer).Connect()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:164 +0x592
machine-0: labix.org/v2/mgo.(*mongoServer).AcquireSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:119 +0x4d7
machine-0: labix.org/v2/mgo.(*mongoServer).pinger()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:275 +0x1fb
machine-0: labix.org/v2/mgo.newServer()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:79 +0x166
machine-0: labix.org/v2/mgo.(*mongoCluster).server()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:375 +0x12c
machine-0: labix.org/v2/mgo.func·004()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:435 +0x72f
machine-0: ==================
machine-0: ==================
machine-0: WARNING: DATA RACE
machine-0: Write by goroutine 290:
machine-0: labix.org/v2/mgo.(*Query).Iter()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2253 +0x5b5
machine-0: labix.org/v2/mgo.(*Query).All()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2557 +0x46
machine-0: launchpad.net/juju-core/state.(*State).PrepareLocalCharmUpload()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/state.go:594 +0x6d1
machine-0: launchpad.net/juju-core/state/apiserver.(*charmsHandler).processPost()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/charms.go:207 +0x913
machine-0: launchpad.net/juju-core/state/apiserver.(*charmsHandler).ServeHTTP()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/charms.go:52 +0x3a7
machine-0: net/http.(*ServeMux).ServeHTTP()
machine-0: /home/ubuntu/go/src/pkg/net/http/server.go:1519 +0x1e7
machine-0: net/http.serverHandler.ServeHTTP()
machine-0: /home/ubuntu/go/src/pkg/net/http/server.go:1681 +0x1e8
machine-0: net/http.(*conn).serve()
machine-0: /home/ubuntu/go/src/pkg/net/http/server.go:1182 +0xf84
machine-0: Previous write by goroutine 24:
machine-0: labix.org/v2/mgo.func·009()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:3168 +0x4e4
machine-0: labix.org/v2/mgo.(*mongoSocket).readLoop()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:538 +0xefd
machine-0: Goroutine 290 (running) created at:
machine-0: net/http.(*Server).Serve()
machine-0: /home/ubuntu/go/src/pkg/net/http/server.go:1729 +0x339
machine-0: net/http.Serve()
machine-0: /home/ubuntu/go/src/pkg/net/http/server.go:1584 +0xb8
machine-0: launchpad.net/juju-core/state/apiserver.(*Server).run()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/apiserver/apiserver.go:182 +0x64c
machine-0: Goroutine 24 (running) created at:
machine-0: labix.org/v2/mgo.newSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:161 +0x3cf
machine-0: labix.org/v2/mgo.(*mongoServer).Connect()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:164 +0x592
machine-0: labix.org/v2/mgo.(*mongoServer).AcquireSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:119 +0x4d7
machine-0: labix.org/v2/mgo.(*mongoServer).pinger()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:275 +0x1fb
machine-0: labix.org/v2/mgo.newServer()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:79 +0x166
machine-0: labix.org/v2/mgo.(*mongoCluster).server()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:375 +0x12c
machine-0: labix.org/v2/mgo.func·004()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:435 +0x72f
machine-0: ==================
machine-0: ==================
machine-0: WARNING: DATA RACE
machine-0: Write by goroutine 24:
machine-0: labix.org/v2/mgo.func·009()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:3168 +0x4e4
machine-0: labix.org/v2/mgo.(*mongoSocket).readLoop()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:538 +0xefd
machine-0: Previous write by goroutine 122:
machine-0: labix.org/v2/mgo.(*Query).Iter()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/session.go:2253 +0x5b5
machine-0: labix.org/v2/mgo/txn.(*Runner).ResumeAll()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/txn/txn.go:317 +0x35d
machine-0: launchpad.net/juju-core/state.(*State).ResumeTransactions()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/state/state.go:1490 +0x63
machine-0: launchpad.net/juju-core/worker/resumer.(*Resumer).loop()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/worker/resumer/resumer.go:67 +0x180
machine-0: launchpad.net/juju-core/worker/resumer.func·001()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/worker/resumer/resumer.go:39 +0x86
machine-0: Goroutine 24 (running) created at:
machine-0: labix.org/v2/mgo.newSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/socket.go:161 +0x3cf
machine-0: labix.org/v2/mgo.(*mongoServer).Connect()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:164 +0x592
machine-0: labix.org/v2/mgo.(*mongoServer).AcquireSocket()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:119 +0x4d7
machine-0: labix.org/v2/mgo.(*mongoServer).pinger()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:275 +0x1fb
machine-0: labix.org/v2/mgo.newServer()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/server.go:79 +0x166
machine-0: labix.org/v2/mgo.(*mongoCluster).server()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:375 +0x12c
machine-0: labix.org/v2/mgo.func·004()
machine-0: /home/ubuntu/src/labix.org/v2/mgo/cluster.go:435 +0x72f
machine-0: Goroutine 122 (running) created at:
machine-0: launchpad.net/juju-core/worker/resumer.NewResumer()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/worker/resumer/resumer.go:40 +0x111
machine-0: main.func·031()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/cmd/jujud/machine.go:454 +0x8c
machine-0: main.func·036()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/cmd/jujud/machine.go:515 +0x143
machine-0: launchpad.net/juju-core/worker.func·005()
machine-0: /home/ubuntu/src/launchpad.net/juju-core/worker/simpleworker.go:25 +0x7c
machine-0: ==================

Related branches

lp:~dave-cheney/juju-core/170-update-to-mgo-latest
Martin Packman (community): Approve
Diff: 12 lines (+1/-1)
1 file modified
dependencies.tsv (+1/-1)
Revision history for this message
John A Meinel (jameinel) wrote :

This seems important, but isn't an explicit bug in itself. Potentially this is the bug that we are seeing with "panic: Session already closed" ?
It is certainly interesting that it is about creating a newSocket and issuing Queries on sockets.

I think we should spend some time on this, but I don't think it blocks a 1.19.0 release (so not actually "Critical")

Changed in juju-core:
milestone: 1.19.0 → 1.19.1
importance: Critical → High
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.19.1 → 1.20.0
Dave Cheney (dave-cheney)
Changed in mgo:
status: New → Invalid
Changed in juju-core:
assignee: nobody → Dave Cheney (dave-cheney)
Dave Cheney (dave-cheney)
Changed in juju-core:
status: Triaged → In Progress
Dave Cheney (dave-cheney)
Changed in juju-core:
milestone: 1.20.0 → 1.19.2
status: In Progress → Fix Committed
Canonical Juju QA Bot (juju-qa-bot)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.