wait-for-state restarts whoopsie every 30sec

Bug #1412719 reported by Michael Blennerhassett
80
This bug affects 15 people
Affects Status Importance Assigned to Milestone
One Hundred Papercuts
Invalid
High
Unassigned
upstart
Invalid
Undecided
Unassigned
upstart (Ubuntu)
Invalid
Wishlist
Unassigned
whoopsie (Ubuntu)
Fix Released
High
Didier Roche-Tolomelli
whoopsie-preferences (Ubuntu)
Fix Released
Undecided
Didier Roche-Tolomelli

Bug Description

wait-for-state is restarting whoopsie every 30sec:

[14103.317598] init: whoopsie main process ended, respawning
[14133.330315] init: wait-for-state (whoopsienetwork-manager) main process (10221) terminated with status 100
[14133.396185] init: whoopsie main process (10227) terminated with status 1
[14133.396205] init: whoopsie main process ended, respawning
[14163.409765] init: wait-for-state (whoopsienetwork-manager) main process (10231) terminated with status 100
[14163.468346] init: whoopsie main process (10237) terminated with status 1
[14163.468366] init: whoopsie main process ended, respawning
[14193.485204] init: wait-for-state (whoopsienetwork-manager) main process (10241) terminated with status 100
[14193.560511] init: whoopsie main process (10249) terminated with status 1
[14193.560531] init: whoopsie main process ended, respawning

The problem in wait-for-state.conf is this line:
status $WAIT_FOR | grep -q "$TARGET_GOAL/$WAIT_STATE" && exit 0

TARGET_GOAL is either "start" or "stop".
WAIT_STATE by default is "started", but there is no job status like "start/started".

so WAIT_STATE needs to be something like "running" (or "waiting" for goal of "stop").

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: upstart 1.13.2-0ubuntu6
ProcVersionSignature: Ubuntu 3.18.0-9.10-generic 3.18.2
Uname: Linux 3.18.0-9-generic x86_64
ApportVersion: 2.15.1-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Jan 20 20:59:02 2015
InstallationDate: Installed on 2014-09-03 (139 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Alpha amd64 (20140902)
ProcKernelCmdline: BOOT_IMAGE=/boot/vmlinuz-3.18.0-9-generic root=UUID=93e046a8-7604-4f60-a5f8-b3be22f948aa ro quiet splash vt.handoff=7
SourcePackage: upstart
UpgradeStatus: No upgrade log present (probably fresh install)
UpstartBugCategory: System
UpstartRunningSessionVersion: upstart 1.13.2
UpstartRunningSystemVersion: init (upstart 1.13.2)
modified.conffile..etc.init.wait.for.state.conf: [modified]
mtime.conffile..etc.init.wait.for.state.conf: 2015-01-20T20:02:17.598668

Related branches

Revision history for this message
Michael Blennerhassett (mjblenner) wrote :
Revision history for this message
Michael Blennerhassett (mjblenner) wrote :

This patch changes the defaults for wait-for-state.conf to something sensible for WAIT_STATE.

tags: added: trusty
tags: added: patch
Changed in upstart (Ubuntu):
status: New → Confirmed
Changed in upstart (Ubuntu):
importance: Undecided → High
Changed in upstart:
status: New → Confirmed
Changed in hundredpapercuts:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Steve Langasek (vorlon) wrote :

The second part of the patch proposed to upstart's wait-for-state job is reasonable, but has nothing to do with the behavior shown here. And the first part of the patch is wrong. It's not appropriate to change the default wait-for-state behavior that other packages are relying on.

/etc/init/whoopsie.conf has:

    status network-manager >/dev/null 2>&1 && start wait-for-state WAITER=whoopsie WAIT_FOR=network-manager GOAL=start || :

- 'GOAL' is incorrect; the variable name is 'TARGET_GOAL', but start is the default so this is a cosmetic issue.
- because network-manager runs a service, WAIT_STATE=running needs to be specified here.
- changing any of this will not cause whoopsie to stop respawning. All of the wait-for-state handling happens in a *pre-start* script which has been written to ignore any failure from wait-for-state. The logs in this bug clearly show that there is a main process which is exiting.

[14103.317598] init: whoopsie main process ended, respawning

This means that whoopsie itself is exiting for some reason and then being respawned. wait-for-state is in no way causing whoopsie to be restarted; the only thing wait-for-state is doing is causing enough of a delay in the pre-start that the respawn limit is never hit, causing this to loop indefinitely. But the real issue is whatever is causing whoopsie to exit in the first place.

Changed in upstart (Ubuntu):
importance: High → Wishlist
Changed in whoopsie (Ubuntu):
status: New → Triaged
assignee: nobody → Didier Roche (didrocks)
Revision history for this message
Steve Langasek (vorlon) wrote :

This is a regression introduced in the last whoopsie upload in vivid. Prior to this, the upstart job included deliberate handling in /etc/init/whoopsie.conf to ensure that if whoopsie were disabled in the config, the job would be stopped without triggering upstart's respawn behavior. Assigning this bug to the developer who made this change.

Revision history for this message
Michael Blennerhassett (mjblenner) wrote :

Looking at man 8 init, there is no job state "started", so having WAIT_STATE=started is exactly equal to WAIT_STATE=foobar, since they both don't exit.

So you may as well change it to something sensible.

Also, the reason that whoopsie exits as soon as it is respawned is that there is already another whoopsie process running, that either was started by something else (apport?), or upstart tost track of it.

Either way, fixing the wait-for-state (this patch or fixing whoopsie.conf), fixes the 30sec respawn bug.

Changed in whoopsie (Ubuntu):
importance: Undecided → High
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Michael, agree that 'started' is the wrong thing to look for and has probably caused a few confusing things through the years since wait-for-state has been in use. I think the patch should be merged.

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

@Steve: I think I understand what happens.
On package upgrade, if the job is disabled, then, there is an upstart .override file written to disable it and so, that behavior shouldn't be triggered. I guess this case happens currently if you disable through ubuntu-control-center after a new installation as I discovered that there is another library (another source package), using that file: whoopsie-preferences that can read/set values in this file.

However, I see that there is another value about report_metrics that we don't use for job enablement status.
I propose to do the following on ubuntu-control-center: report crashes -> add the override and call systemctl disable if installed, change whoopsie-preferences to be aware of those files and detect usptart/systemd for current values.

This should prevent whoopsie to loop when the job is disabled and only have the current enablement state in one place (through libwhoopsie-preferences).

I can set the other changes that you described as well to fix the upstart job, just following what you see as I'm in no way an upstart expert :)

Revision history for this message
Alan (alanjas) wrote :

Tail of dmesg..

[ 4517.664355] init: wait-for-state (whoopsienetwork-manager) main process (7481) terminated with status 100
[ 4517.720627] init: whoopsie main process (7492) terminated with status 1
[ 4517.720645] init: whoopsie main process ended, respawning

Revision history for this message
Chad Miller (cmiller) wrote :

@didrocks, I see this bug on my laptop. Ping me if you want testing or info.

Changed in whoopsie-preferences (Ubuntu):
assignee: nobody → Didier Roche (didrocks)
Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

Fixed, I added as well Michael's upstart job enhancement in this upload (even if that doesn't fix the real issue, still a nice addition). Didn't merge into upstream trunk as I don't have access right, but Brian would merge them back.

Changed in upstart:
status: Confirmed → Invalid
Changed in upstart (Ubuntu):
status: Confirmed → Invalid
Changed in whoopsie (Ubuntu):
status: Triaged → Fix Committed
Changed in whoopsie-preferences (Ubuntu):
status: New → Fix Committed
Changed in hundredpapercuts:
status: Confirmed → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package whoopsie-preferences - 0.14

---------------
whoopsie-preferences (0.14) vivid; urgency=medium

  * Now use service status (upstart and systemd) to notify if crash reporting
    is enabled (LP: #1412719)
  * Default file for whoopsie settings (only used for report_metrics currently)
    is now /etc/whoopsie (LP: #1431432)
  * debian/control: add Breaks to ensure we dep on the right whoopsie version
  * Update Standards-Version.
 -- Didier Roche <email address hidden> Thu, 12 Mar 2015 17:39:30 +0100

Changed in whoopsie-preferences (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package whoopsie - 0.2.47

---------------
whoopsie (0.2.47) vivid; urgency=medium

  [ Didier Roche ]
  * Remove /etc/default/whoopsie conffile and avoid upgrade prompt:
    (LP: #1431432)
    - debian/whoopsie.maintscript: get it removed on upgrade
    - debian/whoopsie.preinst: copy the optional report_metrics value if set
      by gnome-control-center capplet to /etc/whoopsie (not a conffile).
      (LP: #1412719)
      Refactor the logic for caching report_crashes as well.
    - debian/whoopsie.postinst: enhance the logic to ensure we don't have the
      service started after upgrade and keep it disable if it was disabled
    - Makefile, data/whoopsie: remove the conffile so that we don't ship it
  * Bump Standards-Version

  [ Michael Blennerhassett ]
  * Under upstart: Set the WAIT_STATE to get wait-for-state to do something
    useful. GOAL is also not needed.
 -- Didier Roche <email address hidden> Thu, 12 Mar 2015 17:21:58 +0100

Changed in whoopsie (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.