stdout from subprocesses missing from output.txt

As far as I can tell, this isn't a LAVA problem, there is no prompt printed after running "netcfg eth0 dhcp", i.e. it looks like the board hung. Is there something I'm missing?

Oops I gave a wrong link again.
Here is a more recent job which completed succesfully but no stdout logs printed in the log file:
https://validation.linaro.org/lava-server/scheduler/job/48199

But the corresponding logs are available for download under test run details:
https://validation.linaro.org/lava-server/dashboard/attachment/246168/view

It becomes to analyse what went wrong without timestamps.

I tested this and there was a slight error in context.run_command. I made a small change on my branch:

Still needs to be examined to see if there was a reason to not output the process stdout/stderr.

https://code.launchpad.net/~nick-schutt/lava-dispatcher/nick-1130814

nick@neptune:/data/linaro/bzr/lava-dispatcher/nick-1130814-mwhudson-subprocess-output$ bzr diff -c-1
=== modified file 'lava_dispatcher/context.py'
--- lava_dispatcher/context.py 2013-02-18 04:01:52 +0000
+++ lava_dispatcher/context.py 2013-03-08 04:08:58 +0000
@@ -130,11 +130,8 @@
         """run command 'command' with output going to output-dir if specified"""
         if isinstance(command, (str, unicode)):
             command = ['sh', '-c', command]
- output_txt = self.client.context.output.output_txt
- if self.output_txt:
- output_args = {'stdout': output_txt, 'stderr': subprocess.STDOUT}
- else:
- output_args = {}
+ output_txt = self.client.context.output.output_txt
+ output_args = {'stdout': output_txt, 'stderr': subprocess.STDOUT}
         logging.debug("Executing on host : '%r'" % command)
         if failok:
             rc = subprocess.call(command, **output_args)

Hey Nick, I am friendly kidnapping this bug because I had trouble with the current state of the fix and had to put some work on top of it. :-)

Is the bug fixed ?
I still cannot find the logs from CTS execution in this job:
http://validation.linaro.org/lava-server/scheduler/job/50456/log_file
where as the stdout logs for that process is present here:
http://validation.linaro.org/lava-server/dashboard/attachment/266443/download

Here is a job that generated logs for CTS and displayed along with dispatcher logs + serial logs:
https://validation.linaro.org/lava-server/scheduler/job/46327/log_file

Antonio is the process of rolling out the fixes to validation.linaro.org.
I'll check with him on the progress.

On 1 April 2013 07:02, vishal <email address hidden> wrote:

> Is the bug fixed ?
> I still cannot find the logs from CTS execution in this job:
> http://validation.linaro.org/lava-server/scheduler/job/50456/log_file
> where as the stdout logs for that process is present here:
>
> http://validation.linaro.org/lava-server/dashboard/attachment/266443/download
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1127288
>
> Title:
> stdout from subprocesses missing from output.txt
>
> Status in LAVA Dispatcher:
> Fix Released
>
> Bug description:
> Due to the recent changes in the way output is handled/shown in the
> scheduler, output from subprocesses no longer shows up in the
> "complete log" view of the scheduler.
>
> Original description:
>
> stdout logs are not getting printed in recent changes. It makes
> impossible to debug the issues. The logs were getting printed in jobs
> till Jan 28th and have stopped from jobs around feb 1. May be it came
> as part of 13.01 release.
>
> Here is the job where logs are printed:
> https://validation.linaro.org/lava-server/scheduler/job/46327/log_file
>
> Here is a job after few days where logs are not getting printed:
> https://validation.linaro.org/lava-server/scheduler/job/46664/log_file
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1127288/+subscriptions
>

--
Tyler Baker
Technical Architect, Automation & CI
Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

Antonio is working on it now, fix should be deployed shortly.

On 1 April 2013 07:06, vishal <email address hidden> wrote:

> Here is a job that generated logs for CTS and displayed along with
> dispatcher logs + serial logs:
> https://validation.linaro.org/lava-server/scheduler/job/46327/log_file
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Dispatcher.
> https://bugs.launchpad.net/bugs/1127288
>
> Title:
> stdout from subprocesses missing from output.txt
>
> Status in LAVA Dispatcher:
> Fix Released
>
> Bug description:
> Due to the recent changes in the way output is handled/shown in the
> scheduler, output from subprocesses no longer shows up in the
> "complete log" view of the scheduler.
>
> Original description:
>
> stdout logs are not getting printed in recent changes. It makes
> impossible to debug the issues. The logs were getting printed in jobs
> till Jan 28th and have stopped from jobs around feb 1. May be it came
> as part of 13.01 release.
>
> Here is the job where logs are printed:
> https://validation.linaro.org/lava-server/scheduler/job/46327/log_file
>
> Here is a job after few days where logs are not getting printed:
> https://validation.linaro.org/lava-server/scheduler/job/46664/log_file
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-dispatcher/+bug/1127288/+subscriptions
>

--
Tyler Baker
Technical Architect, Automation & CI
Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

Fixes have been rolled into production

On 1 April 2013 07:19, Tyler Baker <email address hidden> wrote:

> Antonio is working on it now, fix should be deployed shortly.
>
>
> On 1 April 2013 07:06, vishal <email address hidden> wrote:
>
>> Here is a job that generated logs for CTS and displayed along with
>> dispatcher logs + serial logs:
>> https://validation.linaro.org/lava-server/scheduler/job/46327/log_file
>>
>> --
>> You received this bug notification because you are a member of Linaro
>> Validation Team, which is subscribed to LAVA Dispatcher.
>> https://bugs.launchpad.net/bugs/1127288
>>
>> Title:
>> stdout from subprocesses missing from output.txt
>>
>> Status in LAVA Dispatcher:
>> Fix Released
>>
>> Bug description:
>> Due to the recent changes in the way output is handled/shown in the
>> scheduler, output from subprocesses no longer shows up in the
>> "complete log" view of the scheduler.
>>
>> Original description:
>>
>> stdout logs are not getting printed in recent changes. It makes
>> impossible to debug the issues. The logs were getting printed in jobs
>> till Jan 28th and have stopped from jobs around feb 1. May be it came
>> as part of 13.01 release.
>>
>> Here is the job where logs are printed:
>> https://validation.linaro.org/lava-server/scheduler/job/46327/log_file
>>
>> Here is a job after few days where logs are not getting printed:
>> https://validation.linaro.org/lava-server/scheduler/job/46664/log_file
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/lava-dispatcher/+bug/1127288/+subscriptions
>>
>
>
>
> --
> Tyler Baker
> Technical Architect, Automation & CI
> Linaro.org | Open source software for ARM SoCs
> Follow Linaro: http://www.facebook.com/pages/Linaro
> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog
>

--
Tyler Baker
Technical Architect, Automation & CI
Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

On Mon, Apr 01, 2013 at 02:19:59PM -0000, Tyler Baker wrote:
> Antonio is working on it now, fix should be deployed shortly.

Just did it.

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

Thanks guys!