> Hi Valentin,
> 
> Thanks for this review.
> 
> > So the case of lhapdf not being used should be better handled.
> 
> 
> We actually do not need to run with lhapdf, just to have the lhapdf (version6)
> available.

I know what I meant was that if I don't have lhapdf (or only lhapdf5) on my system, then systematics crashes with the following message:

UnboundLocalError : local variable 'lhapdf_libdir' referenced before assignment

(  File "/Users/valentin/Documents/Work/MG5/python_syscalc/madgraph/various/misc.py", line 1637, in import_python_lhapdf
    candidates=[dirname for dirname in os.listdir(lhapdf_libdir+'64') \)

It would be nice if this situation could be detected before the crash and if a message would pop up warning the user that systematics are bypassed because he doesn't have lhapdf and that once he will have installed it, he can still rerun systematics independently on his results already produced by using the interactive madevent interface.

> 
> > Also, in the run_card, the commentary of the entry
> > NNPDF23_nlo_as_0116_mc = sys_pdf # matching scales
> > is not correct
> 
> 
> fixed
>
> > It seems that putting the LHAID works as well as putting
> > directly the name of the pdf. Is this also true when using the old C++
> > syscalc?
> > The commentary of this entry should clarify this point.
> 
> This is not a valid input for SysCalc. So I will not mention it here.

Ok, and it would be indeed to complicated to explain in which situation the user can put an lhaid there, I agree. Too bad.

> > Also, when putting a PDF set in 'sys_pdf' that I don't have, the codes
> > doesn't try to download it (unless of course this is also the PDF set that
> > I want to use for my central value).
> > It would be a nice feature if it did, because as of now the code simply
> > crashes when doing the systematics.
> 
> Done. I also automatically load the central if the run was not performed with
> lhapdf.
> (knowing that nnpdf is likely to be slightly off due to their bug in lhapdf.)

Excellent, I tested it and it works.

> > The help of the command 'systematics' is nice. I would maybe rephrase the
> > message
> 
> Done
> 
> >    pdfset = lhapdf.mkPDF(int(data)).set()
> >  File "lhapdf.pyx", line 469, in lhapdf.mkPDF (lhapdf.cpp:7253)
> >  File "lhapdf.pyx", line 454, in lhapdf.mkPDF_lhaid (lhapdf.cpp:7044)
> > RuntimeError: Empty PDF file name given to Info::load
> 
> I do not reproduce this. Looks like either some PDF sets are missing or
> (worse) corrupted
> I have face as well some problem with some lhapdf set so I have use another
> method
> to load them and it fixes the problem that i faced.

Ok, now at NLO, I crash on:

AttributeError : 'Event' object has no attribute 'nloweight'
[...]
  File "/Users/valentin/Documents/Work/MG5/python_syscalc/madgraph/various/systematics.py", line 583, in get_nlo_wgt
    nloinfo = event.parse_nlo_weight()
  File "/Users/valentin/Documents/Work/MG5/python_syscalc/madgraph/various/lhe_parser.py", line 1134, in parse_nlo_weight
    return self.nloweight
AttributeError: 'Event' object has no attribute 'nloweight'

This happened both with and without PDF variations in my generation (I assume you require the user to having generated events with both variations turned on, it would be great if you could safe-check this).

By the way, additional general question:
If there is already a bunch of weights in the event, but only some are missing given the inputs provided to 'systematics', will you recompute all of them or do you automatically filter them out?
It's ok if you recompute all of them, but I was wondering if you had optimized this bit.

Cheers,

> 
> Cheers,
> 
> Olivier
> 
> 
> 
> 
> > On Aug 19, 2016, at 07:21, Valentin Hirschi <email address hidden>
> wrote:
> >
> > Hi Olivier,
> >
> > a)
> >
> > I tried running everything out of the box for p p > e+ ve and python itself
> > crashed (so the error was not handled), because I guess one cannot use your
> > py-syscalc without lhapdf. I got
> >
> > terminate called after throwing an instance of 'LHAPDF::ReadError'
> >  what():  Info file not found for PDF set 'NNPDF23_lo_as_0130_qed'
> >
> > So the case of lhapdf not being used should be better handled.
> >
> > Also, in the run_card, the commentary of the entry
> >
> > NNPDF23_nlo_as_0116_mc = sys_pdf # matching scales
> >
> > is not correct. And it is not clear what the user is supposed to or can put
> > for this field. It seems that putting the LHAID works as well as putting
> > directly the name of the pdf. Is this also true when using the old C++
> > syscalc?
> > The commentary of this entry should clarify this point.
> >
> > Also, when putting a PDF set in 'sys_pdf' that I don't have, the codes
> > doesn't try to download it (unless of course this is also the PDF set that
> > I want to use for my central value).
> > It would be a nice feature if it did, because as of now the code simply
> > crashes when doing the systematics.
> > There should at least be a check that the PDF is available.
> >
> > This comment applies both for LO and NLO.
> >
> > b)
> >
> > The help of the command 'systematics' is nice. I would maybe rephrase the
> > message (which is tricky for users not used to that)
> >
> > --together=mur,muf,dyn # which parameter should we vary together to have
> > all uncorrelated combination of the variations
> > to
> >
> > 'lists the parameter that must be varied simultaneously so as to compute
> > the weights for all combinations of their variations'.
> >
> > but up to you.
> >
> > c)
> >
> > At NLO, I tried
> >
> > systematics run_02 --pdf=21100 --together=mur,muf
> >
> > on p p > e+ ve [QCD] and it crashed with:
> >
> > WARNING: program
> > /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python
> .app/Contents/MacOS/Python
> > /Users/valentin/Documents/Work/MG5/python_syscalc/PROCNLO_loop_sm_0/bin/inte
> rnal/systematics.py
> > events.lhe.gz ./tmp_1_events.lhe.gz --pdf=21100 --together=mur,muf
> > --start_event=2500 --stop_event=5000 --result=./log_sys_1.txt
> > --lhapdf_config=/Users/valentin/Documents/HEP_softs/LHAPDF/LHAPDF-6.1.5_inst
> allation/bin/lhapdf-config
> > launch ends with non zero status: 1. Stop all computation
> >
> > There seems to be a 'launch' too many in the command no?
> >
> > I tried to run it by hand without the 'launch', but I got:
> >
> >    pdfset = lhapdf.mkPDF(int(data)).set()
> >  File "lhapdf.pyx", line 469, in lhapdf.mkPDF (lhapdf.cpp:7253)
> >  File "lhapdf.pyx", line 454, in lhapdf.mkPDF_lhaid (lhapdf.cpp:7044)
> > RuntimeError: Empty PDF file name given to Info::load
> >
> >
> >
> >
> >
> > On Thu, Aug 11, 2016 at 4:55 PM, Olivier Mattelaer <
> > <email address hidden>> wrote:
> >
> >> Olivier Mattelaer has proposed merging
> lp:~maddevelopers/mg5amcnlo/python_syscalc
> >> into lp:~maddevelopers/mg5amcnlo/2.5.0.
> >>
> >> Requested reviews:
> >>  Eleni Vryonidou (evryonidou)
> >>  Rikkert Frederix (frederix)
> >>  Valentin Hirschi (valentin-hirschi)
> >>
> >> For more details, see:
> >> https://code.launchpad.net/~maddevelopers/mg5amcnlo/
> >> python_syscalc/+merge/302735
> >>
> >> This version allows to compute systematics error band on both LO and NLO
> >> sample.
> >>
> >> For LO code, this is run automatically if 'use_syst' is set on True in the
> >> run_card.
> >> For NLO code, this need to be called manually via the command 'systematics
> >> RUN_XX OPTIONS'
> >> The sample need to be generated with the options 'store_rwgt_info=True'
> >>
> >> The computation can also be called outside of madgraph via the code
> >> madgraph/various/systematics.py INPUT.LHE[.GZ] OUTPUT.LHE[.GZ] OPTIONS
> >>
> >> Here is the help of the systematics function such that you can see the
> >> options/make comment on this help if this is not clear/ comment on the
> >> default value.
> >>
> >> INFO: syntax: systematics RUN_NAME [OUTPUT] [options]
> >> INFO: -- Run the systematics run on the run_name run.
> >> INFO:    RUN_NAME can be a path to a lhef file.
> >> INFO:    OUTPUT can be the path to the output lhe file. we overwritte the
> >> input file otherwise
> >> INFO:
> >> INFO: options: (value written are default)
> >> INFO:
> >> INFO:    --mur=0.5,1,2    # specify the values for renormalisation scale
> >> variation
> >> INFO:    --muf=0.5,1,2    # specify the values for factorisation scale
> >> variation
> >> INFO:    --alps=1         # specify the values for MLM emission scale
> >> variation
> >> INFO:    --dyn=-1,1,2,3,4 # specify the dynamical schemes to use.
> >> INFO:                     #   -1 is the one used by the sample.
> >> INFO:                     #   > 0 correspond to options of
> >> dynamical_scale_choice of the run_card.
> >> INFO:    --pdf=errorset   # specify the pdfs to use for pdf variation.
> >> (see below)
> >> INFO:    --together=mur,muf,dyn # which parameter should we vary together
> >> to have all uncorrelated combination of the variations
> >> INFO:    --from_card      # use the information from the run_card (LO
> >> only).
> >> INFO:
> >> INFO:    Allowed value for the pdf options:
> >> INFO:        central  : Do not perform any pdf variation
> >> INFO:        errorset : runs over the  errorset
> >> INFO:        244800   : runs over the associated set and his errorset
> >> INFO:        244800@0 : runs over the associated set ONLY
> >> INFO:        CT10     : runs over the associated set and his errorset
> >> INFO:        CT10@0     : runs over the associated set ONLY
> >> INFO:        CT10@X     : runs over the Xth set of the associated error
> >> set
> >> INFO:        XX,YY,ZZ : runs over the sets for XX,YY,ZZ (those three
> >> follows above syntax)
> >>
> >>
> >> Cheers,
> >>
> >> Olivier
> >>
> >> --
> >> You are requested to review the proposed merge of
> >> lp:~maddevelopers/mg5amcnlo/python_syscalc into
> >> lp:~maddevelopers/mg5amcnlo/2.5.0.
> >>
> >> === modified file 'MadSpin/decay.py'
> >> --- MadSpin/decay.py    2016-06-30 21:49:42 +0000
> >> +++ MadSpin/decay.py    2016-08-11 23:54:34 +0000
> >> @@ -1637,7 +1637,7 @@
> >>                 (' '.join([str(i) for i in opts['particles']]),
> >>                  ' '.join('--%s=%s' % (key,value) for (key,value) in
> >> opts.items()
> >>                         if key not in ['model', 'force', 'particles'] and
> >> value))
> >> -        cmd.exec_cmd('import model %s' % model.get('name'))
> >> +        cmd.exec_cmd('import model %s' % model.get('modelpath+
> >> restriction'))
> >> #        cmd._curr_model = model
> >> #        cmd._curr_fortran_model = helas_call_writers.
> >> FortranUFOHelasCallWriter(model)
> >>         cmd.exec_cmd(line)
> >>
> >> === modified file 'MadSpin/interface_madspin.py'
> >> --- MadSpin/interface_madspin.py        2016-07-02 10:55:33 +0000
> >> +++ MadSpin/interface_madspin.py        2016-08-11 23:54:34 +0000
> >> @@ -209,6 +209,7 @@
> >>             mg_names = True
> >>         model_name = self.banner.get('proc_card', 'model')
> >>         if model_name:
> >> +            model_name = os.path.expanduser(model_name)
> >>             self.load_model(model_name, mg_names, complex_mass)
> >>         else:
> >>             raise self.InvalidCmd('Only UFO model can be loaded in
> >> MadSpin.')
> >> @@ -242,7 +243,7 @@
> >>     def import_model(self, args):
> >>         """syntax: import model NAME CARD_PATH
> >>             args didn't include import model"""
> >> -
> >> +
> >>         bypass_check = False
> >>         if '--bypass_check' in args:
> >>             args.remove('--bypass_check')
> >> @@ -700,8 +701,7 @@
> >>         #   of event to generate for each type of particle.
> >>         # 2. Generate the events requested
> >>         # 3. perform the merge of the events.
> >> -        #    if not enough events. re-generate the missing one.
> >> -
> >> +        #    if not enough events. re-generate the missing one.
> >>         # First define an utility function for generating events when
> >> needed
> >>         def generate_events(pdg, nb_event, mg5, restrict_file=None,
> >> cumul=False):
> >>             """generate new events for this particle
> >> @@ -830,8 +830,9 @@
> >>         # 2. Generate the events requested
> >>         with misc.MuteLogger(["madgraph", "madevent", "ALOHA",
> >> "cmdprint"], [50,50,50,50]):
> >>             mg5 = self.mg5cmd
> >> -            modelpath = self.model.get('modelpath+restriction')
> >> -            mg5.exec_cmd("import model %s" % modelpath)
> >> +            if not self.model:
> >> +                modelpath = self.model.get('modelpath+restriction')
> >> +                mg5.exec_cmd("import model %s" % modelpath)
> >>             to_event = {}
> >>             for pdg, nb_needed in to_decay.items():
> >>                 #check if a splitting is needed
> >> @@ -889,6 +890,12 @@
> >>             else:
> >>                 raise self.InvalidCmd("The bridge mode of MadSpin does
> >> not support event files where events do not *all* share the same set of
> >> final state particles to be decayed.")
> >>         self.branching_ratio = br
> >> +        self.efficiency = 1
> >> +        self.cross, self.error = self.banner.get_cross(witherror=True)
> >> +        self.cross *= br
> >> +        self.error *= br
> >> +
> >> +
> >>         # modify the cross-section in the init block of the banner
> >>         self.banner.scale_init_cross(self.branching_ratio)
> >>
> >> @@ -991,7 +998,6 @@
> >>     def load_model(self, name, use_mg_default, complex_mass=False):
> >>         """load the model"""
> >>
> >> -
> >>         loop = False
> >>         #if (name.startswith('loop_')):
> >>         #    logger.info("The model in the banner is %s" % name)
> >>
> >> === modified file 'Template/LO/SubProcesses/reweight.f'
> >> --- Template/LO/SubProcesses/reweight.f 2016-06-07 09:33:55 +0000
> >> +++ Template/LO/SubProcesses/reweight.f 2016-08-11 23:54:34 +0000
> >> @@ -994,21 +994,21 @@
> >>             pt2ijcl(nexternal-2)=q2fact(1)
> >>             if(nexternal.gt.3) pt2ijcl(nexternal-3)=q2fact(1)
> >>          else
> >> -            q2fact(1)=pt2ijcl(nexternal-2)
> >> -            q2fact(2)=q2fact(1)
> >> +            q2fact(1)=scalefact**2*pt2ijcl(nexternal-2)
> >> +            q2fact(2)=scalefact**2*q2fact(1)
> >>          endif
> >>       elseif(jcentral(1).eq.0)then
> >> -            q2fact(1) = pt2ijcl(jfirst(1))
> >> +            q2fact(1) = scalefact**2*pt2ijcl(jfirst(1))
> >>       elseif(jcentral(2).eq.0)then
> >> -            q2fact(2) = pt2ijcl(jfirst(2))
> >> +            q2fact(2) = scalefact**2*pt2ijcl(jfirst(2))
> >>       elseif(ickkw.eq.2.or.(pdfwgt.and.ickkw.gt.0))then
> >> c     Total pdf weight is f1(x1,pt2E)*fj(x1*z,Q)/fj(x1*z,pt2E)
> >> c     f1(x1,pt2E) is given by DSIG, just need to set scale.
> >> c     Use the minimum scale found for fact scale in ME
> >>          if(jlast(1).gt.0.and.jfirst(1).le.jlast(1))
> >> -     $        q2fact(1)=min(pt2ijcl(jfirst(1)),q2fact(1))
> >> +     $        q2fact(1)=scalefact**2*min(pt2ijcl(jfirst(1)),q2fact(1))
> >>          if(jlast(2).gt.0.and.jfirst(2).le.jlast(2))
> >> -     $        q2fact(2)=min(pt2ijcl(jfirst(2)),q2fact(2))
> >> +     $        q2fact(2)=scalefact**2*min(pt2ijcl(jfirst(2)),q2fact(2))
> >>       endif
> >>
> >> c     Check that factorization scale is >= 2 GeV
> >>
> >> === modified file 'Template/LO/bin/internal/store4grid'
> >> --- Template/LO/bin/internal/store4grid 2012-09-05 07:27:45 +0000
> >> +++ Template/LO/bin/internal/store4grid 2016-08-11 23:54:34 +0000
> >> @@ -32,14 +32,11 @@
> >> if [[ -e default_results.dat ]]; then
> >>     rm -f default_results.dat
> >> fi
> >> -cp -f $1_results.dat default_results.dat
> >> for i in `cat subproc.mg` ; do
> >>     cd $i
> >> -    echo $i
> >>     if [[ -e default_results.dat ]]; then
> >>        rm -f default_results.dat
> >>     fi
> >> -    cp -f $1_results.dat default_results.dat
> >>     for k in G* ; do
> >>        if [[ ! -d $k ]]; then
> >>            continue
> >>
> >> === modified file 'Template/NLO/MCatNLO/PY8Analyzer/py8an_HwU_pp_h.f'
> >> --- Template/NLO/MCatNLO/PY8Analyzer/py8an_HwU_pp_h.f   2016-02-23
> >> 10:48:46 +0000
> >> +++ Template/NLO/MCatNLO/PY8Analyzer/py8an_HwU_pp_h.f   2016-08-11
> >> 23:54:34 +0000
> >> @@ -159,7 +159,7 @@
> >>       common/cww/ww
> >> c
> >>       if(nnn.eq.0)ww(1)=1d0
> >> -      do i=,nnn
> >> +      do i=1,nnn
> >>          ww(i)=xww(i)
> >>       enddo
> >> c
> >>
> >> === modified file 'UpdateNotes.txt'
> >> --- UpdateNotes.txt     2016-06-28 12:22:14 +0000
> >> +++ UpdateNotes.txt     2016-08-11 23:54:34 +0000
> >> @@ -1,6 +1,12 @@
> >> Update notes for MadGraph5_aMC@NLO (in reverse time order)
> >>
> >> -2.4.3 (xx/xx/xx)
> >> +
> >> +X.X.X (xx/xx/xx)
> >> +        OM: NLO Re-weighting works in multi-core
> >> +
> >> +
> >> +
> >> +2.4.3 (01/08/16)
> >>         OM: Reduce the amount of log file/output generated for LO run
> >> (output can use up to three times less output).
> >>         OM: For the LO combination of events (unweighting) pass to the
> >> method previously used for loop-induced.
> >>             This method is faster and requires less I/O operation.
> >> @@ -12,7 +18,9 @@
> >>             were associated to each of those processes.
> >>        RF: For NLO process generation, fix a problem with turning on PDF
> >> reweighting with sets that have only a
> >>            single member. Also, allow for reweighting with up to 25 PDF
> >> sets (and their error members) for a single run.
> >> -
> >> +        OM: Fixing bug allowing to specify a UFO model by his full path
> >> for NLO computation (thanks Zachary Marschal).
> >> +        OM: Fixing bug in LO re-weighting in case of helicity by helicity
> >> re-weighting. Now the events is boost back in
> >> +            the center of mass frame to ensure consistency with the
> >> helicity definition.
> >>
> >> 2.4.2 (10/06/16)
> >>         OM: fix a compilation problem for non standard gfortran system
> >>
> >> === modified file 'VERSION'
> >> --- VERSION     2016-06-10 16:37:06 +0000
> >> +++ VERSION     2016-08-11 23:54:34 +0000
> >> @@ -1,5 +1,6 @@
> >> -version = 2.4.2
> >> -date = 2016-06-10
> >> +version = new_sys
> >> +date = 2016-08-08
> >> +
> >>
> >>
> >>
> >>
> >> === modified file 'aloha/aloha_writers.py'
> >> --- aloha/aloha_writers.py      2015-10-22 18:57:33 +0000
> >> +++ aloha/aloha_writers.py      2016-08-11 23:54:34 +0000
> >> @@ -1,9 +1,11 @@
> >> try:
> >>     import madgraph.iolibs.file_writers as writers
> >>     import madgraph.various.q_polynomial as q_polynomial
> >> +    import madgraph.various.misc as misc
> >> except Exception:
> >>     import aloha.file_writers as writers
> >>     import aloha.q_polynomial as q_polynomial
> >> +    import aloha.misc as misc
> >>
> >> import aloha
> >> import aloha.aloha_lib as aloha_lib
> >> @@ -330,7 +332,11 @@
> >>         file_str = StringIO()
> >>
> >>         if prefactor and obj.prefactor != 1:
> >> -            file_str.write(self.change_number_format(obj.prefactor))
> >> +            formatted = self.change_number_format(obj.prefactor)
> >> +            if formatted.startswith(('+','-')):
> >> +                file_str.write('(%s)' % formatted)
> >> +            else:
> >> +                file_str.write(formatted)
> >>             file_str.write('*(')
> >>         else:
> >>             file_str.write('(')
> >> @@ -756,7 +762,11 @@
> >>
> >>         if not self.offshell:
> >>             if coup_name == 'COUP':
> >> -                out.write(' vertex = COUP*%s\n' %
> >> self.write_obj(numerator.get_rep([0])))
> >> +                formatted = self.write_obj(numerator.get_rep([0]))
> >> +                if formatted.startswith(('+','-')):
> >> +                    out.write(' vertex = COUP*(%s)\n' % formatted)
> >> +                else:
> >> +                    out.write(' vertex = COUP*%s\n' % formatted)
> >>             else:
> >>                 out.write(' vertex = %s\n' % self.write_obj(numerator.get_
> >> rep([0])))
> >>         else:
> >> @@ -790,9 +800,12 @@
> >>                     coeff = ''
> >>             to_order = {}
> >>             for ind in numerator.listindices():
> >> +                formatted = self.write_obj(numerator.get_rep(ind))
> >> +                if formatted.startswith(('+','-')):
> >> +                    formatted = '(%s)*%s' % tuple(formatted.split('*',1))
> >>                 to_order[self.pass_to_HELAS(ind)] = \
> >>                         '    %s(%d)= %s%s\n' % (self.outname,
> >> self.pass_to_HELAS(ind)+1,
> >> -                        coeff, self.write_obj(numerator.get_rep(ind)))
> >> +                        coeff, formatted)
> >>             key = to_order.keys()
> >>             key.sort()
> >>             for i in key:
> >>
> >> === modified file 'madgraph/core/base_objects.py'
> >> --- madgraph/core/base_objects.py       2016-07-20 22:43:24 +0000
> >> +++ madgraph/core/base_objects.py       2016-08-11 23:54:34 +0000
> >> @@ -1145,6 +1145,7 @@
> >>         if name == 'modelpath':
> >>             modeldir = self.get('version_tag').rsplit('##',1)[0]
> >>             if os.path.exists(modeldir):
> >> +                modeldir = os.path.expanduser(modeldir)
> >>                 return modeldir
> >>             else:
> >>                 raise Exception, "path %s not valid anymore." % modeldir
> >> @@ -1160,6 +1161,7 @@
> >>                 raise Exception, "path %s not valid anymore" % modeldir
> >>             modeldir = os.path.dirname(modeldir)
> >>             modeldir = pjoin(modeldir, modelname)
> >> +            modeldir = os.path.expanduser(modeldir)
> >>             return modeldir
> >>         elif name == 'restrict_name':
> >>             modeldir = self.get('version_tag').rsplit('##',1)[0]
> >>
> >> === modified file 'madgraph/interface/amcatnlo_run_interface.py'
> >> --- madgraph/interface/amcatnlo_run_interface.py        2016-07-02
> >> 20:34:59 +0000
> >> +++ madgraph/interface/amcatnlo_run_interface.py        2016-08-11
> >> 23:54:34 +0000
> >> @@ -3267,98 +3267,6 @@
> >>         self.update_status('Run complete', level='shower',
> >> update_results=True)
> >>
> >>
> >> -    ############################################################
> >> ################
> >> -    def set_run_name(self, name, tag=None, level='parton',
> >> reload_card=False):
> >> -        """define the run name, the run_tag, the banner and the
> >> results."""
> >> -
> >> -        # when are we force to change the tag new_run:previous run
> >> requiring changes
> >> -        upgrade_tag = {'parton': ['parton','pythia','pgs','
> >> delphes','shower'],
> >> -                       'pythia': ['pythia','pgs','delphes'],
> >> -                       'shower': ['shower'],
> >> -                       'pgs': ['pgs'],
> >> -                       'delphes':['delphes'],
> >> -                       'plot':[]}
> >> -
> >> -
> >> -
> >> -        if name == self.run_name:
> >> -            if reload_card:
> >> -                run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> -                self.run_card = banner_mod.RunCardNLO(run_card)
> >> -
> >> -            #check if we need to change the tag
> >> -            if tag:
> >> -                self.run_card['run_tag'] = tag
> >> -                self.run_tag = tag
> >> -                self.results.add_run(self.run_name, self.run_card)
> >> -            else:
> >> -                for tag in upgrade_tag[level]:
> >> -                    if getattr(self.results[self.run_name][-1], tag):
> >> -                        tag = self.get_available_tag()
> >> -                        self.run_card['run_tag'] = tag
> >> -                        self.run_tag = tag
> >> -                        self.results.add_run(self.run_name,
> >> self.run_card)
> >> -                        break
> >> -            return # Nothing to do anymore
> >> -
> >> -        # save/clean previous run
> >> -        if self.run_name:
> >> -            self.store_result()
> >> -        # store new name
> >> -        self.run_name = name
> >> -
> >> -        # Read run_card
> >> -        run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> -        self.run_card = banner_mod.RunCardNLO(run_card)
> >> -
> >> -        new_tag = False
> >> -        # First call for this run -> set the banner
> >> -        self.banner = banner_mod.recover_banner(self.results, level,
> >> self.run_name, tag)
> >> -        if tag:
> >> -            self.run_card['run_tag'] = tag
> >> -            new_tag = True
> >> -        elif not self.run_name in self.results and level =='parton':
> >> -            pass # No results yet, so current tag is fine
> >> -        elif not self.run_name in self.results:
> >> -            #This is only for case when you want to trick the interface
> >> -            logger.warning('Trying to run data on unknown run.')
> >> -            self.results.add_run(name, self.run_card)
> >> -            self.results.update('add run %s' % name, 'all', makehtml=True)
> >> -        else:
> >> -            for tag in upgrade_tag[level]:
> >> -
> >> -                if getattr(self.results[self.run_name][-1], tag):
> >> -                    # LEVEL is already define in the last tag -> need to
> >> switch tag
> >> -                    tag = self.get_available_tag()
> >> -                    self.run_card['run_tag'] = tag
> >> -                    new_tag = True
> >> -                    break
> >> -            if not new_tag:
> >> -                # We can add the results to the current run
> >> -                tag = self.results[self.run_name][-1]['tag']
> >> -                self.run_card['run_tag'] = tag # ensure that run_tag is
> >> correct
> >> -
> >> -
> >> -        if name in self.results and not new_tag:
> >> -            self.results.def_current(self.run_name)
> >> -        else:
> >> -            self.results.add_run(self.run_name, self.run_card)
> >> -
> >> -        self.run_tag = self.run_card['run_tag']
> >> -
> >> -        # Return the tag of the previous run having the required data for
> >> this
> >> -        # tag/run to working wel.
> >> -        if level == 'parton':
> >> -            return
> >> -        elif level == 'pythia':
> >> -            return self.results[self.run_name][0]['tag']
> >> -        else:
> >> -            for i in range(-1,-len(self.results[self.run_name])-1,-1):
> >> -                tagRun = self.results[self.run_name][i]
> >> -                if tagRun.pythia:
> >> -                    return tagRun['tag']
> >> -
> >> -
> >>     def store_result(self):
> >>         """ tar the pythia results. This is done when we are quite sure
> >> that
> >>         the pythia output will not be use anymore """
> >> @@ -4898,9 +4806,9 @@
> >>             if '--web' in args:
> >>                 i = args.index('--web')
> >>                 args.pop(i)
> >> -                cmd_line =  aMCatNLOCmd(force_run=True)
> >> +                cmd_line =  aMCatNLOCmd(me_dir=os.path.
> >> dirname(root_path),force_run=True)
> >>             else:
> >> -                cmd_line =  aMCatNLOCmdShell(force_run=True)
> >> +                cmd_line =  aMCatNLOCmdShell(me_dir=os.
> >> path.dirname(root_path),force_run=True)
> >>
> >>             if not hasattr(cmd_line, 'do_%s' % args[0]):
> >>                 if parser_error:
> >>
> >> === modified file 'madgraph/interface/common_run_interface.py'
> >> --- madgraph/interface/common_run_interface.py  2016-07-01 21:32:02 +0000
> >> +++ madgraph/interface/common_run_interface.py  2016-08-11 23:54:34 +0000
> >> @@ -68,6 +68,7 @@
> >>     import internal.files as files
> >>     import internal.save_load_object as save_load_object
> >>     import internal.gen_crossxhtml as gen_crossxhtml
> >> +    import internal.lhe_parser as lhe_parser
> >>     from internal import InvalidCmd, MadGraph5Error
> >>     MADEVENT=True
> >> else:
> >> @@ -78,6 +79,7 @@
> >>     import madgraph.various.misc as misc
> >>     import madgraph.iolibs.files as files
> >>     import madgraph.various.cluster as cluster
> >> +    import madgraph.various.lhe_parser as lhe_parser
> >>     import madgraph.iolibs.save_load_object as save_load_object
> >>     import madgraph.madevent.gen_crossxhtml as gen_crossxhtml
> >>     import models.check_param_card as check_param_card
> >> @@ -450,7 +452,6 @@
> >>         """return the path to the output events
> >>         """
> >>
> >> -
> >>         if self.mode == 'madevent':
> >>             possible_path = [
> >>                 pjoin(self.me_dir,'Events', run_name,
> >> 'unweighted_events.lhe.gz'),
> >> @@ -465,7 +466,10 @@
> >>                 correct_path = path
> >>                 break
> >>         else:
> >> -            raise self.InvalidCmd('No events file corresponding to %s
> >> run. ' % run_name)
> >> +            if os.path.exists(run_name):
> >> +                correct_path = run_name
> >> +            else:
> >> +                raise self.InvalidCmd('No events file corresponding to %s
> >> run. ' % run_name)
> >>         return correct_path
> >>
> >>
> >> @@ -619,6 +623,101 @@
> >>
> >>         return args
> >>
> >> +    ############################################################
> >> ################
> >> +    def set_run_name(self, name, tag=None, level='parton',
> >> reload_card=False,
> >> +                     allow_new_tag=True):
> >> +        """define the run name, the run_tag, the banner and the
> >> results."""
> >> +
> >> +        # when are we force to change the tag new_run:previous run
> >> requiring changes
> >> +        upgrade_tag = {'parton': ['parton','pythia','pgs','delphes'],
> >> +                       'pythia': ['pythia','pgs','delphes'],
> >> +                       'pgs': ['pgs'],
> >> +                       'delphes':['delphes'],
> >> +                       'plot':[],
> >> +                       'syscalc':[]}
> >> +
> >> +
> >> +
> >> +        if name == self.run_name:
> >> +            if reload_card:
> >> +                run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> +                self.run_card = banner_mod.RunCard(run_card)
> >> +
> >> +            #check if we need to change the tag
> >> +            if tag:
> >> +                self.run_card['run_tag'] = tag
> >> +                self.run_tag = tag
> >> +                self.results.add_run(self.run_name, self.run_card)
> >> +            else:
> >> +                for tag in upgrade_tag[level]:
> >> +                    if getattr(self.results[self.run_name][-1], tag):
> >> +                        tag = self.get_available_tag()
> >> +                        self.run_card['run_tag'] = tag
> >> +                        self.run_tag = tag
> >> +                        self.results.add_run(self.run_name,
> >> self.run_card)
> >> +                        break
> >> +            return # Nothing to do anymore
> >> +
> >> +        # save/clean previous run
> >> +        if self.run_name:
> >> +            self.store_result()
> >> +        # store new name
> >> +        self.run_name = name
> >> +
> >> +        new_tag = False
> >> +        # First call for this run -> set the banner
> >> +        self.banner = banner_mod.recover_banner(self.results, level,
> >> name)
> >> +        if 'mgruncard' in self.banner:
> >> +            self.run_card = self.banner.charge_card('run_card')
> >> +        else:
> >> +            # Read run_card
> >> +            run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> +            self.run_card = banner_mod.RunCard(run_card)
> >> +
> >> +        if tag:
> >> +            self.run_card['run_tag'] = tag
> >> +            new_tag = True
> >> +        elif not self.run_name in self.results and level =='parton':
> >> +            pass # No results yet, so current tag is fine
> >> +        elif not self.run_name in self.results:
> >> +            #This is only for case when you want to trick the interface
> >> +            logger.warning('Trying to run data on unknown run.')
> >> +            self.results.add_run(name, self.run_card)
> >> +            self.results.update('add run %s' % name, 'all',
> >> makehtml=False)
> >> +        else:
> >> +            for tag in upgrade_tag[level]:
> >> +
> >> +                if getattr(self.results[self.run_name][-1], tag):
> >> +                    # LEVEL is already define in the last tag -> need to
> >> switch tag
> >> +                    tag = self.get_available_tag()
> >> +                    self.run_card['run_tag'] = tag
> >> +                    new_tag = True
> >> +                    break
> >> +            if not new_tag:
> >> +                # We can add the results to the current run
> >> +                tag = self.results[self.run_name][-1]['tag']
> >> +                self.run_card['run_tag'] = tag # ensure that run_tag is
> >> correct
> >> +
> >> +        if allow_new_tag and (name in self.results and not new_tag):
> >> +            self.results.def_current(self.run_name)
> >> +        else:
> >> +            self.results.add_run(self.run_name, self.run_card)
> >> +
> >> +        self.run_tag = self.run_card['run_tag']
> >> +
> >> +        # Return the tag of the previous run having the required data for
> >> this
> >> +        # tag/run to working wel.
> >> +        if level == 'parton':
> >> +            return
> >> +        elif level == 'pythia':
> >> +            return self.results[self.run_name][0]['tag']
> >> +        else:
> >> +            for i in range(-1,-len(self.results[self.run_name])-1,-1):
> >> +                tagRun = self.results[self.run_name][i]
> >> +                if tagRun.pythia:
> >> +                    return tagRun['tag']
> >> +
> >> +
> >>     @misc.multiple_try(nb_try=5, sleep=2)
> >>     def load_results_db(self):
> >>         """load the current results status"""
> >> @@ -1074,7 +1173,226 @@
> >>         """Dummy routine, to be overwritten by daughter classes"""
> >>
> >>         pass
> >> -
> >> +
> >> +    ############################################################
> >> ################
> >> +    def help_systematics(self):
> >> +        """help for systematics command"""
> >> +        logger.info("syntax: systematics RUN_NAME [OUTPUT]
> >> [options]",'$MG:color:BLACK')
> >> +        logger.info("-- Run the systematics run on the run_name run.")
> >> +        logger.info("   RUN_NAME can be a path to a lhef file.")
> >> +        logger.info("   OUTPUT can be the path to the output lhe file.
> >> we overwritte the input file otherwise")
> >> +        logger.info("")
> >> +        logger.info("options: (value written are default)",
> >> '$MG:color:BLACK')
> >> +        logger.info("")
> >> +        logger.info("   --mur=0.5,1,2 # specify the values for
> >> renormalisation scale variation")
> >> +        logger.info("   --muf=0.5,1,2 # specify the values for
> >> factorisation scale variation")
> >> +        logger.info("   --alps=1      # specify the values for MLM
> >> emission scale variation")
> >> +        logger.info("   --dyn=-1      # specify the dynamical schemes to
> >> use.")
> >> +        logger.info("                 #   -1 is the one used by the
> >> sample.")
> >> +        logger.info("                 #   > 0 correspond to options of
> >> dynamical_scale_choice of the run_card.")
> >> +        logger.info("   --pdf=errorset# specify the pdfs to use for pdf
> >> variation.")
> >> +        logger.info("   --together=mur,muf # which parameter should we
> >> vary together to have all uncorrelated combination of the variations")
> >> +        logger.info("   --from_card        # use the information from
> >> the run_card.")
> >> +
> >> +
> >> +    def complete_systematics(self, text, line, begidx, endidx):
> >> +        """auto completion for the systematics command"""
> >> +
> >> +        args = self.split_arg(line[0:begidx], error=False)
> >> +        options = ['--mur=', '--muf=', '--pdf=', '--dyn=','--alps=','--
> together=','--from_card
> >> ']
> >> +
> >> +        if len(args) == 1 and os.path.sep not in text:
> >> +            #return valid run_name
> >> +            data = misc.glob(pjoin('*','*events.lhe*'),
> >> pjoin(self.me_dir, 'Events'))
> >> +            data = [n.rsplit('/',2)[1] for n in data]
> >> +            return  self.list_completion(text, data, line)
> >> +        elif len(args)==1:
> >> +            #logger.warning('1args')
> >> +            return self.path_completion(text,
> >> +                                        os.path.join('.',*[a for a in
> >> args \
> >> +                                                    if
> >> a.endswith(os.path.sep)]))
> >> +        elif len(args)==2 and os.path.sep in args[1]:
> >> +            #logger.warning('2args %s', args[1])
> >> +            return self.path_completion(text, '.')
> >> +
> >> +        elif not line.endswith(tuple(options)):
> >> +            return self.list_completion(text, options)
> >> +
> >> +
> >> +    ############################################################
> >> ################
> >> +    def do_systematics(self, line):
> >> +        """ syntax is 'systematics [INPUT [OUTPUT]] OPTIONS'
> >> +            --mur=0.5,1,2
> >> +            --muf=0.5,1,2
> >> +            --alps=1
> >> +            --dyn=-1
> >> +            --together=mur,muf #can be repeated
> >> +
> >> +            #special options
> >> +            --from_card=
> >> +        """
> >> +
> >> +        self.update_status('Running Systematic computation',
> >> level='parton')
> >> +        args = self.split_arg(line)
> >> +        #split arguments and option
> >> +        opts= []
> >> +        args = [a for a in args if not a.startswith('-') or
> >> opts.append(a)]
> >> +
> >> +        #check sanity of options
> >> +        if any(not o.startswith(('--mur=', '--muf=',
> >> '--alps=','--dyn=','--together=','--from_card','--pdf='))
> >> +                for o in opts):
> >> +            raise self.InvalidCmd, "command systematics called with
> >> invalid option syntax. Please retry."
> >> +
> >> +        # check that we have define the input
> >> +        if len(args) == 0:
> >> +            if self.run_name:
> >> +                args[0] = self.run_name
> >> +            else:
> >> +                raise self.InvalidCmd, 'no default run. Please specify
> >> the run_name'
> >> +
> >> +        # always pass to a path + get the event size
> >> +        result_file= sys.stdout
> >> +        if not os.path.sep in args[0]:
> >> +            path = [pjoin(self.me_dir, 'Events', args[0],
> >> 'unweighted_events.lhe.gz'),
> >> +                    pjoin(self.me_dir, 'Events', args[0],
> >> 'unweighted_events.lhe'),
> >> +                    pjoin(self.me_dir, 'Events', args[0],
> >> 'events.lhe.gz'),
> >> +                    pjoin(self.me_dir, 'Events', args[0], 'events.lhe')]
> >> +
> >> +            for p in path:
> >> +                if os.path.exists(p):
> >> +                    nb_event = self.results[args[0]].get_
> >> current_info()['nb_event']
> >> +
> >> +
> >> +                    if self.run_name != args[0]:
> >> +                        tag = self.results[args[0]].tags[0]
> >> +                        self.set_run_name(args[0], tag,'parton', False)
> >> +                    result_file = open(pjoin(self.me_dir,'Events',
> >> self.run_name, 'parton_systematics.log'),'w')
> >> +                    args[0] = p
> >> +                    break
> >> +            else:
> >> +                raise self.InvalidCmd, 'Invalid run name. Please retry'
> >> +        elif self.options['nb_core'] != 1:
> >> +            lhe = lhe_parser.EventFile(args)
> >> +            nb_event = len(lhe)
> >> +            lhe.close()
> >> +
> >> +        input = args[0]
> >> +        if len(args)>1:
> >> +            output = pjoin(os.getcwd(),args[1])
> >> +        else:
> >> +            output = input
> >> +
> >> +        if '--from_card' in opts:
> >> +            opts.remove('--from_card')
> >> +            opts.append('--from_card=internal')
> >> +
> >> +        if self.options['run_mode'] ==2:
> >> +            nb_submit = min(self.options['nb_core'], nb_event//2500)
> >> +        elif self.options['run_mode'] ==1:
> >> +            nb_submit = min(self.options['cluster_size'],
> >> nb_event//25000)
> >> +        else:
> >> +            nb_submit =1
> >> +
> >> +        if MADEVENT:
> >> +            import internal.various.systematics as systematics
> >> +        else:
> >> +            import madgraph.various.systematics as systematics
> >> +
> >> +        #one core:
> >> +        if nb_submit in [0,1]:
> >> +            systematics.call_systematics([input, output] + opts,
> >> +                                         log=lambda x: logger.info
> >> (str(x)),
> >> +                                         result=result_file
> >> +                                         )
> >> +
> >> +        elif self.options['run_mode'] in [1,2]:
> >> +            # cluster and multi-core mode
> >> +            if self.options['run_mode'] ==2:
> >> +                nb_submit = min(self.options['nb_core'], nb_event//2500)
> >> +            elif self.options['run_mode'] ==2:
> >> +                nb_submit = min(self.options['cluster_size'],
> >> nb_event//25000)
> >> +            event_per_job = nb_event // nb_submit
> >> +            nb_job_with_plus_one = nb_event % nb_submit
> >> +            start_event, stop_event = 0,0
> >> +            for i in range(nb_submit):
> >> +                #computing start/stop event
> >> +                event_requested = event_per_job
> >> +                if i < nb_job_with_plus_one:
> >> +                    event_requested += 1
> >> +                start_event = stop_event
> >> +                stop_event = start_event + event_requested
> >> +
> >> +                prog = sys.executable
> >> +                input_files = [os.path.basename(input)]
> >> +                output_files = ['./tmp_%s_%s' % (i,
> >> os.path.basename(output)),
> >> +                                './log_sys_%s.txt' % (i)]
> >> +                argument =  [pjoin(self.me_dir, 'bin', 'internal',
> >> 'systematics.py'),
> >> +                             input_files[0], output_files[0]] + opts +\
> >> +                             ['--start_event=%i' % start_event,
> >> +                              '--stop_event=%i' %stop_event,
> >> +                              '--result=./log_sys_%s.txt' %i]
> >> +                required_output = output
> >> +                self.cluster.cluster_submit(prog, argument,
> >> +                                            input_files=input_files,
> >> +                                            output_files=output_files,
> >> +                                            cwd=os.path.dirname(output),
> >> +                                            required_output=required_
> >> output,
> >> +                                            stdout='/dev/null'
> >> +                                            )
> >> +            starttime = time.time()
> >> +            update_status = lambda idle, run, finish: \
> >> +                    self.update_status((idle, run, finish, 'running
> >> systematics'), level=None,
> >> +                                       force=False, starttime=starttime)
> >> +            self.cluster.wait(os.path.dirname(output), update_status,
> >> update_status)
> >> +
> >> +            #collect the data
> >> +            all_cross = []
> >> +            for i in range(nb_submit):
> >> +                pos=0
> >> +                for line in open(pjoin(os.path.dirname(output),
> >> 'log_sys_%s.txt'%i)):
> >> +                    if line.startswith('#'):
> >> +                        continue
> >> +                    split = line.split()
> >> +                    if len(split) in [0,1]:
> >> +                        continue
> >> +                    key = tuple(float(x) for x in split[:-1])
> >> +                    cross= float(split[-1])
> >> +                    if 'event_norm' in self.run_card and \
> >> +                            self.run_card['event_norm'] in ['average',
> >> 'unity']:
> >> +                        cross *= (event_per_job+1 if i
> >> <nb_job_with_plus_one else event_per_job)
> >> +                    if len(all_cross) > pos:
> >> +                        all_cross[pos] += cross
> >> +                    else:
> >> +                        all_cross.append(cross)
> >> +                    pos+=1
> >> +
> >> +            if 'event_norm' in self.run_card and \
> >> +                                       self.run_card['event_norm'] in
> >> ['unity']:
> >> +                all_cross= [cross/nb_event for cross in all_cross]
> >> +
> >> +            sys_obj = systematics.call_systematics([input, None] + opts,
> >> +                                         log=lambda x: logger.info
> >> (str(x)),
> >> +                                         result=result_file,
> >> +                                         running=False
> >> +                                         )
> >> +            sys_obj.print_cross_sections(all_cross, nb_event,
> >> result_file)
> >> +
> >> +            #concatenate the output file
> >> +            subprocess.call(['cat']+\
> >> +                            ['./tmp_%s_%s' % (i,
> >> os.path.basename(output)) for i in range(nb_submit)],
> >> +                            stdout=open(output,'w'),
> >> +                            cwd=os.path.dirname(output))
> >> +            for i in range(nb_submit):
> >> +                os.remove('%s/tmp_%s_%s' %(os.path.dirname(output),i,
> >> os.path.basename(output)))
> >> +            #    os.remove('%s/log_sys_%s.txt' %
> >> (os.path.dirname(output),i))
> >> +
> >> +
> >> +
> >> +
> >> +
> >> +        self.update_status('End of systematics computation',
> >> level='parton', makehtml=False)
> >> +
> >> +
> >>     ############################################################
> >> ################
> >>     def do_reweight(self, line):
> >>         """ syntax is "reweight RUN_NAME"
> >> @@ -1083,9 +1401,51 @@
> >>             cp3.irmp.ucl.ac.be/projects/madgraph/wiki/Reweight
> >>         """
> >>
> >> +        #### Utility function
> >> +        def check_multicore(self):
> >> +            """ determine if the cards are save for multicore use"""
> >> +            card = pjoin(self.me_dir, 'Cards', 'reweight_card.dat')
> >> +
> >> +            multicore = True
> >> +            if self.options['run_mode'] in [0,1]:
> >> +                multicore = False
> >> +
> >> +            lines = [l.strip() for l in open(card) if not
> >> l.strip().startswith('#')]
> >> +            while lines and not lines[0].startswith('launch'):
> >> +                line = lines.pop(0)
> >> +                # if not standard output mode forbid multicore mode
> >> +                if line.startswith('change') and
> >> line[6:].strip().startswith('output'):
> >> +                    return False
> >> +                if line.startswith('change') and
> >> line[6:].strip().startswith('multicore'):
> >> +                    split_line = line.split()
> >> +                    if len(split_line) > 2:
> >> +                        multicore = bool(split_line[2])
> >> +            # we have reached the first launch in the card ensure that no
> >> output change
> >> +            #are done after that point.
> >> +            lines = [line[6:].strip() for line in lines if
> >> line.startswith('change')]
> >> +            for line in lines:
> >> +                if line.startswith(('process','model','output',
> >> 'rwgt_dir')):
> >> +                    return False
> >> +                elif line.startswith('multicore'):
> >> +                    split_line = line.split()
> >> +                    if len(split_line) > 1:
> >> +                        multicore = bool(split_line[1])
> >> +
> >> +            return multicore
> >> +
> >> +
> >> +
> >>         if '-from_cards' in line and not os.path.exists(pjoin(self.me_dir,
> >> 'Cards', 'reweight_card.dat')):
> >>             return
> >> -
> >> +        'print line:', line
> >> +        # option for multicore to avoid that all of them create the same
> >> directory
> >> +        if '--multicore=create' in line:
> >> +            multicore='create'
> >> +        elif '--multicore=wait' in line:
> >> +            multicore='wait'
> >> +        else:
> >> +            multicore=False
> >> +
> >>         # Check that MG5 directory is present .
> >>         if MADEVENT and not self.options['mg5_path']:
> >>             raise self.InvalidCmd, '''The module reweight requires that
> >> MG5 is installed on the system.
> >> @@ -1112,6 +1472,10 @@
> >>             if self.run_name and self.results.current and
> >> self.results.current['cross'] == 0:
> >>                 self.results.delete_run(self.run_name, self.run_tag)
> >>             self.results.save()
> >> +            # ensure that the run_card is present
> >> +            if not hasattr(self, 'run_card'):
> >> +                self.run_card = banner_mod.RunCard(pjoin(self.me_dir,
> >> 'Cards', 'run_card.dat'))
> >> +
> >>             # we want to run this in a separate shell to avoid hard f2py
> >> crash
> >>             command =  [sys.executable]
> >>             if os.path.exists(pjoin(self.me_dir, 'bin', 'madevent')):
> >> @@ -1121,41 +1485,109 @@
> >>             if not isinstance(self, cmd.CmdShell):
> >>                 command.append('--web')
> >>             command.append('reweight')
> >> -            if self.run_name:
> >> -                command.append(self.run_name)
> >> +
> >> +            #########   START SINGLE CORE MODE ############
> >> +            if self.options['nb_core']==1 or self.run_card['nevents'] <
> >> 101 or not check_multicore(self):
> >> +                if self.run_name:
> >> +                    command.append(self.run_name)
> >> +                else:
> >> +                    command += args
> >> +                if '-from_cards' not in command:
> >> +                    command.append('-from_cards')
> >> +                p = misc.Popen(command, stdout = subprocess.PIPE, stderr
> >> = subprocess.STDOUT, cwd=self.me_dir)
> >> +                while p.poll() is None:
> >> +                    line = p.stdout.readline()
> >> +                    if any(t in line for t in ['INFO:', 'WARNING:',
> >> 'CRITICAL:', 'ERROR:', 'root:','KEEP:']) and \
> >> +                       not '***********' in line:
> >> +                            print line[:-1].replace('INFO',
> >> 'REWEIGHT').replace('KEEP:','')
> >> +                    elif __debug__ and line:
> >> +                        logger.debug(line[:-1])
> >> +                if p.returncode !=0:
> >> +                    logger.error("Reweighting failed")
> >> +                    return
> >> +                self.results = self.load_results_db()
> >> +                # forbid this function to create an empty item in results.
> >> +                try:
> >> +                    if self.results[self.run_name][-2]['cross']==0:
> >> +                        self.results.delete_run(self.
> >> run_name,self.results[self.run_name][-2]['tag'])
> >> +                except:
> >> +                    pass
> >> +                try:
> >> +                    if self.results.current['cross'] == 0 and
> >> self.run_name:
> >> +                        self.results.delete_run(self.run_name,
> >> self.run_tag)
> >> +                except:
> >> +                    pass
> >> +                # re-define current run
> >> +                try:
> >> +                    self.results.def_current(self.run_name, self.run_tag)
> >> +                except Exception:
> >> +                    pass
> >> +                return
> >> +                ##########    END SINGLE CORE HANDLING #############
> >>             else:
> >> -                command += args
> >> -            if '-from_cards' not in command:
> >> -                command.append('-from_cards')
> >> -            p = misc.Popen(command, stdout = subprocess.PIPE, stderr =
> >> subprocess.STDOUT, cwd=self.me_dir)
> >> -            while p.poll() is None:
> >> -                line = p.stdout.readline()
> >> -                if any(t in line for t in ['INFO:', 'WARNING:',
> >> 'CRITICAL:', 'ERROR:', 'root:','KEEP:']) and \
> >> -                   not '***********' in line:
> >> -                        print line[:-1].replace('INFO',
> >> 'REWEIGHT').replace('KEEP:','')
> >> -                elif __debug__ and line:
> >> -                    logger.debug(line[:-1])
> >> -            if p.returncode !=0:
> >> -                logger.error("Reweighting failed")
> >> +                ##########    START MULTI-CORE HANDLING #############
> >> +                if not isinstance(self.cluster, cluster.MultiCore):
> >> +                    mycluster = cluster.MultiCore(nb_core=
> >> self.options['nb_core'])
> >> +                else:
> >> +                    mycluster = self.cluster
> >> +
> >> +                new_args=list(args)
> >> +                self.check_decay_events(new_args)
> >> +                # prepare multi-core  job:
> >> +                import madgraph.various.lhe_parser as lhe_parser
> >> +                # args now alway content the path to the valid files
> >> +                if 'nevt_job' in self.run_card and
> >> self.run_card['nevt_job'] !=-1:
> >> +                    nevt_job = self.run_card['nevt_job']
> >> +                else:
> >> +                    nevt_job = max(5000, self.run_card['nevents']/50)
> >> +                logger.info("split the event file in bunch of %s event"
> >> % nevt_job)
> >> +                nb_file = lhe_parser.EventFile(new_args[
> >> 0]).split(nevt_job)
> >> +                starttime = time.time()
> >> +                update_status = lambda idle, run, finish: \
> >> +                    self.update_status((idle, run, finish, 'reweight'),
> >> level=None,
> >> +                                       force=False, starttime=starttime)
> >> +
> >> +                all_lhe = []
> >> +                devnull= open(os.devnull)
> >> +                for i in range(nb_file):
> >> +                    new_command = list(command)
> >> +                    new_command.append('%s_%s.lhe' % (new_args[0],i))
> >> +                    all_lhe.append('%s_%s.lhe' % (new_args[0],i))
> >> +                    if '-from_cards' not in command:
> >> +                        new_command.append('-from_cards')
> >> +                    if i==0:
> >> +                        if __debug__:
> >> +                            stdout = None
> >> +                        else:
> >> +                            stdout = open(pjoin(self.me_dir,'Events',
> >> self.run_name, 'reweight.log'),'w')
> >> +                        new_command.append('--multicore=create')
> >> +                    else:
> >> +                        stdout = devnull
> >> +                        #stdout = open(pjoin(self.me_dir,'Events',
> >> self.run_name, 'reweight%s.log' % i),'w')
> >> +                        new_command.append('--multicore=wait')
> >> +                    mycluster.submit(prog=command[0],
> >> argument=new_command[1:], stdout=stdout, cwd=self.me_dir)
> >> +                mycluster.wait(self.me_dir,update_status)
> >> +                devnull.close()
> >> +
> >> +                lhe = lhe_parser.MultiEventFile(all_lhe, parse=False)
> >> +                nb_event, cross_sections = lhe.write(new_args[0],
> >> get_info=True)
> >> +                if any(os.path.exists('%s_%s_debug.log' % (f,
> >> self.run_tag)) for f in all_lhe):
> >> +                    for f in all_lhe:
> >> +                        if os.path.exists('%s_%s_debug.log' % (f,
> >> self.run_tag)):
> >> +                            raise Exception, "Some of the run failed:
> >> Please read %s_%s_debug.log" % (f, self.run_tag)
> >> +
> >> +
> >> +                if 'event_norm' in self.run_card and
> >> self.run_card['event_norm'] == 'average':
> >> +                    for key, value in cross_sections.items():
> >> +                        cross_sections[key] = value / (nb_event+1)
> >> +                lhe.remove()
> >> +                for key in cross_sections:
> >> +                    if key == 'orig' or key.isdigit():
> >> +                        continue
> >> +                    logger.info('%s : %s pb' % (key,
> >> cross_sections[key]))
> >>                 return
> >> -            self.results = self.load_results_db()
> >> -            # forbid this function to create an empty item in results.
> >> -            try:
> >> -                if self.results[self.run_name][-2]['cross']==0:
> >> -                    self.results.delete_run(self.
> >> run_name,self.results[self.run_name][-2]['tag'])
> >> -            except:
> >> -                pass
> >> -            try:
> >> -                if self.results.current['cross'] == 0 and self.run_name:
> >> -                    self.results.delete_run(self.run_name, self.run_tag)
> >> -            except:
> >> -                pass
> >> -            # re-define current run
> >> -            try:
> >> -                self.results.def_current(self.run_name, self.run_tag)
> >> -            except Exception:
> >> -                pass
> >> -            return
> >> +            ##########    END MULTI-CORE HANDLING #############
> >> +
> >>
> >>         self.to_store.append('event')
> >>         # forbid this function to create an empty item in results.
> >> @@ -1176,6 +1608,8 @@
> >>         path = pjoin(self.me_dir, 'Cards', 'reweight_card.dat')
> >>         reweight_cmd.raw_input=False
> >>         reweight_cmd.me_dir = self.me_dir
> >> +        reweight_cmd.multicore = multicore #allow the directory creation
> >> or not
> >> +        print "We are in mode", multicore
> >>         reweight_cmd.import_command_file(path)
> >>         reweight_cmd.do_quit('')
> >>
> >> @@ -2845,8 +3279,8 @@
> >>
> >>         args = self.split_arg(line)
> >>         # fix some formatting problem
> >> -        if '=' in args[-1]:
> >> -            arg1, arg2 = args.pop(-1).split('=')
> >> +        if len(args)==1 and '=' in args[-1]:
> >> +            arg1, arg2 = args.pop(-1).split('=',1)
> >>             args += [arg1, arg2]
> >>         if '=' in args:
> >>             args.remove('=')
> >>
> >> === modified file 'madgraph/interface/madevent_interface.py'
> >> --- madgraph/interface/madevent_interface.py    2016-06-28 19:24:22 +0000
> >> +++ madgraph/interface/madevent_interface.py    2016-08-11 23:54:34 +0000
> >> @@ -1991,6 +1991,7 @@
> >>             self.exec_cmd('survey  %s %s' % (self.run_name,'
> >> '.join(args)),
> >>                           postcmd=False)
> >>             nb_event = self.run_card['nevents']
> >> +            bypass_run=False
> >>             self.exec_cmd('refine %s' % nb_event, postcmd=False)
> >>             if not float(self.results.current['cross']):
> >>                 # Zero cross-section. Try to guess why
> >> @@ -2002,24 +2003,35 @@
> >>    3) The cuts are too strong.
> >>    Please check/correct your param_card and/or your run_card.'''
> >>                 logger_stderr.critical(text)
> >> -                raise ZeroResult('See https://cp3.irmp.ucl.ac.be/
> >> projects/madgraph/wiki/FAQ-General-14')
> >> -
> >> -            self.exec_cmd('refine %s' % nb_event, postcmd=False)
> >> -
> >> -            self.exec_cmd('combine_events', postcmd=False)
> >> -            self.print_results_in_shell(self.results.current)
> >> -
> >> -
> >> -            self.run_syscalc('parton')
> >> -            self.create_plot('parton')
> >> -            self.exec_cmd('store_events', postcmd=False)
> >> -            self.exec_cmd('reweight -from_cards', postcmd=False)
> >> -            self.exec_cmd('decay_events -from_cards', postcmd=False)
> >> -            if self.run_card['time_of_flight']>=0:
> >> -                self.exec_cmd("add_time_of_flight --threshold=%s" %
> >> self.run_card['time_of_flight'] ,postcmd=False)
> >> -            self.exec_cmd('pythia --no_default', postcmd=False,
> >> printcmd=False)
> >> -            # pythia launches pgs/delphes if needed
> >> -            self.store_result()
> >> +                if not self.param_card_iterator:
> >> +                    raise ZeroResult('See https://cp3.irmp.ucl.ac.be/
> >> projects/madgraph/wiki/FAQ-General-14')
> >> +                else:
> >> +                    bypass_run = True
> >> +
> >> +            #we can bypass the following if scan and first result is zero
> >> +            if not bypass_run:
> >> +                self.exec_cmd('refine %s' % nb_event, postcmd=False)
> >> +
> >> +                self.exec_cmd('combine_events',
> >> postcmd=False,printcmd=False)
> >> +                self.print_results_in_shell(self.results.current)
> >> +
> >> +                if self.run_card['use_syst']:
> >> +                    scdir = self.options['syscalc_path']
> >> +                    if not scdir or not os.path.exists(scdir):
> >> +                        self.exec_cmd('systematics %s --from_card' %
> >> self.run_name, postcmd=False,printcmd=False)
> >> +                    else:
> >> +                        self.run_syscalc('parton')
> >> +
> >> +
> >> +                self.create_plot('parton')
> >> +                self.exec_cmd('store_events', postcmd=False)
> >> +                self.exec_cmd('reweight -from_cards', postcmd=False)
> >> +                self.exec_cmd('decay_events -from_cards', postcmd=False)
> >> +                if self.run_card['time_of_flight']>=0:
> >> +                    self.exec_cmd("add_time_of_flight --threshold=%s" %
> >> self.run_card['time_of_flight'] ,postcmd=False)
> >> +                self.exec_cmd('pythia --no_default', postcmd=False,
> >> printcmd=False)
> >> +                # pythia launches pgs/delphes if needed
> >> +                self.store_result()
> >>
> >>             if self.param_card_iterator:
> >>                 param_card_iterator = self.param_card_iterator
> >> @@ -2031,9 +2043,13 @@
> >>                     for card in param_card_iterator:
> >>                         card.write(pjoin(self.me_dir,'
> >> Cards','param_card.dat'))
> >>                         next_name = param_card_iterator.get_next_
> >> name(self.run_name)
> >> -                        self.exec_cmd("generate_events -f %s" % next_name,
> >> +                        try:
> >> +                            self.exec_cmd("generate_events -f %s" %
> >> next_name,
> >>                                       precmd=True,
> >> postcmd=True,errorhandling=False)
> >> -                        param_card_iterator.store_entry(self.run_name,
> >> self.results.current['cross'])
> >> +                        except ZeroResult:
> >> +                            param_card_iterator.store_entry(self.run_name,
> >> 0)
> >> +                        else:
> >> +                            param_card_iterator.store_entry(self.run_name,
> >> self.results.current['cross'])
> >>                     param_card_iterator.write(pjoin(self.me_dir,'Cards','
> >> param_card.dat'))
> >>                     name = misc.get_scan_name(orig_name, self.run_name)
> >>                     path = pjoin(self.me_dir, 'Events','scan_%s.txt' %
> >> name)
> >> @@ -3960,106 +3976,6 @@
> >>         self.Gdirs = Gdirs
> >>         return self.Gdirs
> >>
> >> -    ############################################################
> >> ################
> >> -    def set_run_name(self, name, tag=None, level='parton',
> >> reload_card=False,
> >> -                     allow_new_tag=True):
> >> -        """define the run name, the run_tag, the banner and the
> >> results."""
> >> -
> >> -        # when are we force to change the tag new_run:previous run
> >> requiring changes
> >> -        upgrade_tag = {'parton': ['parton','pythia','pgs','delphes'],
> >> -                       'pythia': ['pythia','pgs','delphes'],
> >> -                       'pgs': ['pgs'],
> >> -                       'delphes':['delphes'],
> >> -                       'plot':[],
> >> -                       'syscalc':[]}
> >> -
> >> -
> >> -
> >> -        if name == self.run_name:
> >> -            if reload_card:
> >> -                run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> -                self.run_card = banner_mod.RunCard(run_card)
> >> -
> >> -            #check if we need to change the tag
> >> -            if tag:
> >> -                self.run_card['run_tag'] = tag
> >> -                self.run_tag = tag
> >> -                self.results.add_run(self.run_name, self.run_card)
> >> -            else:
> >> -                for tag in upgrade_tag[level]:
> >> -                    if getattr(self.results[self.run_name][-1], tag):
> >> -                        tag = self.get_available_tag()
> >> -                        self.run_card['run_tag'] = tag
> >> -                        self.run_tag = tag
> >> -                        self.results.add_run(self.run_name,
> >> self.run_card)
> >> -                        break
> >> -            return # Nothing to do anymore
> >> -
> >> -        # save/clean previous run
> >> -        if self.run_name:
> >> -            self.store_result()
> >> -        # store new name
> >> -        self.run_name = name
> >> -
> >> -        new_tag = False
> >> -        # First call for this run -> set the banner
> >> -        self.banner = banner_mod.recover_banner(self.results, level,
> >> name)
> >> -        if 'mgruncard' in self.banner:
> >> -            self.run_card = self.banner.charge_card('run_card')
> >> -        else:
> >> -            # Read run_card
> >> -            run_card = pjoin(self.me_dir, 'Cards','run_card.dat')
> >> -            self.run_card = banner_mod.RunCard(run_card)
> >> -
> >> -        if tag:
> >> -            self.run_card['run_tag'] = tag
> >> -            new_tag = True
> >> -        elif not self.run_name in self.results and level =='parton':
> >> -            pass # No results yet, so current tag is fine
> >> -        elif not self.run_name in self.results:
> >> -            #This is only for case when you want to trick the interface
> >> -            logger.warning('Trying to run data on unknown run.')
> >> -            self.results.add_run(name, self.run_card)
> >> -            self.results.update('add run %s' % name, 'all',
> >> makehtml=False)
> >> -        else:
> >> -            for tag in upgrade_tag[level]:
> >> -
> >> -                if getattr(self.results[self.run_name][-1], tag):
> >> -                    # LEVEL is already define in the last tag -> need to
> >> switch tag
> >> -                    tag = self.get_available_tag()
> >> -                    self.run_card['run_tag'] = tag
> >> -                    new_tag = True
> >> -                    break
> >> -            if not new_tag:
> >> -                # We can add the results to the current run
> >> -                tag = self.results[self.run_name][-1]['tag']
> >> -                self.run_card['run_tag'] = tag # ensure that run_tag is
> >> correct
> >> -
> >> -        if allow_new_tag and (name in self.results and not new_tag):
> >> -            self.results.def_current(self.run_name)
> >> -        else:
> >> -            self.results.add_run(self.run_name, self.run_card)
> >> -
> >> -        self.run_tag = self.run_card['run_tag']
> >> -
> >> -        # Return the tag of the previous run having the required data for
> >> this
> >> -        # tag/run to working wel.
> >> -        if level == 'parton':
> >> -            return
> >> -        elif level == 'pythia':
> >> -            return self.results[self.run_name][0]['tag']
> >> -        else:
> >> -            for i in range(-1,-len(self.results[self.run_name])-1,-1):
> >> -                tagRun = self.results[self.run_name][i]
> >> -                if tagRun.pythia:
> >> -                    return tagRun['tag']
> >> -
> >> -
> >> -
> >> -
> >> -
> >> -
> >> -
> >>
> >>     ############################################################
> >> ################
> >>     def find_model_name(self):
> >> @@ -4203,7 +4119,7 @@
> >>         scdir = self.options['syscalc_path']
> >>         if not scdir or not os.path.exists(scdir):
> >>             return
> >> -        logger.info('running syscalc on mode %s' % mode)
> >> +        logger.info('running SysCalc on mode %s' % mode)
> >>
> >>
> >>         # Check that all pdfset are correctly installed
> >> @@ -4253,7 +4169,9 @@
> >>                 if not (os.path.exists(event_path) or
> >> os.path.exists(event_path+".gz")):
> >>                     event_path = pjoin(event_dir, 'unweighted_events.lhe')
> >>                 output = pjoin(event_dir, 'syscalc.lhe')
> >> +                stdout = open(pjoin(event_dir, self.run_name,
> >> '%s_systematics.log' % (mode)),'w')
> >>             elif mode == 'Pythia':
> >> +                stdout = open(pjoin(event_dir, self.run_name,
> >> '%s_%s_syscalc.log' % (tag,mode)),'w')
> >>                 if 'mgpythiacard' in self.banner:
> >>                     pat = re.compile('''^\s*qcut\s*=\s*([\+\-\d.e]*)''',
> >> re.M+re.I)
> >>                     data = pat.search(self.banner['mgpythiacard'])
> >> @@ -4282,7 +4200,7 @@
> >>         try:
> >>             proc = misc.call([os.path.join(scdir, 'sys_calc'),
> >>                                event_path, card, output],
> >> -                            stdout = open(pjoin(event_dir, self.run_name,
> >> '%s_%s_syscalc.log' % (tag,mode)),'w'),
> >> +                            stdout = stdout,
> >>                             stderr = subprocess.STDOUT,
> >>                             cwd=event_dir)
> >>             # Wait 5 s to make sure file is finished writing
> >>
> >> === modified file 'madgraph/interface/reweight_interface.py'
> >> --- madgraph/interface/reweight_interface.py    2016-03-31 10:53:12 +0000
> >> +++ madgraph/interface/reweight_interface.py    2016-08-11 23:54:34 +0000
> >> @@ -82,11 +82,11 @@
> >>         self.model = None
> >>         self.has_standalone_dir = False
> >>         self.mother= mother # calling interface
> >> -
> >> +        self.multicore=False
> >>
> >>         self.options = {'curr_dir': os.path.realpath(os.getcwd()),
> >>                         'rwgt_name':None}
> >> -
> >> +
> >>         self.events_file = None
> >>         self.processes = {}
> >>         self.second_model = None
> >> @@ -358,6 +358,10 @@
> >>             self.rwgt_dir = args[1]
> >>             if not os.path.exists(self.rwgt_dir):
> >>                 os.mkdir(self.rwgt_dir)
> >> +        elif args[0] == 'multicore':
> >> +            pass
> >> +            # this line is meant to be parsed by common_run_interface and
> >> change the way this class is called.
> >> +            #It has no direct impact on this class.
> >>         else:
> >>             logger.critical("unknown option! %s.  Discard line." %
> >> args[0])
> >>
> >> @@ -415,12 +419,25 @@
> >>
> >>         model_line = self.banner.get('proc_card', 'full_model_line')
> >>
> >> -        if not self.has_standalone_dir:
> >> +        if not self.has_standalone_dir:
> >>             if self.rwgt_dir and os.path.exists(pjoin(self.
> >> rwgt_dir,'rw_me','rwgt.pkl')):
> >>                 self.load_from_pickle()
> >> -                self.me_dir = self.rwgt_dir
> >> +                if not self.rwgt_dir:
> >> +                    self.me_dir = self.rwgt_dir
> >> +            elif self.multicore == 'wait':
> >> +                while not os.path.exists(pjoin(self.me_
> >> dir,'rw_me','rwgt.pkl')):
> >> +                    time.sleep(60)
> >> +                    print 'wait for pickle'
> >> +                print "loading from pickle"
> >> +                if not self.rwgt_dir:
> >> +                    self.rwgt_dir = self.me_dir
> >> +                self.load_from_pickle(keep_name=True)
> >>             else:
> >> -                self.create_standalone_directory()
> >> +                self.create_standalone_directory()
> >> +                if self.multicore == 'create':
> >> +                    if not self.rwgt_dir:
> >> +                        self.rwgt_dir = self.me_dir
> >> +                    self.save_to_pickle()
> >>
> >>         if self.rwgt_dir:
> >>             path_me =self.rwgt_dir
> >> @@ -501,7 +518,6 @@
> >>                 rewgtid = maxid
> >>                 if self.options['rwgt_name']:
> >>                     #ensure that the entry is not already define if so
> >> overwrites it
> >> -                    misc.sprint(mg_rwgt_info)
> >>                     for (i, nlotype, diff) in mg_rwgt_info[:]:
> >>                         for flag in type_rwgt:
> >>                             if 'rwgt_%s' % i == '%s%s'
> >> %(self.options['rwgt_name'],flag) or \
> >> @@ -779,10 +795,11 @@
> >>                 results.current.parton.append('lhe')
> >>             logger.info('Event %s is now created with new central
> >> weight' % output2.name)
> >>
> >> -        for name in cross:
> >> -            if name == 'orig':
> >> -                continue
> >> -            logger.info('new cross-section is %s: %g pb (indicative
> >> error: %g pb)' %\
> >> +        if self.multicore != 'create':
> >> +            for name in cross:
> >> +                if name == 'orig':
> >> +                    continue
> >> +                logger.info('new cross-section is %s: %g pb (indicative
> >> error: %g pb)' %\
> >>                         ('(%s)' %name if name else '',cross[name],
> >> error[name]))
> >>
> >>         self.terminate_fortran_executables(new_card_only=True)
> >> @@ -793,7 +810,6 @@
> >>             else:
> >>                 self.all_cross_section[(tag_name,name)] = (cross[name],
> >> error[name])
> >>
> >> -
> >>         # perform the scanning
> >>         if param_card_iterator:
> >>             for i,card in enumerate(param_card_iterator):
> >> @@ -873,6 +889,15 @@
> >>             orig_order, Pdir, hel_dict = self.id_to_path[tag]
> >>             misc.sprint(w_orig, w_new)
> >>             misc.sprint(event)
> >> +            misc.sprint(self.invert_momenta(event.get_momenta(
> >> orig_order)))
> >> +            misc.sprint(event.get_momenta(orig_order))
> >> +            misc.sprint(event.aqcd)
> >> +            hel_order = event.get_helicity(orig_order)
> >> +            if self.helicity_reweighting and 9 not in hel_order:
> >> +                nhel = hel_dict[tuple(hel_order)]
> >> +            else:
> >> +                nhel = 0
> >> +            misc.sprint(nhel, Pdir, hel_dict)
> >>             raise Exception, "Invalid matrix element for original
> >> computation (weight=0)"
> >>
> >>         return {'orig': event.wgt, '': w_new/w_orig*event.wgt}
> >> @@ -1012,6 +1037,11 @@
> >>     def calculate_matrix_element(self, event, hypp_id, space, scale2=0):
> >>         """routine to return the matrix element"""
> >>
> >> +        if self.has_nlo:
> >> +            nb_retry, sleep = 10, 60
> >> +        else:
> >> +            nb_retry, sleep = 5, 20
> >> +
> >>         tag, order = event.get_tag_and_order()
> >>         if isinstance(hypp_id, str) and hypp_id.startswith('V'):
> >>             tag = (tag,'V')
> >> @@ -1057,7 +1087,7 @@
> >>                         dir_to_f2py_free_mod[(Pdir,0)] = (metag,
> >> nb_f2py_module)
> >>                 os.environ['MENUM'] = '2'
> >>                 if not self.rwgt_dir or not os.path.exists(pjoin(Pdir,
> >> 'matrix2py.so')):
> >> -                    misc.compile(['matrix2py.so'], cwd=Pdir)
> >> +
> misc.multiple_try(nb_retry,sleep)(misc.compile)(['matrix2py.so'],
> >> cwd=Pdir)
> >>                 self.rename_f2py_lib(Pdir, 2*metag)
> >>                 try:
> >>                     mymod = __import__('%s.SubProcesses.%s.matrix%spy' %
> >> (base, Pname, 2*metag), globals(), locals(), [],-1)
> >> @@ -1085,10 +1115,9 @@
> >>                 try:
> >>                     mymod = __import__('%s.SubProcesses.%s.matrix%spy' %
> >> (base, Pname, newtag), globals(), locals(), [],-1)
> >>                 except Exception, error:
> >> -                    os.remove(pjoin(Pdir, 'matrix%spy.so' % newtag))
> >>                     newtag  = "L%s" % newtag
> >>                     os.environ['MENUM'] = newtag
> >> -                    misc.compile(['matrix%spy.so' % newtag], cwd=Pdir)
> >> +                    misc.multiple_try(nb_retry,
> >> sleep)(misc.compile)(['matrix%spy.so' % newtag], cwd=Pdir)
> >>                     mymod = __import__('%s.SubProcesses.%s.matrix%spy' %
> >> (base, Pname, newtag), globals(), locals(), [],-1)
> >>
> >>                 S = mymod.SubProcesses
> >> @@ -1110,7 +1139,7 @@
> >>             Pname = os.path.basename(Pdir)
> >>             os.environ['MENUM'] = '2'
> >>             if not self.rwgt_dir or not os.path.exists(pjoin(Pdir,
> >> 'matrix2py.so')):
> >> -                misc.compile(['matrix2py.so'], cwd=pjoin(subdir, Pdir))
> >> +
> misc.multiple_try(nb_retry,sleep)(misc.compile)(['matrix2py.so'],
> >> cwd=pjoin(subdir, Pdir))
> >>             if (Pdir, 1) not in dir_to_f2py_free_mod:
> >>                 metag = 1
> >>                 dir_to_f2py_free_mod[(Pdir,1)] = (metag, nb_f2py_module)
> >> @@ -1123,10 +1152,9 @@
> >>             try:
> >>                 mymod = __import__("%s_second.SubProcesses.%s.matrix%spy"
> >> % (base, Pname, metag))
> >>             except ImportError:
> >> -                os.remove(pjoin(Pdir, 'matrix%spy.so' % metag ))
> >>                 metag = "L%s" % metag
> >>                 os.environ['MENUM'] = str(metag)
> >> -                misc.compile(['matrix%spy.so' % metag], cwd=pjoin(subdir,
> >> Pdir))
> >> +
> misc.multiple_try(nb_retry,sleep)(misc.compile)(['matrix%spy.so'
> >> % metag], cwd=pjoin(subdir, Pdir))
> >>                 mymod = __import__("%s_second.SubProcesses.%s.matrix%spy"
> >> % (base, Pname, metag))
> >>
> >>             reload(mymod)
> >> @@ -1140,17 +1168,24 @@
> >>             space.calculator[(run_id,'module')] = mymod
> >>             external = space.calculator[run_id]
> >>
> >> -        p = self.invert_momenta(event.get_momenta(orig_order))
> >> -
> >>
> >> +
> >> +        p = event.get_momenta(orig_order)
> >>         # add helicity information
> >>
> >>         hel_order = event.get_helicity(orig_order)
> >>         if self.helicity_reweighting and 9 not in hel_order:
> >>             nhel = hel_dict[tuple(hel_order)]
> >> +            if event[1].status == -1: #check if this is a 2 >N processes
> >> +                # need to pass to the rest-frame
> >> +                pboost = lhe_parser.FourMomentum(p[0]) +
> >> lhe_parser.FourMomentum(p[1])
> >> +                for i,thisp in enumerate(p):
> >> +                    p[i] = lhe_parser.FourMomentum(thisp)
> >> .zboost(pboost).get_tuple()
> >>         else:
> >>             nhel = 0
> >>
> >> +        p = self.invert_momenta(p)
> >> +
> >>         with misc.chdir(Pdir):
> >>             with misc.stdchannel_redirected(sys.stdout, os.devnull):
> >>                 if 'V' in tag or \
> >> @@ -1193,18 +1228,21 @@
> >>                 split = line.split()
> >>                 if len(split) == 4:
> >>                     cross, error = float(split[0]), float(split[1])
> >> -        if 'orig' not in self.all_cross_section:
> >> -            logger.info('Original cross-section: %s +- %s pb' % (cross,
> >> error))
> >> -        else:
> >> -            logger.info('Original cross-section: %s +- %s pb
> >> (cross-section from sum of weights: %s)' % (cross, error,
> >> self.all_cross_section['orig'][0]))
> >> -        logger.info('Computed cross-section:')
> >> -        keys = self.all_cross_section.keys()
> >> -        keys.sort()
> >> -        for key in keys:
> >> -            if key == 'orig':
> >> -                continue
> >> -            logger.info('%s : %s +- %s pb' % (key[0] if not key[1] else
> >> '%s%s' % key,
> >> -                self.all_cross_section[key][0]
> >> ,self.all_cross_section[key][1] ))
> >> +
> >> +        if not self.multicore == 'create':
> >> +            # No print of results for the multicore mode for the one
> >> printed on screen
> >> +            if 'orig' not in self.all_cross_section:
> >> +                logger.info('Original cross-section: %s +- %s pb' %
> >> (cross, error))
> >> +            else:
> >> +                logger.info('Original cross-section: %s +- %s pb
> >> (cross-section from sum of weights: %s)' % (cross, error,
> >> self.all_cross_section['orig'][0]))
> >> +            logger.info('Computed cross-section:')
> >> +            keys = self.all_cross_section.keys()
> >> +            keys.sort()
> >> +            for key in keys:
> >> +                if key == 'orig':
> >> +                    continue
> >> +                logger.info('%s : %s +- %s pb' % (key[0] if not key[1]
> >> else '%s%s' % key,
> >> +                    self.all_cross_section[key][0]
> >> ,self.all_cross_section[key][1] ))
> >>         self.terminate_fortran_executables()
> >>
> >>         if self.rwgt_dir:
> >> @@ -1227,7 +1265,7 @@
> >>     @misc.mute_logger()
> >>     def create_standalone_directory(self, second=False):
> >>         """generate the various directory for the weight evaluation"""
> >> -
> >> +
> >>         data={}
> >>         if not second:
> >>             data['paths'] = ['rw_me', 'rw_mevirt']
> >> @@ -1416,6 +1454,18 @@
> >>                          pjoin(path_me, data['paths'][0], 'Cards',
> >> 'MadLoopParams.dat'),
> >>                          commentdefault=False)
> >>
> >> +            if self.multicore == 'create':
> >> +                print "compile OLP", data['paths'][0]
> >> +                misc.compile(['OLP_static'], cwd=pjoin(path_me,
> >> data['paths'][0],'SubProcesses'),
> >> +                             nb_core=self.mother.options['nb_core'])
> >> +
> >> +        if os.path.exists(pjoin(path_me, data['paths'][1], 'Cards',
> >> 'MadLoopParams.dat')):
> >> +            if self.multicore == 'create':
> >> +                print "compile OLP", data['paths'][1]
> >> +                misc.compile(['OLP_static'], cwd=pjoin(path_me,
> >> data['paths'][1],'SubProcesses'),
> >> +                             nb_core=self.mother.options['nb_core'])
> >> +
> >> +
> >>         # 5. create the virtual for NLO reweighting
> >> ---------------------------
> >>         if has_nlo and 'NLO' in self.rwgt_mode:
> >>             # Do not pass here for LO/NLO_tree
> >> @@ -1509,6 +1559,11 @@
> >>                                   self.banner.run_card['lpp2']],
> >>                                  self.banner.run_card.get_lhapdf_id())
> >>                 self.combine_wgt = mymod.get_wgt
> >> +
> >> +            if self.multicore == 'create':
> >> +                print "compile OLP", data['paths'][1]
> >> +                misc.compile(['OLP_static'], cwd=pjoin(path_me,
> >> data['paths'][1],'SubProcesses'),
> >> +                             nb_core=self.mother.options['nb_core'])
> >>
> >>         elif has_nlo and not second and self.rwgt_mode == ['NLO_tree']:
> >>             # We do not have any virtual reweighting to do but we still
> >> have to
> >> @@ -1612,18 +1667,21 @@
> >>         to_save['rwgt_dir'] = self.rwgt_dir
> >>         to_save['has_nlo'] = self.has_nlo
> >>         to_save['rwgt_mode'] = self.rwgt_mode
> >> +        to_save['rwgt_name'] = self.options['rwgt_name']
> >>
> >>         name = pjoin(self.rwgt_dir, 'rw_me', 'rwgt.pkl')
> >>         save_load_object.save_to_file(name, to_save)
> >>
> >> -    def load_from_pickle(self):
> >> +    def load_from_pickle(self, keep_name=False):
> >>         import madgraph.iolibs.save_load_object as save_load_object
> >>
> >>         obj = save_load_object.load_from_file( pjoin(self.rwgt_dir,
> >> 'rw_me', 'rwgt.pkl'))
> >>
> >>         self.has_standalone_dir = True
> >>         self.options = {'curr_dir': os.path.realpath(os.getcwd()),
> >> -                        'rewgt_name': None}
> >> +                        'rwgt_name': None}
> >> +        if keep_name:
> >> +            self.options['rwgt_name'] = obj['rwgt_name']
> >>
> >>         old_rwgt = obj['rwgt_dir']
> >>
> >> @@ -1647,10 +1705,14 @@
> >>         if not self.rwgt_mode:
> >>             self.rwgt_mode = obj['rwgt_mode']
> >>             logger.info("mode set to %s" % self.rwgt_mode)
> >> -        if self.has_nlo:
> >> +        if self.has_nlo and 'NLO' in self.rwgt_mode:
> >>             path = pjoin(obj['rwgt_dir'], 'rw_mevirt','Source')
> >>             sys.path.insert(0, path)
> >> -            mymod = __import__('rwgt2py', globals(), locals())
> >> +            try:
> >> +                mymod = __import__('rwgt2py', globals(), locals())
> >> +            except ImportError:
> >> +                misc.compile(['rwgt2py.so'], cwd=path)
> >> +                mymod = __import__('rwgt2py', globals(), locals())
> >>             with misc.stdchannel_redirected(sys.stdout, os.devnull):
> >>                 mymod.initialise([self.banner.run_card['lpp1'],
> >>                               self.banner.run_card['lpp2']],
> >>
> >> === modified file 'madgraph/iolibs/export_fks.py'
> >> --- madgraph/iolibs/export_fks.py       2016-06-24 01:32:34 +0000
> >> +++ madgraph/iolibs/export_fks.py       2016-08-11 23:54:34 +0000
> >> @@ -268,7 +268,8 @@
> >>           pjoin('various','FO_analyse_card.py'),
> >>           pjoin('various','histograms.py'),
> >>           pjoin('various','banner.py'),
> >> -          pjoin('various','cluster.py'),
> >> +          pjoin('various','cluster.py'),
> >> +          pjoin('various','systematics.py'),
> >>           pjoin('various','lhe_parser.py'),
> >>           pjoin('madevent','sum_html.py'),
> >>           pjoin('madevent','gen_crossxhtml.py'),
> >>
> >> === modified file 'madgraph/iolibs/export_v4.py'
> >> --- madgraph/iolibs/export_v4.py        2016-06-16 09:04:34 +0000
> >> +++ madgraph/iolibs/export_v4.py        2016-08-11 23:54:34 +0000
> >> @@ -351,10 +351,11 @@
> >>
> >>             changer = {"pdf_systemwide": to_add}
> >>
> >> -        ff = open(pjoin(self.dir_path, "Source", "PDF", "opendata.f"),"w")
> >> +
> >> +        ff = writers.FortranWriter(pjoin(self.dir_path, "Source", "PDF",
> >> "opendata.f"))
> >>         template = open(pjoin(MG5DIR, "madgraph", "iolibs",
> >> "template_files", "pdf_opendata.f"),"r").read()
> >> -        ff.write(template % changer)
> >> -
> >> +        ff.writelines(template % changer)
> >> +
> >>         # Do the same for lhapdf set
> >>         if not self.opt["cluster_local_path"]:
> >>             changer = {"cluster_specific_path": ""}
> >> @@ -373,9 +374,10 @@
> >>          """ % {"path" : self.opt["cluster_local_path"]}
> >>             changer = {"cluster_specific_path": to_add}
> >>
> >> -        ff = open(pjoin(self.dir_path, "Source", "PDF",
> >> "pdfwrap_lhapdf.f"),"w")
> >> +        ff = writers.FortranWriter(pjoin(self.dir_path, "Source", "PDF",
> >> "pdfwrap_lhapdf.f"))
> >> +        #ff = open(pjoin(self.dir_path, "Source", "PDF",
> >> "pdfwrap_lhapdf.f"),"w")
> >>         template = open(pjoin(MG5DIR, "madgraph", "iolibs",
> >> "template_files", "pdf_wrap_lhapdf.f"),"r").read()
> >> -        ff.write(template % changer)
> >> +        ff.writelines(template % changer)
> >>
> >>
> >>         return
> >> @@ -779,7 +781,7 @@
> >>         if hasattr(self, 'aloha_model'):
> >>             aloha_model = self.aloha_model
> >>         else:
> >> -            aloha_model = create_aloha.AbstractALOHAModel(model.get('
> >> name'))
> >> +            aloha_model = create_aloha.AbstractALOHAModel(os.path.
> >> basename(model.get('modelpath')))
> >>         aloha_model.add_Lorentz_object(model.get('lorentz'))
> >>
> >>         # Compute the subroutines
> >> @@ -3158,6 +3160,8 @@
> >>                                 self.dir_path+'/bin/internal/
> >> lhe_parser.py')
> >>         cp(_file_path+'/various/banner.py',
> >>                                    self.dir_path+'/bin/internal/
> >> banner.py')
> >> +        cp(_file_path+'/various/systematics.py',
> >> self.dir_path+'/bin/internal/systematics.py')
> >> +
> >>         cp(_file_path+'/various/cluster.py',
> >>                                        self.dir_path+'/bin/internal/
> >> cluster.py')
> >>         cp(_file_path+'/madevent/combine_runs.py',
> >>
> >> === modified file 'madgraph/iolibs/file_writers.py'
> >> --- madgraph/iolibs/file_writers.py     2016-04-05 23:57:17 +0000
> >> +++ madgraph/iolibs/file_writers.py     2016-08-11 23:54:34 +0000
> >> @@ -193,7 +193,7 @@
> >>     # Private variables
> >>     __indent = 0
> >>     __keyword_list = []
> >> -    __comment_pattern = re.compile(r"^(\s*#|c$|(c\s+([^=]|$))|cf2py)",
> >> re.IGNORECASE)
> >> +    __comment_pattern =
> re.compile(r"^(\s*#|c$|(c\s+([^=]|$))|cf2py|c\-\-|c\*\*)",
> >> re.IGNORECASE)
> >>
> >>     def write_line(self, line):
> >>         """Write a fortran line, with correct indent and line splits"""
> >>
> >> === modified file 'madgraph/loop/loop_exporters.py'
> >> --- madgraph/loop/loop_exporters.py     2016-07-02 22:45:19 +0000
> >> +++ madgraph/loop/loop_exporters.py     2016-08-11 23:54:34 +0000
> >> @@ -182,7 +182,7 @@
> >>         same loop exporter.
> >>         """
> >>         if not hasattr(self, 'aloha_model'):
> >> -            self.aloha_model = create_aloha.
> >> AbstractALOHAModel(model.get('name'))
> >> +            self.aloha_model = create_aloha.AbstractALOHAModel(os.path.
> >> basename(model.get('modelpath')))
> >>         return self.aloha_model
> >>
> >>     #===========================================================
> >> ================
> >>
> >> === modified file 'madgraph/madevent/gen_crossxhtml.py'
> >> --- madgraph/madevent/gen_crossxhtml.py 2016-04-15 11:14:25 +0000
> >> +++ madgraph/madevent/gen_crossxhtml.py 2016-08-11 23:54:34 +0000
> >> @@ -812,7 +812,7 @@
> >>                 self.parton.append('param_card')
> >>
> >>             if 'syst' not in self.parton and \
> >> -                                    exists(pjoin(path,
> >> "%s_parton_syscalc.log" %self['tag'])):
> >> +                                    exists(pjoin(path,
> >> "parton_systematics.log")):
> >>                 self.parton.append('syst')
> >>
> >>             for kind in ['top','HwU','pdf','ps']:
> >> @@ -1150,8 +1150,8 @@
> >>                     local_dico['err'] = self['error']
> >>                     local_dico['nb_event'] = self['nb_event']
> >>                     if 'syst' in self.parton:
> >> -                        local_dico['syst'] = '<font
> >> face=symbol>&#177;</font> <a href="./Events/%(run_name)s/%(
> >> tag)s_parton_syscalc.log">systematics</a>' \
> >> -                                             %
> >> {'run_name':self['run_name'], 'tag': self['tag']}
> >> +                        local_dico['syst'] = '<font
> >> face=symbol>&#177;</font> <a href="./Events/%(run_name)s/
> >> parton_systematics.log">systematics</a>' \
> >> +                                             %
> >> {'run_name':self['run_name']}
> >>                 elif self['cross_pythia']:
> >>                     if self.parton:
> >>                         local_dico['cross_span'] = nb_line -1
> >> @@ -1173,7 +1173,7 @@
> >>                     local_dico['err'] = self['error']
> >>                     local_dico['nb_event'] = self['nb_event']
> >>                     if 'syst' in self.parton:
> >> -                        local_dico['syst'] = '<font
> >> face=symbol>&#177;</font> <a href="./Events/%(run_name)s/%(
> >> tag)s_parton_syscalc.log">systematics</a>' \
> >> +                        local_dico['syst'] = '<font
> >> face=symbol>&#177;</font> <a href="./Events/%(run_name)s/
> >> parton_systematics.log">systematics</a>' \
> >>                                              %
> >> {'run_name':self['run_name'], 'tag': self['tag']}
> >>
> >>             elif ttype == 'pythia' and self['cross_pythia']:
> >>
> >> === modified file 'madgraph/various/banner.py'
> >> --- madgraph/various/banner.py  2016-06-28 19:24:22 +0000
> >> +++ madgraph/various/banner.py  2016-08-11 23:54:34 +0000
> >> @@ -170,7 +170,7 @@
> >>             raise Exception, "Not Supported version"
> >>         self.lhe_version = version
> >>
> >> -    def get_cross(self):
> >> +    def get_cross(self, witherror=False):
> >>         """return the cross-section of the file"""
> >>
> >>         if "init" not in self:
> >> @@ -179,41 +179,19 @@
> >>
> >>         text = self["init"].split('\n')
> >>         cross = 0
> >> +        error = 0
> >>         for line in text:
> >>             s = line.split()
> >>             if len(s)==4:
> >>                 cross += float(s[0])
> >> -        return cross
> >> -
> >> -
> >> -
> >> -    def modify_init_cross(self, cross):
> >> -        """modify the init information with the associate cross-section"""
> >> -
> >> -        assert isinstance(cross, dict)
> >> -#        assert "all" in cross
> >> -        assert "init" in self
> >> -
> >> -        all_lines = self["init"].split('\n')
> >> -        new_data = []
> >> -        new_data.append(all_lines[0])
> >> -        for i in range(1, len(all_lines)):
> >> -            line = all_lines[i]
> >> -            split = line.split()
> >> -            if len(split) == 4:
> >> -                xsec, xerr, xmax, pid = split
> >> -            else:
> >> -                new_data += all_lines[i:]
> >> -                break
> >> -            if int(pid) not in cross:
> >> -                raise Exception
> >> -            pid = int(pid)
> >> -            ratio = cross[pid]/float(xsec)
> >> -            line = "   %+13.7e %+13.7e %+13.7e %i" % \
> >> -                (float(cross[pid]), ratio* float(xerr),
> >> ratio*float(xmax), pid)
> >> -            new_data.append(line)
> >> -        self['init'] = '\n'.join(new_data)
> >> -
> >> +                if witherror:
> >> +                    error += float(s[1])**2
> >> +        if not witherror:
> >> +            return cross
> >> +        else:
> >> +            return cross, math.sqrt(error)
> >> +
> >> +
> >>     def scale_init_cross(self, ratio):
> >>         """modify the init information with the associate scale"""
> >>
> >> @@ -237,6 +215,15 @@
> >>             new_data.append(line)
> >>         self['init'] = '\n'.join(new_data)
> >>
> >> +    def get_pdg_beam(self):
> >> +        """return the pdg of each beam"""
> >> +
> >> +        assert "init" in self
> >> +
> >> +        all_lines = self["init"].split('\n')
> >> +        pdg1,pdg2,_ = all_lines[0].split(None, 2)
> >> +        return int(pdg1), int(pdg2)
> >> +
> >>     def load_basic(self, medir):
> >>         """ Load the proc_card /param_card and run_card """
> >>
> >>
> >> === modified file 'madgraph/various/cluster.py'
> >> --- madgraph/various/cluster.py 2016-06-13 08:27:43 +0000
> >> +++ madgraph/various/cluster.py 2016-08-11 23:54:34 +0000
> >> @@ -618,6 +618,8 @@
> >>                     if isinstance(exe,str):
> >>                         if os.path.exists(exe) and not
> >> exe.startswith('/'):
> >>                             exe = './' + exe
> >> +                        if isinstance(opt['stdout'],str):
> >> +                            opt['stdout'] = open(opt['stdout'],'w')
> >>                         if opt['stderr'] == None:
> >>                             opt['stderr'] = subprocess.STDOUT
> >>                         proc = misc.Popen([exe] + arg,  **opt)
> >> @@ -673,7 +675,6 @@
> >>
> >>         tag = (prog, tuple(argument), cwd, nb_submit)
> >>         if isinstance(prog, str):
> >> -
> >>
> >>             opt = {'cwd': cwd,
> >>                    'stdout':stdout,
> >>
> >> === modified file 'madgraph/various/lhe_parser.py'
> >> --- madgraph/various/lhe_parser.py      2016-06-28 11:12:01 +0000
> >> +++ madgraph/various/lhe_parser.py      2016-08-11 23:54:34 +0000
> >> @@ -178,6 +178,8 @@
> >>             if '.gz' in path and isinstance(self, EventFileNoGzip) and\
> >>                 mode == 'r' and os.path.exists(path[:-3]):
> >>                 super(EventFile, self).__init__(path[:-3], mode, *args,
> >> **opt)
> >> +            else:
> >> +                raise
> >>
> >>         self.banner = ''
> >>         if mode == 'r':
> >> @@ -506,7 +508,26 @@
> >>         else:
> >>             return out
> >>
> >> +    def split(self, nb_event=0):
> >> +        """split the file in multiple file. Do not change the weight!"""
> >>
> >> +        nb_file = -1
> >> +        for i, event in enumerate(self):
> >> +            if i % nb_event == 0:
> >> +                if i:
> >> +                    #close previous file
> >> +                    current.write('</LesHouchesEvent>\n')
> >> +                    current.close()
> >> +                # create the new file
> >> +                nb_file +=1
> >> +                current = open('%s_%s.lhe' % (self.name, nb_file),'w')
> >> +                current.write(self.banner)
> >> +            current.write(str(event))
> >> +        if i!=0:
> >> +            current.write('</LesHouchesEvent>\n')
> >> +            current.close()
> >> +        return nb_file +1
> >> +
> >>     def update_Hwu(self, hwu, fct, name='lhe', keep_wgt=True):
> >>
> >>         first=True
> >> @@ -620,13 +641,14 @@
> >>        The number of events in each file need to be provide in advance
> >>        (if not provide the file is first read to find that number"""
> >>
> >> -    def __new__(cls, start_list=[]):
> >> +    def __new__(cls, start_list=[],parse=True):
> >>         return object.__new__(MultiEventFile)
> >>
> >> -    def __init__(self, start_list=[]):
> >> +    def __init__(self, start_list=[], parse=True):
> >>         """if trunc_error is define here then this allow
> >>         to only read all the files twice and not three times."""
> >>         self.files = []
> >> +        self.parsefile = parse #if self.files is formatted or just the
> >> path
> >>         self.banner = ''
> >>         self.initial_nb_events = []
> >>         self.total_event_in_files = 0
> >> @@ -636,8 +658,11 @@
> >>         self.across = []
> >>         self.scales = []
> >>         if start_list:
> >> -            for p in start_list:
> >> -                self.add(p)
> >> +            if parse:
> >> +                for p in start_list:
> >> +                    self.add(p)
> >> +            else:
> >> +                self.files = start_list
> >>         self._configure = False
> >>
> >>     def add(self, path, cross, error, across):
> >> @@ -881,6 +906,66 @@
> >>
> >>         return super(MultiEventFile, self).unweight(outputpath,
> >> get_wgt_multi, **opts)
> >>
> >> +    def write(self, path, random=False, banner=None, get_info=False):
> >> +        """ """
> >> +
> >> +        if isinstance(path, str):
> >> +            out = EventFile(path, 'w')
> >> +            if self.parsefile and not banner:
> >> +                banner = self.files[0].banner
> >> +            elif not banner:
> >> +                firstlhe = EventFile(self.files[0])
> >> +                banner = firstlhe.banner
> >> +        else:
> >> +            out = path
> >> +        if banner:
> >> +            out.write(banner)
> >> +        nb_event = 0
> >> +        info = collections.defaultdict(float)
> >> +        if random and self.open:
> >> +            for event in self:
> >> +                nb_event +=1
> >> +                out.write(event)
> >> +                if get_info:
> >> +                    event.parse_reweight()
> >> +                    for key, value in event.reweight_data.items():
> >> +                        info[key] += value
> >> +                    info['central'] += event.wgt
> >> +        elif not random:
> >> +            for i,f in enumerate(self.files):
> >> +                #check if we need to parse the file or not
> >> +                if not self.parsefile:
> >> +                    if i==0:
> >> +                        try:
> >> +                            lhe = firstlhe
> >> +                        except:
> >> +                            lhe = EventFile(f)
> >> +                    else:
> >> +                        lhe = EventFile(f)
> >> +                else:
> >> +                    lhe = f
> >> +                for event in lhe:
> >> +                    nb_event +=1
> >> +                    if get_info:
> >> +                        event.parse_reweight()
> >> +                        for key, value in event.reweight_data.items():
> >> +                            info[key] += value
> >> +                        info['central'] += event.wgt
> >> +                    out.write(str(event))
> >> +                lhe.close()
> >> +        out.write("</LesHouchesEvents>\n")
> >> +        return nb_event, info
> >> +
> >> +    def remove(self):
> >> +        """ """
> >> +        if self.parsefile:
> >> +            for f in self.files:
> >> +                os.remove(f.name)
> >> +        else:
> >> +            for f in self.files:
> >> +                os.remove(f)
> >> +
> >> +
> >>
> >> class Event(list):
> >>     """Class storing a single event information (list of particles +
> >> global information)"""
> >> @@ -1025,7 +1110,7 @@
> >>         self.reweight_order = []
> >>         start, stop = self.tag.find('<rwgt>'), self.tag.find('</rwgt>')
> >>         if start != -1 != stop :
> >> -            pattern = re.compile(r'''<\s*wgt
> >> id=(?:\'|\")(?P<id>[^\'\"]+)(?:\'|\")\s*>\s*(?P<val>[\ded+-.
> >> ]*)\s*</wgt>''')
> >> +            pattern = re.compile(r'''<\s*wgt
> >> id=(?:\'|\")(?P<id>[^\'\"]+)(?:\'|\")\s*>\s*(?P<val>[\ded+-.
> >> ]*)\s*</wgt>''',re.I)
> >>             data = pattern.findall(self.tag)
> >>             try:
> >>                 self.reweight_data = dict([(pid, float(value)) for (pid,
> >> value) in data
> >> @@ -1046,7 +1131,54 @@
> >>
> >>             text = self.tag[start+8:stop]
> >>             self.nloweight = NLO_PARTIALWEIGHT(text, self)
> >> -
> >> +        return self.nloweight
> >> +
> >> +    def parse_lo_weight(self):
> >> +        """ """
> >> +        if hasattr(self, 'loweight'):
> >> +            return self.loweight
> >> +
> >> +        start, stop = self.tag.find('<mgrwt>'), self.tag.find('</mgrwt>')
> >> +
> >> +        if start != -1 != stop :
> >> +            text = self.tag[start+8:stop]
> >> +#<rscale>  3 0.29765919e+03</rscale>
> >> +#<asrwt>0</asrwt>
> >> +#<pdfrwt beam="1">  1       21 0.15134321e+00 0.29765919e+03</pdfrwt>
> >> +#<pdfrwt beam="2">  1       21 0.38683649e-01 0.29765919e+03</pdfrwt>
> >> +#<totfact> 0.17315115e+03</totfact>
> >> +            self.loweight={}
> >> +            for line in text.split('\n'):
> >> +                line = line.replace('<', ' <').replace("'",'"')
> >> +                if 'rscale' in line:
> >> +                    _, nqcd, scale, _ = line.split()
> >> +                    self.loweight['n_qcd'] = int(nqcd)
> >> +                    self.loweight['ren_scale'] = float(scale)
> >> +                elif '<pdfrwt beam="1"' in line:
> >> +                    args = line.split()
> >> +                    self.loweight['n_pdfrw1'] = int(args[2])
> >> +                    npdf = self.loweight['n_pdfrw1']
> >> +                    self.loweight['pdf_pdg_code1'] = [int(i) for i in
> >> args[3:3+npdf]]
> >> +                    self.loweight['pdf_x1'] = [float(i) for i in
> >> args[3+npdf:3+2*npdf]]
> >> +                    self.loweight['pdf_q1'] = [float(i) for i in
> >> args[3+2*npdf:3+3*npdf]]
> >> +                elif '<pdfrwt beam="2"' in line:
> >> +                    args = line.split()
> >> +                    self.loweight['n_pdfrw2'] = int(args[2])
> >> +                    npdf = self.loweight['n_pdfrw2']
> >> +                    self.loweight['pdf_pdg_code2'] = [int(i) for i in
> >> args[3:3+npdf]]
> >> +                    self.loweight['pdf_x2'] = [float(i) for i in
> >> args[3+npdf:3+2*npdf]]
> >> +                    self.loweight['pdf_q2'] = [float(i) for i in
> >> args[3+2*npdf:3+3*npdf]]
> >> +                elif '<asrwt>' in line:
> >> +                    args = line.replace('>','> ').split()
> >> +                    nalps = int(args[1])
> >> +                    self.loweight['asrwt'] = [float(a) for a in
> >> args[2:2+nalps]]
> >> +
> >> +                elif 'totfact' in line:
> >> +                    args = line.split()
> >> +                    self.loweight['tot_fact'] = float(args[1])
> >> +        else:
> >> +            return None
> >> +        return self.loweight
> >>
> >>     def parse_matching_scale(self):
> >>         """Parse the line containing the starting scale for the shower"""
> >> @@ -1347,8 +1479,6 @@
> >>
> >>     def get_helicity(self, get_order, allow_reversed=True):
> >>         """return a list with the helicities in the order asked for"""
> >> -
> >> -
> >>
> >>         #avoid to modify the input
> >>         order = [list(get_order[0]), list(get_order[1])]
> >> @@ -1610,10 +1740,40 @@
> >>         for particle in self:
> >>             if particle.status != 1:
> >>                 continue
> >> -            scale += particle.mass**2 + particle.momentum.pt**2
> >> +            p = FourMomentum(particle)
> >> +            scale += math.sqrt(p.mass_sqr + p.pt**2)
> >>
> >>         return prefactor * scale
> >>
> >> +
> >> +    def get_et_scale(self, prefactor=1):
> >> +
> >> +        scale = 0
> >> +        for particle in self:
> >> +            if particle.status != 1:
> >> +                continue
> >> +            p = FourMomentum(particle)
> >> +            pt = p.pt
> >> +            if (pt>0):
> >> +                scale += p.E*pt/math.sqrt(pt**2+p.pz**2)
> >> +
> >> +        return prefactor * scale
> >> +
> >> +    def get_sqrts_scale(self, prefactor=1):
> >> +
> >> +        scale = 0
> >> +        init = []
> >> +        for particle in self:
> >> +            if particle.status == -1:
> >> +                init.append(FourMomentum(particle))
> >> +        if len(init) == 1:
> >> +            return init[0].mass
> >> +        elif len(init)==2:
> >> +            return math.sqrt((init[0]+init[1])**2)
> >> +
> >> +
> >> +
> >> +
> >>     def get_momenta_str(self, get_order, allow_reversed=True):
> >>         """return the momenta str in the order asked for"""
> >>
> >> @@ -1686,7 +1846,7 @@
> >>             py = obj.py
> >>             pz = obj.pz
> >>             E = obj.E
> >> -        elif isinstance(obj, list):
> >> +        elif isinstance(obj, (list, tuple)):
> >>             assert len(obj) ==4
> >>             E = obj[0]
> >>             px = obj[1]
> >> @@ -1787,11 +1947,14 @@
> >>         if power == 1:
> >>             return FourMomentum(self)
> >>         elif power == 2:
> >> -            return self.mass_sqr()
> >> +            return self.mass_sqr
> >>
> >>     def __repr__(self):
> >>         return 'FourMomentum(%s,%s,%s,%s)' % (self.E, self.px,
> >> self.py,self.pz)
> >>
> >> +    def get_tuple(self):
> >> +        return (self.E, self.px, self.py,self.pz)
> >> +
> >>     def boost(self, mom):
> >>         """mom 4-momenta is suppose to be given in the rest frame of this
> >> 4-momenta.
> >>         the output is the 4-momenta in the frame of this 4-momenta
> >> @@ -1809,6 +1972,36 @@
> >>                            pz=mom.pz + self.pz * lf)
> >>         else:
> >>             return FourMomentum(mom)
> >> +
> >> +    def zboost(self, pboost=None, E=0, pz=0):
> >> +        """Both momenta should be in the same frame.
> >> +           The boost perform correspond to the boost required to set
> >> pboost at
> >> +           rest (only z boost applied).
> >> +        """
> >> +        if isinstance(pboost, FourMomentum):
> >> +            E = pboost.E
> >> +            pz = pboost.pz
> >> +
> >> +        #beta = pz/E
> >> +        gamma = E / math.sqrt(E**2-pz**2)
> >> +        gammabeta = pz  / math.sqrt(E**2-pz**2)
> >> +
> >> +        out =  FourMomentum([gamma*self.E - gammabeta*self.pz,
> >> +                            self.py,
> >> +                            self.pz,
> >> +                            gamma*self.pz - gammabeta*self.E])
> >> +
> >> +        if abs(out.pz) < 1e-6 * out.E:
> >> +            out.pz = 0
> >> +        return out
> >> +
> >> +
> >> +
> >> +
> >> +
> >> +        """mom 4-momenta is suppose to be given in the rest frame of this
> >> 4-momenta.
> >> +        the output is the 4-momenta in the frame of this 4-momenta
> >> +        function copied from HELAS routine."""
> >>
> >>
> >> class OneNLOWeight(object):
> >> @@ -1819,6 +2012,20 @@
> >>         if isinstance(input, str):
> >>             self.parse(input)
> >>
> >> +    def __str__(self):
> >> +
> >> +        out = """        pwgt: %(pwgt)s
> >> +        born, real : %(born)s %(real)s
> >> +        pdgs : %(pdgs)s
> >> +        bjks : %(bjks)s
> >> +        scales**2, gs: %(scales2)s %(gs)s
> >> +        born/real related : %(born_related)s %(real_related)s
> >> +        type / nfks : %(type)s  %(nfks)s
> >> +        to merge : %(to_merge_pdg)s in %(merge_new_pdg)s
> >> +        ref_wgt :  %(ref_wgt)s""" % self.__dict__
> >> +        return out
> >> +
> >> +
> >>     def parse(self, text):
> >>         """parse the line and create the related object"""
> >>         #0.546601845792D+00 0.000000000000D+00 0.000000000000D+00
> >> 0.119210435309D+02 0.000000000000D+00  5 -1 2 -11 12 21 0 0.24546101D-01
> >> 0.15706890D-02 0.12586055D+04 0.12586055D+04 0.12586055D+04  1  2  2  2  5
> >> 2  2 0.539995789976D+04
> >> @@ -2055,6 +2262,42 @@
> >>         @property
> >>         def aqcd(self):
> >>             return self.event.aqcd
> >> +
> >> +        def get_ht_scale(self, prefactor=1):
> >> +
> >> +            scale = 0
> >> +            for particle in self:
> >> +                p = particle
> >> +                scale += math.sqrt(p.mass_sqr + p.pt**2)
> >> +
> >> +            return prefactor * scale
> >> +
> >> +        def get_et_scale(self, prefactor=1):
> >> +
> >> +            scale = 0
> >> +            for particle in self:
> >> +                p = particle
> >> +                pt = p.pt
> >> +                if (pt>0):
> >> +                    scale += p.E*pt/math.sqrt(pt**2+p.pz**2)
> >> +
> >> +            return prefactor * scale
> >> +
> >> +
> >> +        def get_sqrts_scale(self, event,prefactor=1):
> >> +
> >> +            scale = 0
> >> +            nb_init = 0
> >> +            for particle in event:
> >> +                if particle.status == -1:
> >> +                    nb_init+=1
> >> +            if nb_init == 1:
> >> +                return self[0].mass
> >> +            elif nb_init==2:
> >> +                return math.sqrt((self[0]+self[1])**2)
> >> +
> >> +
> >> +
> >>
> >>     def __init__(self, input, event):
> >>
> >>
> >> === modified file 'madgraph/various/misc.py'
> >> --- madgraph/various/misc.py    2016-05-26 00:13:49 +0000
> >> +++ madgraph/various/misc.py    2016-08-11 23:54:34 +0000
> >> @@ -1329,6 +1329,7 @@
> >>     if not __debug__:
> >>         return
> >>
> >> +    use_print = False
> >>     import inspect
> >>     if opt.has_key('log'):
> >>         log = opt['log']
> >> @@ -1338,6 +1339,8 @@
> >>         level = opt['level']
> >>     else:
> >>         level = logging.getLogger('madgraph').level
> >> +        if level == 0:
> >> +            use_print = True
> >>         #print  "madgraph level",level
> >>         #if level == 20:
> >>         #    level = 10 #avoid info level
> >> @@ -1365,9 +1368,12 @@
> >>     else:
> >>         intro = ''
> >>
> >> -
> >> -    log.log(level, ' '.join([intro]+[str(a) for a in args]) + \
> >> +    if not use_print:
> >> +        log.log(level, ' '.join([intro]+[str(a) for a in args]) + \
> >>                    ' \033[1;30m[%s at line %s]\033[0m' %
> >> (os.path.basename(filename), lineno))
> >> +    else:
> >> +        print ' '.join([intro]+[str(a) for a in args]) + \
> >> +                   ' \033[1;30m[%s at line %s]\033[0m' %
> >> (os.path.basename(filename), lineno)
> >>     return
> >>
> >> ############################################################
> >> ####################
> >> @@ -1590,3 +1596,72 @@
> >> except:
> >>     def apple_notify(subtitle, info_text, userInfo={}):
> >>         return
> >> +
> >> +
> >> +python_lhapdf=None
> >> +def import_python_lhapdf(lhapdfconfig):
> >> +    """load the python module of lhapdf return None if it can not be
> >> loaded"""
> >> +
> >> +    #save the result to have it faster and avoid the segfault at the
> >> second try if lhapdf is not compatible
> >> +    global python_lhapdf
> >> +    if python_lhapdf:
> >> +        if python_lhapdf == -1:
> >> +            return None
> >> +        else:
> >> +            return python_lhapdf
> >> +
> >> +    use_lhapdf=False
> >> +    try:
> >> +        lhapdf_libdir=subprocess.Popen([lhapdfconfig,'--libdir'],\
> >> +                                           stdout=subprocess.PIPE).
> >> stdout.read().strip()
> >> +    except:
> >> +        use_lhapdf=False
> >> +    else:
> >> +        try:
> >> +            candidates=[dirname for dirname in os.listdir(lhapdf_libdir) \
> >> +                            if os.path.isdir(os.path.join(
> >> lhapdf_libdir,dirname))]
> >> +        except OSError:
> >> +            candidates=[]
> >> +        for candidate in candidates:
> >> +            if os.path.isfile(os.path.join(lhapdf_libdir,candidate,'site-
> >> packages','lhapdf.so')):
> >> +                sys.path.insert(0,os.path.join(lhapdf_libdir,candidate,'
> >> site-packages'))
> >> +                try:
> >> +                    import lhapdf
> >> +                    use_lhapdf=True
> >> +                    break
> >> +                except ImportError:
> >> +                    sys.path.pop(0)
> >> +                    continue
> >> +    if not use_lhapdf:
> >> +        try:
> >> +            candidates=[dirname for dirname in
> >> os.listdir(lhapdf_libdir+'64') \
> >> +                            if os.path.isdir(os.path.join(
> >> lhapdf_libdir+'64',dirname))]
> >> +        except OSError:
> >> +            candidates=[]
> >> +        for candidate in candidates:
> >> +            if os.path.isfile(os.path.join(lhapdf_libdir+'64',candidate,'
> >> site-packages','lhapdf.so')):
> >> +                sys.path.insert(0,os.path.join(lhapdf_libdir+'64',
> >> candidate,'site-packages'))
> >> +                try:
> >> +                    import lhapdf
> >> +                    use_lhapdf=True
> >> +                    break
> >> +                except ImportError:
> >> +                    sys.path.pop(0)
> >> +                    continue
> >> +        if not use_lhapdf:
> >> +            try:
> >> +                import lhapdf
> >> +                use_lhapdf=True
> >> +            except ImportError:
> >> +                print 'fail'
> >> +                logger.warning("Failed to access python version of
> >> LHAPDF: "\
> >> +                                   "If the python interface to LHAPDF is
> >> available on your system, try "\
> >> +                                   "adding its location to the PYTHONPATH
> >> environment variable and the"\
> >> +                                   "LHAPDF library location to
> >> LD_LIBRARY_PATH (linux) or DYLD_LIBRARY_PATH (mac os x).")
> >> +
> >> +    if use_lhapdf:
> >> +        python_lhapdf = lhapdf
> >> +    else:
> >> +        python_lhapdf = None
> >> +    python_lhapdf.setVerbosity(0)
> >> +    return python_lhapdf
> >>
> >> === added file 'madgraph/various/systematics.py'
> >> --- madgraph/various/systematics.py     1970-01-01 00:00:00 +0000
> >> +++ madgraph/various/systematics.py     2016-08-11 23:54:34 +0000
> >> @@ -0,0 +1,696 @@
> >> +###########################################################
> >> #####################
> >> +#
> >> +# Copyright (c) 2016 The MadGraph5_aMC@NLO Development team and
> >> Contributors
> >> +#
> >> +# This file is a part of the MadGraph5_aMC@NLO project, an application
> >> which
> >> +# automatically generates Feynman diagrams and matrix elements for
> >> arbitrary
> >> +# high-energy processes in the Standard Model and beyond.
> >> +#
> >> +# It is subject to the MadGraph5_aMC@NLO license which should accompany
> >> this
> >> +# distribution.
> >> +#
> >> +# For more information, visit madgraph.phys.ucl.ac.be and
> >> amcatnlo.web.cern.ch
> >> +#
> >> +###########################################################
> >> #####################
> >> +from __future__ import division
> >> +if __name__ == "__main__":
> >> +    import sys
> >> +    import os
> >> +    root = os.path.dirname(__file__)
> >> +    if os.path.basename(root) == 'internal':
> >> +        sys.path.append(os.path.dirname(root))
> >> +    else:
> >> +        sys.path.append(os.path.dirname(os.path.dirname(root)))
> >> +
> >> +import lhe_parser
> >> +import banner
> >> +import banner as banner_mod
> >> +import itertools
> >> +import misc
> >> +import math
> >> +import os
> >> +import re
> >> +import sys
> >> +import time
> >> +import StringIO
> >> +
> >> +pjoin = os.path.join
> >> +
> >> +class SystematicsError(Exception):
> >> +    pass
> >> +
> >> +class Systematics(object):
> >> +
> >> +    def __init__(self, input_file, output_file,
> >> +                 start_event=0, stop_event=sys.maxint, write_banner=False,
> >> +                 mur=[0.5,1,2],
> >> +                 muf=[0.5,1,2],
> >> +                 alps=[1],
> >> +                 pdf='errorset', #[(id, subset)]
> >> +                 dyn=[-1],
> >> +                 together=[('mur', 'muf')],
> >> +                 lhapdf_config=misc.which('lhapdf-config'),
> >> +                 log=lambda x: sys.stdout.write(str(x)+'\n')
> >> +                 ):
> >> +
> >> +        # INPUT/OUTPUT FILE
> >> +        if isinstance(input_file, str):
> >> +            self.input = lhe_parser.EventFile(input_file)
> >> +        else:
> >> +            self.input = input_file
> >> +        self.output_path = output_file
> >> +        if output_file != None:
> >> +            if isinstance(output_file, str):
> >> +                if output_file == input_file:
> >> +                    directory,name = os.path.split(output_file)
> >> +                    new_name = pjoin(directory, '.tmp_'+name)
> >> +                    self.output =  lhe_parser.EventFile(new_name, 'w')
> >> +                else:
> >> +                    self.output =  lhe_parser.EventFile(output_file, 'w')
> >> +            else:
> >> +                self.output = output_file
> >> +        self.log = log
> >> +
> >> +        #get some information from the run_card.
> >> +        self.banner = banner_mod.Banner(self.input.banner)
> >> +        self.force_write_banner = bool(write_banner)
> >> +        self.orig_dyn = self.banner.get('run_card',
> >> 'dynamical_scale_choice')
> >> +        self.orig_pdf = self.banner.run_card.get_lhapdf_id()
> >> +
> >> +        #check for beam
> >> +        beam1, beam2 = self.banner.get_pdg_beam()
> >> +        if abs(beam1) != 2212 and abs(beam2) != 2212:
> >> +            raise SystematicsError, 'can only reweight proton beam'
> >> +        elif abs(beam1) != 2212:
> >> +            self.b1 = 0
> >> +            self.b2 = beam2//2212
> >> +        elif abs(beam2) != 2212:
> >> +            self.b1 = beam1//2212
> >> +            self.b2 = 0
> >> +        else:
> >> +            self.b1 = beam1//2212
> >> +            self.b2 = beam2//2212
> >> +
> >> +        if isinstance(self.banner.run_card, banner_mod.RunCardLO):
> >> +            self.is_lo = True
> >> +        else:
> >> +            self.is_lo = False
> >> +
> >> +        # MUR/MUF/ALPS PARSING
> >> +        if isinstance(mur, str):
> >> +            mur = mur.split(',')
> >> +        self.mur=[float(i) for i in mur]
> >> +        if isinstance(muf, str):
> >> +            muf = muf.split(',')
> >> +        self.muf=[float(i) for i in muf]
> >> +
> >> +        if isin
> >> ...
> >>
> >> [Message clipped]
> >
> >
> >
> >
> > --
> > Valentin
> >
> > https://code.launchpad.net/~maddevelopers/mg5amcnlo/python_syscalc/+merge/30
> 2735
> > Your team MadDevelopers is subscribed to branch
> lp:~maddevelopers/mg5amcnlo/2.5.0.