Merge lp:~maddevelopers/mg5amcnlo/FixMA5 into lp:~mg5core2/mg5amcnlo/2.6.1

[Valentin Hirschi]

1. Fixed the handling of MA5 output paths so as to comply with the new standards introduced in MA5 v1.6.32.

2. The installation and handling of MA5 has been adjusted so as to make sure that MG5 and MA5 version are compatible.
     Namely:
        MG5 v2.6+ requires MA5 v1.6.32+
        MG5 v2.6- requires MA5 v1.6.21-
This will require Benjamin to set up the online links I asked him to setup and for us to update the online version on the servers of the HEPToolsInstaller.
The links you should setup Benj are:

http://madanalysis.irmp.ucl.ac.be/raw-attachment/wiki/MA5SandBox/ma5_latest.tgz

with v1.6.32 and

http://madanalysis.irmp.ucl.ac.be/raw-attachment/wiki/MA5SandBox/ma5_v_1_6_21.tgz

with v1.6.21

And as soon as we are done with this merge, you can remove the following old link:

http://madanalysis.irmp.ucl.ac.be/raw-attachment/wiki/MA5SandBox/ma5.tgz

For now Olivier, you can test the new MA5 version 1.6.32 by installing it with:
   install madanalysis5 --madanalysis5_tarball=https://madanalysis.irmp.ucl.ac.be/raw-attachment/wiki/MA5SandBox/ma5_new.tgz

And Benj, please check that the handling of the recasting output (path-wise) is correctly handled with this new version (that is, you can find the CLs link in the MG5 HTML summary).

Let us please have this in for the next release, also because Kentarou needs it for a school.

[lp:~maddevelopers/mg5amcnlo/FixMA5]

300. By Valentin Hirschi on 2017-11-20

1. Small Fix, I should require MG5aMC version 2.6.1+ not 2.6.0+

[Valentin Hirschi (valentin-hirschi)]

Sorry, the above should read:

     Namely:
        MG5 v2.6.1+ requires MA5 v1.6.32+
        MG5 v2.6.1- requires MA5 v1.6.21-

I just modified and pushed the code so as to match the above.

[lp:~maddevelopers/mg5amcnlo/FixMA5]

301. By Valentin Hirschi on 2017-11-21

1. Reworked the version check for MA5 so as to apply it both at set_configuration and do_set stage.

[Olivier Mattelaer (olivier-mattelaer)]

Hi Valentin,

Perfect, nicely done. just one minor comment to fix before merging.

1)
@@ -2755,6 +2760,7 @@
                 shutil.copy(target, pjoin(self.me_dir,'Events',self.run_name,carboncopy_name))
             else:
                 logger.error('MadAnalysis5 failed to create PDF output')
+ stop

stop is not a valid python syntax. Is that a debug statement that you forget to remove?

review: Approve

[lp:~maddevelopers/mg5amcnlo/FixMA5]

302. By Valentin Hirschi on 2017-11-23

1. Removal of a left-over 'stop' debug statement.

[Valentin Hirschi (valentin-hirschi)]

On Thu, Nov 23, 2017 at 12:53 PM, Olivier Mattelaer <
<email address hidden>> wrote:

> Review: Approve
>
> Hi Valentin,
>
> Perfect, nicely done. just one minor comment to fix before merging.
>
> 1)
> @@ -2755,6 +2760,7 @@
> shutil.copy(target,
> pjoin(self.me_dir,'Events',self.run_name,carboncopy_name))
> else:
> logger.error('MadAnalysis5 failed to create PDF output')
> + stop
>
> stop is not a valid python syntax. Is that a debug statement that you
> forget to remove?
>

Yes, thank you for having spotted this. I pushed the removal of this line.

>
>
>
>
>
> Diff comments:
>
> >
> > === modified file 'madgraph/interface/common_run_interface.py'
> > --- madgraph/interface/common_run_interface.py 2017-11-17
> 12:35:40 +0000
> > +++ madgraph/interface/common_run_interface.py 2017-11-21
> 13:27:59 +0000
> > @@ -2755,6 +2760,7 @@
> > shutil.copy(target,
> pjoin(self.me_dir,'Events',self.run_name,carboncopy_name))
> > else:
> > logger.error('MadAnalysis5 failed to create PDF output')
> > + stop
>
> Well stop is not a valid python syntax -> should be raise Exception
> "MadAnalysis5 failed to create PDF output" if you really want to make the
> code to crash (but do we want that?)
>
> > if MA5_runtag!='default':
> > logger.info("MadAnalysis5 successfully completed the "+
> > "%s. Reported results are placed in:"%("analysis
> '%s'"%MA5_runtag
>
>
> --
> https://code.launchpad.net/~maddevelopers/mg5amcnlo/FixMA5/+merge/333963
> Your team MadDevelopers is subscribed to branch
> lp:~maddevelopers/mg5amcnlo/FixMA5.
>

--
Valentin
--
Valentin

=== modified file 'VERSION'
--- VERSION	2017-08-16 21:28:30 +0000
+++ VERSION	2017-11-23 12:02:36 +0000
@@ -1,4 +1,4 @@
-version = 2.6.0
+version = 2.6.1
 date = 2017-08-16
 
 

=== modified file 'madgraph/interface/common_run_interface.py'
--- madgraph/interface/common_run_interface.py	2017-11-17 12:35:40 +0000
+++ madgraph/interface/common_run_interface.py	2017-11-23 12:02:36 +0000
@@ -2661,8 +2661,11 @@
         # Make sure to only run over one analysis over each fifo.
         used_up_fifos = []
         # Now loop over the different MA5_runs
-        for MA5_runtag, MA5_cmds in MA5_cmds_list:
+        for MA5_run_number, (MA5_runtag, MA5_cmds) in enumerate(MA5_cmds_list):
             
+            # Since we place every MA5 run in a fresh new folder, the MA5_run_number
+            # is always zero.
+            MA5_run_number = 0
             # Bypass the banner.
             MA5_interpreter.setLogLevel(100)
             # Make sure to properly initialize MA5 interpreter
@@ -2689,7 +2692,7 @@
                 pjoin(self.me_dir,'Events',self.run_name,'%s_MA5_%s.log'%(self.run_tag,MA5_runtag))):
                 # Unsuccessful MA5 run, we therefore stop here.
                 return
-            
+
             if MA5_runtag.startswith('_reco_'):
                 # When doing a reconstruction we must first link the event file
                 # created with MA5 reconstruction and then directly proceed to the
@@ -2712,8 +2715,8 @@
                     reco_output = pjoin(self.me_dir,
                            'MA5_%s_ANALYSIS%s_%d'%(mode.upper(),MA5_runtag,i+1))
                     # Look for either a root or .lhe.gz output
-                    reco_event_file = misc.glob('*.lhe.gz',pjoin(reco_output,'Output','_reco_events'))+\
-                                      misc.glob('*.root',pjoin(reco_output,'Output','_reco_events'))
+                    reco_event_file = misc.glob('*.lhe.gz',pjoin(reco_output,'Output','_reco_events','lheEvents0_%d'%MA5_run_number))+\
+                                       misc.glob('*.root',pjoin(reco_output,'Output','_reco_events', 'RecoEvents0_%d'%MA5_run_number))
                     if len(reco_event_file)==0:
                         raise MadGraph5Error, "MadAnalysis5 failed to produce the "+\
                   "reconstructed event file for reconstruction '%s'."%MA5_runtag[6:]
@@ -2722,13 +2725,15 @@
                     shutil.move(reco_output,pjoin(self.me_dir,'HTML',
                                  self.run_name,'%s_MA5_%s_ANALYSIS%s_%d'%
                                     (self.run_tag,mode.upper(),MA5_runtag,i+1)))
+                    
                     # link the reconstructed event file to the run directory
                     links_created.append(os.path.basename(reco_event_file))
+                    parent_dir_name = os.path.basename(os.path.dirname(reco_event_file))
                     files.ln(pjoin(self.me_dir,'HTML',self.run_name,
                       '%s_MA5_%s_ANALYSIS%s_%d'%(self.run_tag,mode.upper(),
-                      MA5_runtag,i+1),'Output','_reco_events',links_created[-1]),
+                      MA5_runtag,i+1),'Output','_reco_events',parent_dir_name,links_created[-1]),
                                       pjoin(self.me_dir,'Events',self.run_name))
-                    
+
                 logger.info("MadAnalysis5 successfully completed the reconstruction "+
                   "'%s'. Links to the reconstructed event files are:"%MA5_runtag[6:])
                 for link in links_created:
@@ -2740,7 +2745,7 @@
               %(mode.upper(),MA5_runtag),'Output','CLs_output_summary.dat')
             else:
                 target = pjoin(self.me_dir,'MA5_%s_ANALYSIS_%s'\
-                                  %(mode.upper(),MA5_runtag),'PDF','main.pdf')
+                    %(mode.upper(),MA5_runtag),'Output','PDF','MadAnalysis5job_%d'%MA5_run_number,'main.pdf')
             has_pdf = True
             if not os.path.isfile(target):
                 has_pdf = False

=== modified file 'madgraph/interface/madgraph_interface.py'
--- madgraph/interface/madgraph_interface.py	2017-11-02 10:22:25 +0000
+++ madgraph/interface/madgraph_interface.py	2017-11-23 12:02:36 +0000
@@ -39,7 +39,6 @@
 import urllib
 import random
 
-
 #useful shortcut
 pjoin = os.path.join
 
@@ -6527,7 +6526,13 @@
                     if not os.path.isfile(pjoin(path,'bin','ma5')):
                         self.options['madanalysis5_path'] = None
                     else:
-                        continue
+                        ma5path = pjoin(MG5DIR, path) if os.path.isfile(pjoin(MG5DIR, path)) else path
+                        message = misc.is_MA5_compatible_with_this_MG5(ma5path)
+                        if not message is None:
+                            self.options['madanalysis5_path'] = None
+                            logger.warning(message)
+                            continue
+ 
                 #this is for hw++
                 if key == 'hwpp_path' and not os.path.isfile(pjoin(MG5DIR, path, 'include', 'Herwig++', 'Analysis', 'BasicConsistency.hh')):
                     if not os.path.isfile(pjoin(path, 'include', 'Herwig++', 'Analysis', 'BasicConsistency.hh')):
@@ -7209,6 +7214,14 @@
 
             self.options[args[0]] = (int(first), int(second))
 
+        elif args[0] == 'madanalysis5_path':
+            ma5path = pjoin(MG5DIR, args[1]) if os.path.isfile(pjoin(MG5DIR, args[1])) else args[1]
+            message = misc.is_MA5_compatible_with_this_MG5(ma5path)
+            if message is None:
+                self.options['madanalysis5_path'] = args[1]
+            else:
+                logger.warning(message)
+
         elif args[0] == 'OLP':
             # Reset the amplitudes, MatrixElements and exporter as they might
             # depend on this option

=== modified file 'madgraph/madevent/gen_crossxhtml.py'
--- madgraph/madevent/gen_crossxhtml.py	2017-08-10 13:09:28 +0000
+++ madgraph/madevent/gen_crossxhtml.py	2017-11-23 12:02:36 +0000
@@ -850,7 +850,7 @@
                 self.parton.append('ma5_plot')                
 
             if 'ma5_html' not in self.parton and \
-               misc.glob(pjoin('%s_MA5_PARTON_ANALYSIS_*'%self['tag'],'HTML','index.html'),html_path):
+               misc.glob(pjoin('%s_MA5_PARTON_ANALYSIS_*'%self['tag'],'Output','HTML','MadAnalysis5job_0','index.html'),html_path):
                 self.parton.append('ma5_html')                
             
             if 'ma5_card' not in self.parton and \
@@ -868,7 +868,7 @@
                 self.madanalysis5_hadron.append('ma5_plot')                
 
             if 'ma5_html' not in self.madanalysis5_hadron and \
-               misc.glob(pjoin('%s_MA5_HADRON_ANALYSIS_*'%self['tag'],'HTML','index.html'),html_path):
+               misc.glob(pjoin('%s_MA5_HADRON_ANALYSIS_*'%self['tag'],'Output','HTML','MadAnalysis5job_0','index.html'),html_path):
                 self.madanalysis5_hadron.append('ma5_html')                
 
             if 'ma5_cls' not in self.madanalysis5_hadron and \
@@ -1055,7 +1055,7 @@
             if 'ma5_html' in self.parton:
                 for result in misc.glob(pjoin('%s_MA5_PARTON_ANALYSIS_*'%self['tag']),
                                         pjoin(self.me_dir,'HTML',self['run_name'])):
-                    target    = pjoin(os.curdir,os.path.relpath(result,self.me_dir),'HTML','index.html')
+                    target    = pjoin(os.curdir,os.path.relpath(result,self.me_dir),'Output','HTML','MadAnalysis5job_0','index.html')
                     link_name = os.path.basename(result).split('PARTON_ANALYSIS')[-1]
                     out += """ <a href="%s">MA5_report%s</a> """%(target, link_name)
                         
@@ -1164,7 +1164,7 @@
                 linked_analysis = False
                 for result in misc.glob(pjoin('%s_MA5_HADRON_ANALYSIS_*'%self['tag']),
                                         pjoin(self.me_dir,'HTML',self['run_name'])):
-                    target    = pjoin(os.curdir,os.path.relpath(result,self.me_dir),'HTML','index.html')
+                    target    = pjoin(os.curdir,os.path.relpath(result,self.me_dir),'Output','HTML','MadAnalysis5job_0','index.html')
                     link_name = os.path.basename(result).split('HADRON_ANALYSIS')[-1]
                     if link_name.startswith('_reco_'):
                         continue
@@ -1180,7 +1180,7 @@
                     for result in misc.glob(pjoin('%s_MA5_HADRON_ANALYSIS_*'%self['tag']),
                                             pjoin(self.me_dir,'HTML',self['run_name'])):
                         target    = pjoin(os.curdir,os.path.relpath(
-                                        result,self.me_dir),'HTML','index.html')
+                                        result,self.me_dir),'Output','HTML','MadAnalysis5job_0','index.html')
                         link_name = os.path.basename(result).split('HADRON_ANALYSIS')[-1]
                         if not link_name.startswith('_reco_'):
                             continue

=== modified file 'madgraph/various/misc.py'
--- madgraph/various/misc.py	2017-10-05 15:19:01 +0000
+++ madgraph/various/misc.py	2017-11-23 12:02:36 +0000
@@ -29,6 +29,7 @@
 import shutil
 import traceback
 import gzip as ziplib
+from distutils.version import LooseVersion, StrictVersion
 
 try:
     # Use in MadGraph
@@ -149,6 +150,53 @@
 
     return time_info
 
+
+#===============================================================================
+# Test the compatibility of a given version of MA5 with this version of MG5
+#===============================================================================
+def is_MA5_compatible_with_this_MG5(ma5path):
+    """ Returns None if compatible or, it not compatible, a string explaining 
+    why it is so."""
+
+    ma5_version = None
+    try:
+        for line in open(pjoin(ma5path,'version.txt'),'r').read().split('\n'):
+            if line.startswith('MA5 version :'):
+                ma5_version=LooseVersion(line[13:].strip())
+                break
+    except:
+        ma5_version = None
+
+    if ma5_version is None:
+        reason = "No MadAnalysis5 version number could be read from the path supplied '%s'."%ma5path
+        reason += "\nThe specified version of MadAnalysis5 will not be active in your session."
+        return reason
+        
+    mg5_version = None
+    try:
+        info = get_pkg_info()        
+        mg5_version = LooseVersion(info['version'])
+    except:
+        mg5_version = None
+    
+    # If version not reckognized, then carry on as it's probably a development version
+    if not mg5_version:
+        return None
+    
+    if mg5_version < LooseVersion("2.6.1") and ma5_version >= LooseVersion("1.6.32"):
+        reason =  "This active MG5aMC version is too old (v%s) for your selected version of MadAnalysis5 (v%s)"%(mg5_version,ma5_version)
+        reason += "\nUpgrade MG5aMC or re-install MA5 from within MG5aMC to fix this compatibility issue."
+        reason += "\nThe specified version of MadAnalysis5 will not be active in your session."
+        return reason
+
+    if mg5_version >= LooseVersion("2.6.1") and ma5_version < LooseVersion("1.6.32"):
+        reason = "Your selected version of MadAnalysis5 (v%s) is too old for this active version of MG5aMC (v%s)."%(ma5_version,mg5_version)
+        reason += "\nRe-install MA5 from within MG5aMC to fix this compatibility issue."
+        reason += "\nThe specified version of MadAnalysis5 will not be active in your session."
+        return reason
+
+    return None
+
 #===============================================================================
 # Find the subdirectory which includes the files ending with a given extension 
 #===============================================================================