Merge lp:~g-gorman/fluidity/fluidity-petsc-3.3 into lp:fluidity

[Gerard]

This is a branch of fluidity-petsc-dev specifically intended to support PETSc 3.3.

[Stephan Kramer (s-kramer)]

Looks good. This can go once a buildbot queue has been set up and returned green.

review: Approve

[Lawrence Mitchell (wence)]

femtools/tests/test_multigrid.F90 needs a fix too:

#include "petscversion.h" needs to happen before the use statements too

In general, I don't like all this sprinkling of petsc version knowledge everywhere in the source tree. Can we do something like this:

file_that_needs_petsc.F90:

module foo

  use bar
#include "petsc_modules_maybe.h"
  implicit none
#include "petsc_headers_maybe.h"

  ...
  Write code that is petsc version-agnostic
end module foo

petsc_modules_maybe.h is something like

#ifdef HAVE_PETSC
#ifdef HAVE_PETSC_MODULES
   use petsc
#include "petscversion.h"
#if PETSC_VERSION_MINOR == 0
  use ...
#endif
#endif
#endif

and so forth.

Can we then get away without sprinkling all the code with VecSqrt/VecSqrtAbs etc...?

[James Robert Percival (j-percival)]

While I remember and before this gets merged, I think femtools/tests/test_petsc_csr_matrix.F90 has slipped through without being changed for support. Post petsc-3.2 the first call to error handlers must be an MPI_Comm communicator. I have the code which does this but it isn't petsc-3.1 safe at the moment.

[Stephan Kramer (s-kramer)]

Just to recap for those not present at the dev-meeting: we decided to go ahead with this merge as is, so that people depending on petsc 3.3 don't have to wait. A cleanup of the petsc-version specific hackery is still very much appreciated and shall be done in someone's copious free time. Also we discussed possibly dropping support for petsc 3.1 and petsc 3.2, and roll our own petsc 3.3 packages on a ppa as long as Ubuntu's catching up. I'll try to friendly nudge the Debian package maintainer and ask for an upgrade of their package (Debian unstable is on 3.2), so we could simply back(trans?)-port it to Ubuntu.

James: would you mind committing that fix to the unit-test with another ugly "#if PETSC_VERSION_MINOR>=2"...#else ... to make it work with 3.1. Or commit what you have and I'll add the hacks? This stuff will go away as soon as we drop support for 3.1

[AMCG Buildbot (amcg-buildbot)]

Let me know if you get any movement from Debian and if so I'll push this to the PPA. If not I suspect it's not too tricky to roll our own.

[Jon Hill (jon-hill)]

Just a note that HECToR currently supports:

petsc/3.0.0.10
petsc/3.1.09(default)
petsc/3.2.01
petsc/3.2.02
petsc-complex/3.0.0.10
petsc-complex/3.1.09(default)
petsc-complex/3.2.01
petsc-complex/3.2.02

So let me know in plenty of time if we drop 3.1 and 3.2 - it'll try and switch the default petsc we use to 3.2.02 soon.

[Rhodri Davies (rhodri-davies)]

Following on from John's note - I have asked the HECToR team to install a module for PETSC 3.3. This was around 6 weeks back. There is still no module available, but have just given them another poke for an update.

[James Robert Percival (j-percival)]

Stephan, just to confirm, I don't seem to be able upload to this branch since it's private, so I'm emailing my attempt at a protected version of the test supporting both versions directly to Gerard and you.

[Stephan Kramer (s-kramer)]

This branch has now been moved to ~/fluidity-core/fluidity/fluidity-petsc-3.3 so other people can commit.

[Rhodri Davies (rhodri-davies)]

Note that this branch should no longer be considered for merge. The newer version under fluidity-core should be. I will add a further comment there.

=== modified file 'aclocal.m4'
--- aclocal.m4	2012-06-15 12:17:04 +0000
+++ aclocal.m4	2012-07-21 21:13:21 +0000
@@ -497,7 +497,7 @@
       PetscInt  i,j,II,JJ,m,n,its
       PetscInt  Istart,Iend,ione
       PetscErrorCode ierr
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
       PetscBool flg
 #else
       PetscTruth  flg

=== modified file 'configure'
--- configure	2012-07-20 07:26:35 +0000
+++ configure	2012-07-21 21:13:21 +0000
@@ -13002,7 +13002,7 @@
       PetscInt  i,j,II,JJ,m,n,its
       PetscInt  Istart,Iend,ione
       PetscErrorCode ierr
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
       PetscBool flg
 #else
       PetscTruth  flg

=== modified file 'configure.in'
=== modified file 'femtools/Multigrid.F90'
--- femtools/Multigrid.F90	2011-10-26 14:37:06 +0000
+++ femtools/Multigrid.F90	2012-07-21 21:13:21 +0000
@@ -40,9 +40,15 @@
 #include "finclude/petscsys.h"
 #endif
 #endif
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
 #define KSP_NORM_NO KSP_NORM_NONE
 #endif
+#if PETSC_VERSION_MINOR>=3
+#define MatCreateSeqAIJ myMatCreateSeqAIJ
+#define MatCreateMPIAIJ myMatCreateMPIAIJ
+#define MatCreateSeqBAIJ myMatCreateSeqBAIJ
+#define MatCreateMPIBAIJ myMatCreateMPIBAIJ
+#endif
 
 !! Some parameters that change the behaviour of 
 !! the smoothed aggregation method. All of
@@ -687,13 +693,17 @@
   PC:: pc
   PetscErrorCode:: ierr
   
+#if PETSC_VERSION_MINOR>=3
+  call KSPSetType(ksp, KSPCHEBYSHEV, ierr)
+#else
   call KSPSetType(ksp, KSPCHEBYCHEV, ierr)
+#endif
   call KSPSetOperators(ksp, matrix, matrix, SAME_PRECONDITIONER, ierr)
   call KSPSetTolerances(ksp, PETSC_DEFAULT_DOUBLE_PRECISION, &
     PETSC_DEFAULT_DOUBLE_PRECISION, PETSC_DEFAULT_DOUBLE_PRECISION, &
     iterations, ierr)
-#ifdef DOUBLEP
-  call KSPChebychevSetEigenvalues(ksp, emax, emin, ierr)
+#if PETSC_VERSION_MINOR>=3
+  call KSPChebyshevSetEigenvalues(ksp, emax, emin, ierr)
 #else
   call KSPChebychevSetEigenvalues(ksp, emax, emin, ierr)
 #endif
@@ -710,7 +720,7 @@
 integer, intent(out):: maxlevels, coarsesize
 integer, intent(out):: nosmd, nosmu, clustersize
 
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
   PetscBool flag
 #else
   PetscTruth flag
@@ -811,7 +821,7 @@
   
   !
   call VecCopy(diag, sqrt_diag, ierr)
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=3
   call VecSqrtAbs(sqrt_diag, ierr)
 #else
   call VecSqrt(sqrt_diag, ierr)

=== modified file 'femtools/Petsc_Tools.F90'
--- femtools/Petsc_Tools.F90	2012-05-14 08:49:05 +0000
+++ femtools/Petsc_Tools.F90	2012-07-21 21:13:21 +0000
@@ -139,7 +139,13 @@
   public addup_global_assembly
   ! for petsc_numbering:
   public incref, decref, addref
-  
+#if PETSC_VERSION_MINOR>=3
+#define MatCreateSeqAIJ myMatCreateSeqAIJ
+#define MatCreateMPIAIJ myMatCreateMPIAIJ
+#define MatCreateSeqBAIJ myMatCreateSeqBAIJ
+#define MatCreateMPIBAIJ myMatCreateMPIBAIJ
+  public MatCreateSeqAIJ, MatCreateMPIAIJ, MatCreateSeqBAIJ, MatCreateMPIBAIJ
+#endif
 contains
 
   ! Note about definitions in this module:
@@ -1467,7 +1473,69 @@
     PetscErrorCode :: ierr
     call PetscInitialize(PETSC_NULL_CHARACTER, ierr); CHKERRQ(ierr);
   end subroutine Initialize_Petsc
-  
+
+! In petsc-3.3 the MatCreate[B]{Seq|MPI}() routines have changed to MatCreate[B]Aij
+! and MatSetup always needs to be called
+#if PETSC_VERSION_MINOR>=3
+  subroutine MatCreateSeqAIJ(MPI_Comm, nrows, ncols, &
+      nz, nnz, M, ierr)
+    integer, intent(in):: MPI_Comm
+    PetscInt, intent(in):: nrows, ncols, nz
+    PetscInt, dimension(:), intent(in):: nnz
+    Mat, intent(out):: M
+    PetscErrorCode, intent(out):: ierr
+
+    call MatCreateAij(MPI_Comm, nrows, ncols, nrows, ncols, &
+      nz, nnz, 0, PETSC_NULL_INTEGER, M, ierr)
+    call MatSetup(M, ierr)
+
+  end subroutine MatCreateSeqAIJ
+
+  subroutine MatCreateMPIAIJ(MPI_Comm, nprows, npcols, &
+      nrows, ncols, &
+      dnz, dnnz, onz, onnz, M, ierr)
+    integer, intent(in):: MPI_Comm
+    PetscInt, intent(in):: nprows, npcols,nrows, ncols, dnz, onz
+    PetscInt, dimension(:), intent(in):: dnnz, onnz
+    Mat, intent(out):: M
+    PetscErrorCode, intent(out):: ierr
+
+    call MatCreateAij(MPI_Comm, nprows, npcols, nrows, ncols, &
+      dnz, dnnz, onz, onnz, M, ierr)
+    call MatSetup(M, ierr)
+
+  end subroutine MatCreateMPIAIJ
+
+  subroutine MatCreateSeqBAIJ(MPI_Comm, bs, nrows, ncols, &
+      nz, nnz, M, ierr)
+    integer, intent(in):: MPI_Comm
+    PetscInt, intent(in):: bs, nrows, ncols, nz
+    PetscInt, dimension(:), intent(in):: nnz
+    Mat, intent(out):: M
+    PetscErrorCode, intent(out):: ierr
+
+    call MatCreateBAij(MPI_Comm, bs, nrows, ncols, nrows, ncols, &
+      nz, nnz, 0, PETSC_NULL_INTEGER, M, ierr)
+    call MatSetup(M, ierr)
+
+  end subroutine MatCreateSeqBAIJ
+
+  subroutine MatCreateMPIBAIJ(MPI_Comm, bs, nprows, npcols, &
+      nrows, ncols, &
+      dnz, dnnz, onz, onnz, M, ierr)
+    integer, intent(in):: MPI_Comm
+    PetscInt, intent(in):: bs, nprows, npcols,nrows, ncols, dnz, onz
+    PetscInt, dimension(:), intent(in):: dnnz, onnz
+    Mat, intent(out):: M
+    PetscErrorCode, intent(out):: ierr
+
+    call MatCreateBAij(MPI_Comm, bs, nprows, npcols, nrows, ncols, &
+      dnz, dnnz, onz, onnz, M, ierr)
+    call MatSetup(M, ierr)
+
+  end subroutine MatCreateMPIBAIJ
+#endif
+
 #include "Reference_count_petsc_numbering_type.F90"
 end module Petsc_Tools
 

=== modified file 'femtools/Solvers.F90'
--- femtools/Solvers.F90	2012-04-16 16:41:59 +0000
+++ femtools/Solvers.F90	2012-07-21 21:13:21 +0000
@@ -1700,7 +1700,7 @@
          FLAbort("Need petsc_numbering for monitor")
        end if
        call petsc_monitor_setup(petsc_numbering, max_its)
-       call KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL_INTEGER, &
+       call KSPMonitorSet(ksp,MyKSPMonitor,PETSC_NULL_OBJECT, &
             &                     PETSC_NULL_FUNCTION,ierr)
     end if
 
@@ -1842,7 +1842,7 @@
     PCType:: pctype
     PetscReal:: rtol, atol, dtol
     PetscInt:: maxits
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
     PetscBool:: flag
 #else
     PetscTruth:: flag
@@ -2165,9 +2165,6 @@
   PC:: pc
   Vec:: dummy_vec, r, rhs
 
-  ! Stop warnings.
-  ierr = dummy  
-
   !  Build the solution vector  
   call VecZeroEntries(petsc_monitor_x,ierr)
   ! don't pass PETSC_NULL_OBJECT instead of dummy_vec, as petsc

=== modified file 'femtools/Sparse_Tools_Petsc.F90'
--- femtools/Sparse_Tools_Petsc.F90	2012-04-17 08:52:34 +0000
+++ femtools/Sparse_Tools_Petsc.F90	2012-07-21 21:13:21 +0000
@@ -73,6 +73,12 @@
 #include "finclude/petscis.h"
 #endif
 #endif
+#if PETSC_VERSION_MINOR>=3
+#define MatCreateSeqAIJ myMatCreateSeqAIJ
+#define MatCreateMPIAIJ myMatCreateMPIAIJ
+#define MatCreateSeqBAIJ myMatCreateSeqBAIJ
+#define MatCreateMPIBAIJ myMatCreateMPIBAIJ
+#endif
 
   private
   

=== modified file 'femtools/tests/test_multigrid.F90'
--- femtools/tests/test_multigrid.F90	2011-02-28 16:03:39 +0000
+++ femtools/tests/test_multigrid.F90	2012-07-21 21:13:21 +0000
@@ -12,6 +12,7 @@
   use solvers
   use fields
   use parallel_tools
+#include "petscversion.h"
 #ifdef HAVE_PETSC
 #ifdef HAVE_PETSC_MODULES
   use petsc 

=== modified file 'tools/petsc_readnsolve.F90'
--- tools/petsc_readnsolve.F90	2011-11-17 11:59:39 +0000
+++ tools/petsc_readnsolve.F90	2012-07-21 21:13:21 +0000
@@ -70,7 +70,7 @@
 #endif
 #endif
 ! hack around PetscTruth->PetscBool change in petsc 3.2
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
 #define PetscTruth PetscBool
 #endif
   ! options read from command-line (-prns_... options)
@@ -93,7 +93,7 @@
   ! read in the matrix equation and init. guess:
   call PetscViewerBinaryOpen(MPI_COMM_FEMTOOLS, trim(filename), &
      FILE_MODE_READ, viewer, ierr)
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
   ! in petsc 3.2 MatLoad and VecLoad do no longer create a new vector
   call MatCreate(MPI_COMM_FEMTOOLS, matrix, ierr)
   call VecCreate(MPI_COMM_FEMTOOLS, rhs, ierr)
@@ -753,7 +753,7 @@
     ncomponents=size(petsc_numbering%gnn2unn, 2)
     allocate(unns(1:n*ncomponents))
     unns=reshape( petsc_numbering%gnn2unn(1:n,:), (/ n*ncomponents /))
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
     ! for petsc 3.2 we have an extra PetscCopyMode argument
     call ISCreateGeneral(MPI_COMM_FEMTOOLS, &
        size(unns), unns, PETSC_COPY_VALUES, row_indexset, ierr)

=== modified file 'tools/petsc_readnsolve_main.cpp'
--- tools/petsc_readnsolve_main.cpp	2011-10-27 09:56:35 +0000
+++ tools/petsc_readnsolve_main.cpp	2012-07-21 21:13:21 +0000
@@ -40,7 +40,7 @@
   char flml_extension[]=".flml";
   char *flml_file=NULL;
   PetscErrorCode ierr;
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
   PetscBool      flg;
 #else
   PetscTruth     flg;

=== modified file 'tools/test_pressure_solve.F90'
--- tools/test_pressure_solve.F90	2011-10-26 14:37:06 +0000
+++ tools/test_pressure_solve.F90	2012-07-21 21:13:21 +0000
@@ -320,7 +320,7 @@
 #include "finclude/petscpc.h"
 #endif
 
-#if PETSC_VERSION_MINOR==2
+#if PETSC_VERSION_MINOR>=2
     PetscBool:: flag
 #else
     PetscTruth:: flag