Merge lp:~fluidity-core/fluidity/fix-lagr-detectors into lp:fluidity

[Michael Lange]

Fixing several bugs in lagrangian detectors:

 * Bug #1262890: Single lagrangian detectors are initialised with wrong names, causing detector I/O to fail in parallel
 * Bug #1260019: Prevent buffer corruption in exchange_detectors through use of multiple send/receive buffers
 * Re-initialisation of detector names from checkpoint

[Michael Lange (michael-lange)]

Buildbot queue is green.

[Rhodri Davies (rhodri-davies)]

Hi Michael,

The changes looks good to me, although it may be worth somebody that better understands how detectors are handled in parallel taking a look at the MPI changes before the merge is approved.

One issue: there don't seem to be any modifications to tests to verify that these errors will not crop up again in the future. Is there any way some of the lagrangian detector cases could be modified to test these things? I guess a test for the individual lagrangian detector I/O bug could easily be added to an existing test, but how about the other bug?

Rhod

[Jon Hill (jon-hill)]

Yep - these look good. We do need a short test, if possible to verify these changes do fix the bug (I'm positive they do!) and do make the detectors more robust in general. Otherwise, it's an approve from me.

[lp:~fluidity-core/fluidity/fix-lagr-detectors]

4305. By Michael Lange on 2014-01-22

Adding a test for single lagrangian detectors.

4306. By Michael Lange on 2014-01-22

Adding a test for detector checkpointing based on the lagrangian_detectors test.

4307. By Michael Lange on 2014-01-22

Bugfix to allow detector output files to grow larger than 2.1GB. Thanks to Jean Mensa for reporting and providing the fix for this.

4308. By Michael Lange on 2014-01-22

Merge from trunk

[Michael Lange (michael-lange)]

I added tests for detector checkpointing and the single lagrangian detector fix. The buffer corruption issue, however, cannot reliably be triggered and requires a lot of detectors; so I think it is impractical to test for this in small/medium tests.

I have also added Jean Mensa's fix for large detector output files, but again this seems like a corner case that is very hard to catch in buildbot.

[Jon Hill (jon-hill)]

Fine - we can't add tests for 3Gb files on a buildbot, so approved.

review: Approve

[Rhodri Davies (rhodri-davies)]

Hi Michael/Jon,

Sorry - I emailed Michal this earlier rather than add it to the review... At present, there remains work to be done before the merge (email copied below)...

------------

Hi Michael,

Whilst looking over your most recent changes to the fix_lagr_detectors branch, I tried to run the lagrangian_detectors test case locally. I get the following error (with backtrace):

*** FLUIDITY ERROR ***
Source location: (Diagnostic_variables.F90, 2785)
Error message: Failed assertion ierror == MPI_SUCCESS
Backtrace will follow if it is available:
../../bin/fluidity(fprint_backtrace_+0x1f) [0x57be1f]
../../bin/fluidity(__fldebug_MOD_flabort_pinpoint+0x304) [0x5701c3]
../../bin/fluidity(__diagnostic_variables_MOD_write_mpi_out+0x7a0) [0x627c29]
../../bin/fluidity(__diagnostic_variables_MOD_write_detectors+0x1c7d) [0x62af98]
../../bin/fluidity(__diagnostic_variables_MOD_write_diagnostics+0x577d) [0x635d9e]
../../bin/fluidity(__fluids_module_MOD_fluids+0x72a5) [0x57b17c]
../../bin/fluidity(mainfl+0x11d) [0x56d889]
../../bin/fluidity(main+0x177) [0x5641ad]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f785ded476d]
../../bin/fluidity() [0x563d51]
Use addr2line -e <binary> <address> to decipher.
Error is terminal.

I configured as follows:

./configure --enable-debugging --enable-2d-adaptivity ; make clean ; make -j ; make -j fltools

Do you get this error locally?

Buildbot on your branch is also red right now (but apparently not for this test, which confuses me slightly!)... Any ideas?

Rhod

[Jon Hill (jon-hill)]

then we will fix the approve :)

review: Needs Fixing

[lp:~fluidity-core/fluidity/fix-lagr-detectors]

4309. By Michael Lange on 2014-01-24

Reverting the previous bugfix for large detector output files, since it causes buildbot failures. We'll have to re-visit this problem in a separate merge.

[Michael Lange (michael-lange)]

Hi Rhodri,

All tests pass locally for me, so the buildbot error is indeed confusing. Your local error though should to be due to the filesize bugfix I just reverted. Can you please confirm that this works now?

Thanks
Michael

[Rhodri Davies (rhodri-davies)]

Hi Michael,

I can confirm that the error is fixed on my local system, whilst detectors_parallel_adaptivity also passes locally. Sadly the buildbot branch is still red for the detectors_parallel_adaptivity case, however, which I think is a tolerance issue. Are you happy to reduce the final_location_error tolerance slightly (if this is indeed the issue on buildbot of course)? Once this test passes on buildbot I think this merge is good to go.

Rhod

[Xiaohu Guo (xiaohu-guo)]

Hi Michael,
   Just a thought.

   in femtools/Detector_Parallel.F90:35

   should MPI_ISEND also covered by "if (ndet_to_send>0)", otherwise you would have zero messages ?

Cheers
Xiaohu

[lp:~fluidity-core/fluidity/fix-lagr-detectors]

4310. By Michael Lange on 2014-01-29

Avoid errors due to Python import caching in all lagrangian detector tests.

When runnning all test in one Python instance, like buildbot does, Python
would load the wrong readtool.py due to import caching. This can be avoided
by moving the test code into the xml.

[Michael Lange (michael-lange)]

Buildbot is green now.

[Rhodri Davies (rhodri-davies)]

In that case - I think it's good to go. Nice work.

review: Approve

[Jon Hill (jon-hill)]

review: Approve

=== modified file 'femtools/Detector_Parallel.F90'
--- femtools/Detector_Parallel.F90	2011-10-14 07:58:31 +0000
+++ femtools/Detector_Parallel.F90	2014-01-29 10:13:58 +0000
@@ -50,6 +50,11 @@
   type(detector_list_ptr), dimension(:), allocatable, target, save :: detector_list_array
   integer :: num_detector_lists = 0
 
+  type detector_buffer
+     !!< Container type for MPI data buffers
+     real, dimension(:,:), pointer :: ptr
+  end type detector_buffer
+
 contains
 
   subroutine register_detector_list(detector_list)
@@ -301,7 +306,7 @@
     type(detector_linked_list), intent(inout) :: detector_list
     type(detector_linked_list), dimension(:), intent(inout) :: send_list_array
 
-    real, dimension(:,:), allocatable :: detector_buffer
+    type(detector_buffer), dimension(:), allocatable :: send_buffer, recv_buffer
     type(detector_type), pointer :: detector, detector_received
     type(vector_field), pointer :: xfield
     type(halo_type), pointer :: ele_halo
@@ -342,9 +347,10 @@
     end if
     
     ! Send to all procs
+    allocate(send_buffer(nprocs))
     do target_proc=1, nprocs
        ndet_to_send=send_list_array(target_proc)%length
-       allocate(detector_buffer(ndet_to_send,det_size))
+       allocate(send_buffer(target_proc)%ptr(ndet_to_send,det_size))
 
        if (ndet_to_send>0) then
           ewrite(2,*) " Sending", ndet_to_send, "detectors to process", target_proc
@@ -360,9 +366,9 @@
              detector%element = halo_universal_number(ele_halo, detector%element)
 
              if (have_update_vector) then
-                call pack_detector(detector, detector_buffer(j,1:det_size), dim, nstages=n_stages)
+                call pack_detector(detector, send_buffer(target_proc)%ptr(j,1:det_size), dim, nstages=n_stages)
              else
-                call pack_detector(detector, detector_buffer(j,1:det_size), dim)
+                call pack_detector(detector, send_buffer(target_proc)%ptr(j,1:det_size), dim)
              end if
 
              ! delete also advances detector
@@ -372,18 +378,17 @@
        end if
 
        ! getprocno() returns the rank of the processor + 1, hence target_proc-1
-       call MPI_ISEND(detector_buffer,size(detector_buffer), &
+       call MPI_ISEND(send_buffer(target_proc)%ptr,size(send_buffer(target_proc)%ptr(:,:)), &
             & getpreal(), target_proc-1, TAG, MPI_COMM_FEMTOOLS, sendRequest(target_proc), IERROR)
        assert(ierror == MPI_SUCCESS)
 
-       ! deallocate buffer after sending
-       deallocate(detector_buffer)
     end do
 
     allocate( status(MPI_STATUS_SIZE) )
     call get_universal_numbering_inverse(ele_halo, ele_numbering_inverse)
 
     ! Receive from all procs
+    allocate(recv_buffer(nprocs))
     do receive_proc=1, nprocs
        call MPI_PROBE(receive_proc-1, TAG, MPI_COMM_FEMTOOLS, status(:), IERROR) 
        assert(ierror == MPI_SUCCESS)
@@ -392,13 +397,13 @@
        assert(ierror == MPI_SUCCESS)
 
        ndet_received=count/det_size
-       allocate(detector_buffer(ndet_received,det_size))
+       allocate(recv_buffer(receive_proc)%ptr(ndet_received,det_size))
 
        if (ndet_received>0) then
           ewrite(2,*) " Receiving", ndet_received, "detectors from process", receive_proc
        end if
 
-       call MPI_Recv(detector_buffer,count, getpreal(), status(MPI_SOURCE), TAG, MPI_COMM_FEMTOOLS, MPI_STATUS_IGNORE, IERROR)
+       call MPI_Recv(recv_buffer(receive_proc)%ptr,count, getpreal(), status(MPI_SOURCE), TAG, MPI_COMM_FEMTOOLS, MPI_STATUS_IGNORE, IERROR)
        assert(ierror == MPI_SUCCESS)
 
        do j=1, ndet_received
@@ -407,10 +412,10 @@
           ! Unpack routine uses ele_numbering_inverse to translate universal element 
           ! back to local detector element
           if (have_update_vector) then
-             call unpack_detector(detector_received,detector_buffer(j,1:det_size),dim,&
+             call unpack_detector(detector_received,recv_buffer(receive_proc)%ptr(j,1:det_size),dim,&
                     global_to_local=ele_numbering_inverse,coordinates=xfield,nstages=n_stages)
           else
-             call unpack_detector(detector_received,detector_buffer(j,1:det_size),dim,&
+             call unpack_detector(detector_received,recv_buffer(receive_proc)%ptr(j,1:det_size),dim,&
                     global_to_local=ele_numbering_inverse,coordinates=xfield)
           end if
 
@@ -423,8 +428,15 @@
 
           call insert(detector_received, detector_list)           
        end do
-       deallocate(detector_buffer)
-    end do    
+    end do
+
+    ! Deallocate buffers after exchange
+    do target_proc=1, nprocs
+       deallocate(send_buffer(target_proc)%ptr)
+       deallocate(recv_buffer(target_proc)%ptr)
+    end do
+    deallocate(send_buffer)
+    deallocate(recv_buffer)
 
     call MPI_WAITALL(nprocs, sendRequest, MPI_STATUSES_IGNORE, IERROR)
     assert(ierror == MPI_SUCCESS)

=== modified file 'femtools/Diagnostic_variables.F90'
--- femtools/Diagnostic_variables.F90	2013-10-10 07:35:33 +0000
+++ femtools/Diagnostic_variables.F90	2014-01-29 10:13:58 +0000
@@ -1533,7 +1533,7 @@
           default_stat%detector_list%detector_names(static_dete+i)=detector_name
 
           call create_single_detector(default_stat%detector_list, xfield, &
-                detector_location, static_dete+1, LAGRANGIAN_DETECTOR, trim(detector_name))
+                detector_location, static_dete+i, LAGRANGIAN_DETECTOR, trim(detector_name))
        end do
 
        k=static_dete+lagrangian_dete+1
@@ -1682,6 +1682,7 @@
                    read(default_stat%detector_checkpoint_unit) detector_location
                    call create_single_detector(default_stat%detector_list, xfield, &
                           detector_location, k, type_det, trim(detector_name)) 
+                   default_stat%detector_list%detector_names(k)=trim(detector_name)
                    k=k+1           
                 end do
              else                     

=== modified file 'tests/detectors_parallel_adaptivity/detectors_parallel_adaptivity.xml'
--- tests/detectors_parallel_adaptivity/detectors_parallel_adaptivity.xml	2013-08-21 19:58:53 +0000
+++ tests/detectors_parallel_adaptivity/detectors_parallel_adaptivity.xml	2014-01-29 10:13:58 +0000
@@ -18,8 +18,18 @@
 ElapsedTime=s["ElapsedTime"]["value"]
     </variable>
     <variable name="last_locations_error" language="python">
-from readtool import readstat
-last_locations_error = readstat("detectors.detectors")
+from fluidity_tools import stat_parser
+from numpy import zeros,fromfile
+
+s = stat_parser('detectors.detectors')
+last_locations_error = zeros((2,100))
+X = fromfile('Xvals.txt',sep=' ')
+Y = fromfile('Yvals.txt',sep=' ')
+for i in range(100):
+    name = 'Steve_' + str(i+1).zfill(3)
+    # +0.5 due to domain change from the lagrangian_detectors setup
+    last_locations_error[0,i] = s[name]['position'][0][-1] - X[i] + 0.5
+    last_locations_error[1,i] = s[name]['position'][1][-1] - Y[i] + 0.5
     </variable>
   </variables>
   <pass_tests>

=== removed file 'tests/detectors_parallel_adaptivity/readtool.py'
--- tests/detectors_parallel_adaptivity/readtool.py	2011-06-08 19:33:56 +0000
+++ tests/detectors_parallel_adaptivity/readtool.py	1970-01-01 00:00:00 +0000
@@ -1,21 +0,0 @@
-from fluidity_tools import stat_parser
-from numpy import zeros,fromfile
-def readstat(file_name):
-    s = stat_parser(file_name)
-
-    last_locations_error = zeros((2,100))
-    for i in range(100):
-        n = i + 1
-        padding = ''
-        if(n<100):
-            padding = '0'
-            if(n<10):
-                padding ='00'
-        # +0.5 due to domain change from the lagrangian_detectors setup
-        last_locations_error[0,i] = s['Steve_'+padding+str(n)]['position'][0][-1]+0.5
-        last_locations_error[1,i] = s['Steve_'+padding+str(n)]['position'][1][-1]+0.5
-    X = fromfile('Xvals.txt',sep=' ')
-    Y = fromfile('Yvals.txt',sep=' ')
-    last_locations_error[0,:] = last_locations_error[0,:] - X
-    last_locations_error[1,:] = last_locations_error[1,:] - Y
-    return last_locations_error

=== modified file 'tests/lagrangian_detectors/lagrangian_detectors.xml'
--- tests/lagrangian_detectors/lagrangian_detectors.xml	2013-08-21 19:58:53 +0000
+++ tests/lagrangian_detectors/lagrangian_detectors.xml	2014-01-29 10:13:58 +0000
@@ -11,12 +11,38 @@
     </command_line>
   </problem_definition>
   <variables>
+    <variable name="last_single_locations_error" language="python">
+import numpy as np
+from fluidity_tools import stat_parser
+s = stat_parser("lagrangian_detectors.detectors")
+last_single_locations_error = np.zeros((2,3))
+X = np.fromfile('Xvals.txt',sep=' ')
+Y = np.fromfile('Yvals.txt',sep=' ')
+last_single_locations_error[0,0] = s['Single_Lagr_1']['position'][0][-1] - X[0]
+last_single_locations_error[1,0] = s['Single_Lagr_1']['position'][1][-1] - Y[0]
+last_single_locations_error[0,1] = s['Single_Lagr_2']['position'][0][-1] - X[50]
+last_single_locations_error[1,1] = s['Single_Lagr_2']['position'][1][-1] - Y[50]
+last_single_locations_error[0,2] = s['Single_Lagr_3']['position'][0][-1] - X[99]
+last_single_locations_error[1,2] = s['Single_Lagr_3']['position'][1][-1] - Y[99]
+    </variable>
     <variable name="last_locations_error" language="python">
-from readtool import readstat
-last_locations_error = readstat()
+from fluidity_tools import stat_parser
+from numpy import zeros,fromfile
+
+s = stat_parser('lagrangian_detectors.detectors')
+last_locations_error = zeros((2,100))
+X = fromfile('Xvals.txt',sep=' ')
+Y = fromfile('Yvals.txt',sep=' ')
+for i in range(100):
+    name = 'Steve_' + str(i+1).zfill(3)
+    last_locations_error[0,i] = s[name]['position'][0][-1] - X[i]
+    last_locations_error[1,i] = s[name]['position'][1][-1] - Y[i]
     </variable>
   </variables>
   <pass_tests>
+    <test name="CorrectSingleLagrangianDetectors" language="python">
+assert abs(last_single_locations_error).max() &lt; 1.0e-6
+    </test>
     <test name="CorrectLagrangianTrajectory" language="python">
 assert abs(last_locations_error).max() &lt; 1.0e-6
     </test>

=== modified file 'tests/lagrangian_detectors/lagrangian_detectors_rk.flml'
--- tests/lagrangian_detectors/lagrangian_detectors_rk.flml	2012-03-11 02:25:56 +0000
+++ tests/lagrangian_detectors/lagrangian_detectors_rk.flml	2014-01-29 10:13:58 +0000
@@ -52,6 +52,21 @@
     <output_mesh name="VelocityMesh"/>
     <stat/>
     <detectors>
+      <lagrangian_detector name="Single_Lagr_1">
+        <location>
+          <real_value shape="2" dim1="dim" rank="1">0.5 0.5</real_value>
+        </location>
+      </lagrangian_detector>
+      <lagrangian_detector name="Single_Lagr_2">
+        <location>
+          <real_value shape="2" dim1="dim" rank="1">0.625 0.5</real_value>
+        </location>
+      </lagrangian_detector>
+      <lagrangian_detector name="Single_Lagr_3">
+        <location>
+          <real_value shape="2" dim1="dim" rank="1">0.7475 0.5</real_value>
+        </location>
+      </lagrangian_detector>
       <detector_array name="Steve">
         <number_of_detectors>
           <integer_value rank="0">100</integer_value>

=== removed file 'tests/lagrangian_detectors/readtool.py'
--- tests/lagrangian_detectors/readtool.py	2011-02-16 16:29:58 +0000
+++ tests/lagrangian_detectors/readtool.py	1970-01-01 00:00:00 +0000
@@ -1,20 +0,0 @@
-from fluidity_tools import stat_parser
-from numpy import zeros,fromfile
-def readstat():
-    s = stat_parser("lagrangian_detectors.detectors")
-
-    last_locations_error = zeros((2,100))
-    for i in range(100):
-        n = i + 1
-        padding = ''
-        if(n<100):
-            padding = '0'
-            if(n<10):
-                padding ='00'
-        last_locations_error[0,i] = s['Steve_'+padding+str(n)]['position'][0][-1]
-        last_locations_error[1,i] = s['Steve_'+padding+str(n)]['position'][1][-1]
-    X = fromfile('Xvals.txt',sep=' ')
-    Y = fromfile('Yvals.txt',sep=' ')
-    last_locations_error[0,:] = last_locations_error[0,:] - X
-    last_locations_error[1,:] = last_locations_error[1,:] - Y
-    return last_locations_error

=== added directory 'tests/lagrangian_detectors_checkpoint'
=== added file 'tests/lagrangian_detectors_checkpoint/Makefile'
--- tests/lagrangian_detectors_checkpoint/Makefile	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/Makefile	2014-01-29 10:13:58 +0000
@@ -0,0 +1,9 @@
+input: default
+default: clean
+	gmsh -2 -bin square.geo -o square.msh
+
+clean: 
+	rm -rf *.msh *.halo *.vtu *.pvtu *.log* *.err* *.stat blob_[012] *.detectors *.dat *.groups *flredecomp* *checkpoint.flml
+
+logs: 
+	../../bin/fluidity -v3 -l lagrangian_detectors_rk.flml

=== added file 'tests/lagrangian_detectors_checkpoint/Xvals.txt'
--- tests/lagrangian_detectors_checkpoint/Xvals.txt	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/Xvals.txt	2014-01-29 10:13:58 +0000
@@ -0,0 +1,1 @@
+0.5 0.499391156266 0.498782312532 0.498173468799 0.497564625065 0.496955781331 0.496346937597 0.495738093863 0.49512925013 0.494520406396 0.493911562662 0.493302718928 0.492693875194 0.492085031461 0.491476187727 0.490867343993 0.490258500259 0.489649656525 0.489040812792 0.488431969058 0.487823125324 0.48721428159 0.486605437856 0.485996594123 0.485387750389 0.484778906655 0.484170062921 0.483561219187 0.482952375453 0.48234353172 0.481734687986 0.481125844252 0.480517000518 0.479908156784 0.479299313051 0.478690469317 0.478081625583 0.477472781849 0.476863938115 0.476255094382 0.475646250648 0.475037406914 0.47442856318 0.473819719446 0.473210875713 0.472602031979 0.471993188245 0.471384344511 0.470775500777 0.470166657044 0.46955781331 0.468948969576 0.468340125842 0.467731282108 0.467122438375 0.466513594641 0.465904750907 0.465295907173 0.464687063439 0.464078219706 0.463469375972 0.462860532238 0.462251688504 0.46164284477 0.461034001037 0.460425157303 0.459816313569 0.459207469835 0.458598626101 0.457989782368 0.457380938634 0.4567720949 0.456163251166 0.455554407432 0.454945563699 0.454336719965 0.453727876231 0.453119032497 0.452510188763 0.45190134503 0.451292501296 0.450683657562 0.450074813828 0.449465970094 0.44885712636 0.448248282627 0.447639438893 0.447030595159 0.446421751425 0.445812907691 0.445204063958 0.444595220224 0.44398637649 0.443377532756 0.442768689022 0.442159845289 0.441551001555 0.440942157821 0.440333314087 0.439724470353
\ No newline at end of file

=== added file 'tests/lagrangian_detectors_checkpoint/Yvals.txt'
--- tests/lagrangian_detectors_checkpoint/Yvals.txt	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/Yvals.txt	2014-01-29 10:13:58 +0000
@@ -0,0 +1,1 @@
+0.5 0.50242472726 0.50484945452 0.50727418178 0.50969890904 0.5121236363 0.51454836356 0.51697309082 0.51939781808 0.52182254534 0.5242472726 0.52667199986 0.52909672712 0.53152145438 0.53394618164 0.5363709089 0.53879563616 0.54122036342 0.54364509068 0.54606981794 0.5484945452 0.55091927246 0.55334399972 0.55576872698 0.55819345424 0.5606181815 0.56304290876 0.56546763602 0.56789236328 0.57031709054 0.5727418178 0.57516654506 0.57759127232 0.58001599958 0.58244072684 0.5848654541 0.58729018136 0.58971490862 0.59213963588 0.59456436314 0.5969890904 0.59941381766 0.60183854492 0.60426327218 0.60668799944 0.6091127267 0.61153745396 0.61396218122 0.61638690848 0.61881163574 0.621236363 0.62366109026 0.62608581752 0.62851054478 0.63093527204 0.6333599993 0.63578472656 0.63820945382 0.64063418108 0.64305890834 0.6454836356 0.64790836286 0.65033309012 0.65275781738 0.65518254464 0.6576072719 0.66003199916 0.662456726419 0.664881453679 0.667306180939 0.669730908199 0.672155635459 0.674580362719 0.677005089979 0.679429817239 0.681854544499 0.684279271759 0.686703999019 0.689128726279 0.691553453539 0.693978180799 0.696402908059 0.698827635319 0.701252362579 0.703677089839 0.706101817099 0.708526544359 0.710951271619 0.713375998879 0.715800726139 0.718225453399 0.720650180659 0.723074907919 0.725499635179 0.727924362439 0.730349089699 0.732773816959 0.735198544219 0.737623271479 0.740047998739
\ No newline at end of file

=== added file 'tests/lagrangian_detectors_checkpoint/lagrangian_detectors.flml'
--- tests/lagrangian_detectors_checkpoint/lagrangian_detectors.flml	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/lagrangian_detectors.flml	2014-01-29 10:13:58 +0000
@@ -0,0 +1,178 @@
+<?xml version='1.0' encoding='utf-8'?>
+<fluidity_options>
+  <simulation_name>
+    <string_value lines="1">lagrangian_detectors</string_value>
+  </simulation_name>
+  <problem_type>
+    <string_value lines="1">fluids</string_value>
+  </problem_type>
+  <geometry>
+    <dimension>
+      <integer_value rank="0">2</integer_value>
+    </dimension>
+    <mesh name="CoordinateMesh">
+      <from_file file_name="square">
+        <format name="gmsh"/>
+        <stat>
+          <include_in_stat/>
+        </stat>
+      </from_file>
+    </mesh>
+    <mesh name="VelocityMesh">
+      <from_mesh>
+        <mesh name="CoordinateMesh"/>
+        <mesh_shape>
+          <polynomial_degree>
+            <integer_value rank="0">1</integer_value>
+          </polynomial_degree>
+        </mesh_shape>
+        <mesh_continuity>
+          <string_value>discontinuous</string_value>
+        </mesh_continuity>
+        <stat>
+          <exclude_from_stat/>
+        </stat>
+      </from_mesh>
+    </mesh>
+    <quadrature>
+      <degree>
+        <integer_value rank="0">3</integer_value>
+      </degree>
+    </quadrature>
+  </geometry>
+  <io>
+    <dump_format>
+      <string_value>vtk</string_value>
+    </dump_format>
+    <dump_period>
+      <constant>
+        <real_value rank="0">4.0</real_value>
+      </constant>
+    </dump_period>
+    <output_mesh name="VelocityMesh"/>
+    <checkpointing>
+      <checkpoint_period_in_dumps>
+        <integer_value rank="0">1</integer_value>
+      </checkpoint_period_in_dumps>
+    </checkpointing>
+    <stat/>
+    <detectors>
+      <detector_array name="Steve">
+        <number_of_detectors>
+          <integer_value rank="0">100</integer_value>
+        </number_of_detectors>
+        <lagrangian/>
+        <python>
+          <string_value lines="20" type="code" language="python">def val(t):
+  from numpy import arange,zeros,reshape,concatenate
+  x = 0.5+0.25*arange(0,100.)/100.
+  y = zeros(100) + 0.5
+  return reshape(concatenate((x,y)),(2,100)).T</string_value>
+        </python>
+      </detector_array>
+      <fail_outside_domain/>
+      <lagrangian_timestepping>
+        <subcycles>
+          <integer_value rank="0">1</integer_value>
+        </subcycles>
+        <search_tolerance>
+          <real_value rank="0">1.0e-10</real_value>
+        </search_tolerance>
+        <explicit_runge_kutta_guided_search>
+          <n_stages>
+            <integer_value rank="0">4</integer_value>
+          </n_stages>
+          <stage_weights>
+            <real_value shape="6" rank="1">0.5 0. 0.5 0. 0. 1</real_value>
+          </stage_weights>
+          <timestep_weights>
+            <real_value shape="4" rank="1">0.16666666666666666 0.3333333333333333
+ 0.3333333333333333 0.16666666666666666</real_value>
+          </timestep_weights>
+        </explicit_runge_kutta_guided_search>
+      </lagrangian_timestepping>
+    </detectors>
+  </io>
+  <timestepping>
+    <current_time>
+      <real_value rank="0">0.0</real_value>
+    </current_time>
+    <timestep>
+      <real_value rank="0">0.1</real_value>
+    </timestep>
+    <finish_time>
+      <real_value rank="0">8</real_value>
+    </finish_time>
+  </timestepping>
+  <physical_parameters>
+    <gravity>
+      <magnitude>
+        <real_value rank="0">1</real_value>
+      </magnitude>
+      <vector_field name="GravityDirection" rank="1">
+        <prescribed>
+          <mesh name="CoordinateMesh"/>
+          <value name="WholeMesh">
+            <constant>
+              <real_value shape="2" dim1="dim" rank="1">0.0 -1.0</real_value>
+            </constant>
+          </value>
+          <output/>
+          <stat>
+            <include_in_stat/>
+          </stat>
+          <detectors>
+            <exclude_from_detectors/>
+          </detectors>
+        </prescribed>
+      </vector_field>
+    </gravity>
+  </physical_parameters>
+  <material_phase name="BoussinesqFluid">
+    <vector_field name="Velocity" rank="1">
+      <prescribed>
+        <mesh name="VelocityMesh"/>
+        <value name="WholeMesh">
+          <python>
+            <string_value lines="20" type="code" language="python">def val(X,t):
+  x = X[0]
+  y = X[1]
+  return [-(y-0.5),x-0.5]</string_value>
+          </python>
+        </value>
+        <output/>
+        <stat>
+          <include_in_stat/>
+        </stat>
+        <detectors>
+          <exclude_from_detectors/>
+        </detectors>
+      </prescribed>
+    </vector_field>
+  </material_phase>
+  <mesh_adaptivity>
+    <hr_adaptivity>
+      <period>
+        <real_value rank="0">4.</real_value>
+      </period>
+      <maximum_number_of_nodes>
+        <integer_value rank="0">200</integer_value>
+      </maximum_number_of_nodes>
+      <enable_gradation/>
+      <tensor_field name="MinimumEdgeLengths">
+        <anisotropic_symmetric>
+          <constant>
+            <real_value symmetric="true" dim2="dim" shape="2 2" dim1="dim" rank="2">0.05 0.0 0.0 0.05</real_value>
+          </constant>
+        </anisotropic_symmetric>
+      </tensor_field>
+      <tensor_field name="MaximumEdgeLengths">
+        <anisotropic_symmetric>
+          <constant>
+            <real_value symmetric="true" dim2="dim" shape="2 2" dim1="dim" rank="2">0.06 0.0 0.0 0.06</real_value>
+          </constant>
+        </anisotropic_symmetric>
+      </tensor_field>
+    </hr_adaptivity>
+  </mesh_adaptivity>
+</fluidity_options>

=== added file 'tests/lagrangian_detectors_checkpoint/lagrangian_detectors_checkpoint.xml'
--- tests/lagrangian_detectors_checkpoint/lagrangian_detectors_checkpoint.xml	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/lagrangian_detectors_checkpoint.xml	2014-01-29 10:13:58 +0000
@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+
+<testproblem>
+  <name>lagrangian_detectors_checkpoint</name>
+  <owner userid="mlange"/>
+  <tags>flml detectors checkpoint</tags>
+  <problem_definition length="short" nprocs="4">
+    <command_line>
+../../bin/fluidity -v2 -l lagrangian_detectors.flml;
+../../bin/fluidity -v2 -l lagrangian_detectors_1_checkpoint.flml
+    </command_line>
+  </problem_definition>
+  <variables>
+    <variable name="last_locations_error" language="python">
+from fluidity_tools import stat_parser
+from numpy import zeros,fromfile
+
+s = stat_parser('lagrangian_detectors_checkpoint.detectors')
+last_locations_error = zeros((2,100))
+X = fromfile('Xvals.txt',sep=' ')
+Y = fromfile('Yvals.txt',sep=' ')
+for i in range(100):
+    name = 'Steve_' + str(i+1).zfill(3)
+    last_locations_error[0,i] = s[name]['position'][0][-1] - X[i]
+    last_locations_error[1,i] = s[name]['position'][1][-1] - Y[i]
+    </variable>
+  </variables>
+  <pass_tests>
+    <test name="CorrectLagrangianTrajectory" language="python">
+assert abs(last_locations_error).max() &lt; 1.0e-6
+    </test>
+  </pass_tests>
+  <warn_tests>
+  </warn_tests>
+</testproblem>

=== added file 'tests/lagrangian_detectors_checkpoint/square.geo'
--- tests/lagrangian_detectors_checkpoint/square.geo	1970-01-01 00:00:00 +0000
+++ tests/lagrangian_detectors_checkpoint/square.geo	2014-01-29 10:13:58 +0000
@@ -0,0 +1,13 @@
+Point(1) = {0,0,0,0.03};
+Extrude {1,0,0} {
+  Point{1};
+}
+Extrude {0,1,0} {
+  Line{1};
+}
+Physical Line(8) = {2};
+Physical Line(9) = {4};
+Physical Line(10) = {3};
+Physical Line(11) = {1};
+Physical Surface(7) = {5};
+