Merge lp:~garsua/siesta/trunk-elsi-dm into lp:siesta

Proposed by Victor GS
Status: Superseded
Proposed branch: lp:~garsua/siesta/trunk-elsi-dm
Merge into: lp:siesta
Diff against target: 4436 lines (+3744/-70) (has conflicts)
32 files modified
Docs/siesta.tex (+119/-2)
Src/MPI/mpi_siesta.F90 (+18/-1)
Src/Makefile (+26/-18)
Src/compute_dm.F (+25/-1)
Src/dhscf.F (+10/-0)
Src/fold_auxcell.f90 (+85/-0)
Src/fsiesta_mpi.F90 (+4/-0)
Src/initparallel.F (+1/-1)
Src/m_elsi_interface.F90 (+1601/-0)
Src/m_ncdf_siesta.F90 (+3/-0)
Src/m_pexsi_dos.F90 (+1/-1)
Src/m_pexsi_driver.F90 (+1/-1)
Src/m_pexsi_local_dos.F90 (+1/-1)
Src/m_redist_spmatrix.F90 (+187/-21)
Src/parallel.F (+1/-1)
Src/post_scf_work.F (+33/-2)
Src/read_options.F90 (+11/-0)
Src/siesta.F (+9/-9)
Src/siesta_forces.F90 (+30/-5)
Src/siesta_init.F (+23/-5)
Src/siesta_options.F90 (+1/-0)
Src/siesta_tddft.F90 (+1/-1)
Src/state_init.F (+4/-0)
Src/write_subs.F (+1/-0)
Tests/si-quantum-dot/coords.Si987H372.fdf (+1368/-0)
Tests/si-quantum-dot/makefile (+7/-0)
Tests/si-quantum-dot/si-quantum-dot.fdf (+39/-0)
Tests/si-quantum-dot/si-quantum-dot.pseudos (+1/-0)
Tests/sih-elsi/makefile (+6/-0)
Tests/sih-elsi/sih-elsi.fdf (+121/-0)
Tests/sih-elsi/sih-elsi.pseudos (+1/-0)
version.info (+5/-0)
Text conflict in Src/siesta_forces.F90
Text conflict in version.info
To merge this branch: bzr merge lp:~garsua/siesta/trunk-elsi-dm
Reviewer Review Type Date Requested Status
Alberto Garcia Needs Resubmitting
Review via email: mp+361820@code.launchpad.net

Commit message

Cosmetic changes on m_elsi_interface.F90
- Removed "use fdf, only: fdf_get" from elsi_real_solver and elsi_complex_solver subroutines
- Unindented lines from 401 to 510 (in elsi_real_solver) and from 1351 to 1460 (in elsi_complex_solver)

To post a comment you must log in.
Revision history for this message
Alberto Garcia (albertog) wrote :

Victor: As a test of the MR process it is fine, but you should specify the target branch as
~albertog/siesta/trunk-elsi-dm

review: Needs Resubmitting
Revision history for this message
Victor GS (garsua) wrote :

Gracias Alberto, se me pasó incluirlo. Vuelvo a hacer el merge request, a
ver si ahora sale bien.

On Wed, Jan 16, 2019 at 3:34 PM Alberto Garcia <email address hidden> wrote:

> Review: Resubmit
>
> Victor: As a test of the MR process it is fine, but you should specify the
> target branch as
> ~albertog/siesta/trunk-elsi-dm
>
> --
> https://code.launchpad.net/~garsua/siesta/trunk-elsi-dm/+merge/361820
> You are the owner of lp:~garsua/siesta/trunk-elsi-dm.
>

Unmerged revisions

696. By Victor GS

Cosmetic changes on m_elsi_interface.F90
- Removed "use fdf, only: fdf_get" from elsi_real_solver and elsi_complex_solver subroutines
- Unindented lines from 401 to 510 (in elsi_real_solver) and from 1351 to 1460 (in elsi_complex_solver)

695. By Alberto Garcia

Do not get the EDM during the SCF cycle + NTPoly support

* In compute_dm, the call to the ELSI ('get_dm') interface uses a
flag to disable the generation of the EDM matrix. In this case
only the DM is computed.

After the scf loop, in 'post_scf_work', a further call to the ELSI
interface requests (only) the EDM.

Note that this is achieved through a logical flag 'Get_EDM_Only',
which is not really appropriate. It would be better to have a
more explicit flag, such as:

           output_mode="DM|EDM"

The ELSI shutdown has been moved towards the end of 'siesta_forces',
and the 'Siesta_Worker' logic preserved in that routine (even though
it is not working elsewhere).

* NTpoly support has been added to the interface

(Thanks to Victor M. Garcia-Suarez and Victor Yu)

694. By Alberto Garcia

Wrap MPI check on number of procs for ELSI solver

693. By Alberto Garcia

Add Si quantum dot test case for ELSI

692. By Alberto Garcia

Fix bug in computation of Delta-V for PEXSI mu bracketing

A new array is needed to properly update the value of the
change in potential.

(+ added a declaration for 'date_stamp' in the auxiliary
code for version compatibility)

691. By Alberto Garcia

Change the spin-reduction convention for entropy

For ELSI versions after 2018-08-17, there is no need to reduce over spins
when computing the entropy.

To use older versions (such as the 2.0.2 release), use the preprocessor symbol

    SIESTA__ELSI__OLD_SPIN_CONVENTION

If the version is 'old', but the above symbol is not used, a compilation error will
be triggered. (This works by means of a call to 'elsi_get_datestamp', which was added just
around the same time as the spin-reduction-convention change: Aug 17 vs Aug 21, 2018.
ELSI versions in between will not work without further intervention.

690. By Alberto Garcia

Update documentation. Check that Nodes is a multiple of nk*nspin

689. By Alberto Garcia

Fix weight of DM and EDM. Working

688. By Alberto Garcia

Do not make shadow copies of elsi_global_comm

687. By Alberto Garcia

Change some ptrs to allocs. Fix communicators

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Docs/siesta.tex'
2--- Docs/siesta.tex 2019-01-04 07:01:47 +0000
3+++ Docs/siesta.tex 2019-01-16 11:00:58 +0000
4@@ -952,6 +952,33 @@
5 LIBS += -Wl,--allow-multiple-definition -lpexsi_linux <>
6 \end{shellexample}
7
8+ \item[\href{http://elsi-interchange.org}{ELSI}]%
9+ \index{ELSI}%
10+ \index{External library!ELSI}%
11+ The ELSI library may be used with \siesta\ for access to a range of
12+ solvers, see Sec.~\ref{SolverELSI}. Currently the interface is
13+ implemented (tested) as of ELSI version 2.0.2. If
14+ newer versions retain the same interface they may also be used.
15+
16+ Add these flags to your \file{arch.make} file to enable ELSI
17+\begin{shellexample}
18+ INCFLAGS += -I/opt/elsi/include
19+ LIBS += -L/opt/elsi/lib <>
20+ FPPFLAGS += -DSIESTA__ELSI
21+\end{shellexample}
22+ \index{compile!pre-processor!-DSIESTA\_\_ELSI}
23+ where \shell{<>} represents the appropriate ELSI libraries (see the
24+ ELSI installation instructions).
25+ %
26+ If one experiences linker failures, one possible solution that may
27+ help is
28+\begin{shellexample}
29+ LIBS += -lmpi_cxx -lstdc++
30+\end{shellexample}
31+ which is due to ELSI containing some parts written in C++, while the
32+ linking is done by the Fortran compiler. The exact library name for
33+ your MPI vendor may vary.
34+
35
36 \item[CheSS]%
37 \index{CheSS}%
38@@ -6021,6 +6048,11 @@
39 of the density matrix via an efficient scheme of selected
40 inversion (see Sec~\ref{SolverPEXSI}).
41
42+Additionally, \siesta\ can use the ELSI library (see
43+Sec.~\ref{SolverELSI}) of solvers (which provides versions of OMM
44+and PEXSI, in addition to diagonalization), and the new linear-scaling
45+CheSS library (see Sec.~\ref{SolverCheSS})
46+
47 The calculation of the H and S matrix elements is always done with an
48 O(N) method. The actual scaling is not linear for small systems, but
49 it becomes O(N) when the system dimensions are larger than the scale
50@@ -6044,8 +6076,8 @@
51 Character string to choose among diagonalization (\fdf*{diagon}),
52 cubic-scaling minimization (\fdf*{OMM}), Order-N (\fdf*{OrderN})
53 solution of the Kohn-Sham Hamiltonian, \fdf*{transiesta}, the
54- PEXSI method (\fdf*{PEXSI}) or the \fdf*{CheSS} solver.
55-
56+ PEXSI method (\fdf*{PEXSI}), ELSI solver (\fdf*{ELSI}) or the \fdf*{CheSS} solver.
57+
58 \end{fdfentry}
59
60
61@@ -6867,6 +6899,91 @@
62
63 \end{fdflogicalF}
64
65+\subsection{The ELSI solver family}
66+\label{SolverELSI}
67+\index{ELSI solvers}
68+
69+The ELSI library provides a unified interface to a range of solvers,
70+including ELPA (diagonalization), PEXSI, OMM, and SIPS (based on spectrum
71+slicing). It can be downloaded and installed freely from
72+\url{http://elsi-interchange.org}.
73+
74+The current interface in \siesta\ is based on the ``DM'' mode in ELSI:
75+sparse H and S matrices are passed to the library, and sparse DM and
76+EDM matrices are returned. Internally, the library performs any extra
77+operations needed. K-points and/or spin-polarization (in collinear
78+form) are also supported, in fully parallel mode. This means that
79+k-points/spin-polarized calculations will be carried out splitting the
80+communicator for orbital distribution into nk$\times$nspin copies,
81+where nk is the number of k-points and nspin the number of spin
82+components. The total number of MPI processors must be a multiple of
83+nk$\times$nspin.
84+
85+See Sec.~\ref{sec:libs} for details on installing \siesta\ with ELSI support.
86+
87+\subsubsection{Input parameters}
88+
89+The most important parameter is the solver to be used. Most other
90+options are given default values by the program which are nearly
91+always appropriate.
92+
93+\begin{fdfentry}{ELSI!Solver}[string]<ELPA>
94+ A specification of the solver wanted. Possible values are
95+ \begin{itemize}
96+ \item ELPA
97+ \item OMM
98+ \item PEXSI
99+ \item SIPS
100+ \end{itemize}
101+\end{fdfentry}
102+
103+\begin{fdfentry}{ELSI!Broadening!Method}[string]<''fermi''>
104+The scheme for level broadening. Valid options are ``fermi'',
105+``gauss'', ``mp'' (Methfessel-Paxton), ``cold''
106+(Marzari-Vanderbilt cold smearing).
107+\end{fdfentry}
108+
109+\begin{fdfentry}{ELSI!Output!Level}[integer]<0>
110+ The level of detail provided in the output. Possible values are 0, 1,
111+ 2, and 3.
112+\end{fdfentry}
113+
114+\begin{fdfentry}{ELSI!Output!Json}[integer]<1>
115+ If not 0, a separate log file in JSON format will be written out.
116+\end{fdfentry}
117+
118+Other fdf options are mapped in a direct way to the options described in the
119+ELSI manual:
120+
121+\begin{fdfentry}{ELSI!Broadening!MPOrder}[integer]<1>
122+\end{fdfentry}
123+
124+\begin{fdfentry}{ELSI!ELPA!Flavor}[integer]<2>
125+\end{fdfentry}
126+
127+\begin{fdfentry}{ELSI!OMM!Flavor}[integer]<0>
128+\end{fdfentry}
129+
130+\begin{fdfentry}{ELSI!OMM!ELPA!Steps}[integer]<3>
131+\end{fdfentry}
132+
133+\begin{fdfentry}{ELSI!OMM!Tolerance}[real]<10e-9>
134+\end{fdfentry}
135+
136+\begin{fdfentry}{ELSI!PEXSI!TasksPerPole}[integer]< {\bf no default}>
137+ See the discussion on the three-level PEXSI parallelism in the ELSI manual.
138+\end{fdfentry}
139+
140+\begin{fdfentry}{ELSI!PEXSI!TasksSymbolic}[integer]<1>
141+\end{fdfentry}
142+
143+\begin{fdfentry}{ELSI!SIPS!Slices}[integer]<{\bf no default}>
144+ See the discussion in the ELSI manual.
145+\end{fdfentry}
146+
147+\begin{fdfentry}{ELSI!SIPS!ELPA!Steps}[integer]<2>
148+\end{fdfentry}
149+
150
151 \subsection{The CheSS solver}
152 \label{SolverCheSS}
153
154=== modified file 'Src/MPI/mpi_siesta.F90'
155--- Src/MPI/mpi_siesta.F90 2016-09-16 12:25:04 +0000
156+++ Src/MPI/mpi_siesta.F90 2019-01-16 11:00:58 +0000
157@@ -37,12 +37,29 @@
158 USE MPI__INCLUDE, true_MPI_Comm_World => MPI_Comm_World
159 #endif /* NO_MPI_INTERFACES */
160
161+ ! The reason to define true_MPI_Comm_World as a pointer to the
162+ ! (inmutable) MPI_Comm_World parameter in the MPI module is to keep
163+ ! it as reference. The MPI_Comm_World *variable* defined below can
164+ ! be redefined at will. This practice is needed because most Siesta
165+ ! routines have hard-wired references to MPI_Comm_World as their
166+ ! communicator. Instead of generalizing the interfaces and pass the
167+ ! communicator as an argument (work in progress), we use the kludge
168+ ! of re-defining MPI_Comm_World.
169+
170 ! The following construction allows to supplant MPI_Comm_World within SIESTA,
171 ! and to use it as a subroutine with its own internal MPI communicator.
172
173+ ! Siesta-instances should refer to MPI_Comm_DFT
174+ ! This is the case, for example, when a driver program splits the
175+ ! global communicator to dispatch several simultaneous Siesta instances
176+ ! Typically, comm_world = comm_dft, but not for calculations (PEXSI, ELSI)
177+ ! for which only a subset of processors carry out the core Siesta operations
178+ ! that have hard-wired references to mpi_comm_world
179+
180 integer, public :: MPI_Comm_World = true_MPI_Comm_World
181+ integer, public :: MPI_Comm_DFT = true_MPI_Comm_World
182
183- public :: true_MPI_Comm_World
184+ public :: true_MPI_Comm_World
185
186
187 #ifdef GRID_SP
188
189=== modified file 'Src/Makefile'
190--- Src/Makefile 2018-12-06 09:30:22 +0000
191+++ Src/Makefile 2019-01-16 11:00:58 +0000
192@@ -156,6 +156,8 @@
193 write_orb_indx.o \
194 die.o m_pexsi.o m_pexsi_driver.o m_pexsi_dos.o m_pexsi_local_dos.o\
195 m_chess.o \
196+ m_elsi_interface.o \
197+ fold_auxcell.o \
198 m_redist_spmatrix.o \
199 class_Distribution.o\
200 m_dminim.o m_zminim.o \
201@@ -293,7 +295,7 @@
202 "FFLAGS=$(FFLAGS:$(IPO_FLAG)=)" module )
203
204 #--------------------------------------------------------------
205-MS=MatrixSwitch.a
206+MS?=MatrixSwitch.a
207 $(MS): $(MPI_INTERFACE)
208 (cd MatrixSwitch/src ; \
209 $(MAKE) -j 1 \
210@@ -641,11 +643,11 @@
211 class_TriMat.o: alloc.o intrinsic_missing.o
212 coceri.o: files.o periodic_table.o precision.o units.o
213 compute_dm.o: atomlist.o class_SpData1D.o compute_ebs_shift.o diagon.o
214-compute_dm.o: iodmhs_netcdf.o kpoint_scf.o m_chess.o m_dminim.o m_energies.o
215-compute_dm.o: m_eo.o m_hsx.o m_pexsi_driver.o m_rmaxh.o m_spin.o m_steps.o
216-compute_dm.o: m_transiesta.o m_ts_global_vars.o m_zminim.o normalize_dm.o
217-compute_dm.o: ordern.o parallel.o precision.o siesta_geom.o siesta_options.o
218-compute_dm.o: sparse_matrices.o sys.o units.o
219+compute_dm.o: iodmhs_netcdf.o kpoint_scf.o m_chess.o m_dminim.o
220+compute_dm.o: m_elsi_interface.o m_energies.o m_eo.o m_hsx.o m_pexsi_driver.o
221+compute_dm.o: m_rmaxh.o m_spin.o m_steps.o m_transiesta.o m_ts_global_vars.o
222+compute_dm.o: m_zminim.o normalize_dm.o ordern.o parallel.o precision.o
223+compute_dm.o: siesta_geom.o siesta_options.o sparse_matrices.o sys.o units.o
224 compute_ebs_shift.o: m_mpi_utils.o parallel.o precision.o
225 compute_energies.o: atomlist.o class_SpData1D.o class_SpData2D.o
226 compute_energies.o: class_SpData2D.o dhscf.o files.o m_dipol.o m_energies.o
227@@ -683,11 +685,12 @@
228 dfscf.o: meshphi.o parallel.o parallelsubs.o precision.o sys.o
229 dhscf.o: alloc.o atmfuncs.o bsc_xcmod.o cellxc_mod.o delk.o dfscf.o
230 dhscf.o: doping_uniform.o files.o forhar.o iogrid_netcdf.o m_charge_add.o
231-dhscf.o: m_efield.o m_hartree_add.o m_iorho.o m_iotddft.o m_mesh_node.o
232-dhscf.o: m_ncdf_io.o m_ncdf_siesta.o m_partial_charges.o m_rhog.o m_spin.o
233-dhscf.o: m_ts_global_vars.o m_ts_hartree.o m_ts_options.o m_ts_voltage.o mesh.o
234-dhscf.o: meshdscf.o meshsubs.o moremeshsubs.o parallel.o parsing.o precision.o
235-dhscf.o: rhofft.o rhoofd.o siesta_options.o sys.o units.o vmat.o
236+dhscf.o: m_efield.o m_elsi_interface.o m_hartree_add.o m_iorho.o m_iotddft.o
237+dhscf.o: m_mesh_node.o m_ncdf_io.o m_ncdf_siesta.o m_partial_charges.o m_rhog.o
238+dhscf.o: m_spin.o m_ts_global_vars.o m_ts_hartree.o m_ts_options.o
239+dhscf.o: m_ts_voltage.o mesh.o meshdscf.o meshsubs.o moremeshsubs.o parallel.o
240+dhscf.o: parsing.o precision.o rhofft.o rhoofd.o siesta_options.o sys.o units.o
241+dhscf.o: vmat.o
242 diag.o: alloc.o diag_option.o parallel.o precision.o sys.o
243 diag2g.o: fermid.o intrinsic_missing.o parallel.o parallelsubs.o precision.o
244 diag2g.o: sys.o
245@@ -838,6 +841,8 @@
246 m_dscfcomm.o: alloc.o parallel.o parallelsubs.o precision.o schecomm.o
247 m_efield.o: atmfuncs.o m_cite.o m_ts_global_vars.o mesh.o parallel.o
248 m_efield.o: precision.o siesta_cml.o siesta_geom.o sys.o units.o
249+m_elsi_interface.o: alloc.o class_Distribution.o fold_auxcell.o m_mpi_utils.o
250+m_elsi_interface.o: m_redist_spmatrix.o parallel.o precision.o units.o
251 m_energies.o: precision.o
252 m_eo.o: precision.o
253 m_evolve.o: atomlist.o cranknic_evolg.o cranknic_evolk.o m_spin.o precision.o
254@@ -1194,8 +1199,9 @@
255 post_scf_work.o: atomlist.o class_Fstack_Pair_Geometry_SpData2D.o
256 post_scf_work.o: class_Geometry.o class_Pair_Geometry_SpData2D.o compute_dm.o
257 post_scf_work.o: diagon.o final_H_f_stress.o kpoint_scf.o m_dminim.o
258-post_scf_work.o: m_energies.o m_eo.o m_spin.o m_steps.o m_zminim.o mneighb.o
259-post_scf_work.o: parallel.o siesta_geom.o siesta_options.o sparse_matrices.o
260+post_scf_work.o: m_elsi_interface.o m_energies.o m_eo.o m_spin.o m_steps.o
261+post_scf_work.o: m_zminim.o mneighb.o parallel.o siesta_geom.o siesta_options.o
262+post_scf_work.o: sparse_matrices.o
263 print_spin.o: atomlist.o m_mpi_utils.o m_spin.o parallel.o precision.o
264 print_spin.o: siesta_cml.o sparse_matrices.o
265 printmatrix.o: alloc.o
266@@ -1288,11 +1294,12 @@
267 siesta_forces.o: atomlist.o class_Fstack_Data1D.o class_SpData2D.o compute_dm.o
268 siesta_forces.o: compute_energies.o compute_max_diff.o densematrix.o
269 siesta_forces.o: dm_charge.o files.o final_H_f_stress.o flook_siesta.o
270-siesta_forces.o: kpoint_scf.o m_check_walltime.o m_convergence.o m_energies.o
271-siesta_forces.o: m_forces.o m_initwf.o m_iodm_old.o m_mixing.o m_mixing_scf.o
272-siesta_forces.o: m_mpi_utils.o m_ncdf_siesta.o m_pexsi.o m_pexsi_driver.o
273-siesta_forces.o: m_rhog.o m_spin.o m_steps.o m_stress.o m_transiesta.o
274-siesta_forces.o: m_ts_charge.o m_ts_electype.o m_ts_global_vars.o m_ts_method.o
275+siesta_forces.o: kpoint_scf.o m_check_walltime.o m_convergence.o
276+siesta_forces.o: m_elsi_interface.o m_energies.o m_forces.o m_initwf.o
277+siesta_forces.o: m_iodm_old.o m_mixing.o m_mixing_scf.o m_mpi_utils.o
278+siesta_forces.o: m_ncdf_siesta.o m_pexsi.o m_pexsi_driver.o m_rhog.o m_spin.o
279+siesta_forces.o: m_steps.o m_stress.o m_transiesta.o m_ts_charge.o
280+siesta_forces.o: m_ts_electype.o m_ts_global_vars.o m_ts_method.o
281 siesta_forces.o: m_ts_options.o mixer.o parallel.o post_scf_work.o precision.o
282 siesta_forces.o: save_density_matrix.o scfconvergence_test.o setup_H0.o
283 siesta_forces.o: setup_hamiltonian.o siesta_cml.o siesta_dicts.o siesta_geom.o
284@@ -1493,6 +1500,7 @@
285 m_final_h_f_stress.o: final_H_f_stress.o
286 m_find_kgrid.o: find_kgrid.o
287 m_fire_optim.o: fire_optim.o
288+m_fold_auxcell.o: fold_auxcell.o
289 m_forhar.o: forhar.o
290 m_get_kpoints_scale.o: get_kpoints_scale.o
291 m_gradient.o: gradient.o
292
293=== modified file 'Src/compute_dm.F'
294--- Src/compute_dm.F 2018-11-09 09:36:46 +0000
295+++ Src/compute_dm.F 2019-01-16 11:00:58 +0000
296@@ -36,6 +36,9 @@
297 #ifdef SIESTA__PEXSI
298 use m_pexsi_solver, only: pexsi_solver
299 #endif
300+#ifdef SIESTA__ELSI
301+ use m_elsi_interface, only: elsi_getdm
302+#endif
303 use m_hsx, only : write_hs_formatted
304 #ifdef MPI
305 use mpi_siesta
306@@ -75,7 +78,7 @@
307 if (SIESTA_worker) call timer( 'compute_dm', 1 )
308
309 #ifdef MPI
310- call MPI_Bcast(isolve,1,MPI_integer,0,true_MPI_Comm_World,mpierr)
311+ call MPI_Bcast(isolve,1,MPI_integer,0,MPI_Comm_DFT,mpierr)
312 #endif
313
314 if (SIESTA_worker) then
315@@ -137,6 +140,27 @@
316 endif
317 if (.not. SIESTA_worker) RETURN
318 #endif
319+#ifdef SIESTA__ELSI
320+ if (isolve .eq. SOLVE_ELSI) then
321+ ! This test done in node 0 since NonCol and SpOrb
322+ ! are not set for ELSI-solver-only processes
323+ if (ionode) then
324+ if (spin%NCol .or. spin%SO) call die(
325+ $ "The ELSI solver does not implement "//
326+ $ "non-coll spins or Spin-orbit yet")
327+ endif
328+ call elsi_getdm(iscf, no_s, spin%spinor,
329+ & no_l, maxnh, no_u,
330+ & numh, listhptr, listh,
331+ & H, S, qtot, temp,
332+ & xijo,
333+ & kpoint_scf%N, kpoint_scf%k, kpoint_scf%w,
334+ & eo, qo, Dscf, Escf, ef, Entropy,
335+ & occtol, neigwanted, Get_EDM_Only=.false.)
336+ Ecorrec = 0.0_dp
337+ endif
338+ if (.not. SIESTA_worker) RETURN
339+#endif
340 ! Here we decide if we want to calculate one or more SCF steps by
341 ! diagonalization before proceeding with the OMM routine
342 CallDiagon=.false.
343
344=== modified file 'Src/dhscf.F'
345--- Src/dhscf.F 2019-01-04 07:01:47 +0000
346+++ Src/dhscf.F 2019-01-16 11:00:58 +0000
347@@ -694,6 +694,10 @@
348 use m_ts_voltage, only: ts_voltage
349 use m_ts_hartree, only: ts_hartree_fix
350
351+#ifdef SIESTA__ELSI
352+ use m_elsi_interface, only: elsi_save_potential
353+#endif
354+
355 implicit none
356
357 integer
358@@ -1653,6 +1657,12 @@
359 #endif
360 endif
361
362+#ifdef SIESTA__ELSI
363+ ! This routine, and the associated data, live
364+ ! in the ELSI interface module
365+ call elsi_save_potential(ntpl,nspin,Vscf,comm=MPI_Comm_World)
366+#endif
367+
368 C ----------------------------------------------------------------------
369 C Print vacuum level
370 C ----------------------------------------------------------------------
371
372=== added file 'Src/fold_auxcell.f90'
373--- Src/fold_auxcell.f90 1970-01-01 00:00:00 +0000
374+++ Src/fold_auxcell.f90 2019-01-16 11:00:58 +0000
375@@ -0,0 +1,85 @@
376+module m_fold_auxcell
377+ public :: fold_sparse_arrays
378+ private
379+ CONTAINS
380+ subroutine fold_sparse_arrays(no_l,no_u,numh,listhptr,nnz,listh, &
381+ numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u)
382+
383+! Fold-in a sparse matrix from the auxiliary supercell into the
384+! unit cell (see picture)
385+
386+ ! Number of orbitals handled by this node
387+ integer, intent(in) :: no_l
388+ ! Number of orbitals in unit cell
389+ integer, intent(in) :: no_u
390+
391+ ! Number of interactions per orbital; pointer to beginning of sparse array data
392+ integer, intent(in) :: numh(no_l), listhptr(no_l)
393+ ! Total number of interactions
394+ integer, intent(in) :: nnz
395+ ! Columns of the (rectangular) supercell-based matrix
396+ integer, intent(in) :: listh(nnz)
397+
398+ ! Output: All interactions are collapsed to the unit cell orbitals
399+
400+ ! Number of interactions per orbital; pointer to beginning of sparse array data
401+ integer, intent(out) :: numh_u(no_l), listhptr_u(no_l)
402+ ! Total number of interactions
403+ integer, intent(out) :: nnz_u
404+ ! Columns of the (square) unit cell-based matrix
405+ integer, allocatable, intent(out) :: listh_u(:)
406+ ! Mapper of indexes from supercell-sparse-array to unit-cell sparse array
407+ integer, allocatable, intent(out) :: ind2ind_u(:)
408+
409+
410+ ! Local variables
411+ integer :: iuo, ind, j, jo, j_u, ind_u, juo
412+ integer, allocatable :: mask(:)
413+
414+ allocate(mask(no_u))
415+
416+ ! Find out how many "folded" interactions there are
417+ nnz_u = 0
418+ do iuo = 1,no_l
419+ mask = 0
420+ do j = 1,numh(iuo)
421+ ind = listhptr(iuo) + j
422+ jo = listh(ind)
423+ juo = mod(jo-1,no_u) + 1
424+ mask(juo) = 1 ! Found one
425+ enddo
426+ numh_u(iuo) = sum(mask)
427+ nnz_u = nnz_u + numh_u(iuo)
428+ enddo
429+
430+ allocate(listh_u(nnz_u))
431+ allocate(ind2ind_u(nnz))
432+
433+ ! Generate folded pointer array
434+ listhptr_u(1) = 0
435+ do iuo = 2, no_l
436+ listhptr_u(iuo) = listhptr_u(iuo-1) + numh_u(iuo-1)
437+ enddo
438+
439+ ! Complete the mapping
440+ do iuo = 1,no_l
441+ mask(:) = 0
442+ ind_u = listhptr_u(iuo)
443+ ind = listhptr(iuo)
444+ do j = 1,numh(iuo)
445+ ind = ind + 1
446+ jo = listh(ind)
447+ juo = mod(jo-1,no_u) + 1
448+ if (mask(juo) > 0) then
449+ ! juo already seen and its place recorded
450+ ind2ind_u(ind) = mask(juo)
451+ else
452+ ind_u = ind_u + 1
453+ listh_u(ind_u) = juo
454+ ind2ind_u(ind) = ind_u ! map
455+ mask(juo) = ind_u ! record
456+ endif
457+ enddo
458+ enddo
459+ end subroutine fold_sparse_arrays
460+end module m_fold_auxcell
461
462=== modified file 'Src/fsiesta_mpi.F90'
463--- Src/fsiesta_mpi.F90 2018-12-03 22:23:36 +0000
464+++ Src/fsiesta_mpi.F90 2019-01-16 11:00:58 +0000
465@@ -197,6 +197,7 @@
466 ! MPI_COMM_WORLD
467 use mpi_siesta, only: MPI_Comm_Siesta => MPI_Comm_World ! What siesta uses
468 ! as MPI_Comm_World
469+ use mpi_siesta, only: MPI_Comm_DFT ! Siesta-instance (dft-calc)
470 use mpi_siesta, only: MPI_Integer ! Integer data type
471 use mpi_siesta, only: MPI_Character ! Character data type
472 use mpi_siesta, only: MPI_Double_Precision ! Real double precision type
473@@ -372,6 +373,9 @@
474 // trim(label) // ' 2> /dev/null')
475 endif
476
477+ ! Set the whole-siesta-instance (dft calc) communicator
478+ MPI_Comm_DFT = MPI_Comm_Siesta
479+
480 #endif
481
482 ! Initialize flags and counters
483
484=== modified file 'Src/initparallel.F'
485--- Src/initparallel.F 2018-11-23 10:32:56 +0000
486+++ Src/initparallel.F 2019-01-16 11:00:58 +0000
487@@ -16,7 +16,7 @@
488 use fdf
489 use siesta_options, only: want_domain_decomposition
490 use siesta_options, only: want_spatial_decomposition
491- use siesta_options, only: isolve, SOLVE_ORDERN, SOLVE_PEXSI
492+ use siesta_options, only: isolve, SOLVE_ORDERN
493 use siesta_options, only: rcoor
494 use parallel, only : Node, Nodes, BlockSize, ProcessorY
495 use parallelsubs, only : WhichNodeOrb, GetNodeOrbs
496
497=== added file 'Src/m_elsi_interface.F90'
498--- Src/m_elsi_interface.F90 1970-01-01 00:00:00 +0000
499+++ Src/m_elsi_interface.F90 2019-01-16 11:00:58 +0000
500@@ -0,0 +1,1601 @@
501+!
502+! Alberto Garcia, February-July 2018
503+! Modified by Victor M. Garcia-Suarez. January 2019
504+! (Introduction of an extra flag to only get the EDM instead of carrying
505+! out a full computation --- to be refactored)
506+!
507+! ELSI DM-based interface to Siesta. It uses the sparse matrices from Siesta,
508+! and obtains the DM (and optionally the EDM) matrices in sparse form.
509+!
510+! The elsi_getdm routine is in principle able to perform (spin-polarized)
511+! calculations for real matrices (i.e., at the Gamma point), and periodic
512+! calculations for complex matrices (i.e., multiple k-points), including spin.
513+!
514+! The structure of the solver routine is such that it will detect when it is
515+! called for the first scf step, so it can perform any needed initialization.
516+! The same idea can be used for the diagonalization mode, with (more extensive)
517+! appropriate changes.
518+!
519+! The module also exports the "elsi_finalize_scfloop" routine, to be called from
520+! the appropriate place. Some variables are kept at the module level for this.
521+!
522+! Usage: Compile Siesta with -DSIESTA__ELSI
523+! Define
524+! SolutionMethod ELSI
525+! in the fdf file
526+! TODO:
527+! - MPI.Nprocs.SIESTA is not working -- maybe we will remove that feature
528+! - Add more documentation
529+! - Maybe refactor a few things
530+!
531+module m_elsi_interface
532+
533+#if SIESTA__ELSI
534+
535+ use precision, only: dp
536+ use elsi
537+
538+ implicit none
539+
540+ private
541+
542+ integer, parameter :: ELPA_SOLVER = 1 ! solver
543+ integer, parameter :: OMM_SOLVER = 2 ! solver
544+ integer, parameter :: PEXSI_SOLVER = 3 ! solver
545+ integer, parameter :: SIPS_SOLVER = 5 ! solver
546+ integer, parameter :: NTPOLY_SOLVER = 6 ! solver
547+ integer, parameter :: MULTI_PROC = 1 ! parallel_mode
548+ integer, parameter :: SIESTA_CSC = 2 ! distribution
549+ integer, parameter :: GAUSSIAN = 0 ! broadening
550+ integer, parameter :: FERMI = 1 ! broadening
551+ integer, parameter :: METHFESSEL_PAXTON = 2 ! broadening
552+ integer, parameter :: CUBIC = 3 ! broadening
553+ integer, parameter :: COLD = 4 ! broadening
554+ integer, parameter :: ELSI_NOT_SET = -910910
555+
556+ type(elsi_handle) :: elsi_h
557+
558+ integer :: elsi_global_comm ! Used by all routines. Freed at end of scf loop
559+ integer :: which_solver
560+ integer :: which_broad
561+ integer :: out_level
562+ integer :: out_json
563+ integer :: mp_order
564+ integer :: elpa_flavor
565+ integer :: omm_flavor
566+ integer :: omm_n_elpa
567+ integer :: pexsi_tasks_per_pole
568+ integer :: pexsi_tasks_symbolic
569+ integer :: sips_n_slice
570+ integer :: sips_n_elpa
571+ integer :: ntpoly_method
572+
573+ real(dp) :: omm_tol
574+ real(dp) :: ntpoly_filter
575+ real(dp) :: ntpoly_tol
576+
577+ character(len=6) :: solver_string
578+ character(len=5) :: broad_string
579+
580+ real(dp), allocatable :: v_old(:,:) ! For mu update in PEXSI solver
581+ real(dp), allocatable :: delta_v(:,:) ! For mu update in PEXSI solver
582+
583+ public :: elsi_getdm
584+ public :: elsi_finalize_scfloop
585+ public :: elsi_save_potential
586+
587+CONTAINS
588+
589+subroutine elsi_getdm(iscf, no_s, nspin, no_l, maxnh, no_u, &
590+ numh, listhptr, listh, H, S, qtot, temp, &
591+ xijo, nkpnt, kpoint, kweight, &
592+ eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only)
593+
594+ !
595+ ! Analogous to 'diagon', it dispatches ELSI solver routines as needed
596+ !
597+
598+ use m_fold_auxcell, only: fold_sparse_arrays ! Could be called in state_init
599+
600+! Missing for now
601+! eo(no_u,nspin,nk) : Eigenvalues
602+! qo(no_u,nspin,nk) : Occupations of eigenstates
603+
604+
605+ real(dp), intent(inout) :: H(:,:), S(:) ! Note: we might overwrite these
606+ integer, intent(in) :: iscf, maxnh, no_u, no_l, no_s, nkpnt
607+ integer, intent(in) :: neigwanted, nspin
608+ integer, intent(in) :: listh(maxnh), numh(no_l), listhptr(no_l)
609+ real(dp), intent(in) :: kpoint(3,nkpnt), qtot, temp, kweight(nkpnt), occtol,xijo(3,maxnh)
610+
611+ real(dp), intent(out) :: Dscf(maxnh,nspin), ef, Escf(maxnh,nspin), Entropy
612+ real(dp), intent(out) :: eo(no_u,nspin,nkpnt), qo(no_u,nspin,nkpnt)
613+
614+ ! Interim flag to just get the EDM
615+ ! Note that, if .true., the DM is NOT obtained
616+ ! This needs to be refactored
617+
618+ logical, intent(in) :: Get_EDM_Only
619+
620+
621+ logical :: gamma, using_aux_cell
622+ integer, allocatable :: numh_u(:), listhptr_u(:), listh_u(:)
623+ integer, allocatable :: ind2ind_u(:)
624+
625+ !
626+ real(dp), allocatable, dimension(:,:) :: Dscf_u, Escf_u, H_u
627+ real(dp), allocatable, dimension(:) :: S_u
628+
629+ integer :: iuo, ispin, j, ind, ind_u, nnz_u
630+
631+ external die
632+
633+ gamma = ((nkpnt == 1) .and. (sum(abs(kpoint(:,1))) == 0.0_dp))
634+ using_aux_cell = (no_s /= no_u)
635+
636+ if (gamma) then
637+
638+ if (.not. using_aux_cell) then
639+
640+ call elsi_real_solver(iscf, no_u, no_l, nspin, &
641+ maxnh, listhptr, listh, qtot, temp, &
642+ H, S, Dscf, Escf, ef, Entropy, Get_EDM_Only)
643+
644+ else
645+
646+ ! We fold-in the sparse arrays so that they add up all the
647+ ! matrix elements whose 'column' supercell index maps to the same
648+ ! unit-cell column. Note that they are still sparse.
649+
650+ ! First, determine the appropriate index arrays (ending in _u), and
651+ ! a mapper ind2ind_u
652+
653+ allocate(numh_u(no_l), listhptr_u(no_l))
654+ call fold_sparse_arrays(no_l,no_u, &
655+ numh,listhptr,maxnh,listh, &
656+ numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u)
657+
658+ allocate(H_u(nnz_u,nspin), S_u(nnz_u))
659+ S_u = 0
660+ H_u = 0
661+ do iuo = 1,no_l
662+ do j = 1,numh(iuo)
663+ ind = listhptr(iuo) + j
664+ ind_u = ind2ind_u(ind)
665+ S_u(ind_u) = S_u(ind_u) + S(ind)
666+ do ispin = 1, nspin
667+ H_u(ind_u,ispin) = H_u(ind_u,ispin) + H(ind,ispin)
668+ enddo
669+ enddo
670+ enddo
671+
672+ ! We can now call the standard real solver routine
673+ allocate(Dscf_u(nnz_u,nspin), Escf_u(nnz_u,nspin))
674+
675+ call elsi_real_solver(iscf, no_u, no_l, nspin, &
676+ nnz_u, listhptr_u, listh_u, qtot, temp, &
677+ H_u, S_u, Dscf_u, Escf_u, ef, Entropy, Get_EDM_Only)
678+
679+ deallocate(H_u, S_u)
680+ deallocate(numh_u, listhptr_u, listh_u)
681+
682+
683+ ! Unfold. We put the same '_u' DM entry into all image slots.
684+ do iuo = 1,no_l
685+ do j = 1,numh(iuo)
686+ ind = listhptr(iuo) + j
687+ ind_u = ind2ind_u(ind)
688+ do ispin = 1, nspin
689+ Dscf(ind,ispin) = Dscf_u(ind_u,ispin)
690+ Escf(ind,ispin) = Escf_u(ind_u,ispin)
691+ enddo
692+ enddo
693+ enddo
694+ deallocate(Dscf_u, Escf_u)
695+ deallocate(ind2ind_u)
696+
697+ endif ! using auxiliary supercell with Gamma sampling
698+
699+ else
700+
701+ ! We need more preparation
702+ call elsi_kpoints_dispatcher(iscf, no_s, nspin, no_l, maxnh, no_u, &
703+ numh, listhptr, listh, H, S, qtot, temp, &
704+ xijo, nkpnt, kpoint, kweight, &
705+ eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only)
706+
707+ endif
708+
709+end subroutine elsi_getdm
710+
711+subroutine elsi_get_opts()
712+
713+ use fdf, only: fdf_get
714+
715+ solver_string = fdf_get("ELSISolver", "elpa")
716+ out_level = fdf_get("ELSIOutputLevel", 0)
717+ out_json = fdf_get("ELSIOutputJson", 1)
718+ broad_string = fdf_get("ELSIBroadeningMethod", "fermi")
719+ mp_order = fdf_get("ELSIBroadeningMPOrder", 1)
720+ elpa_flavor = fdf_get("ELSIELPAFlavor", 2)
721+ omm_flavor = fdf_get("ELSIOMMFlavor", 0)
722+ omm_n_elpa = fdf_get("ELSIOMMELPASteps", 3)
723+ omm_tol = fdf_get("ELSIOMMTolerance", 1.0e-9_dp)
724+ pexsi_tasks_per_pole = fdf_get("ELSIPEXSITasksPerPole", ELSI_NOT_SET)
725+ pexsi_tasks_symbolic = fdf_get("ELSIPEXSITasksSymbolic", 1)
726+ sips_n_slice = fdf_get("ELSISIPSSlices", ELSI_NOT_SET)
727+ sips_n_elpa = fdf_get("ELSISIPSELPASteps", 2)
728+ ntpoly_method = fdf_get("ELSINTPOLYMethod", 2)
729+ ntpoly_filter = fdf_get("ELSINTPOLYFilter", 1.0e-9_dp)
730+ ntpoly_tol = fdf_get("ELSINTPOLYTolerance", 1.0e-6_dp)
731+
732+ select case (solver_string)
733+ case ("elpa", "ELPA")
734+ which_solver = ELPA_SOLVER
735+ case ("omm", "OMM")
736+ which_solver = OMM_SOLVER
737+ case ("pexsi", "PEXSI")
738+ which_solver = PEXSI_SOLVER
739+ case ("sips", "SIPS", "SIPs")
740+ which_solver = SIPS_SOLVER
741+ case ("ntpoly", "NTPOLY", "NTPoly")
742+ which_solver = NTPOLY_SOLVER
743+ case default
744+ which_solver = ELPA_SOLVER
745+ end select
746+
747+ select case (broad_string)
748+ case ("gauss")
749+ which_broad = GAUSSIAN
750+ case ("fermi")
751+ which_broad = FERMI
752+ case ("mp")
753+ which_broad = METHFESSEL_PAXTON
754+ case ("cubic")
755+ which_broad = CUBIC
756+ case ("cold")
757+ which_broad = COLD
758+ case default
759+ which_broad = FERMI
760+ end select
761+
762+end subroutine elsi_get_opts
763+
764+!======================================================================================
765+! ELSI takes 1D block-cyclic distributed CSC/CSR matrices as its
766+! input/output.
767+!
768+! But note taht this version assumes *the same* distributions for Siesta (setup_H et al) and ELSI
769+! operations.
770+!
771+subroutine elsi_real_solver(iscf, n_basis, n_basis_l, n_spin, nnz_l, row_ptr, &
772+ col_idx, qtot, temp, ham, ovlp, dm, edm, ef, ets, Get_EDM_Only)
773+
774+ use m_mpi_utils, only: globalize_sum
775+ use parallel, only: BlockSize
776+#ifdef MPI
777+ use mpi_siesta
778+#endif
779+ use class_Distribution
780+ use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix
781+ use alloc
782+
783+ implicit none
784+
785+ integer, intent(in) :: iscf ! SCF step counter
786+ integer, intent(in) :: n_basis ! Global basis
787+ integer, intent(in) :: n_basis_l ! Local basis
788+ integer, intent(in) :: n_spin
789+ integer, intent(in) :: nnz_l ! Local nonzero
790+ integer, intent(in) :: row_ptr(n_basis_l)
791+ integer, intent(in), target :: col_idx(nnz_l)
792+ real(dp), intent(in) :: qtot
793+ real(dp), intent(in) :: temp
794+ real(dp), intent(inout), target :: ham(nnz_l,n_spin)
795+ real(dp), intent(inout), target :: ovlp(nnz_l)
796+ real(dp), intent(out) :: dm(nnz_l,n_spin)
797+ real(dp), intent(out) :: edm(nnz_l,n_spin)
798+ real(dp), intent(out) :: ef ! Fermi energy
799+ real(dp), intent(out) :: ets ! Entropy/k, dimensionless
800+
801+ integer :: ierr
802+ integer :: n_state
803+ integer :: nnz_g
804+
805+ real(dp) :: energy
806+
807+ integer, allocatable, dimension(:) :: row_ptr2
808+ integer, allocatable, dimension(:), target :: numh
809+
810+ integer :: elsi_Spatial_comm, elsi_Spin_comm
811+
812+ type(distribution) :: dist_global
813+ type(distribution) :: dist_spin(2)
814+
815+ type(aux_matrix) :: pkg_global, pkg_spin ! Packages for transfer
816+
817+ integer :: my_spin
818+
819+ integer :: my_no_l
820+ integer :: my_nnz_l
821+ integer :: my_nnz
822+ integer, pointer :: my_row_ptr2(:) => null()
823+ integer :: i, ih, ispin, spin_rank
824+ real(dp) :: ets_spin
825+
826+ integer, pointer :: my_col_idx(:)
827+ real(dp), pointer :: my_S(:)
828+ real(dp), pointer :: my_H(:)
829+ real(dp), pointer :: my_DM(:) => null()
830+ real(dp), pointer :: my_EDM(:) => null()
831+
832+ integer :: date_stamp
833+
834+ logical :: Get_EDM_Only
835+
836+ external :: timer
837+
838+#ifndef MPI
839+ call die("This ELSI solver interface needs MPI")
840+#endif
841+
842+ ! Global communicator is a duplicate of passed communicator
843+ call MPI_Comm_Dup(MPI_Comm_DFT, elsi_global_comm, ierr)
844+
845+ call timer("elsi", 1)
846+
847+ ! Initialization
848+ if (iscf == 1) then
849+
850+ ! Get ELSI options
851+ call elsi_get_opts()
852+
853+ ! Number of states to solve when calling an eigensolver
854+ n_state = min(n_basis, n_basis/2+5)
855+
856+ ! Now we have all ingredients to initialize ELSI
857+ call elsi_init(elsi_h, which_solver, MULTI_PROC, SIESTA_CSC, n_basis, &
858+ qtot, n_state)
859+
860+ ! Output
861+ call elsi_set_output(elsi_h, out_level)
862+ call elsi_set_output_log(elsi_h, out_json)
863+ call elsi_set_write_unit(elsi_h, 6)
864+
865+ ! Broadening
866+ call elsi_set_mu_broaden_scheme(elsi_h, which_broad)
867+ call elsi_set_mu_broaden_width(elsi_h, temp)
868+ call elsi_set_mu_mp_order(elsi_h, mp_order)
869+
870+ ! Solver settings
871+ call elsi_set_elpa_solver(elsi_h, elpa_flavor)
872+
873+ call elsi_set_omm_flavor(elsi_h, omm_flavor)
874+ call elsi_set_omm_n_elpa(elsi_h, omm_n_elpa)
875+ call elsi_set_omm_tol(elsi_h, omm_tol)
876+
877+ if (pexsi_tasks_per_pole /= ELSI_NOT_SET) then
878+ call elsi_set_pexsi_np_per_pole(elsi_h, pexsi_tasks_per_pole)
879+ end if
880+
881+ call elsi_set_pexsi_np_symbo(elsi_h, pexsi_tasks_symbolic)
882+ call elsi_set_pexsi_temp(elsi_h, temp)
883+! call elsi_set_pexsi_gap(elsi_h, gap)
884+! call elsi_set_pexsi_delta_e(elsi_h, delta_e)
885+ call elsi_set_pexsi_mu_min(elsi_h, -10.0_dp)
886+ call elsi_set_pexsi_mu_max(elsi_h, 10.0_dp)
887+
888+ if (sips_n_slice /= ELSI_NOT_SET) then
889+ call elsi_set_sips_n_slice(elsi_h, sips_n_slice)
890+ end if
891+
892+ call elsi_set_sips_n_elpa(elsi_h, sips_n_elpa)
893+ call elsi_set_sips_interval(elsi_h, -10.0_dp, 10.0_dp)
894+
895+ call elsi_set_ntpoly_method(elsi_h, ntpoly_method)
896+ call elsi_set_ntpoly_filter(elsi_h, ntpoly_filter)
897+ call elsi_set_ntpoly_tol(elsi_h, ntpoly_tol)
898+
899+ endif ! iscf == 1
900+
901+ if (n_spin == 1) then
902+
903+ ! Sparsity pattern
904+ call globalize_sum(nnz_l, nnz_g, comm=elsi_global_comm)
905+
906+ allocate(row_ptr2(n_basis_l+1))
907+ row_ptr2(1:n_basis_l) = row_ptr(1:n_basis_l)+1
908+ row_ptr2(n_basis_l+1) = nnz_l+1
909+
910+ call elsi_set_csc(elsi_h, nnz_g, nnz_l, n_basis_l, col_idx, row_ptr2)
911+ deallocate(row_ptr2)
912+
913+ call elsi_set_csc_blk(elsi_h, BlockSize)
914+ call elsi_set_mpi(elsi_h, elsi_global_comm)
915+
916+ else
917+
918+ ! MPI logic for spin polarization
919+
920+ ! Re-create numh, as we use it in the transfer
921+ allocate(numh(n_basis_l))
922+ numh(1) = row_ptr(2)
923+ do i = 2, n_basis_l-1
924+ numh(i) = row_ptr(i+1)-row_ptr(i)
925+ enddo
926+ numh(n_basis_l) = nnz_l - row_ptr(n_basis_l)
927+
928+ ! Split the communicator in spins and get distribution objects
929+ ! for the data redistribution needed
930+ ! Note that dist_spin is an array
931+ call get_spin_comms_and_dists(elsi_global_comm,elsi_global_comm, &
932+ blocksize, n_spin, &
933+ dist_global,dist_spin, elsi_spatial_comm, elsi_spin_comm)
934+
935+ ! Find out which spin team we are in, and tag the spin we work on
936+ call mpi_comm_rank( elsi_Spin_Comm, spin_rank, ierr )
937+ my_spin = spin_rank+1 ! {1,2}
938+
939+
940+ ! This is done serially, each time filling one spin set
941+ ! Note that **all processes** need to have the same pkg_global
942+
943+ do ispin = 1, n_spin
944+
945+ ! Load pkg_global data package
946+ pkg_global%norbs = n_basis
947+ pkg_global%no_l = n_basis_l
948+ pkg_global%nnzl = nnz_l
949+ pkg_global%numcols => numh
950+ pkg_global%cols => col_idx
951+
952+ allocate(pkg_global%vals(2))
953+ ! Link the vals items to the appropriate arrays (no extra memory here)
954+ pkg_global%vals(1)%data => ovlp(:)
955+ ! Note that we *cannot* say => ham(:,my_spin)
956+ ! and avoid the sequential loop, as then half the processors will send
957+ ! the information for 'spin up' and the other half the information for 'spin down',
958+ ! which is *not* what we want.
959+ pkg_global%vals(2)%data => ham(:,ispin)
960+
961+ call timer("redist_orbs_fwd", 1)
962+
963+ ! We are doing the transfers sequentially. One spin team is
964+ ! 'idle' (in the receiving side) in each pass, as the dist_spin(ispin) distribution
965+ ! does not involve them.
966+
967+ call redistribute_spmatrix(n_basis,pkg_global,dist_global, &
968+ pkg_spin,dist_spin(ispin),elsi_global_Comm)
969+
970+ call timer("redist_orbs_fwd", 2)
971+
972+ if (my_spin == ispin) then ! Each team gets their own data
973+
974+ !nrows = pkg_spin%norbs ! or simply 'norbs'
975+ my_no_l = pkg_spin%no_l
976+ my_nnz_l = pkg_spin%nnzl
977+ call MPI_AllReduce(my_nnz_l,my_nnz,1,MPI_integer,MPI_sum,elsi_Spatial_Comm,ierr)
978+ ! generate off-by-one row pointer
979+ call re_alloc(my_row_ptr2,1,my_no_l+1,"my_row_ptr2","elsi_solver")
980+ my_row_ptr2(1) = 1
981+ do ih = 1,my_no_l
982+ my_row_ptr2(ih+1) = my_row_ptr2(ih) + pkg_spin%numcols(ih)
983+ enddo
984+
985+ my_col_idx => pkg_spin%cols
986+ my_S => pkg_spin%vals(1)%data
987+ my_H => pkg_spin%vals(2)%data
988+
989+ call re_alloc(my_DM,1,my_nnz_l,"my_DM","elsi_solver")
990+ call re_alloc(my_EDM,1,my_nnz_l,"my_EDM","elsi_solver")
991+ endif
992+
993+ ! Clean pkg_global
994+ nullify(pkg_global%vals(1)%data)
995+ nullify(pkg_global%vals(2)%data)
996+ deallocate(pkg_global%vals)
997+ nullify(pkg_global%numcols)
998+ nullify(pkg_global%cols)
999+
1000+ enddo
1001+
1002+ call elsi_set_csc(elsi_h, my_nnz, my_nnz_l, my_no_l, my_col_idx, my_row_ptr2)
1003+ call de_alloc(my_row_ptr2,"my_row_ptr2","elsi_solver")
1004+
1005+ call elsi_set_csc_blk(elsi_h, BlockSize)
1006+ call elsi_set_spin(elsi_h, n_spin, my_spin)
1007+ call elsi_set_mpi(elsi_h, elsi_Spatial_comm)
1008+ call elsi_set_mpi_global(elsi_h, elsi_global_comm)
1009+
1010+ endif ! n_spin
1011+
1012+ call timer("elsi-solver", 1)
1013+
1014+ if (n_spin == 1) then
1015+ if (.not.Get_EDM_Only) then
1016+ call elsi_dm_real_sparse(elsi_h, ham, ovlp, DM, energy)
1017+ call elsi_get_entropy(elsi_h, ets)
1018+ else
1019+ call elsi_get_edm_real_sparse(elsi_h, EDM)
1020+ endif
1021+ else
1022+ ! Solve DM, and get (at every step for now) EDM, Fermi energy, and entropy
1023+ ! Energy is already summed over spins
1024+ if (.not.Get_EDM_Only) then
1025+ call elsi_dm_real_sparse(elsi_h, my_H, my_S, my_DM, energy)
1026+ call elsi_get_entropy(elsi_h, ets_spin)
1027+ else
1028+ call elsi_get_edm_real_sparse(elsi_h, my_EDM)
1029+ endif
1030+#ifdef SIESTA__ELSI__OLD_SPIN_CONVENTION
1031+ ! NOTE**
1032+ ! For ELSI versions before 2018-08-17 we still need to sum the entropy over spins
1033+ ! ... but to figure out the version is not trivial, as the relevant routines changed...
1034+ call globalize_sum(ets_spin, ets, comm=elsi_Spin_comm)
1035+#else
1036+ ! If this gives an error, then your version of ELSI is old, and you should
1037+ ! be using the pre-processor symbol above
1038+ ! (This works because 'elsi_get_datestamp' was added just
1039+ ! around the same time as the spin-reduction-convention change) (Aug 17 vs Aug 21, 2018...)
1040+ ! If you are unlucky enough to have an ELSI version in between, get rid of it.
1041+ call elsi_get_datestamp(date_stamp)
1042+#endif
1043+ endif
1044+
1045+ call elsi_get_mu(elsi_h, ef)
1046+ ets = ets/temp
1047+
1048+ ! Ef, energy, and ets are known to all nodes
1049+
1050+ call timer("elsi-solver", 2)
1051+
1052+ if ( n_spin == 2) then
1053+ ! Now we need to redistribute back
1054+
1055+ do ispin = 1, n_spin
1056+
1057+ if (my_spin == ispin) then
1058+ ! Prepare pkg_spin to transfer the right spin information
1059+ ! The other fields (numcols, cols) are the same and are still there
1060+ ! Deallocate my_S and my_H
1061+ call de_alloc(pkg_spin%vals(1)%data,"pkg_spin%vals(1)%data","elsi_solver")
1062+ call de_alloc(pkg_spin%vals(2)%data,"pkg_spin%vals(2)%data","elsi_solver")
1063+
1064+ pkg_spin%vals(1)%data => my_DM(1:my_nnz_l)
1065+ pkg_spin%vals(2)%data => my_EDM(1:my_nnz_l)
1066+
1067+ endif
1068+
1069+ ! pkg_global is clean now
1070+ call timer("redist_orbs_bck", 1)
1071+ call redistribute_spmatrix(n_basis,pkg_spin,dist_spin(ispin) &
1072+ ,pkg_global,dist_global,elsi_global_Comm)
1073+ call timer("redist_orbs_bck", 2)
1074+
1075+ ! Clean pkg_spin
1076+ if (my_spin == ispin) then
1077+ ! Each team deallocates during "its" spin cycle
1078+ call de_alloc(my_DM, "my_DM", "elsi_solver")
1079+ call de_alloc(my_EDM,"my_EDM","elsi_solver")
1080+
1081+ nullify(pkg_spin%vals(1)%data) ! formerly pointing to DM
1082+ nullify(pkg_spin%vals(2)%data) ! formerly pointing to EDM
1083+ deallocate(pkg_spin%vals)
1084+ ! allocated in the direct transfer
1085+ call de_alloc(pkg_spin%numcols,"pkg_spin%numcols","elsi_solver")
1086+ call de_alloc(pkg_spin%cols, "pkg_spin%cols", "elsi_solver")
1087+ endif
1088+
1089+
1090+ ! In future, pkg_global%vals(1,2) could be pointing to DM and EDM,
1091+ ! and the 'redistribute' routine check whether the vals arrays are
1092+ ! associated, to use them instead of allocating them.
1093+ DM(:,ispin) = pkg_global%vals(1)%data(:)
1094+ EDM(:,ispin) = pkg_global%vals(2)%data(:)
1095+ ! Check no_l
1096+ if (n_basis_l /= pkg_global%no_l) then
1097+ call die("Mismatch in no_l")
1098+ endif
1099+ ! Check listH
1100+ if (any(col_idx(:) /= pkg_global%cols(:))) then
1101+ call die("Mismatch in listH")
1102+ endif
1103+
1104+ ! Clean pkg_global
1105+ ! allocated by the transfer routine, but we did not actually
1106+ ! look at them
1107+ call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver")
1108+ call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver")
1109+
1110+ call de_alloc(pkg_global%vals(1)%data,"pkg_global%vals(1)%data","elsi_solver")
1111+ call de_alloc(pkg_global%vals(2)%data,"pkg_global%vals(2)%data","elsi_solver")
1112+ deallocate(pkg_global%vals)
1113+
1114+ enddo
1115+
1116+ call MPI_Comm_Free(elsi_Spatial_comm, ierr)
1117+ call MPI_Comm_Free(elsi_Spin_comm, ierr)
1118+
1119+ endif
1120+
1121+ call timer("elsi", 2)
1122+
1123+
1124+end subroutine elsi_real_solver
1125+
1126+subroutine elsi_kpoints_dispatcher(iscf, no_s, nspin, no_l, maxnh, no_u, &
1127+ numh, listhptr, listh, H, S, qtot, temp, &
1128+ xijo, nkpnt, kpoint, kweight, &
1129+ eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only)
1130+
1131+ use mpi_siesta, only: mpi_comm_dft
1132+ use mpi
1133+ use parallel, only: blocksize
1134+ use class_Distribution
1135+ use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix
1136+ use alloc
1137+
1138+ !
1139+ ! K-point redistribution, Hk and Sk building, and call to complex ELSI solver
1140+ !
1141+
1142+ use m_fold_auxcell, only: fold_sparse_arrays ! Could be called in state_init
1143+
1144+! Missing for now
1145+! eo(no_u,nspin,nk) : Eigenvalues
1146+! qo(no_u,nspin,nk) : Occupations of eigenstates
1147+
1148+
1149+ real(dp), intent(in), target :: H(:,:), S(:) ! Note that now we do not change them
1150+ integer, intent(in) :: iscf, maxnh, no_u, no_l, no_s, nkpnt
1151+ integer, intent(in) :: neigwanted, nspin
1152+ integer, intent(in) :: listhptr(no_l)
1153+ integer, intent(in), target :: listh(maxnh), numh(no_l)
1154+
1155+ real(dp), intent(out) :: Dscf(maxnh,nspin), ef, Escf(maxnh,nspin), Entropy
1156+ real(dp), intent(out) :: eo(no_u,nspin,nkpnt), qo(no_u,nspin,nkpnt)
1157+ real(dp), intent(in) :: kpoint(3,nkpnt), qtot, temp, kweight(nkpnt), occtol,xijo(3,maxnh)
1158+
1159+
1160+ integer :: mpirank, kcolrank, npGlobal
1161+ integer :: npPerK, color, my_kpt_n
1162+
1163+ integer :: kpt_comm, kpt_col_comm
1164+ integer :: Global_Group, kpt_Group
1165+
1166+ integer, allocatable :: ranks_in_world(:), ranks_in_world_AllK(:,:)
1167+ type(distribution) :: dist_global
1168+ type(distribution), allocatable :: dist_k(:)
1169+
1170+ type(aux_matrix) :: pkg_global, pkg_k
1171+ integer :: nvals
1172+ real(dp), allocatable, target :: xijo_transp(:,:)
1173+ real(dp), allocatable :: my_xijo_transp(:,:)
1174+ real(dp), allocatable :: my_xij(:,:)
1175+
1176+ integer :: my_no_l, my_nnz_l
1177+ integer, pointer :: my_numh(:)
1178+ integer, pointer :: my_listh(:)
1179+ integer, pointer :: my_listhptr(:) => null()
1180+ real(dp), allocatable :: my_S(:)
1181+ real(dp), allocatable :: my_H(:,:)
1182+ real(dp), pointer :: buffer(:) ! for unpacking help
1183+
1184+ real(dp), allocatable :: my_Escf(:,:)
1185+ real(dp), allocatable :: my_Dscf(:,:)
1186+ real(dp), allocatable, target :: my_Escf_reduced(:,:)
1187+ real(dp), allocatable, target :: my_Dscf_reduced(:,:)
1188+
1189+ complex(dp), allocatable :: DM_k(:,:)
1190+ complex(dp), allocatable :: EDM_k(:,:)
1191+ complex(dp), allocatable :: Hk(:,:)
1192+ complex(dp), allocatable :: Sk(:)
1193+
1194+ integer :: iuo, j, ind, ind_u, ispin
1195+ real(dp) :: kxij, my_kpoint(3)
1196+ complex(dp) :: kphs
1197+
1198+ integer :: i, ih, ik, ierr
1199+
1200+ integer, allocatable :: numh_u(:), listhptr_u(:), listh_u(:), ind2ind_u(:)
1201+ integer :: nnz_u
1202+
1203+ logical :: Get_EDM_Only
1204+
1205+ external die
1206+
1207+ ! Split the global communicator
1208+ ! Re-distribute H and S to the k-point (and spin) teams
1209+ ! Generate Hk, Sk
1210+ ! Call elsi_complex_solver
1211+ ! Construct and re-distribute global DM and EDM
1212+
1213+ ! Global communicator is a duplicate of passed communicator
1214+ call MPI_Comm_Dup(MPI_Comm_DFT, elsi_global_comm, ierr)
1215+ call MPI_Comm_Rank(elsi_global_comm, mpirank, ierr)
1216+ call MPI_Comm_Size(elsi_global_comm, npGlobal, ierr)
1217+
1218+ ! Split elsi_global_comm
1219+
1220+ npPerK = npGlobal/nkpnt
1221+
1222+ ! Options for split: As many groups as nkpnt, so numbering is trivial
1223+ color = mpirank/npPerK ! : color 0: 0,1,2,3 ; color 1: 4,5,6,7
1224+ call MPI_Comm_Split(elsi_global_comm, color, mpirank, kpt_Comm, ierr)
1225+ my_kpt_n = 1 + color ! 1 + mpirank/npPerK
1226+ ! Column communicator
1227+ color = mod(mpirank, npPerK) ! : color 0: 0,4 1: 1,5 2: 2,6 3: 3,7
1228+ call MPI_Comm_Split(elsi_global_comm, color, mpirank, kpt_col_Comm, ierr) ! OK to use mpirank: just ordering
1229+ call MPI_Comm_Rank(kpt_col_comm, kcolrank, ierr)
1230+
1231+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " rank in col:", kcolrank
1232+
1233+ ! Create distribution for k-point group
1234+ call MPI_Comm_Group(elsi_global_comm, Global_Group, Ierr)
1235+ call MPI_Comm_Group(kpt_Comm, kpt_Group, Ierr)
1236+ allocate(ranks_in_world(npPerK))
1237+ call MPI_Group_translate_ranks( kpt_Group, npPerK, &
1238+ (/ (i,i=0,npPerK-1) /), &
1239+ Global_Group, ranks_in_world, ierr )
1240+
1241+ call MPI_Group_Free(kpt_Group, ierr)
1242+
1243+ allocate (ranks_in_World_AllK(npPerK,nkpnt))
1244+ ! We need everybody to have this information, as all nodes
1245+ ! are part of the global distribution side of the communication
1246+ ! (global_comm is the common-address space)
1247+
1248+ ! Use the k-column communicator
1249+ call MPI_AllGather(ranks_in_world,npPerK,MPI_integer,&
1250+ Ranks_in_World_AllK(1,1),npPerK, &
1251+ MPI_integer,kpt_col_Comm,ierr)
1252+
1253+ allocate(dist_k(nkpnt))
1254+ ! Create distributions known to all nodes (again, those in base_comm)
1255+ do ik = 1, nkpnt
1256+ call newDistribution(dist_k(ik), elsi_global_Comm, &
1257+ Ranks_in_World_AllK(:,ik), &
1258+ TYPE_BLOCK_CYCLIC, blockSize, "kpt dist")
1259+ enddo
1260+ deallocate(ranks_in_world,Ranks_in_World_AllK)
1261+ call MPI_Barrier(elsi_global_Comm,ierr)
1262+
1263+ call newDistribution(dist_global,elsi_global_Comm, (/ (i, i=0, npGlobal-1) /), &
1264+ TYPE_BLOCK_CYCLIC,BlockSize,"global dist")
1265+ call MPI_Barrier(elsi_global_Comm,ierr)
1266+
1267+ ! Redistribute arrays
1268+
1269+ ! No need to go sequentially over k-points
1270+ ! Load pkg_global data package
1271+ pkg_global%norbs = no_u
1272+ pkg_global%no_l = no_l
1273+ pkg_global%nnzl = maxnh
1274+ pkg_global%numcols => numh
1275+ pkg_global%cols => listh
1276+
1277+ nvals = 1 + 3 + nspin ! S, xijo (tranposed), H(:,nspin)
1278+ allocate(pkg_global%vals(nvals))
1279+ ! Link the vals items to the appropriate arrays (no extra memory here)
1280+ pkg_global%vals(1)%data => S(:)
1281+ call transpose(xijo,xijo_transp)
1282+ do i = 1, 3
1283+ pkg_global%vals(1+i)%data => xijo_transp(:,i)
1284+ enddo
1285+ do ispin = 1, nspin
1286+ pkg_global%vals(4+ispin)%data => H(:,ispin)
1287+ enddo
1288+
1289+ ! Do sequential
1290+ do ik=1, nkpnt
1291+
1292+ call timer("redist_orbs_fwd", 1)
1293+ call redistribute_spmatrix(no_u,pkg_global,dist_global, &
1294+ pkg_k,dist_k(ik),elsi_global_Comm)
1295+ call timer("redist_orbs_fwd", 2)
1296+
1297+ if (my_kpt_n == ik) then
1298+ !------------------------------------------
1299+ ! Unpack info: real S and H (and index arrays) distributed over each kpt_comm
1300+ my_no_l = pkg_k%no_l
1301+ my_nnz_l = pkg_k%nnzl
1302+ my_numh => pkg_k%numcols
1303+
1304+ my_listh => pkg_k%cols
1305+
1306+ allocate(my_S(my_nnz_l))
1307+ my_S(:) = pkg_k%vals(1)%data(:)
1308+ call de_alloc(pkg_k%vals(1)%data)
1309+
1310+ allocate(my_xijo_transp(my_nnz_l,3))
1311+ do i = 1, 3
1312+ buffer => pkg_k%vals(1+i)%data
1313+ my_xijo_transp(:,i) = buffer(:)
1314+ call de_alloc(pkg_k%vals(1+i)%data)
1315+ enddo
1316+
1317+ allocate(my_H(my_nnz_l,nspin))
1318+ do ispin = 1, nspin
1319+ buffer => pkg_k%vals(4+ispin)%data
1320+ my_H(:,ispin) = buffer(:)
1321+ call de_alloc(pkg_k%vals(4+ispin)%data)
1322+ enddo
1323+ deallocate(pkg_k%vals)
1324+
1325+ ! Now we could clear the rest of pkg_k--
1326+ ! nullify(pkg_k%numcols)
1327+ ! nullify(pkg_k%cols)
1328+ !
1329+ ! but we need to remember to deallocate the actual arrays after use
1330+ ! it is probably safer to keep the pkg references
1331+ !---------------------------
1332+
1333+ endif
1334+ enddo !ik
1335+
1336+ ! Clean pkg_global -- This is safe, as we use pointers to data only
1337+ do i = 1, size(pkg_global%vals)
1338+ nullify(pkg_global%vals(i)%data)
1339+ enddo
1340+ deallocate(pkg_global%vals)
1341+ nullify(pkg_global%numcols)
1342+ nullify(pkg_global%cols)
1343+
1344+ deallocate(xijo_transp) ! Auxiliary array used for sending
1345+
1346+ ! generate listhptr for folding/unfolding operations
1347+ call re_alloc(my_listhptr,1,my_no_l,"my_listhptr","elsi_solver")
1348+ my_listhptr(1) = 0
1349+ do ih = 2,my_no_l
1350+ my_listhptr(ih) = my_listhptr(ih-1) + my_numh(ih-1)
1351+ enddo
1352+
1353+ ! The folded variables (*_u) are local to each team
1354+
1355+ allocate(numh_u(my_no_l), listhptr_u(my_no_l))
1356+ call fold_sparse_arrays(my_no_l,no_u, &
1357+ my_numh,my_listhptr,my_nnz_l,my_listh, &
1358+ numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u)
1359+
1360+ !
1361+ call transpose(my_xijo_transp,my_xij)
1362+ deallocate(my_xijo_transp)
1363+
1364+ my_kpoint(:) = kpoint(:,my_kpt_n)
1365+
1366+ allocate(Hk(nnz_u,nspin), Sk(nnz_u))
1367+ Sk = 0
1368+ Hk = 0
1369+ do iuo = 1,my_no_l
1370+ do j = 1,my_numh(iuo)
1371+ ind = my_listhptr(iuo) + j
1372+ ind_u = ind2ind_u(ind)
1373+
1374+ kxij = my_kpoint(1) * my_xij(1,ind) + &
1375+ my_kpoint(2) * my_xij(2,ind) + &
1376+ my_kpoint(3) * my_xij(3,ind)
1377+ kphs = cdexp(dcmplx(0.0_dp, -1.0_dp)*kxij)
1378+
1379+ Sk(ind_u) = Sk(ind_u) + my_S(ind) *kphs
1380+ do ispin = 1, nspin
1381+ Hk(ind_u,ispin) = Hk(ind_u,ispin) + my_H(ind,ispin) *kphs
1382+ enddo
1383+ enddo
1384+ enddo
1385+
1386+ deallocate(my_S,my_H)
1387+ !
1388+
1389+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done folding"
1390+ ! Prepare arrays for holding results
1391+ allocate(DM_k(nnz_u,nspin))
1392+ allocate(EDM_k(nnz_u,nspin))
1393+
1394+ call elsi_complex_solver(iscf, no_u, my_no_l, nspin, nnz_u, numh_u, listhptr_u, &
1395+ listh_u, qtot, temp, Hk, Sk, DM_k, EDM_k, Ef, Entropy, &
1396+ nkpnt, my_kpt_n, kpoint(:,my_kpt_n), kweight(my_kpt_n), &
1397+ kpt_Comm, Get_EDM_Only )
1398+
1399+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done elsi_complex_solver"
1400+ deallocate(listhptr_u, numh_u, listh_u)
1401+ deallocate(Hk,Sk)
1402+
1403+ ! Re-create DM and EDM:
1404+ ! Unfold within a given k
1405+ ! Add up all the k contributions with the appropriate phases
1406+
1407+ ! Prepare arrays for holding results: Note sizes: these are folded out
1408+ allocate(my_Dscf(my_nnz_l,nspin))
1409+ allocate(my_Escf(my_nnz_l,nspin))
1410+
1411+ do iuo = 1, my_no_l
1412+ do j = 1, my_numh(iuo)
1413+ ind = my_listhptr(iuo) + j
1414+ ind_u = ind2ind_u(ind)
1415+
1416+ kxij = my_kpoint(1) * my_xij(1,ind) + &
1417+ my_kpoint(2) * my_xij(2,ind) + &
1418+ my_kpoint(3) * my_xij(3,ind)
1419+ kphs = cdexp(dcmplx(0.0_dp, +1.0_dp)*kxij)
1420+
1421+ do ispin = 1, nspin
1422+ my_Dscf(ind,ispin) = real( DM_k(ind_u,ispin) * kphs, kind=dp )
1423+ my_Escf(ind,ispin) = real( EDM_k(ind_u,ispin) * kphs, kind=dp )
1424+ enddo
1425+ enddo
1426+ enddo
1427+
1428+ ! Apply the k-point weight to the (apparently normalized) DM and EDM that
1429+ ! come out of ELSI
1430+
1431+ my_Dscf = kweight(my_kpt_n) * my_Dscf
1432+ my_Escf = kweight(my_kpt_n) * my_Escf
1433+
1434+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done generating my_Dscf"
1435+ deallocate(my_xij)
1436+ deallocate(ind2ind_u)
1437+ deallocate(DM_k,EDM_k)
1438+
1439+ if (my_kpt_n == 1 ) then
1440+ ! Prepare arrays for holding reduced data
1441+ allocate(my_Dscf_reduced(my_nnz_l,nspin))
1442+ allocate(my_Escf_reduced(my_nnz_l,nspin))
1443+ if (kcolrank /= 0) call die("Rank 0 in kpt_comm not doing kpt 1")
1444+ else
1445+ ! These should not be referenced
1446+ allocate(my_Dscf_reduced(1,1))
1447+ allocate(my_Escf_reduced(1,1))
1448+ endif
1449+
1450+ ! Use k-point column communicator, and reduce to rank 0,
1451+ ! which *should* correspond to kpt=1... (checked above)
1452+ call MPI_Reduce( my_Dscf, my_Dscf_reduced, nspin*my_nnz_l, MPI_Double_Precision, &
1453+ MPI_Sum, 0, kpt_col_comm, ierr )
1454+ call MPI_Reduce( my_Escf, my_Escf_reduced, nspin*my_nnz_l, MPI_Double_Precision, &
1455+ MPI_Sum, 0, kpt_col_comm, ierr )
1456+
1457+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done reducing my_Dscf"
1458+ deallocate(my_Dscf, my_Escf)
1459+
1460+ ! redistribute to global distribution, only from the first k-point
1461+
1462+ if (my_kpt_n == 1) then
1463+
1464+ ! These bits are still there *** update if we ever clean pkgs after use
1465+ ! pkg_k%norbs = no_u
1466+ ! pkg_k%no_l = my_no_l
1467+ ! pkg_k%nnzl = my_nnz_l
1468+ ! pkg_k%numcols => my_numh
1469+ ! pkg_k%cols => my_listh
1470+
1471+ nvals = 2*nspin ! DM, EDM
1472+ allocate(pkg_k%vals(nvals))
1473+ do ispin = 1, nspin
1474+ pkg_k%vals(ispin)%data => my_Dscf_reduced(:,ispin)
1475+ pkg_k%vals(nspin+ispin)%data => my_Escf_reduced(:,ispin)
1476+ enddo
1477+
1478+ endif
1479+
1480+ ! Everybody participates in the transfer in the receiving side, but
1481+ ! only kpt=1 in the sending side, because we are using dist_k(1)
1482+ call timer("redist_dm-edm_bck", 1)
1483+ call redistribute_spmatrix(no_u,pkg_k,dist_k(1), &
1484+ pkg_global,dist_global,elsi_global_Comm)
1485+ call timer("redist_dm-edm_bck", 2)
1486+
1487+ ! Deallocate aux arrays
1488+ deallocate(my_Dscf_reduced, my_Escf_reduced)
1489+ if (my_kpt_n == 1) then
1490+ !Clean pkg_k in sender
1491+ deallocate(pkg_k%vals) ! just pointers
1492+ endif
1493+
1494+ !Clean all pkg_k: These were actually allocated in the forward transfer
1495+ call de_alloc(pkg_k%numcols,"pkg_k%numcols")
1496+ call de_alloc(pkg_k%cols,"pkg_k%cols")
1497+
1498+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done cleaning pkg_k"
1499+
1500+ ! Unpack data (all processors, original global distribution)
1501+ ! In future, pkg_global%vals(:) could be pointing to DM and EDM,
1502+ ! and the 'redistribute' routine check whether the vals arrays are
1503+ ! associated, to use them instead of allocating them.
1504+ do ispin = 1, nspin
1505+ Dscf(:,ispin) = pkg_global%vals(ispin)%data(:)
1506+ Escf(:,ispin) = pkg_global%vals(nspin+ispin)%data(:)
1507+ enddo
1508+ ! Check no_l
1509+ if (no_l /= pkg_global%no_l) then
1510+ call die("Mismatch in no_l at end of kpoints-dispatcher")
1511+ endif
1512+ ! Check listH
1513+ if (any(listh(:) /= pkg_global%cols(:))) then
1514+ call die("Mismatch in listH at end of kpoints-dispatcher")
1515+ endif
1516+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done Dscf"
1517+
1518+ ! Clean pkg_global
1519+ call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver")
1520+ call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver")
1521+ do i = 1, size(pkg_global%vals)
1522+ call de_alloc(pkg_global%vals(i)%data,"pkg_global%vals%data","kpoints_dispatcher")
1523+ enddo
1524+ deallocate(pkg_global%vals)
1525+ !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done cleaning pkg_global"
1526+
1527+ ! Reduction of entropy over kpt_col_comm is not necessary
1528+
1529+ call MPI_Comm_Free(kpt_comm, ierr)
1530+ call MPI_Comm_Free(kpt_col_comm, ierr)
1531+
1532+end subroutine elsi_kpoints_dispatcher
1533+
1534+! Clean up: Finalize ELSI instance and free MPI communicator.
1535+!
1536+subroutine elsi_finalize_scfloop()
1537+
1538+ integer :: ierr
1539+
1540+ ! Make which_solver invalid
1541+ which_solver = ELSI_NOT_SET
1542+
1543+ if (allocated(v_old)) then
1544+ deallocate(v_old)
1545+ deallocate(delta_v)
1546+ end if
1547+
1548+ call elsi_finalize(elsi_h)
1549+
1550+ call MPI_Comm_Free(elsi_global_comm, ierr)
1551+
1552+end subroutine elsi_finalize_scfloop
1553+
1554+! Save Hartree + XC potential and find minimum and maximum change of it between
1555+! two SCF iterations.
1556+!
1557+subroutine elsi_save_potential(n_pts, n_spin, v_scf, comm)
1558+
1559+ use m_mpi_utils, only: globalize_min, globalize_max
1560+ use parallel, only: ionode, node
1561+ use units, only: eV
1562+
1563+ integer, intent(in) :: n_pts
1564+ integer, intent(in) :: n_spin
1565+ real(dp), intent(in) :: v_scf(n_pts,n_spin)
1566+ integer , intent(in) :: comm ! The Siesta communicator used for grid operations
1567+ ! since this routine is called from dhscf...
1568+
1569+ real(dp) :: mu_min
1570+ real(dp) :: mu_max
1571+ real(dp) :: dv_min
1572+ real(dp) :: dv_max
1573+ real(dp) :: tmp
1574+
1575+ if (which_solver == PEXSI_SOLVER) then
1576+ if (.not. allocated(v_old)) then
1577+ allocate(v_old(n_pts,n_spin))
1578+ allocate(delta_v(n_pts,n_spin))
1579+
1580+ v_old = v_scf
1581+
1582+ mu_min = -10.0_dp
1583+ mu_max = 10.0_dp
1584+ else
1585+ call elsi_get_pexsi_mu_min(elsi_h, mu_min)
1586+ call elsi_get_pexsi_mu_max(elsi_h, mu_max)
1587+ if (ionode) then
1588+ print *, "*-- solver mu_min, mu_max:", mu_min/eV, mu_max/eV
1589+ endif
1590+
1591+ delta_v = v_scf - v_old
1592+ v_old = v_scf
1593+
1594+ ! Get minimum and maximum of change of total potential
1595+ tmp = minval(delta_v)
1596+
1597+ call globalize_min(tmp, dv_min, comm=comm)
1598+
1599+ tmp = maxval(delta_v)
1600+
1601+ call globalize_max(tmp, dv_max, comm=comm)
1602+ if (ionode) then
1603+ print *, " *---- min and max Delta-V: ", dv_min/eV, dv_max/eV
1604+ endif
1605+
1606+ mu_min = mu_min+dv_min
1607+ mu_max = mu_max+dv_max
1608+ end if
1609+
1610+ ! Adjust chemical potential range for PEXSI
1611+ call elsi_set_pexsi_mu_min(elsi_h, mu_min)
1612+ call elsi_set_pexsi_mu_max(elsi_h, mu_max)
1613+ if (ionode) then
1614+ print *, "*-- updated mu_min, mu_max:", mu_min/eV, mu_max/eV
1615+ endif
1616+ end if
1617+
1618+end subroutine elsi_save_potential
1619+
1620+! To-Do: Check the implications of distinguishing between 'global_comm'
1621+! (common space for communications) and 'base_comm' (to be split)
1622+! For the real case they can be the same
1623+!
1624+! This routine should be called by everybody in global_comm
1625+
1626+subroutine get_spin_comms_and_dists(global_comm, base_comm, blocksize, n_spin,&
1627+ dist_global, dist_spin, spatial_comm, spin_comm)
1628+
1629+ use mpi_siesta
1630+ use class_Distribution ! distribution, newDistribution, types
1631+
1632+ integer, intent(in) :: global_comm ! Communicator for global distribution
1633+ ! Needed to include the actual global ranks
1634+ ! in the distribution objects
1635+
1636+ integer, intent(in) :: base_comm ! Communicator to be split
1637+ integer, intent(out) :: spatial_comm ! "Orbital" comm (one for each spin team)
1638+ integer, intent(out) :: spin_comm ! "Inner" spin comm (for reductions)
1639+
1640+ integer, intent(in) :: blocksize
1641+ integer, intent(in) :: n_spin ! should be 2 in this version
1642+
1643+ ! Note inout for bud objects
1644+ type(distribution), intent(inout) :: dist_global ! global distribution object
1645+ type(distribution), intent(inout) :: dist_spin(2) ! per-spin objects
1646+
1647+ ! Local variables
1648+ integer :: color, base_rank
1649+ integer :: npGlobal, npPerSpin
1650+ integer, allocatable, dimension(:) :: global_ranks_in_world
1651+ integer, allocatable, dimension(:) :: ranks_in_world ! should be 'spatial_ranks...'
1652+ integer, allocatable, dimension(:,:) :: ranks_in_World_Spin
1653+ integer :: global_group, Spatial_group
1654+ integer :: i, ispin, ierr
1655+
1656+ call MPI_Comm_Size(global_comm, npGlobal, ierr)
1657+ ! This distribution could be created by the caller...
1658+ allocate(global_ranks_in_world(npGlobal))
1659+ global_ranks_in_world = (/ (i, i=0, npGlobal-1) /)
1660+ call newDistribution(dist_global,global_Comm,global_ranks_in_world, &
1661+ TYPE_BLOCK_CYCLIC,BlockSize,"global dist")
1662+ deallocate(global_ranks_in_world)
1663+ call MPI_Barrier(global_Comm,ierr)
1664+
1665+ ! Split the base communicator
1666+ call MPI_Comm_Rank(base_comm, base_rank, ierr)
1667+ ! "Row" communicator for independent operations on each spin
1668+ ! The name refers to "spatial" degrees of freedom.
1669+ color = mod(base_rank,n_spin) ! {0,1} for n_spin = 2, or {0} for n_spin = 1
1670+ call MPI_Comm_Split(base_comm, color, base_rank, Spatial_Comm, ierr)
1671+ ! "Column" communicator for spin reductions
1672+ color = base_rank/n_spin
1673+ call MPI_Comm_Split(base_comm, color, base_rank, Spin_Comm, ierr)
1674+
1675+ call MPI_Comm_Size(spatial_comm, npPerSpin, ierr) ! number of procs in each spin team
1676+
1677+ call MPI_Comm_Group(Spatial_Comm, Spatial_Group, Ierr)
1678+ allocate(ranks_in_world(npPerSpin))
1679+ call MPI_Comm_Group(global_comm, Global_Group, Ierr)
1680+ call MPI_Group_translate_ranks( Spatial_Group, npPerSpin, &
1681+ (/ (i,i=0,npPerSpin-1) /), &
1682+ Global_Group, ranks_in_world, ierr )
1683+
1684+ call MPI_Group_Free(Spatial_Group, ierr)
1685+ call MPI_Group_Free(Global_Group, ierr)
1686+
1687+ allocate (ranks_in_World_Spin(npPerSpin,n_spin))
1688+ ! We need everybody to have this information, as all nodes
1689+ ! are part of the global distribution side of the communication
1690+ ! (global_comm is the common-address space)
1691+
1692+ ! But note that this will tell only those in base_comm...
1693+ ! This routine might not be the end of the story for cascading splits
1694+ ! (k-points and spin)
1695+ ! We might need an extra 'broadcast' over the k-point 'column' comm.
1696+ call MPI_AllGather(ranks_in_world,npPerSpin,MPI_integer,&
1697+ Ranks_in_World_Spin(1,1),npPerSpin, &
1698+ MPI_integer,Spin_Comm,ierr)
1699+
1700+ ! Create distributions known to all nodes (again, those in base_comm)
1701+ do ispin = 1, n_spin
1702+ call newDistribution(dist_spin(ispin), global_Comm, &
1703+ Ranks_in_World_Spin(:,ispin), &
1704+ TYPE_BLOCK_CYCLIC, blockSize, "SPATIAL dist")
1705+ enddo
1706+ deallocate(ranks_in_world,Ranks_in_World_Spin)
1707+ call MPI_Barrier(global_Comm,ierr)
1708+
1709+end subroutine get_spin_comms_and_dists
1710+
1711+#endif /* SIESTA__ELSI */
1712+
1713+#if SIESTA__ELSI || SIESTA__PEXSI
1714+
1715+subroutine elsi_complex_solver(iscf, n_basis, n_basis_l, n_spin, nnz_l, numh, row_ptr, &
1716+ col_idx, qtot, temp, ham, ovlp, dm, edm, ef, ets, &
1717+ nkpnt, kpt_n, kpt, weight, kpt_comm, Get_EDM_Only)
1718+
1719+ use m_mpi_utils, only: globalize_sum
1720+ use parallel, only: BlockSize
1721+#ifdef MPI
1722+ use mpi_siesta
1723+#endif
1724+ use class_Distribution
1725+ use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix
1726+ use alloc
1727+ use m_mpi_utils, only: globalize_sum
1728+
1729+ implicit none
1730+
1731+ integer, intent(in) :: iscf ! SCF step counter
1732+ integer, intent(in) :: n_basis ! Global basis
1733+ integer, intent(in) :: n_basis_l ! Local basis
1734+ integer, intent(in) :: n_spin
1735+ integer, intent(in) :: nnz_l ! Local nonzero
1736+ integer, intent(in), target :: numh(n_basis_l)
1737+ integer, intent(in) :: row_ptr(n_basis_l)
1738+ integer, intent(in), target :: col_idx(nnz_l)
1739+ real(dp), intent(in) :: qtot
1740+ real(dp), intent(in) :: temp
1741+ complex(dp), intent(inout), target :: ham(nnz_l,n_spin)
1742+ complex(dp), intent(inout), target :: ovlp(nnz_l)
1743+ complex(dp), intent(out) :: dm(nnz_l,n_spin)
1744+ complex(dp), intent(out) :: edm(nnz_l,n_spin)
1745+ real(dp), intent(out) :: ef ! Fermi energy
1746+ real(dp), intent(out) :: ets ! Entropy/k, dimensionless
1747+ integer, intent(in) :: nkpnt ! number of k-points
1748+ integer, intent(in) :: kpt_n
1749+ real(dp), intent(in) :: kpt(3:)
1750+ real(dp), intent(in) :: weight
1751+ integer, intent(in) :: kpt_comm
1752+
1753+ integer :: ierr
1754+ integer :: n_state
1755+ integer :: nnz_g
1756+
1757+ real(dp) :: energy
1758+
1759+ integer, allocatable, dimension(:) :: row_ptr2
1760+
1761+ integer :: elsi_Spatial_comm, elsi_Spin_comm
1762+
1763+ type(distribution) :: dist_global
1764+ type(distribution) :: dist_spin(2)
1765+
1766+ type(aux_matrix) :: pkg_global, pkg_spin ! Packages for transfer
1767+
1768+ integer :: my_spin
1769+
1770+ integer :: my_no_l
1771+ integer :: my_nnz_l
1772+ integer :: my_nnz
1773+ integer, pointer :: my_row_ptr2(:) => null()
1774+ integer :: i, ih, ispin, spin_rank, global_rank
1775+ real(dp) :: ets_spin
1776+
1777+ integer, pointer :: my_col_idx(:)
1778+ complex(dp), pointer :: my_S(:)
1779+ complex(dp), pointer :: my_H(:)
1780+ complex(dp), pointer :: my_DM(:) => null()
1781+ complex(dp), pointer :: my_EDM(:) => null()
1782+
1783+ integer :: date_stamp
1784+
1785+ logical :: Get_EDM_Only
1786+
1787+ external :: timer
1788+
1789+#ifndef MPI
1790+ call die("This ELSI solver interface needs MPI")
1791+#endif
1792+
1793+ call timer("elsi-complex-solver", 1)
1794+
1795+ ! Initialization
1796+ if (iscf == 1) then
1797+
1798+ ! Get ELSI options
1799+ call elsi_get_opts()
1800+
1801+ ! Number of states to solve when calling an eigensolver
1802+ n_state = min(n_basis, n_basis/2+5)
1803+
1804+ ! Now we have all ingredients to initialize ELSI
1805+ call elsi_init(elsi_h, which_solver, MULTI_PROC, SIESTA_CSC, n_basis, &
1806+ qtot, n_state)
1807+
1808+ ! Output
1809+ call elsi_set_output(elsi_h, out_level)
1810+ call elsi_set_output_log(elsi_h, out_json)
1811+ call elsi_set_write_unit(elsi_h, 6)
1812+
1813+ ! Broadening
1814+ call elsi_set_mu_broaden_scheme(elsi_h, which_broad)
1815+ call elsi_set_mu_broaden_width(elsi_h, temp)
1816+ call elsi_set_mu_mp_order(elsi_h, mp_order)
1817+
1818+ ! Solver settings
1819+ call elsi_set_elpa_solver(elsi_h, elpa_flavor)
1820+
1821+ call elsi_set_omm_flavor(elsi_h, omm_flavor)
1822+ call elsi_set_omm_n_elpa(elsi_h, omm_n_elpa)
1823+ call elsi_set_omm_tol(elsi_h, omm_tol)
1824+
1825+ if (pexsi_tasks_per_pole /= ELSI_NOT_SET) then
1826+ call elsi_set_pexsi_np_per_pole(elsi_h, pexsi_tasks_per_pole)
1827+ end if
1828+
1829+ call elsi_set_pexsi_np_symbo(elsi_h, pexsi_tasks_symbolic)
1830+ call elsi_set_pexsi_temp(elsi_h, temp)
1831+! call elsi_set_pexsi_gap(elsi_h, gap)
1832+! call elsi_set_pexsi_delta_e(elsi_h, delta_e)
1833+ call elsi_set_pexsi_mu_min(elsi_h, -10.0_dp)
1834+ call elsi_set_pexsi_mu_max(elsi_h, 10.0_dp)
1835+
1836+ if (sips_n_slice /= ELSI_NOT_SET) then
1837+ call elsi_set_sips_n_slice(elsi_h, sips_n_slice)
1838+ end if
1839+
1840+ call elsi_set_sips_n_elpa(elsi_h, sips_n_elpa)
1841+ call elsi_set_sips_interval(elsi_h, -10.0_dp, 10.0_dp)
1842+
1843+ call elsi_set_ntpoly_method(elsi_h, ntpoly_method)
1844+ call elsi_set_ntpoly_filter(elsi_h, ntpoly_filter)
1845+ call elsi_set_ntpoly_tol(elsi_h, ntpoly_tol)
1846+
1847+ endif ! iscf == 1
1848+
1849+ !print *, global_rank, "| ", " Entering elsi_complex_solver"
1850+
1851+ if (n_spin == 1) then
1852+
1853+ ! Sparsity pattern
1854+ call globalize_sum(nnz_l, nnz_g, comm=kpt_comm)
1855+
1856+ allocate(row_ptr2(n_basis_l+1))
1857+ row_ptr2(1:n_basis_l) = row_ptr(1:n_basis_l)+1
1858+ row_ptr2(n_basis_l+1) = nnz_l+1
1859+
1860+ call elsi_set_csc(elsi_h, nnz_g, nnz_l, n_basis_l, col_idx, row_ptr2)
1861+ deallocate(row_ptr2)
1862+
1863+ call elsi_set_csc_blk(elsi_h, BlockSize)
1864+ call elsi_set_kpoint(elsi_h, nkpnt, kpt_n, weight)
1865+ call elsi_set_mpi(elsi_h, kpt_comm)
1866+ call elsi_set_mpi_global(elsi_h, elsi_global_comm)
1867+
1868+ else
1869+
1870+ call mpi_comm_rank( elsi_global_Comm, global_rank, ierr )
1871+
1872+ ! MPI logic for spin polarization
1873+
1874+ ! Split the communicator in spins and get distribution objects
1875+ ! for the data redistribution needed
1876+ ! Note that dist_spin is an array
1877+ call get_spin_comms_and_dists(kpt_comm,kpt_comm, & !! **** kpt_comm as global?
1878+ blocksize, n_spin, &
1879+ dist_global,dist_spin, elsi_spatial_comm, elsi_spin_comm)
1880+
1881+ ! Find out which spin team we are in, and tag the spin we work on
1882+ call mpi_comm_rank( elsi_Spin_Comm, spin_rank, ierr )
1883+ my_spin = spin_rank+1 ! {1,2}
1884+
1885+ !print *, global_rank, "| ", "spin ", my_spin, " After spin splitting"
1886+
1887+ ! This is done serially, each time filling one spin set
1888+ ! Note that **all processes** need to have the same pkg_global
1889+
1890+ do ispin = 1, n_spin
1891+
1892+ ! Load pkg_global data package
1893+ pkg_global%norbs = n_basis
1894+ pkg_global%no_l = n_basis_l
1895+ pkg_global%nnzl = nnz_l
1896+ pkg_global%numcols => numh
1897+ pkg_global%cols => col_idx
1898+
1899+ allocate(pkg_global%complex_vals(2))
1900+ ! Link the vals items to the appropriate arrays (no extra memory here)
1901+ pkg_global%complex_vals(1)%data => ovlp(:)
1902+ ! Note that we *cannot* say => ham(:,my_spin)
1903+ ! and avoid the sequential loop, as then half the processors will send
1904+ ! the information for 'spin up' and the other half the information for 'spin down',
1905+ ! which is *not* what we want.
1906+ pkg_global%complex_vals(2)%data => ham(:,ispin)
1907+
1908+ call timer("redist_orbs_fwd", 1)
1909+
1910+ ! We are doing the transfers sequentially. One spin team is
1911+ ! 'idle' (in the receiving side) in each pass, as the dist_spin(ispin) distribution
1912+ ! does not involve them.
1913+
1914+ call redistribute_spmatrix(n_basis,pkg_global,dist_global, &
1915+ pkg_spin,dist_spin(ispin),kpt_Comm)
1916+
1917+ call timer("redist_orbs_fwd", 2)
1918+
1919+ if (my_spin == ispin) then ! Each team gets their own data
1920+
1921+ !nrows = pkg_spin%norbs ! or simply 'norbs'
1922+ my_no_l = pkg_spin%no_l
1923+ my_nnz_l = pkg_spin%nnzl
1924+ call MPI_AllReduce(my_nnz_l,my_nnz,1,MPI_integer,MPI_sum,elsi_Spatial_Comm,ierr)
1925+ ! generate off-by-one row pointer
1926+ call re_alloc(my_row_ptr2,1,my_no_l+1,"my_row_ptr2","elsi_solver")
1927+ my_row_ptr2(1) = 1
1928+ do ih = 1,my_no_l
1929+ my_row_ptr2(ih+1) = my_row_ptr2(ih) + pkg_spin%numcols(ih)
1930+ enddo
1931+
1932+ my_col_idx => pkg_spin%cols
1933+ my_S => pkg_spin%complex_vals(1)%data
1934+ my_H => pkg_spin%complex_vals(2)%data
1935+
1936+ call re_alloc(my_DM,1,my_nnz_l,"my_DM","elsi_solver")
1937+ call re_alloc(my_EDM,1,my_nnz_l,"my_EDM","elsi_solver")
1938+ endif
1939+
1940+ ! Clean pkg_global
1941+ nullify(pkg_global%complex_vals(1)%data)
1942+ nullify(pkg_global%complex_vals(2)%data)
1943+ deallocate(pkg_global%complex_vals)
1944+ nullify(pkg_global%numcols)
1945+ nullify(pkg_global%cols)
1946+
1947+ enddo
1948+
1949+ !print *, global_rank, "| ", "spin ", my_spin, "Done spin transfers"
1950+
1951+ call elsi_set_csc(elsi_h, my_nnz, my_nnz_l, my_no_l, my_col_idx, my_row_ptr2)
1952+ call de_alloc(my_row_ptr2,"my_row_ptr2","elsi_solver")
1953+
1954+ call elsi_set_csc_blk(elsi_h, BlockSize)
1955+ call elsi_set_spin(elsi_h, n_spin, my_spin)
1956+ call elsi_set_kpoint(elsi_h, nkpnt, kpt_n, weight)
1957+ call elsi_set_mpi(elsi_h, elsi_Spatial_comm)
1958+ call elsi_set_mpi_global(elsi_h, elsi_global_comm)
1959+
1960+ endif ! n_spin
1961+
1962+ call timer("elsi-solver", 1)
1963+
1964+ !print *, global_rank, "| ", "About to call elsi_dm"
1965+ if (n_spin == 1) then
1966+ if (.not.Get_EDM_Only) then
1967+ call elsi_dm_complex_sparse(elsi_h, ham, ovlp, DM, energy)
1968+ call elsi_get_entropy(elsi_h, ets)
1969+ else
1970+ call elsi_get_edm_complex_sparse(elsi_h, EDM)
1971+ endif
1972+ else
1973+ ! Solve DM, and get (at every step for now) EDM, Fermi energy, and entropy
1974+ ! Energy is already summed over spins
1975+ if (.not.Get_EDM_Only) then
1976+ call elsi_dm_complex_sparse(elsi_h, my_H, my_S, my_DM, energy)
1977+ !... but we still need to sum the entropy over spins
1978+ call elsi_get_entropy(elsi_h, ets_spin)
1979+ else
1980+ call elsi_get_edm_complex_sparse(elsi_h, my_EDM)
1981+ endif
1982+#ifdef SIESTA__ELSI__OLD_SPIN_CONVENTION
1983+ ! NOTE**
1984+ ! For ELSI versions before 2018-08-17 we still need to sum the entropy over spins
1985+ ! ... but to figure out the version is not trivial, as the relevant routines changed...
1986+ call globalize_sum(ets_spin, ets, comm=elsi_Spin_comm)
1987+#else
1988+ ! If this gives an error, then your version of ELSI is old, and you should
1989+ ! be using the pre-processor symbol above
1990+ ! (This works because 'elsi_get_datestamp' was added just
1991+ ! around the same time as the spin-reduction-convention change) (Aug 17 vs Aug 21, 2018...)
1992+ ! If you are unlucky enough to have an ELSI version in between, get rid of it.
1993+ call elsi_get_datestamp(date_stamp)
1994+#endif
1995+ ! And over kpoints?? -- not necessary, ever
1996+ endif
1997+
1998+ call elsi_get_mu(elsi_h, ef)
1999+ ets = ets/temp
2000+
2001+ ! Ef, energy, and ets are known to all nodes
2002+
2003+ call timer("elsi-solver", 2)
2004+ !print *, global_rank, "| ", "Done elsi_dm"
2005+
2006+ if ( n_spin == 2) then
2007+ ! Now we need to redistribute back
2008+
2009+ do ispin = 1, n_spin
2010+
2011+ if (my_spin == ispin) then
2012+ ! Prepare pkg_spin to transfer the right spin information
2013+ ! The other fields (numcols, cols) are the same and are still there
2014+ ! Deallocate my_S and my_H
2015+ call de_alloc(pkg_spin%complex_vals(1)%data,"pkg_spin%vals(1)%data","elsi_solver")
2016+ call de_alloc(pkg_spin%complex_vals(2)%data,"pkg_spin%vals(2)%data","elsi_solver")
2017+
2018+ pkg_spin%complex_vals(1)%data => my_DM(1:my_nnz_l)
2019+ pkg_spin%complex_vals(2)%data => my_EDM(1:my_nnz_l)
2020+
2021+ endif
2022+
2023+ ! pkg_global is clean now
2024+ call timer("redist_orbs_bck", 1)
2025+ call redistribute_spmatrix(n_basis,pkg_spin,dist_spin(ispin) &
2026+ ,pkg_global,dist_global,kpt_Comm)
2027+ call timer("redist_orbs_bck", 2)
2028+
2029+ ! Clean pkg_spin
2030+ if (my_spin == ispin) then
2031+ ! Each team deallocates during "its" spin cycle
2032+ call de_alloc(my_DM, "my_DM", "elsi_solver")
2033+ call de_alloc(my_EDM,"my_EDM","elsi_solver")
2034+
2035+ nullify(pkg_spin%complex_vals(1)%data) ! formerly pointing to DM
2036+ nullify(pkg_spin%complex_vals(2)%data) ! formerly pointing to EDM
2037+ deallocate(pkg_spin%complex_vals)
2038+ ! allocated in the direct transfer
2039+ call de_alloc(pkg_spin%numcols,"pkg_spin%numcols","elsi_solver")
2040+ call de_alloc(pkg_spin%cols, "pkg_spin%cols", "elsi_solver")
2041+ endif
2042+
2043+
2044+ ! In future, pkg_global%vals(1,2) could be pointing to DM and EDM,
2045+ ! and the 'redistribute' routine check whether the vals arrays are
2046+ ! associated, to use them instead of allocating them.
2047+ DM(:,ispin) = pkg_global%complex_vals(1)%data(:)
2048+ EDM(:,ispin) = pkg_global%complex_vals(2)%data(:)
2049+ ! Check no_l
2050+ if (n_basis_l /= pkg_global%no_l) then
2051+ call die("Mismatch in no_l")
2052+ endif
2053+ ! Check listH
2054+ if (any(col_idx(:) /= pkg_global%cols(:))) then
2055+ call die("Mismatch in listH")
2056+ endif
2057+
2058+ ! Clean pkg_global
2059+ ! allocated by the transfer routine, but we did not actually
2060+ ! look at them
2061+ call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver")
2062+ call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver")
2063+
2064+ call de_alloc(pkg_global%complex_vals(1)%data,"pkg_global%vals(1)%data","elsi_solver")
2065+ call de_alloc(pkg_global%complex_vals(2)%data,"pkg_global%vals(2)%data","elsi_solver")
2066+ deallocate(pkg_global%complex_vals)
2067+
2068+ enddo
2069+
2070+ call MPI_Comm_Free(elsi_Spatial_comm, ierr)
2071+ call MPI_Comm_Free(elsi_Spin_comm, ierr)
2072+
2073+ else ! n_spin == 1
2074+
2075+ ! Nothing else to do
2076+ endif
2077+
2078+ call timer("elsi-complex-solver", 2)
2079+
2080+
2081+end subroutine elsi_complex_solver
2082+
2083+subroutine transpose(a,b)
2084+ real(dp), intent(in) :: a(:,:)
2085+ real(dp), allocatable, intent(out) :: b(:,:)
2086+
2087+ integer n, m, i, j
2088+
2089+ n = size(a,1)
2090+ m = size(a,2)
2091+ allocate(b(m,n))
2092+ do i = 1, n
2093+ do j = 1, m
2094+ b(j, i) = a(i, j)
2095+ enddo
2096+ enddo
2097+end subroutine transpose
2098+
2099+# endif
2100+
2101+end module m_elsi_interface
2102
2103=== modified file 'Src/m_ncdf_siesta.F90'
2104--- Src/m_ncdf_siesta.F90 2019-01-04 07:01:47 +0000
2105+++ Src/m_ncdf_siesta.F90 2019-01-16 11:00:58 +0000
2106@@ -51,6 +51,7 @@
2107 use siesta_options, only: SOLVE_DIAGON, SOLVE_ORDERN
2108 use siesta_options, only: SOLVE_MINIM, SOLVE_TRANSI
2109 use siesta_options, only: SOLVE_PEXSI
2110+ use siesta_options, only: SOLVE_ELSI
2111 use siesta_options, only: savehs
2112 use siesta_options, only: fixspin, total_spin
2113 use siesta_options, only: ia1, ia2, dx ! FC information
2114@@ -361,6 +362,8 @@
2115 dic = dic//('method'.kv.'transiesta')
2116 else if ( isolve == SOLVE_PEXSI ) then
2117 dic = dic//('method'.kv.'pexsi')
2118+ else if ( isolve == SOLVE_ELSI ) then
2119+ dic = dic//('method'.kv.'elsi')
2120 end if
2121
2122 ! Attributes are collective
2123
2124=== modified file 'Src/m_pexsi_dos.F90'
2125--- Src/m_pexsi_dos.F90 2016-08-04 09:03:32 +0000
2126+++ Src/m_pexsi_dos.F90 2019-01-16 11:00:58 +0000
2127@@ -99,7 +99,7 @@
2128 !
2129 ! Our global communicator is a duplicate of the passed communicator
2130 !
2131-call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr)
2132+call MPI_Comm_Dup(mpi_comm_dft, World_Comm, ierr)
2133 call mpi_comm_rank( World_Comm, mpirank, ierr )
2134
2135 call timer("pexsi_dos", 1)
2136
2137=== modified file 'Src/m_pexsi_driver.F90'
2138--- Src/m_pexsi_driver.F90 2016-08-04 09:03:32 +0000
2139+++ Src/m_pexsi_driver.F90 2019-01-16 11:00:58 +0000
2140@@ -123,7 +123,7 @@
2141 !
2142 ! Our global communicator is a duplicate of the passed communicator
2143 !
2144-call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr)
2145+call MPI_Comm_Dup(MPI_Comm_DFT, World_Comm, ierr)
2146 call mpi_comm_rank( World_Comm, mpirank, ierr )
2147
2148 ! NOTE: fdf calls will assign values to the whole processor set,
2149
2150=== modified file 'Src/m_pexsi_local_dos.F90'
2151--- Src/m_pexsi_local_dos.F90 2018-04-27 09:35:52 +0000
2152+++ Src/m_pexsi_local_dos.F90 2019-01-16 11:00:58 +0000
2153@@ -144,7 +144,7 @@
2154 !
2155 ! Our global communicator is a duplicate of the passed communicator
2156 !
2157- call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr)
2158+ call MPI_Comm_Dup(MPI_Comm_DFT, World_Comm, ierr)
2159 call mpi_comm_rank( World_Comm, mpirank, ierr )
2160
2161 call timer("pexsi-ldos", 1)
2162
2163=== modified file 'Src/m_redist_spmatrix.F90'
2164--- Src/m_redist_spmatrix.F90 2016-09-14 09:35:16 +0000
2165+++ Src/m_redist_spmatrix.F90 2019-01-16 11:00:58 +0000
2166@@ -1,18 +1,44 @@
2167-
2168-! --- Tangled code
2169+!
2170+! This module implements functionality to change the MPI parallel
2171+! distribution of a sparse matrix stored in the modified
2172+! block-compressed form used in Siesta.
2173+!
2174+! The immediate goal is to exchange matrices between Siesta and the
2175+! PEXSI library. Siesta uses a "block-cyclic" distribution over
2176+! N_Siesta processors, and PEXSI uses a simple "greedy" blocked
2177+! distribution on each pole-group, with npPerPole processors. The
2178+! code works in principle with any distribution (subject to some
2179+! conditions spelled out below). The source and target process groups
2180+! are arbitrary, and can overlap. In fact, in SIESTA-PEXSI they
2181+! *will* overlap, as otherwise we would need extra processors for the
2182+! Siesta operations. This overlap forces the code to re-invent much
2183+! of the logic involved in MPI intercommunicators, which cannot be
2184+! created for non-disjoint groups.
2185+
2186+! Written by Alberto Garcia
2187+
2188+! --- Tangled code (using Org mode in emacs)
2189+! Alas, the Org version is no longer maintained.
2190+!
2191 module m_redist_spmatrix
2192-#ifdef SIESTA__PEXSI
2193+#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI)
2194 implicit none
2195+ integer, parameter, private :: dp = selected_real_kind(10,100)
2196+
2197+ ! Some auxiliary derived types
2198+
2199 type, public :: comm_t
2200 integer :: src, dst, i1, i2, nitems
2201 end type comm_t
2202-
2203- integer, parameter, private :: dp = selected_real_kind(10,100)
2204-
2205+
2206 type, public :: dp_pointer
2207 ! boxed array pointer type
2208 real(dp), pointer :: data(:) => null()
2209 end type dp_pointer
2210+ type, public :: complex_dp_pointer
2211+ ! boxed array pointer type
2212+ complex(dp), pointer :: data(:) => null()
2213+ end type complex_dp_pointer
2214
2215 type, public :: aux_matrix
2216 integer :: norbs = -1
2217@@ -20,14 +46,21 @@
2218 integer :: nnzl = -1
2219 integer, pointer :: numcols(:) => null()
2220 integer, pointer :: cols(:) => null()
2221- ! array of 1D pointers
2222+ ! arrays of 1D pointers
2223 type(dp_pointer), dimension(:), pointer :: vals(:) => null()
2224+ type(complex_dp_pointer), dimension(:), pointer :: complex_vals(:) => null()
2225 end type aux_matrix
2226+
2227+
2228 public :: redistribute_spmatrix
2229+
2230 CONTAINS
2231+
2232 subroutine redistribute_spmatrix(norbs,m1,dist1,m2,dist2,bridge_comm)
2233
2234- use mpi
2235+ use mpi ! Note that this is fine, as we do not use any Siesta-specific
2236+ ! parameter, such as a modified mpi_comm_world
2237+
2238 use class_Distribution
2239 use alloc, only: re_alloc, de_alloc
2240
2241@@ -47,10 +80,11 @@
2242 logical :: proc_in_set1, proc_in_set2
2243 integer :: ierr
2244
2245- integer :: i, io, g1, g2, j, nvals
2246+ integer :: i, io, g1, g2, j, nvals, nvals_complex
2247 integer :: comparison, n1, n2, c1, c2
2248 integer, parameter :: dp = selected_real_kind(10,100)
2249 real(dp), dimension(:), pointer :: data1 => null(), data2 => null()
2250+ complex(dp), dimension(:), pointer :: cdata1 => null(), cdata2 => null()
2251
2252 integer, allocatable :: ranks1(:), ranks2(:)
2253
2254@@ -109,10 +143,10 @@
2255 proc_in_set1 = (myrank1 /= MPI_UNDEFINED)
2256 proc_in_set2 = (myrank2 /= MPI_UNDEFINED)
2257
2258- if (proc_in_set1 .or. proc_in_set2) then
2259- print "(a,3i6,2l2)", "world_rank, rank1, rank2, ing1?, ing2?", myid, &
2260- myrank1, myrank2, proc_in_set1, proc_in_set2
2261- endif
2262+! if (proc_in_set1 .or. proc_in_set2) then
2263+! print "(a,3i6,2l2)", "world_rank, rank1, rank2, ing1?, ing2?", myid, &
2264+! myrank1, myrank2, proc_in_set1, proc_in_set2
2265+! endif
2266
2267 ! Figure out the communication needs
2268 call analyze_comms()
2269@@ -130,10 +164,10 @@
2270 call re_alloc(m2%numcols,1,m2%no_l,"m2%numcols","redistribute_spmatrix")
2271 endif
2272
2273- if (myid == 0) print *, "About to transfer numcols..."
2274+! if (myid == 0) print *, "About to transfer numcols..."
2275 call do_transfers_int(comms,m1%numcols,m2%numcols, &
2276 g1,g2,bridge_comm)
2277- if (myid == 0) print *, "Transferred numcols."
2278+! if (myid == 0) print *, "Transferred numcols."
2279
2280 ! We need to tell the processes in set 2 how many
2281 ! "vals" to expect.
2282@@ -143,12 +177,18 @@
2283 else
2284 nvals = 0
2285 endif
2286+ if (associated(m1%complex_vals)) then
2287+ nvals_complex = size(m1%complex_vals)
2288+ else
2289+ nvals_complex = 0
2290+ endif
2291 endif
2292 ! Now do a broadcast within bridge_comm, using as root one
2293 ! process in the first set. Let's say the one with rank 0
2294 ! in g1, the first in the set, which will have rank=ranks1(1)
2295 ! in bridge_comm
2296 call MPI_Bcast(nvals,1,MPI_Integer,ranks1(1),bridge_comm,ierr)
2297+ call MPI_Bcast(nvals_complex,1,MPI_Integer,ranks1(1),bridge_comm,ierr)
2298
2299 ! Now we can figure out how many non-zeros there are
2300 if (proc_in_set2) then
2301@@ -163,6 +203,14 @@
2302 enddo
2303 endif
2304
2305+ if (nvals_complex > 0) then
2306+ allocate(m2%complex_vals(nvals_complex))
2307+ do j=1,nvals_complex
2308+ call re_alloc(m2%complex_vals(j)%data,1,m2%nnzl, &
2309+ "m2%complex_vals(j)%data","redistribute_spmatrix")
2310+ enddo
2311+ endif
2312+
2313 endif
2314
2315 ! Generate a new comms-structure with new start/count indexes
2316@@ -202,12 +250,12 @@
2317 !!$ endif
2318 !!$ enddo
2319
2320- if (myid == 0) print *, "About to transfer cols..."
2321+! if (myid == 0) print *, "About to transfer cols..."
2322 ! Transfer the cols arrays
2323 call do_transfers_int(commsnnz,m1%cols,m2%cols, &
2324 g1, g2, bridge_comm)
2325
2326- if (myid == 0) print *, "About to transfer values..."
2327+! if (myid == 0) print *, "About to transfer values..."
2328 ! Transfer the values arrays
2329 do j=1, nvals
2330 if (proc_in_set1) data1 => m1%vals(j)%data
2331@@ -216,7 +264,17 @@
2332 g1,g2,bridge_comm)
2333 enddo
2334 nullify(data1,data2)
2335- if (myid == 0) print *, "Done transfers."
2336+
2337+! if (myid == 0) print *, "About to transfer complex values..."
2338+ ! Transfer the values arrays
2339+ do j=1, nvals_complex
2340+ if (proc_in_set1) cdata1 => m1%complex_vals(j)%data
2341+ if (proc_in_set2) cdata2 => m2%complex_vals(j)%data
2342+ call do_transfers_complex_dp(commsnnz,cdata1,cdata2, &
2343+ g1,g2,bridge_comm)
2344+ enddo
2345+ nullify(cdata1,cdata2)
2346+! if (myid == 0) print *, "Done transfers."
2347
2348 deallocate(commsnnz)
2349 deallocate(comms)
2350@@ -312,9 +370,9 @@
2351 if (myid == 0 .and. comms_not_printed) then
2352 do i = 1, ncomms
2353 c => comms(i)
2354- write(6,"(a,i5,a,2i5,2i7,i5)") &
2355- "comm: ", i, " src, dst, i1, i2, n:", &
2356- c%src, c%dst, c%i1, c%i2, c%nitems
2357+! write(6,"(a,i5,a,2i5,2i7,i5)") &
2358+! "comm: ", i, " src, dst, i1, i2, n:", &
2359+! c%src, c%dst, c%i1, c%i2, c%nitems
2360 enddo
2361 comms_not_printed = .false.
2362 endif
2363@@ -543,6 +601,114 @@
2364 deallocate(local_reqR, local_reqS, statuses)
2365
2366 end subroutine do_transfers_dp
2367+ !--------------------------------------------------
2368+ subroutine do_transfers_complex_dp(comms,data1,data2,g1,g2,bridge_comm)
2369+
2370+ use mpi
2371+ integer, parameter :: dp = selected_real_kind(10,100)
2372+
2373+ type(comm_t), intent(in), target :: comms(:)
2374+ complex(dp), dimension(:), pointer :: data1
2375+ complex(dp), dimension(:), pointer :: data2
2376+ integer, intent(in) :: g1
2377+ integer, intent(in) :: g2
2378+ integer, intent(in) :: bridge_comm
2379+
2380+ integer :: basegroup, nsize1, nsize2, ierr
2381+ integer, allocatable :: comm_rank1(:), comm_rank2(:)
2382+
2383+
2384+ integer :: ncomms
2385+ integer :: i
2386+ integer :: nrecvs_local, nsends_local
2387+ integer, allocatable :: statuses(:,:), local_reqR(:), local_reqS(:)
2388+ integer :: src_in_comm, dst_in_comm
2389+ integer :: myrank1, myrank2, myid
2390+ type(comm_t), pointer :: c
2391+
2392+ call MPI_Comm_Rank( bridge_comm, myid, ierr )
2393+ ! print *, "Entering transfer_complex_dp"
2394+ ! print *, "rank, Associated data1: ", myid, associated(data1)
2395+ ! print *, "rank, Associated data2: ", myid, associated(data2)
2396+
2397+ ! Find the rank correspondences, in case
2398+ ! there is implicit renumbering at the time of group creation
2399+
2400+ call MPI_Comm_group( bridge_comm, basegroup, ierr )
2401+ call MPI_Group_Size( g1, nsize1, ierr )
2402+ call MPI_Group_Size( g2, nsize2, ierr )
2403+ allocate(comm_rank1(0:nsize1-1))
2404+ call MPI_Group_translate_ranks( g1, nsize1, (/ (i,i=0,nsize1-1) /), &
2405+ basegroup, comm_rank1, ierr )
2406+ ! print "(a,10i3)", "Ranks of g1 in base group:", comm_rank1
2407+ allocate(comm_rank2(0:nsize2-1))
2408+ call MPI_Group_translate_ranks( g2, nsize2, (/ (i,i=0,nsize2-1) /), &
2409+ basegroup, comm_rank2, ierr )
2410+ ! print "(a,10i3)", "Ranks of g2 in base group:", comm_rank2
2411+
2412+ call mpi_group_rank(g1,myrank1,ierr)
2413+ call mpi_group_rank(g2,myrank2,ierr)
2414+
2415+ ! Do the actual transfers.
2416+ ! This version with non-blocking communications
2417+
2418+ ncomms = size(comms)
2419+
2420+ ! Some bookkeeping for the requests
2421+ nrecvs_local = 0
2422+ nsends_local = 0
2423+ do i=1,ncomms
2424+ c => comms(i)
2425+ if (myrank2 == c%dst) then
2426+ nrecvs_local = nrecvs_local + 1
2427+ endif
2428+ if (myrank1 == c%src) then
2429+ nsends_local = nsends_local + 1
2430+ endif
2431+ enddo
2432+ allocate(local_reqR(nrecvs_local))
2433+ allocate(local_reqS(nsends_local))
2434+ allocate(statuses(mpi_status_size,nrecvs_local))
2435+
2436+ ! First, post the receives
2437+ nrecvs_local = 0
2438+ do i=1,ncomms
2439+ c => comms(i)
2440+ if (myrank2 == c%dst) then
2441+ nrecvs_local = nrecvs_local + 1
2442+ src_in_comm = comm_rank1(c%src)
2443+ call MPI_irecv(data2(c%i2),c%nitems,MPI_Double_Complex,src_in_comm, &
2444+ i,bridge_comm,local_reqR(nrecvs_local),ierr)
2445+ endif
2446+ enddo
2447+
2448+ ! Post the sends
2449+ nsends_local = 0
2450+ do i=1,ncomms
2451+ c => comms(i)
2452+ if (myrank1 == c%src) then
2453+ nsends_local = nsends_local + 1
2454+ dst_in_comm = comm_rank2(c%dst)
2455+ call MPI_isend(data1(c%i1),c%nitems,MPI_Double_Complex,dst_in_comm, &
2456+ i,bridge_comm,local_reqS(nsends_local),ierr)
2457+ endif
2458+ enddo
2459+
2460+ ! A former loop of waits can be substituted by a "waitall",
2461+ ! with every processor keeping track of the actual number of
2462+ ! requests in which it is involved.
2463+
2464+ ! Should we wait also on the sends?
2465+
2466+ call MPI_waitall(nrecvs_local, local_reqR, statuses, ierr)
2467+
2468+
2469+ ! This barrier is needed, I think
2470+ call MPI_Barrier(bridge_comm,ierr)
2471+
2472+ deallocate(local_reqR, local_reqS, statuses)
2473+
2474+ end subroutine do_transfers_complex_dp
2475 #endif
2476 end module m_redist_spmatrix
2477 ! --- End of tangled code
2478
2479=== modified file 'Src/parallel.F'
2480--- Src/parallel.F 2017-07-14 12:17:28 +0000
2481+++ Src/parallel.F 2019-01-16 11:00:58 +0000
2482@@ -1,5 +1,5 @@
2483 ! ---
2484-! Copyright (C) 1996-2016 The SIESTA group
2485+! Copyright- (C) 1996-2016 The SIESTA group
2486 ! This file is distributed under the terms of the
2487 ! GNU General Public License: see COPYING in the top directory
2488 ! or http://www.gnu.org/copyleft/gpl.txt .
2489
2490=== modified file 'Src/post_scf_work.F'
2491--- Src/post_scf_work.F 2018-09-12 13:49:21 +0000
2492+++ Src/post_scf_work.F 2019-01-16 11:00:58 +0000
2493@@ -27,12 +27,13 @@
2494 use class_Fstack_Pair_Geometry_dSpData2D, only: new, max_size
2495 use class_Fstack_Pair_Geometry_dSpData2D, only: push
2496
2497- use parallel, only: ionode
2498+ use parallel, only : IOnode, SIESTA_worker
2499+
2500 use atomlist, only: lasto, rmaxo, datm, indxuo, no_s, no_u,
2501 & iphorb, no_l, qtot, qtots
2502 use m_energies
2503 use neighbour, only : maxna=>maxnna ! For plcharge...
2504- use m_spin, only : nspin, h_spin_dim
2505+ use m_spin, only : nspin, h_spin_dim, spin
2506 use m_spin, only : spinor_dim
2507 use m_diagon, only : diagon
2508 use m_dminim, only : dminim
2509@@ -41,6 +42,9 @@
2510 use m_compute_dm, only : PreviousCallDiagon
2511 use m_eo
2512 use kpoint_scf_m, only: kpoint_scf, gamma_scf
2513+#ifdef SIESTA__ELSI
2514+ use m_elsi_interface, only: elsi_getdm
2515+#endif
2516 implicit none
2517
2518 ! MD-step, SCF-step
2519@@ -51,10 +55,37 @@
2520 type(Pair_Geometry_dSpData2D) :: pair
2521 type(Geometry) :: geom
2522
2523+ real(dp), allocatable :: Dscf_dummy(:,:)
2524+ real(dp) :: ef_dummy
2525+ real(dp) :: Entropy_dummy
2526+
2527 call timer( 'PostSCF', 1 )
2528
2529 ! If converged, make one last iteration to find forces and stress
2530
2531+! If we use the ELSI interface, the energy-density
2532+! matrix is not calculated inside the SCF step, and
2533+! so this must be done now
2534+
2535+#ifdef SIESTA__ELSI
2536+ if (isolve .eq. SOLVE_ELSI) then
2537+ allocate(Dscf_dummy(maxnh,spin%DM))
2538+ ! Note that we do not care about the content of H and S
2539+ ! Only that the appropriate elsi_h handle is still active
2540+ ! and allows us to get the appropriate EDM
2541+ call elsi_getdm(iscf, no_s, spin%spinor,
2542+ & no_l, maxnh, no_u,
2543+ & numh, listhptr, listh,
2544+ & H, S, qtot, temp,
2545+ & xijo,
2546+ & kpoint_scf%N, kpoint_scf%k, kpoint_scf%w,
2547+ & eo, qo, Dscf_dummy, Escf, ef_dummy, Entropy_dummy,
2548+ & occtol, neigwanted, Get_EDM_Only=.true.)
2549+ deallocate(Dscf_dummy)
2550+ endif
2551+ if (.not. SIESTA_worker) RETURN
2552+#endif
2553+
2554 ! If we use the minimization routine, the energy-density
2555 ! matrix is not calculated inside the SCF step, and
2556 ! so this must be done now
2557
2558=== modified file 'Src/read_options.F90'
2559--- Src/read_options.F90 2018-12-02 00:15:31 +0000
2560+++ Src/read_options.F90 2019-01-16 11:00:58 +0000
2561@@ -76,6 +76,7 @@
2562 ! 4 = PEXSI
2563 ! 5 = (Matrix write)
2564 ! 6 = CheSS
2565+ ! 7 = ELSI
2566 ! real*8 temp : Temperature for Fermi smearing (Ry)
2567 ! logical fixspin : Fix the spin of the system?
2568 ! real*8 total_spin : Total spin of the system
2569@@ -706,6 +707,16 @@
2570 #else
2571 call die("PEXSI solver is not compiled in. Use -DSIESTA__PEXSI")
2572 #endif
2573+ else if (leqi(method,"elsi")) then
2574+#ifdef SIESTA__ELSI
2575+ isolve = SOLVE_ELSI
2576+ if (ionode) then
2577+ call add_citation("ELSI PAPER***")
2578+ write(*,3) 'redata: Method of Calculation', 'ELSI'
2579+ endif
2580+#else
2581+ call die("ELSI solver is not compiled in. Use -DSIESTA__ELSI")
2582+#endif
2583
2584 #ifdef SIESTA__CHESS
2585 else if (leqi(method,'chess')) then
2586
2587=== modified file 'Src/siesta.F'
2588--- Src/siesta.F 2018-05-30 09:15:30 +0000
2589+++ Src/siesta.F 2019-01-16 11:00:58 +0000
2590@@ -21,7 +21,7 @@
2591 use parallel, only: SIESTA_worker ! whether part of Siesta's communicator
2592 use parallel, only: ionode
2593 #ifdef MPI
2594- use mpi_siesta, only: true_mpi_comm_world
2595+ use mpi_siesta, only: mpi_comm_dft ! Includes Solver nodes
2596 #endif
2597 use m_mpi_utils, only: broadcast
2598
2599@@ -35,12 +35,12 @@
2600 integer :: istep
2601 logical :: relaxd
2602
2603-! Notes for PEXSI operation
2604+! Notes for PEXSI and ELSI operation (extra procs for Solver)
2605 !
2606-! A subset of nodes carries out non-PEXSI Siesta operations
2607+! A subset of nodes carries out non-Solver Siesta operations
2608 ! (i.e., setting up H, moving atoms, diagonalizing...).
2609 ! These are tagged as "SIESTA_worker" (admittedly, a bad name)
2610-! All nodes are involved in the PEXSI electronic-structure solver,
2611+! All nodes are involved in the (PEXSI or ELSI) electronic-structure solver,
2612 ! and in the new LocalDOS computation based on selected inversion.
2613 !
2614 ! 'siesta_init', 'siesta_forces', and 'siesta_analysis' need to
2615@@ -66,11 +66,11 @@
2616 C Begin of coordinate relaxation iteration
2617 relaxd = .false.
2618
2619-#ifdef SIESTA__PEXSI
2620+#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI)
2621 ! Broadcast relevant things for program logic
2622 ! These were set in siesta_options, called only by "SIESTA_workers".
2623- call broadcast(inicoor,comm=true_MPI_Comm_World)
2624- call broadcast(fincoor,comm=true_MPI_Comm_World)
2625+ call broadcast(inicoor,comm=mpi_comm_dft)
2626+ call broadcast(fincoor,comm=mpi_comm_dft)
2627 #endif
2628 istep = inicoor
2629 DO WHILE ((istep.le.fincoor) .AND. (.not. relaxd))
2630@@ -82,8 +82,8 @@
2631 end if
2632
2633 if (SIESTA_worker) call siesta_move( istep, relaxd )
2634-#ifdef SIESTA__PEXSI
2635- call broadcast(relaxd,comm=true_MPI_Comm_World)
2636+#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI)
2637+ call broadcast(relaxd,comm=mpi_comm_dft)
2638 #endif
2639 if (.not. relaxd) then
2640 istep = istep + 1
2641
2642=== modified file 'Src/siesta_forces.F90'
2643--- Src/siesta_forces.F90 2018-09-24 07:34:04 +0000
2644+++ Src/siesta_forces.F90 2019-01-16 11:00:58 +0000
2645@@ -84,6 +84,9 @@
2646 #ifdef SIESTA__PEXSI
2647 use m_pexsi, only: pexsi_finalize_scfloop
2648 #endif
2649+#ifdef SIESTA__ELSI
2650+ use m_elsi_interface, only: elsi_finalize_scfloop
2651+#endif
2652 use m_check_walltime
2653
2654 use m_energies, only: DE_NEGF
2655@@ -140,10 +143,10 @@
2656 call write_debug( ' PRE siesta_forces' )
2657 #endif
2658
2659-#ifdef SIESTA__PEXSI
2660+#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI)
2661 ! Broadcast relevant things for program logic
2662 ! These were set in read_options, called only by "SIESTA_workers".
2663- call broadcast(nscf, comm=true_MPI_Comm_World)
2664+ call broadcast(nscf, comm=mpi_comm_dft)
2665 #endif
2666
2667 if ( SIESTA_worker ) then
2668@@ -484,9 +487,9 @@
2669
2670 end if
2671
2672-#ifdef SIESTA__PEXSI
2673- call broadcast(iscf, comm=true_MPI_Comm_World)
2674- call broadcast(SCFconverged, comm=true_MPI_Comm_World)
2675+#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI)
2676+ call broadcast(iscf, comm=mpi_comm_dft)
2677+ call broadcast(SCFconverged, comm=mpi_comm_dft)
2678 #endif
2679
2680 ! Exit if converged
2681@@ -500,9 +503,15 @@
2682 end if
2683 #endif
2684
2685+<<<<<<< TREE
2686 if ( .not. SIESTA_worker ) return
2687
2688 call end_of_cycle_save_operations(SCFconverged)
2689+=======
2690+ if ( SIESTA_worker ) then
2691+!===
2692+ call end_of_cycle_save_operations()
2693+>>>>>>> MERGE-SOURCE
2694
2695 if ( .not. SCFconverged ) then
2696 if ( SCFMustConverge ) then
2697@@ -538,11 +547,18 @@
2698 ! Clean-up here to limit memory usage
2699 call mixers_scf_history_init( )
2700
2701+!===
2702+ endif ! Siesta_Worker
2703+
2704 ! End of standard SCF loop.
2705 ! Do one more pass to compute forces and stresses
2706
2707 ! Note that this call will no longer overwrite H while computing the
2708 ! final energies, forces and stresses...
2709+
2710+ ! If we want to preserve the "Siesta_Worker" subset implementation for ELSI,
2711+ ! this block needs to be executed by everybody
2712+ ! as it contains a call to the ELSI interface to get the EDM
2713
2714 if ( fdf_get("compute-forces",.true.) ) then
2715 call post_scf_work( istep, iscf , SCFconverged )
2716@@ -550,6 +566,15 @@
2717 if (ionode) call memory_snapshot("after post_scf_work")
2718 #endif
2719 end if
2720+
2721+#ifdef SIESTA__ELSI
2722+ ! Recall that there could be an extra call after the scf loop to get the EDM
2723+ if ( isolve == SOLVE_ELSI ) then
2724+ call elsi_finalize_scfloop()
2725+ end if
2726+#endif
2727+
2728+ if (.not. Siesta_Worker) RETURN
2729
2730 ! ... so H at this point is the latest generator of the DM, except
2731 ! if mixing H beyond self-consistency or terminating the scf loop
2732
2733=== modified file 'Src/siesta_init.F'
2734--- Src/siesta_init.F 2018-11-27 14:28:55 +0000
2735+++ Src/siesta_init.F 2019-01-16 11:00:58 +0000
2736@@ -267,8 +267,11 @@
2737 call reinit( sname )
2738
2739 #ifdef MPI
2740+
2741+#ifdef SIESTA__PEXSI
2742 ! Optionally, restrict the number of processors that SIESTA uses, out
2743- ! of the total pool
2744+ ! of the total pool (but only for PEXSI, not yet for other solvers)
2745+
2746 npSIESTA = fdf_get("MPI.Nprocs.SIESTA",Nodes)
2747 call MPI_Comm_Group(MPI_Comm_World, World_Group, MPIerror)
2748 call MPI_Group_incl(World_Group, npSIESTA,
2749@@ -278,8 +281,16 @@
2750 $ SIESTA_Comm, MPIerror)
2751
2752 SIESTA_worker = (Node < npSIESTA)
2753-
2754- ! Swap communicator
2755+#else
2756+ SIESTA_Comm = MPI_Comm_World
2757+ SIESTA_worker = .true.
2758+#endif
2759+ ! Swap communicators
2760+ ! This is needed in case that the whole program is run with
2761+ ! a subset of the global processes. In that case, the DFT instance
2762+ ! communicator (a better name than 'true world' communicator)
2763+ ! is at this point MPI_COMM_WORLD
2764+ MPI_COMM_DFT = MPI_COMM_WORLD
2765 ! This is needed since MPI_COMM_WORLD is used implicitly by
2766 ! Siesta for all operations.
2767 MPI_COMM_WORLD = SIESTA_Comm
2768@@ -287,7 +298,7 @@
2769 call MPI_Comm_Rank( MPI_Comm_World, Node, MPIerror )
2770 call MPI_Comm_Size( MPI_Comm_World, Nodes, MPIerror )
2771 endif
2772-#else
2773+#else /* not MPI */
2774 SIESTA_worker = .true.
2775 Node = 0
2776 Nodes = 1
2777@@ -586,7 +597,14 @@
2778 ! auxiliary supercell for the calculation of matrix elements.
2779 call setup_kpoint_scf( ucell )
2780 not_using_auxcell = gamma_scf
2781-
2782+#ifdef SIESTA__ELSI
2783+ if (isolve == SOLVE_ELSI) then
2784+ if (mod(Nodes,kpoint_scf%N*spin%spinor) /=0) then
2785+ call die("The number of MPI processes " //
2786+ $ "must be a multiple of nk*nspin")
2787+ endif
2788+ endif
2789+#endif
2790 ! Call initialisation of PDOS here since we need to check if
2791 ! the auxiliary supercell is needed for a non-gamma calculation
2792 call init_projected_DOS( ucell )
2793
2794=== modified file 'Src/siesta_options.F90'
2795--- Src/siesta_options.F90 2018-09-12 13:49:21 +0000
2796+++ Src/siesta_options.F90 2019-01-16 11:00:58 +0000
2797@@ -246,6 +246,7 @@
2798 integer, parameter :: SOLVE_PEXSI = 4
2799 integer, parameter :: MATRIX_WRITE = 5
2800 integer, parameter :: SOLVE_CHESS = 6
2801+ integer, parameter :: SOLVE_ELSI = 7
2802
2803 #ifdef SIESTA__FLOOK
2804 ! LUA-handle
2805
2806=== modified file 'Src/siesta_tddft.F90'
2807--- Src/siesta_tddft.F90 2018-06-26 17:09:38 +0000
2808+++ Src/siesta_tddft.F90 2019-01-16 11:00:58 +0000
2809@@ -61,7 +61,7 @@
2810 #ifdef SIESTA__PEXSI
2811 ! Broadcast relevant things for program logic
2812 ! These were set in read_options, called only by "SIESTA_workers".
2813- call broadcast(nscf,comm=true_MPI_Comm_World)
2814+ call broadcast(nscf,comm=MPI_Comm_DFT)
2815 #endif
2816
2817 if ( SIESTA_worker ) then
2818
2819=== modified file 'Src/state_init.F'
2820--- Src/state_init.F 2018-10-23 09:46:30 +0000
2821+++ Src/state_init.F 2019-01-16 11:00:58 +0000
2822@@ -811,6 +811,10 @@
2823 #endif
2824 call timer( 'state_init', 2 )
2825
2826+! Call BLACS setup here if needed (all solvers except PEXSI)
2827+! Setup S in 2d Scalapack distribution if needed (ELSI ELPA solver)
2828+! Initialize ELSI
2829+
2830 END subroutine state_init
2831
2832 subroutine check_cohp()
2833
2834=== modified file 'Src/write_subs.F'
2835--- Src/write_subs.F 2018-06-04 14:08:20 +0000
2836+++ Src/write_subs.F 2019-01-16 11:00:58 +0000
2837@@ -441,6 +441,7 @@
2838 write(6,"(/a,f14.6,/)") 'siesta: Eharris(eV) = ', Eharrs/eV
2839 ! No need for further cml output
2840 elseif ((isolve==SOLVE_DIAGON) .or. (isolve==SOLVE_PEXSI)
2841+ . .or. (isolve==SOLVE_ELSI)
2842 . .or. (isolve==SOLVE_CHESS)
2843 . .or. (isolve==SOLVE_TRANSI)) then
2844 if (cml_p) then
2845
2846=== added directory 'Tests/si-quantum-dot'
2847=== added file 'Tests/si-quantum-dot/coords.Si987H372.fdf'
2848--- Tests/si-quantum-dot/coords.Si987H372.fdf 1970-01-01 00:00:00 +0000
2849+++ Tests/si-quantum-dot/coords.Si987H372.fdf 2019-01-16 11:00:58 +0000
2850@@ -0,0 +1,1368 @@
2851+number-of-atoms 1359
2852+number-of-species 2
2853+#
2854+# Si987H372 from CSIRO Silicon Quantum Dot dataset
2855+# https://doi.org/10.4225/08/5721BB609EDB0
2856+#
2857+atomic-coordinates-format Ang
2858+%block atomic-coordinates-and-atomic-species
2859+-12.27486761 -6.80192861 -4.06859976 1 # Si
2860+-12.25680064 -9.54338724 -1.35303719 1 # Si
2861+-15.00302593 -6.81988895 -1.33325565 1 # Si
2862+-13.61801556 -8.18242131 -0.02610591 1 # Si
2863+-12.27960089 -6.82966552 1.35920843 1 # Si
2864+-12.28371979 -9.54137501 4.12509126 1 # Si
2865+-13.62454044 -8.34100436 5.61139268 1 # Si
2866+-12.30649445 -6.84800235 6.84800235 1 # Si
2867+-12.25680064 -1.35303719 -9.54338724 1 # Si
2868+-12.27486761 -4.06859976 -6.80192861 1 # Si
2869+-15.00302593 -1.33325565 -6.81988895 1 # Si
2870+-13.66848041 -2.72110232 -5.45900176 1 # Si
2871+-13.66848041 -5.45900176 -2.72110232 1 # Si
2872+-16.37023166 -2.73136988 -2.73136988 1 # Si
2873+-15.01032944 -4.07390228 -4.07390228 1 # Si
2874+-12.30121531 -1.35797308 -4.09496434 1 # Si
2875+-12.30121531 -4.09496434 -1.35797308 1 # Si
2876+-15.04397328 -1.36373167 -1.36373167 1 # Si
2877+-16.34134070 -5.63195305 0.16771494 1 # Si
2878+-13.68153993 -2.73058729 0.00300340 1 # Si
2879+-13.61156377 -5.42510828 2.69385812 1 # Si
2880+-16.34634363 -2.90187977 2.90187977 1 # Si
2881+-15.03413728 -4.10489826 1.37590571 1 # Si
2882+-12.30671934 -1.37011664 1.37011664 1 # Si
2883+-12.27225867 -4.09402380 4.09402380 1 # Si
2884+-15.03413728 -1.37590571 4.10489826 1 # Si
2885+-13.61156377 -2.69385812 5.42510828 1 # Si
2886+-13.62454044 -5.61139268 8.34100436 1 # Si
2887+-12.27960089 -1.35920843 6.82966552 1 # Si
2888+-12.28371979 -4.12509126 9.54137501 1 # Si
2889+-13.61801556 -0.02610591 -8.18242131 1 # Si
2890+-12.28371979 4.12509126 -9.54137501 1 # Si
2891+-12.27960089 1.35920843 -6.82966552 1 # Si
2892+-16.34134070 0.16771494 -5.63195305 1 # Si
2893+-13.61156377 2.69385812 -5.42510828 1 # Si
2894+-13.68153993 0.00300340 -2.73058729 1 # Si
2895+-16.34634363 2.90187977 -2.90187977 1 # Si
2896+-15.03413728 1.37590571 -4.10489826 1 # Si
2897+-12.27225867 4.09402380 -4.09402380 1 # Si
2898+-12.30671934 1.37011664 -1.37011664 1 # Si
2899+-15.03413728 4.10489826 -1.37590571 1 # Si
2900+-16.37821316 0.00000000 0.00000000 1 # Si
2901+-13.68153993 2.73058729 -0.00300340 1 # Si
2902+-13.68153993 -0.00300340 2.73058729 1 # Si
2903+-16.37023166 2.73136988 2.73136988 1 # Si
2904+-15.04397328 1.36373167 1.36373167 1 # Si
2905+-12.30121531 4.09496434 1.35797308 1 # Si
2906+-12.30121531 1.35797308 4.09496434 1 # Si
2907+-15.01032944 4.07390228 4.07390228 1 # Si
2908+-16.34134070 -0.16771494 5.63195305 1 # Si
2909+-13.66848041 2.72110232 5.45900176 1 # Si
2910+-13.61801556 0.02610591 8.18242131 1 # Si
2911+-15.00302593 1.33325565 6.81988895 1 # Si
2912+-12.27486761 4.06859976 6.80192861 1 # Si
2913+-12.25680064 1.35303719 9.54338724 1 # Si
2914+-13.62454044 5.61139268 -8.34100436 1 # Si
2915+-12.30649445 6.84800235 -6.84800235 1 # Si
2916+-13.62454044 8.34100436 -5.61139268 1 # Si
2917+-13.61156377 5.42510828 -2.69385812 1 # Si
2918+-12.28371979 9.54137501 -4.12509126 1 # Si
2919+-12.27960089 6.82966552 -1.35920843 1 # Si
2920+-16.34134070 5.63195305 -0.16771494 1 # Si
2921+-13.61801556 8.18242131 0.02610591 1 # Si
2922+-13.66848041 5.45900176 2.72110232 1 # Si
2923+-15.00302593 6.81988895 1.33325565 1 # Si
2924+-12.25680064 9.54338724 1.35303719 1 # Si
2925+-12.27486761 6.80192861 4.06859976 1 # Si
2926+-6.80192861 -12.27486761 -4.06859976 1 # Si
2927+-6.81988895 -15.00302593 -1.33325565 1 # Si
2928+-9.54338724 -12.25680064 -1.35303719 1 # Si
2929+-8.18242131 -13.61801556 -0.02610591 1 # Si
2930+-6.82966552 -12.27960089 1.35920843 1 # Si
2931+-9.54137501 -12.28371979 4.12509126 1 # Si
2932+-8.34100436 -13.62454044 5.61139268 1 # Si
2933+-6.84800235 -12.30649445 6.84800235 1 # Si
2934+-6.80203106 -6.80203106 -9.53854106 1 # Si
2935+-6.80203106 -9.53854106 -6.80203106 1 # Si
2936+-9.53854106 -6.80203106 -6.80203106 1 # Si
2937+-8.18684496 -8.18684496 -5.46061302 1 # Si
2938+-8.18958251 -10.91859845 -2.73303284 1 # Si
2939+-10.91859845 -8.18958251 -2.73303284 1 # Si
2940+-9.53373688 -9.53373688 -4.08178398 1 # Si
2941+-6.82463094 -6.82463094 -4.09496380 1 # Si
2942+-6.82852418 -9.55456223 -1.36492084 1 # Si
2943+-9.55456223 -6.82852418 -1.36492084 1 # Si
2944+-10.90149235 -10.90149235 -0.01304838 1 # Si
2945+-8.19611273 -8.19611273 0.01202455 1 # Si
2946+-8.19260795 -10.93410102 2.74437054 1 # Si
2947+-10.93410102 -8.19260795 2.74437054 1 # Si
2948+-9.56265344 -9.56265344 1.38318968 1 # Si
2949+-6.82894278 -6.82894278 1.37205363 1 # Si
2950+-6.82705083 -9.56489675 4.10298675 1 # Si
2951+-9.56489675 -6.82705083 4.10298675 1 # Si
2952+-10.90291341 -10.90291341 5.43284542 1 # Si
2953+-8.19345007 -8.19345007 5.46312108 1 # Si
2954+-8.17530342 -10.89255734 8.17530342 1 # Si
2955+-10.89255734 -8.17530342 8.17530342 1 # Si
2956+-9.55790920 -9.55790920 6.82271744 1 # Si
2957+-6.82700831 -6.82700831 6.82700831 1 # Si
2958+-6.82271744 -9.55790920 9.55790920 1 # Si
2959+-9.55790920 -6.82271744 9.55790920 1 # Si
2960+-8.17530342 -8.17530342 10.89255734 1 # Si
2961+-6.84800235 -6.84800235 12.30649445 1 # Si
2962+-6.81988895 -1.33325565 -15.00302593 1 # Si
2963+-6.80192861 -4.06859976 -12.27486761 1 # Si
2964+-9.54338724 -1.35303719 -12.25680064 1 # Si
2965+-8.18958251 -2.73303284 -10.91859845 1 # Si
2966+-8.18684496 -5.46061302 -8.18684496 1 # Si
2967+-10.91859845 -2.73303284 -8.18958251 1 # Si
2968+-9.53373688 -4.08178398 -9.53373688 1 # Si
2969+-6.82852418 -1.36492084 -9.55456223 1 # Si
2970+-6.82463094 -4.09496380 -6.82463094 1 # Si
2971+-9.55456223 -1.36492084 -6.82852418 1 # Si
2972+-10.92814020 -5.45537137 -5.45537137 1 # Si
2973+-8.19145655 -2.72890018 -5.45886981 1 # Si
2974+-8.19145655 -5.45886981 -2.72890018 1 # Si
2975+-10.93548204 -2.72665137 -2.72665137 1 # Si
2976+-9.56309566 -4.09195916 -4.09195916 1 # Si
2977+-6.82665093 -1.36387577 -4.09489509 1 # Si
2978+-6.82665093 -4.09489509 -1.36387577 1 # Si
2979+-9.56805296 -1.36256401 -1.36256401 1 # Si
2980+-10.91922202 -5.46070202 -0.00117430 1 # Si
2981+-8.19459074 -2.72767707 -0.00218763 1 # Si
2982+-8.19802839 -5.46221414 2.73481248 1 # Si
2983+-10.91829649 -2.72813440 2.72813440 1 # Si
2984+-9.55550648 -4.09408870 1.36300571 1 # Si
2985+-6.82823155 -1.36467215 1.36467215 1 # Si
2986+-6.82965269 -4.09758805 4.09758805 1 # Si
2987+-9.55550648 -1.36300571 4.09408870 1 # Si
2988+-10.94234231 -5.47569502 5.47569502 1 # Si
2989+-8.19802839 -2.73481248 5.46221414 1 # Si
2990+-8.19345007 -5.46312108 8.19345007 1 # Si
2991+-10.93410102 -2.74437054 8.19260795 1 # Si
2992+-9.56489675 -4.10298675 6.82705083 1 # Si
2993+-6.82894278 -1.37205363 6.82894278 1 # Si
2994+-6.82705083 -4.10298675 9.56489675 1 # Si
2995+-9.56265344 -1.38318968 9.56265344 1 # Si
2996+-10.90291341 -5.43284542 10.90291341 1 # Si
2997+-8.19260795 -2.74437054 10.93410102 1 # Si
2998+-8.34100436 -5.61139268 13.62454044 1 # Si
2999+-9.54137501 -4.12509126 12.28371979 1 # Si
3000+-6.82966552 -1.35920843 12.27960089 1 # Si
3001+-8.18242131 -0.02610591 -13.61801556 1 # Si
3002+-6.82966552 1.35920843 -12.27960089 1 # Si
3003+-9.54137501 4.12509126 -12.28371979 1 # Si
3004+-10.90149235 -0.01304838 -10.90149235 1 # Si
3005+-8.19260795 2.74437054 -10.93410102 1 # Si
3006+-8.19611273 0.01202455 -8.19611273 1 # Si
3007+-10.93410102 2.74437054 -8.19260795 1 # Si
3008+-9.56265344 1.38318968 -9.56265344 1 # Si
3009+-6.82705083 4.10298675 -9.56489675 1 # Si
3010+-6.82894278 1.37205363 -6.82894278 1 # Si
3011+-9.56489675 4.10298675 -6.82705083 1 # Si
3012+-10.91922202 -0.00117430 -5.46070202 1 # Si
3013+-8.19802839 2.73481248 -5.46221414 1 # Si
3014+-8.19459074 -0.00218763 -2.72767707 1 # Si
3015+-10.91829649 2.72813440 -2.72813440 1 # Si
3016+-9.55550648 1.36300571 -4.09408870 1 # Si
3017+-6.82965269 4.09758805 -4.09758805 1 # Si
3018+-6.82823155 1.36467215 -1.36467215 1 # Si
3019+-9.55550648 4.09408870 -1.36300571 1 # Si
3020+-10.94349563 0.00000000 -0.00000000 1 # Si
3021+-8.19459074 2.72767707 0.00218763 1 # Si
3022+-8.19459074 0.00218763 2.72767707 1 # Si
3023+-10.93548204 2.72665137 2.72665137 1 # Si
3024+-9.56805296 1.36256401 1.36256401 1 # Si
3025+-6.82665093 4.09489509 1.36387577 1 # Si
3026+-6.82665093 1.36387577 4.09489509 1 # Si
3027+-9.56309566 4.09195916 4.09195916 1 # Si
3028+-10.91922202 0.00117430 5.46070202 1 # Si
3029+-8.19145655 2.72890018 5.45886981 1 # Si
3030+-8.19611273 -0.01202455 8.19611273 1 # Si
3031+-10.91859845 2.73303284 8.18958251 1 # Si
3032+-9.55456223 1.36492084 6.82852418 1 # Si
3033+-6.82463094 4.09496380 6.82463094 1 # Si
3034+-6.82852418 1.36492084 9.55456223 1 # Si
3035+-9.53373688 4.08178398 9.53373688 1 # Si
3036+-10.90149235 0.01304838 10.90149235 1 # Si
3037+-8.18958251 2.73303284 10.91859845 1 # Si
3038+-8.18242131 0.02610591 13.61801556 1 # Si
3039+-9.54338724 1.35303719 12.25680064 1 # Si
3040+-6.80192861 4.06859976 12.27486761 1 # Si
3041+-6.81988895 1.33325565 15.00302593 1 # Si
3042+-8.34100436 5.61139268 -13.62454044 1 # Si
3043+-6.84800235 6.84800235 -12.30649445 1 # Si
3044+-10.90291341 5.43284542 -10.90291341 1 # Si
3045+-8.17530342 8.17530342 -10.89255734 1 # Si
3046+-8.19345007 5.46312108 -8.19345007 1 # Si
3047+-10.89255734 8.17530342 -8.17530342 1 # Si
3048+-9.55790920 6.82271744 -9.55790920 1 # Si
3049+-6.82271744 9.55790920 -9.55790920 1 # Si
3050+-6.82700831 6.82700831 -6.82700831 1 # Si
3051+-9.55790920 9.55790920 -6.82271744 1 # Si
3052+-10.94234231 5.47569502 -5.47569502 1 # Si
3053+-8.19345007 8.19345007 -5.46312108 1 # Si
3054+-8.19802839 5.46221414 -2.73481248 1 # Si
3055+-10.93410102 8.19260795 -2.74437054 1 # Si
3056+-9.56489675 6.82705083 -4.10298675 1 # Si
3057+-6.82705083 9.56489675 -4.10298675 1 # Si
3058+-6.82894278 6.82894278 -1.37205363 1 # Si
3059+-9.56265344 9.56265344 -1.38318968 1 # Si
3060+-10.91922202 5.46070202 0.00117430 1 # Si
3061+-8.19611273 8.19611273 -0.01202455 1 # Si
3062+-8.19145655 5.45886981 2.72890018 1 # Si
3063+-10.91859845 8.18958251 2.73303284 1 # Si
3064+-9.55456223 6.82852418 1.36492084 1 # Si
3065+-6.82852418 9.55456223 1.36492084 1 # Si
3066+-6.82463094 6.82463094 4.09496380 1 # Si
3067+-9.53373688 9.53373688 4.08178398 1 # Si
3068+-10.92814020 5.45537137 5.45537137 1 # Si
3069+-8.18684496 8.18684496 5.46061302 1 # Si
3070+-8.18684496 5.46061302 8.18684496 1 # Si
3071+-9.53854106 6.80203106 6.80203106 1 # Si
3072+-6.80203106 9.53854106 6.80203106 1 # Si
3073+-6.80203106 6.80203106 9.53854106 1 # Si
3074+-8.17530342 10.89255734 -8.17530342 1 # Si
3075+-6.84800235 12.30649445 -6.84800235 1 # Si
3076+-10.90291341 10.90291341 -5.43284542 1 # Si
3077+-8.34100436 13.62454044 -5.61139268 1 # Si
3078+-8.19260795 10.93410102 -2.74437054 1 # Si
3079+-9.54137501 12.28371979 -4.12509126 1 # Si
3080+-6.82966552 12.27960089 -1.35920843 1 # Si
3081+-10.90149235 10.90149235 0.01304838 1 # Si
3082+-8.18242131 13.61801556 0.02610591 1 # Si
3083+-8.18958251 10.91859845 2.73303284 1 # Si
3084+-9.54338724 12.25680064 1.35303719 1 # Si
3085+-6.81988895 15.00302593 1.33325565 1 # Si
3086+-6.80192861 12.27486761 4.06859976 1 # Si
3087+-1.35303719 -12.25680064 -9.54338724 1 # Si
3088+-1.33325565 -15.00302593 -6.81988895 1 # Si
3089+-4.06859976 -12.27486761 -6.80192861 1 # Si
3090+-2.72110232 -13.66848041 -5.45900176 1 # Si
3091+-2.73136988 -16.37023166 -2.73136988 1 # Si
3092+-5.45900176 -13.66848041 -2.72110232 1 # Si
3093+-4.07390228 -15.01032944 -4.07390228 1 # Si
3094+-1.35797308 -12.30121531 -4.09496434 1 # Si
3095+-1.36373167 -15.04397328 -1.36373167 1 # Si
3096+-4.09496434 -12.30121531 -1.35797308 1 # Si
3097+-5.63195305 -16.34134070 0.16771494 1 # Si
3098+-2.73058729 -13.68153993 0.00300340 1 # Si
3099+-2.90187977 -16.34634363 2.90187977 1 # Si
3100+-5.42510828 -13.61156377 2.69385812 1 # Si
3101+-4.10489826 -15.03413728 1.37590571 1 # Si
3102+-1.37011664 -12.30671934 1.37011664 1 # Si
3103+-1.37590571 -15.03413728 4.10489826 1 # Si
3104+-4.09402380 -12.27225867 4.09402380 1 # Si
3105+-2.69385812 -13.61156377 5.42510828 1 # Si
3106+-5.61139268 -13.62454044 8.34100436 1 # Si
3107+-1.35920843 -12.27960089 6.82966552 1 # Si
3108+-4.12509126 -12.28371979 9.54137501 1 # Si
3109+-1.33325565 -6.81988895 -15.00302593 1 # Si
3110+-1.35303719 -9.54338724 -12.25680064 1 # Si
3111+-4.06859976 -6.80192861 -12.27486761 1 # Si
3112+-2.73303284 -8.18958251 -10.91859845 1 # Si
3113+-2.73303284 -10.91859845 -8.18958251 1 # Si
3114+-5.46061302 -8.18684496 -8.18684496 1 # Si
3115+-4.08178398 -9.53373688 -9.53373688 1 # Si
3116+-1.36492084 -6.82852418 -9.55456223 1 # Si
3117+-1.36492084 -9.55456223 -6.82852418 1 # Si
3118+-4.09496380 -6.82463094 -6.82463094 1 # Si
3119+-5.45537137 -10.92814020 -5.45537137 1 # Si
3120+-2.72890018 -8.19145655 -5.45886981 1 # Si
3121+-2.72665137 -10.93548204 -2.72665137 1 # Si
3122+-5.45886981 -8.19145655 -2.72890018 1 # Si
3123+-4.09195916 -9.56309566 -4.09195916 1 # Si
3124+-1.36387577 -6.82665093 -4.09489509 1 # Si
3125+-1.36256401 -9.56805296 -1.36256401 1 # Si
3126+-4.09489509 -6.82665093 -1.36387577 1 # Si
3127+-5.46070202 -10.91922202 -0.00117430 1 # Si
3128+-2.72767707 -8.19459074 -0.00218763 1 # Si
3129+-2.72813440 -10.91829649 2.72813440 1 # Si
3130+-5.46221414 -8.19802839 2.73481248 1 # Si
3131+-4.09408870 -9.55550648 1.36300571 1 # Si
3132+-1.36467215 -6.82823155 1.36467215 1 # Si
3133+-1.36300571 -9.55550648 4.09408870 1 # Si
3134+-4.09758805 -6.82965269 4.09758805 1 # Si
3135+-5.47569502 -10.94234231 5.47569502 1 # Si
3136+-2.73481248 -8.19802839 5.46221414 1 # Si
3137+-2.74437054 -10.93410102 8.19260795 1 # Si
3138+-5.46312108 -8.19345007 8.19345007 1 # Si
3139+-4.10298675 -9.56489675 6.82705083 1 # Si
3140+-1.37205363 -6.82894278 6.82894278 1 # Si
3141+-1.38318968 -9.56265344 9.56265344 1 # Si
3142+-4.10298675 -6.82705083 9.56489675 1 # Si
3143+-5.43284542 -10.90291341 10.90291341 1 # Si
3144+-2.74437054 -8.19260795 10.93410102 1 # Si
3145+-5.61139268 -8.34100436 13.62454044 1 # Si
3146+-4.12509126 -9.54137501 12.28371979 1 # Si
3147+-1.35920843 -6.82966552 12.27960089 1 # Si
3148+-2.73136988 -2.73136988 -16.37023166 1 # Si
3149+-2.72110232 -5.45900176 -13.66848041 1 # Si
3150+-5.45900176 -2.72110232 -13.66848041 1 # Si
3151+-4.07390228 -4.07390228 -15.01032944 1 # Si
3152+-1.36373167 -1.36373167 -15.04397328 1 # Si
3153+-1.35797308 -4.09496434 -12.30121531 1 # Si
3154+-4.09496434 -1.35797308 -12.30121531 1 # Si
3155+-5.45537137 -5.45537137 -10.92814020 1 # Si
3156+-2.72665137 -2.72665137 -10.93548204 1 # Si
3157+-2.72890018 -5.45886981 -8.19145655 1 # Si
3158+-5.45886981 -2.72890018 -8.19145655 1 # Si
3159+-4.09195916 -4.09195916 -9.56309566 1 # Si
3160+-1.36256401 -1.36256401 -9.56805296 1 # Si
3161+-1.36387577 -4.09489509 -6.82665093 1 # Si
3162+-4.09489509 -1.36387577 -6.82665093 1 # Si
3163+-5.45846237 -5.45846237 -5.45846237 1 # Si
3164+-2.72992539 -2.72992539 -5.46104379 1 # Si
3165+-2.72992539 -5.46104379 -2.72992539 1 # Si
3166+-5.46104379 -2.72992539 -2.72992539 1 # Si
3167+-4.09449230 -4.09449230 -4.09449230 1 # Si
3168+-1.36499559 -1.36499559 -4.09571959 1 # Si
3169+-1.36499559 -4.09571959 -1.36499559 1 # Si
3170+-4.09571959 -1.36499559 -1.36499559 1 # Si
3171+-5.46373779 -5.46373779 0.00319897 1 # Si
3172+-2.73149619 -2.73149619 0.00070150 1 # Si
3173+-2.73166810 -5.46412728 2.73166810 1 # Si
3174+-5.46412728 -2.73166810 2.73166810 1 # Si
3175+-4.09725659 -4.09725659 1.36658639 1 # Si
3176+-1.36585704 -1.36585704 1.36585704 1 # Si
3177+-1.36658639 -4.09725659 4.09725659 1 # Si
3178+-4.09725659 -1.36658639 4.09725659 1 # Si
3179+-5.46236779 -5.46236779 5.46236779 1 # Si
3180+-2.73166810 -2.73166810 5.46412728 1 # Si
3181+-2.73481248 -5.46221414 8.19802839 1 # Si
3182+-5.46221414 -2.73481248 8.19802839 1 # Si
3183+-4.09758805 -4.09758805 6.82965269 1 # Si
3184+-1.36467215 -1.36467215 6.82823155 1 # Si
3185+-1.36300571 -4.09408870 9.55550648 1 # Si
3186+-4.09408870 -1.36300571 9.55550648 1 # Si
3187+-5.47569502 -5.47569502 10.94234231 1 # Si
3188+-2.72813440 -2.72813440 10.91829649 1 # Si
3189+-2.69385812 -5.42510828 13.61156377 1 # Si
3190+-5.42510828 -2.69385812 13.61156377 1 # Si
3191+-4.09402380 -4.09402380 12.27225867 1 # Si
3192+-1.37011664 -1.37011664 12.30671934 1 # Si
3193+-1.37590571 -4.10489826 15.03413728 1 # Si
3194+-4.10489826 -1.37590571 15.03413728 1 # Si
3195+-2.90187977 -2.90187977 16.34634363 1 # Si
3196+-5.63195305 0.16771494 -16.34134070 1 # Si
3197+-2.90187977 2.90187977 -16.34634363 1 # Si
3198+-2.73058729 0.00300340 -13.68153993 1 # Si
3199+-5.42510828 2.69385812 -13.61156377 1 # Si
3200+-4.10489826 1.37590571 -15.03413728 1 # Si
3201+-1.37590571 4.10489826 -15.03413728 1 # Si
3202+-1.37011664 1.37011664 -12.30671934 1 # Si
3203+-4.09402380 4.09402380 -12.27225867 1 # Si
3204+-5.46070202 -0.00117430 -10.91922202 1 # Si
3205+-2.72813440 2.72813440 -10.91829649 1 # Si
3206+-2.72767707 -0.00218763 -8.19459074 1 # Si
3207+-5.46221414 2.73481248 -8.19802839 1 # Si
3208+-4.09408870 1.36300571 -9.55550648 1 # Si
3209+-1.36300571 4.09408870 -9.55550648 1 # Si
3210+-1.36467215 1.36467215 -6.82823155 1 # Si
3211+-4.09758805 4.09758805 -6.82965269 1 # Si
3212+-5.46373779 0.00319897 -5.46373779 1 # Si
3213+-2.73166810 2.73166810 -5.46412728 1 # Si
3214+-2.73149619 0.00070150 -2.73149619 1 # Si
3215+-5.46412728 2.73166810 -2.73166810 1 # Si
3216+-4.09725659 1.36658639 -4.09725659 1 # Si
3217+-1.36658639 4.09725659 -4.09725659 1 # Si
3218+-1.36585704 1.36585704 -1.36585704 1 # Si
3219+-4.09725659 4.09725659 -1.36658639 1 # Si
3220+-5.46151579 -0.00000000 0.00000000 1 # Si
3221+-2.73149619 2.73149619 -0.00070150 1 # Si
3222+-2.73149619 -0.00070150 2.73149619 1 # Si
3223+-5.46104379 2.72992539 2.72992539 1 # Si
3224+-4.09571959 1.36499559 1.36499559 1 # Si
3225+-1.36499559 4.09571959 1.36499559 1 # Si
3226+-1.36499559 1.36499559 4.09571959 1 # Si
3227+-4.09449230 4.09449230 4.09449230 1 # Si
3228+-5.46373779 -0.00319897 5.46373779 1 # Si
3229+-2.72992539 2.72992539 5.46104379 1 # Si
3230+-2.72767707 0.00218763 8.19459074 1 # Si
3231+-5.45886981 2.72890018 8.19145655 1 # Si
3232+-4.09489509 1.36387577 6.82665093 1 # Si
3233+-1.36387577 4.09489509 6.82665093 1 # Si
3234+-1.36256401 1.36256401 9.56805296 1 # Si
3235+-4.09195916 4.09195916 9.56309566 1 # Si
3236+-5.46070202 0.00117430 10.91922202 1 # Si
3237+-2.72665137 2.72665137 10.93548204 1 # Si
3238+-2.73058729 -0.00300340 13.68153993 1 # Si
3239+-5.45900176 2.72110232 13.66848041 1 # Si
3240+-4.09496434 1.35797308 12.30121531 1 # Si
3241+-1.35797308 4.09496434 12.30121531 1 # Si
3242+-1.36373167 1.36373167 15.04397328 1 # Si
3243+-4.07390228 4.07390228 15.01032944 1 # Si
3244+-5.63195305 -0.16771494 16.34134070 1 # Si
3245+-2.73136988 2.73136988 16.37023166 1 # Si
3246+-2.69385812 5.42510828 -13.61156377 1 # Si
3247+-5.61139268 8.34100436 -13.62454044 1 # Si
3248+-1.35920843 6.82966552 -12.27960089 1 # Si
3249+-4.12509126 9.54137501 -12.28371979 1 # Si
3250+-5.47569502 5.47569502 -10.94234231 1 # Si
3251+-2.74437054 8.19260795 -10.93410102 1 # Si
3252+-2.73481248 5.46221414 -8.19802839 1 # Si
3253+-5.46312108 8.19345007 -8.19345007 1 # Si
3254+-4.10298675 6.82705083 -9.56489675 1 # Si
3255+-1.38318968 9.56265344 -9.56265344 1 # Si
3256+-1.37205363 6.82894278 -6.82894278 1 # Si
3257+-4.10298675 9.56489675 -6.82705083 1 # Si
3258+-5.46236779 5.46236779 -5.46236779 1 # Si
3259+-2.73481248 8.19802839 -5.46221414 1 # Si
3260+-2.73166810 5.46412728 -2.73166810 1 # Si
3261+-5.46221414 8.19802839 -2.73481248 1 # Si
3262+-4.09758805 6.82965269 -4.09758805 1 # Si
3263+-1.36300571 9.55550648 -4.09408870 1 # Si
3264+-1.36467215 6.82823155 -1.36467215 1 # Si
3265+-4.09408870 9.55550648 -1.36300571 1 # Si
3266+-5.46373779 5.46373779 -0.00319897 1 # Si
3267+-2.72767707 8.19459074 0.00218763 1 # Si
3268+-2.72992539 5.46104379 2.72992539 1 # Si
3269+-5.45886981 8.19145655 2.72890018 1 # Si
3270+-4.09489509 6.82665093 1.36387577 1 # Si
3271+-1.36256401 9.56805296 1.36256401 1 # Si
3272+-1.36387577 6.82665093 4.09489509 1 # Si
3273+-4.09195916 9.56309566 4.09195916 1 # Si
3274+-5.45846237 5.45846237 5.45846237 1 # Si
3275+-2.72890018 8.19145655 5.45886981 1 # Si
3276+-2.72890018 5.45886981 8.19145655 1 # Si
3277+-5.46061302 8.18684496 8.18684496 1 # Si
3278+-4.09496380 6.82463094 6.82463094 1 # Si
3279+-1.36492084 9.55456223 6.82852418 1 # Si
3280+-1.36492084 6.82852418 9.55456223 1 # Si
3281+-4.08178398 9.53373688 9.53373688 1 # Si
3282+-5.45537137 5.45537137 10.92814020 1 # Si
3283+-2.73303284 8.18958251 10.91859845 1 # Si
3284+-2.72110232 5.45900176 13.66848041 1 # Si
3285+-4.06859976 6.80192861 12.27486761 1 # Si
3286+-1.35303719 9.54338724 12.25680064 1 # Si
3287+-1.33325565 6.81988895 15.00302593 1 # Si
3288+-5.43284542 10.90291341 -10.90291341 1 # Si
3289+-2.74437054 10.93410102 -8.19260795 1 # Si
3290+-5.61139268 13.62454044 -8.34100436 1 # Si
3291+-4.12509126 12.28371979 -9.54137501 1 # Si
3292+-1.35920843 12.27960089 -6.82966552 1 # Si
3293+-5.47569502 10.94234231 -5.47569502 1 # Si
3294+-2.69385812 13.61156377 -5.42510828 1 # Si
3295+-2.72813440 10.91829649 -2.72813440 1 # Si
3296+-5.42510828 13.61156377 -2.69385812 1 # Si
3297+-4.09402380 12.27225867 -4.09402380 1 # Si
3298+-1.37590571 15.03413728 -4.10489826 1 # Si
3299+-1.37011664 12.30671934 -1.37011664 1 # Si
3300+-4.10489826 15.03413728 -1.37590571 1 # Si
3301+-5.46070202 10.91922202 0.00117430 1 # Si
3302+-2.73058729 13.68153993 -0.00300340 1 # Si
3303+-2.72665137 10.93548204 2.72665137 1 # Si
3304+-5.45900176 13.66848041 2.72110232 1 # Si
3305+-4.09496434 12.30121531 1.35797308 1 # Si
3306+-1.36373167 15.04397328 1.36373167 1 # Si
3307+-1.35797308 12.30121531 4.09496434 1 # Si
3308+-4.07390228 15.01032944 4.07390228 1 # Si
3309+-5.45537137 10.92814020 5.45537137 1 # Si
3310+-2.72110232 13.66848041 5.45900176 1 # Si
3311+-2.73303284 10.91859845 8.18958251 1 # Si
3312+-4.06859976 12.27486761 6.80192861 1 # Si
3313+-1.33325565 15.00302593 6.81988895 1 # Si
3314+-1.35303719 12.25680064 9.54338724 1 # Si
3315+-2.90187977 16.34634363 -2.90187977 1 # Si
3316+-5.63195305 16.34134070 -0.16771494 1 # Si
3317+-2.73136988 16.37023166 2.73136988 1 # Si
3318+-0.02610591 -13.61801556 -8.18242131 1 # Si
3319+4.12509126 -12.28371979 -9.54137501 1 # Si
3320+1.35920843 -12.27960089 -6.82966552 1 # Si
3321+0.16771494 -16.34134070 -5.63195305 1 # Si
3322+2.69385812 -13.61156377 -5.42510828 1 # Si
3323+2.90187977 -16.34634363 -2.90187977 1 # Si
3324+0.00300340 -13.68153993 -2.73058729 1 # Si
3325+1.37590571 -15.03413728 -4.10489826 1 # Si
3326+4.09402380 -12.27225867 -4.09402380 1 # Si
3327+4.10489826 -15.03413728 -1.37590571 1 # Si
3328+1.37011664 -12.30671934 -1.37011664 1 # Si
3329+-0.00000000 -16.37821316 -0.00000000 1 # Si
3330+2.73058729 -13.68153993 -0.00300340 1 # Si
3331+2.73136988 -16.37023166 2.73136988 1 # Si
3332+-0.00300340 -13.68153993 2.73058729 1 # Si
3333+1.36373167 -15.04397328 1.36373167 1 # Si
3334+4.09496434 -12.30121531 1.35797308 1 # Si
3335+4.07390228 -15.01032944 4.07390228 1 # Si
3336+1.35797308 -12.30121531 4.09496434 1 # Si
3337+-0.16771494 -16.34134070 5.63195305 1 # Si
3338+2.72110232 -13.66848041 5.45900176 1 # Si
3339+0.02610591 -13.61801556 8.18242131 1 # Si
3340+1.33325565 -15.00302593 6.81988895 1 # Si
3341+4.06859976 -12.27486761 6.80192861 1 # Si
3342+1.35303719 -12.25680064 9.54338724 1 # Si
3343+-0.02610591 -8.18242131 -13.61801556 1 # Si
3344+4.12509126 -9.54137501 -12.28371979 1 # Si
3345+1.35920843 -6.82966552 -12.27960089 1 # Si
3346+-0.01304838 -10.90149235 -10.90149235 1 # Si
3347+2.74437054 -8.19260795 -10.93410102 1 # Si
3348+2.74437054 -10.93410102 -8.19260795 1 # Si
3349+0.01202455 -8.19611273 -8.19611273 1 # Si
3350+1.38318968 -9.56265344 -9.56265344 1 # Si
3351+4.10298675 -6.82705083 -9.56489675 1 # Si
3352+4.10298675 -9.56489675 -6.82705083 1 # Si
3353+1.37205363 -6.82894278 -6.82894278 1 # Si
3354+-0.00117430 -10.91922202 -5.46070202 1 # Si
3355+2.73481248 -8.19802839 -5.46221414 1 # Si
3356+2.72813440 -10.91829649 -2.72813440 1 # Si
3357+-0.00218763 -8.19459074 -2.72767707 1 # Si
3358+1.36300571 -9.55550648 -4.09408870 1 # Si
3359+4.09758805 -6.82965269 -4.09758805 1 # Si
3360+4.09408870 -9.55550648 -1.36300571 1 # Si
3361+1.36467215 -6.82823155 -1.36467215 1 # Si
3362+0.00000000 -10.94349563 0.00000000 1 # Si
3363+2.72767707 -8.19459074 0.00218763 1 # Si
3364+2.72665137 -10.93548204 2.72665137 1 # Si
3365+0.00218763 -8.19459074 2.72767707 1 # Si
3366+1.36256401 -9.56805296 1.36256401 1 # Si
3367+4.09489509 -6.82665093 1.36387577 1 # Si
3368+4.09195916 -9.56309566 4.09195916 1 # Si
3369+1.36387577 -6.82665093 4.09489509 1 # Si
3370+0.00117430 -10.91922202 5.46070202 1 # Si
3371+2.72890018 -8.19145655 5.45886981 1 # Si
3372+2.73303284 -10.91859845 8.18958251 1 # Si
3373+-0.01202455 -8.19611273 8.19611273 1 # Si
3374+1.36492084 -9.55456223 6.82852418 1 # Si
3375+4.09496380 -6.82463094 6.82463094 1 # Si
3376+4.08178398 -9.53373688 9.53373688 1 # Si
3377+1.36492084 -6.82852418 9.55456223 1 # Si
3378+0.01304838 -10.90149235 10.90149235 1 # Si
3379+2.73303284 -8.18958251 10.91859845 1 # Si
3380+0.02610591 -8.18242131 13.61801556 1 # Si
3381+1.35303719 -9.54338724 12.25680064 1 # Si
3382+4.06859976 -6.80192861 12.27486761 1 # Si
3383+1.33325565 -6.81988895 15.00302593 1 # Si
3384+0.16771494 -5.63195305 -16.34134070 1 # Si
3385+2.90187977 -2.90187977 -16.34634363 1 # Si
3386+2.69385812 -5.42510828 -13.61156377 1 # Si
3387+0.00300340 -2.73058729 -13.68153993 1 # Si
3388+1.37590571 -4.10489826 -15.03413728 1 # Si
3389+4.10489826 -1.37590571 -15.03413728 1 # Si
3390+4.09402380 -4.09402380 -12.27225867 1 # Si
3391+1.37011664 -1.37011664 -12.30671934 1 # Si
3392+-0.00117430 -5.46070202 -10.91922202 1 # Si
3393+2.72813440 -2.72813440 -10.91829649 1 # Si
3394+2.73481248 -5.46221414 -8.19802839 1 # Si
3395+-0.00218763 -2.72767707 -8.19459074 1 # Si
3396+1.36300571 -4.09408870 -9.55550648 1 # Si
3397+4.09408870 -1.36300571 -9.55550648 1 # Si
3398+4.09758805 -4.09758805 -6.82965269 1 # Si
3399+1.36467215 -1.36467215 -6.82823155 1 # Si
3400+0.00319897 -5.46373779 -5.46373779 1 # Si
3401+2.73166810 -2.73166810 -5.46412728 1 # Si
3402+2.73166810 -5.46412728 -2.73166810 1 # Si
3403+0.00070150 -2.73149619 -2.73149619 1 # Si
3404+1.36658639 -4.09725659 -4.09725659 1 # Si
3405+4.09725659 -1.36658639 -4.09725659 1 # Si
3406+4.09725659 -4.09725659 -1.36658639 1 # Si
3407+1.36585704 -1.36585704 -1.36585704 1 # Si
3408+-0.00000000 -5.46151579 0.00000000 1 # Si
3409+2.73149619 -2.73149619 -0.00070150 1 # Si
3410+2.72992539 -5.46104379 2.72992539 1 # Si
3411+-0.00070150 -2.73149619 2.73149619 1 # Si
3412+1.36499559 -4.09571959 1.36499559 1 # Si
3413+4.09571959 -1.36499559 1.36499559 1 # Si
3414+4.09449230 -4.09449230 4.09449230 1 # Si
3415+1.36499559 -1.36499559 4.09571959 1 # Si
3416+-0.00319897 -5.46373779 5.46373779 1 # Si
3417+2.72992539 -2.72992539 5.46104379 1 # Si
3418+2.72890018 -5.45886981 8.19145655 1 # Si
3419+0.00218763 -2.72767707 8.19459074 1 # Si
3420+1.36387577 -4.09489509 6.82665093 1 # Si
3421+4.09489509 -1.36387577 6.82665093 1 # Si
3422+4.09195916 -4.09195916 9.56309566 1 # Si
3423+1.36256401 -1.36256401 9.56805296 1 # Si
3424+0.00117430 -5.46070202 10.91922202 1 # Si
3425+2.72665137 -2.72665137 10.93548204 1 # Si
3426+2.72110232 -5.45900176 13.66848041 1 # Si
3427+-0.00300340 -2.73058729 13.68153993 1 # Si
3428+1.35797308 -4.09496434 12.30121531 1 # Si
3429+4.09496434 -1.35797308 12.30121531 1 # Si
3430+4.07390228 -4.07390228 15.01032944 1 # Si
3431+1.36373167 -1.36373167 15.04397328 1 # Si
3432+-0.16771494 -5.63195305 16.34134070 1 # Si
3433+2.73136988 -2.73136988 16.37023166 1 # Si
3434+-0.00000000 -0.00000000 -16.37821316 1 # Si
3435+2.73136988 2.73136988 -16.37023166 1 # Si
3436+2.73058729 -0.00300340 -13.68153993 1 # Si
3437+-0.00300340 2.73058729 -13.68153993 1 # Si
3438+1.36373167 1.36373167 -15.04397328 1 # Si
3439+4.07390228 4.07390228 -15.01032944 1 # Si
3440+4.09496434 1.35797308 -12.30121531 1 # Si
3441+1.35797308 4.09496434 -12.30121531 1 # Si
3442+0.00000000 0.00000000 -10.94349563 1 # Si
3443+2.72665137 2.72665137 -10.93548204 1 # Si
3444+2.72767707 0.00218763 -8.19459074 1 # Si
3445+0.00218763 2.72767707 -8.19459074 1 # Si
3446+1.36256401 1.36256401 -9.56805296 1 # Si
3447+4.09195916 4.09195916 -9.56309566 1 # Si
3448+4.09489509 1.36387577 -6.82665093 1 # Si
3449+1.36387577 4.09489509 -6.82665093 1 # Si
3450+-0.00000000 0.00000000 -5.46151579 1 # Si
3451+2.72992539 2.72992539 -5.46104379 1 # Si
3452+2.73149619 -0.00070150 -2.73149619 1 # Si
3453+-0.00070150 2.73149619 -2.73149619 1 # Si
3454+1.36499559 1.36499559 -4.09571959 1 # Si
3455+4.09449230 4.09449230 -4.09449230 1 # Si
3456+4.09571959 1.36499559 -1.36499559 1 # Si
3457+1.36499559 4.09571959 -1.36499559 1 # Si
3458+-0.00000000 0.00000000 -0.00000000 1 # Si
3459+2.73149619 2.73149619 0.00070150 1 # Si
3460+2.73149619 0.00070150 2.73149619 1 # Si
3461+0.00070150 2.73149619 2.73149619 1 # Si
3462+1.36585704 1.36585704 1.36585704 1 # Si
3463+4.09725659 4.09725659 1.36658639 1 # Si
3464+4.09725659 1.36658639 4.09725659 1 # Si
3465+1.36658639 4.09725659 4.09725659 1 # Si
3466+-0.00000000 -0.00000000 5.46151579 1 # Si
3467+2.73166810 2.73166810 5.46412728 1 # Si
3468+2.72767707 -0.00218763 8.19459074 1 # Si
3469+-0.00218763 2.72767707 8.19459074 1 # Si
3470+1.36467215 1.36467215 6.82823155 1 # Si
3471+4.09758805 4.09758805 6.82965269 1 # Si
3472+4.09408870 1.36300571 9.55550648 1 # Si
3473+1.36300571 4.09408870 9.55550648 1 # Si
3474+-0.00000000 -0.00000000 10.94349563 1 # Si
3475+2.72813440 2.72813440 10.91829649 1 # Si
3476+2.73058729 0.00300340 13.68153993 1 # Si
3477+0.00300340 2.73058729 13.68153993 1 # Si
3478+1.37011664 1.37011664 12.30671934 1 # Si
3479+4.09402380 4.09402380 12.27225867 1 # Si
3480+4.10489826 1.37590571 15.03413728 1 # Si
3481+1.37590571 4.10489826 15.03413728 1 # Si
3482+-0.00000000 -0.00000000 16.37821316 1 # Si
3483+2.90187977 2.90187977 16.34634363 1 # Si
3484+-0.16771494 5.63195305 -16.34134070 1 # Si
3485+2.72110232 5.45900176 -13.66848041 1 # Si
3486+0.02610591 8.18242131 -13.61801556 1 # Si
3487+1.33325565 6.81988895 -15.00302593 1 # Si
3488+4.06859976 6.80192861 -12.27486761 1 # Si
3489+1.35303719 9.54338724 -12.25680064 1 # Si
3490+0.00117430 5.46070202 -10.91922202 1 # Si
3491+2.73303284 8.18958251 -10.91859845 1 # Si
3492+2.72890018 5.45886981 -8.19145655 1 # Si
3493+-0.01202455 8.19611273 -8.19611273 1 # Si
3494+1.36492084 6.82852418 -9.55456223 1 # Si
3495+4.08178398 9.53373688 -9.53373688 1 # Si
3496+4.09496380 6.82463094 -6.82463094 1 # Si
3497+1.36492084 9.55456223 -6.82852418 1 # Si
3498+-0.00319897 5.46373779 -5.46373779 1 # Si
3499+2.72890018 8.19145655 -5.45886981 1 # Si
3500+2.72992539 5.46104379 -2.72992539 1 # Si
3501+0.00218763 8.19459074 -2.72767707 1 # Si
3502+1.36387577 6.82665093 -4.09489509 1 # Si
3503+4.09195916 9.56309566 -4.09195916 1 # Si
3504+4.09489509 6.82665093 -1.36387577 1 # Si
3505+1.36256401 9.56805296 -1.36256401 1 # Si
3506+-0.00000000 5.46151579 0.00000000 1 # Si
3507+2.72767707 8.19459074 -0.00218763 1 # Si
3508+2.73166810 5.46412728 2.73166810 1 # Si
3509+-0.00218763 8.19459074 2.72767707 1 # Si
3510+1.36467215 6.82823155 1.36467215 1 # Si
3511+4.09408870 9.55550648 1.36300571 1 # Si
3512+4.09758805 6.82965269 4.09758805 1 # Si
3513+1.36300571 9.55550648 4.09408870 1 # Si
3514+0.00319897 5.46373779 5.46373779 1 # Si
3515+2.73481248 8.19802839 5.46221414 1 # Si
3516+2.73481248 5.46221414 8.19802839 1 # Si
3517+0.01202455 8.19611273 8.19611273 1 # Si
3518+1.37205363 6.82894278 6.82894278 1 # Si
3519+4.10298675 9.56489675 6.82705083 1 # Si
3520+4.10298675 6.82705083 9.56489675 1 # Si
3521+1.38318968 9.56265344 9.56265344 1 # Si
3522+-0.00117430 5.46070202 10.91922202 1 # Si
3523+2.74437054 8.19260795 10.93410102 1 # Si
3524+2.69385812 5.42510828 13.61156377 1 # Si
3525+-0.02610591 8.18242131 13.61801556 1 # Si
3526+1.35920843 6.82966552 12.27960089 1 # Si
3527+4.12509126 9.54137501 12.28371979 1 # Si
3528+0.16771494 5.63195305 16.34134070 1 # Si
3529+0.01304838 10.90149235 -10.90149235 1 # Si
3530+2.73303284 10.91859845 -8.18958251 1 # Si
3531+0.02610591 13.61801556 -8.18242131 1 # Si
3532+1.35303719 12.25680064 -9.54338724 1 # Si
3533+4.06859976 12.27486761 -6.80192861 1 # Si
3534+1.33325565 15.00302593 -6.81988895 1 # Si
3535+0.00117430 10.91922202 -5.46070202 1 # Si
3536+2.72110232 13.66848041 -5.45900176 1 # Si
3537+2.72665137 10.93548204 -2.72665137 1 # Si
3538+-0.00300340 13.68153993 -2.73058729 1 # Si
3539+1.35797308 12.30121531 -4.09496434 1 # Si
3540+4.07390228 15.01032944 -4.07390228 1 # Si
3541+4.09496434 12.30121531 -1.35797308 1 # Si
3542+1.36373167 15.04397328 -1.36373167 1 # Si
3543+-0.00000000 10.94349563 0.00000000 1 # Si
3544+2.73058729 13.68153993 0.00300340 1 # Si
3545+2.72813440 10.91829649 2.72813440 1 # Si
3546+0.00300340 13.68153993 2.73058729 1 # Si
3547+1.37011664 12.30671934 1.37011664 1 # Si
3548+4.10489826 15.03413728 1.37590571 1 # Si
3549+4.09402380 12.27225867 4.09402380 1 # Si
3550+1.37590571 15.03413728 4.10489826 1 # Si
3551+-0.00117430 10.91922202 5.46070202 1 # Si
3552+2.69385812 13.61156377 5.42510828 1 # Si
3553+2.74437054 10.93410102 8.19260795 1 # Si
3554+-0.02610591 13.61801556 8.18242131 1 # Si
3555+1.35920843 12.27960089 6.82966552 1 # Si
3556+4.12509126 12.28371979 9.54137501 1 # Si
3557+-0.01304838 10.90149235 10.90149235 1 # Si
3558+-0.16771494 16.34134070 -5.63195305 1 # Si
3559+2.73136988 16.37023166 -2.73136988 1 # Si
3560+0.00000000 16.37821316 -0.00000000 1 # Si
3561+2.90187977 16.34634363 2.90187977 1 # Si
3562+0.16771494 16.34134070 5.63195305 1 # Si
3563+5.61139268 -13.62454044 -8.34100436 1 # Si
3564+6.84800235 -12.30649445 -6.84800235 1 # Si
3565+8.34100436 -13.62454044 -5.61139268 1 # Si
3566+5.42510828 -13.61156377 -2.69385812 1 # Si
3567+9.54137501 -12.28371979 -4.12509126 1 # Si
3568+6.82966552 -12.27960089 -1.35920843 1 # Si
3569+5.63195305 -16.34134070 -0.16771494 1 # Si
3570+8.18242131 -13.61801556 0.02610591 1 # Si
3571+5.45900176 -13.66848041 2.72110232 1 # Si
3572+6.81988895 -15.00302593 1.33325565 1 # Si
3573+9.54338724 -12.25680064 1.35303719 1 # Si
3574+6.80192861 -12.27486761 4.06859976 1 # Si
3575+5.61139268 -8.34100436 -13.62454044 1 # Si
3576+6.84800235 -6.84800235 -12.30649445 1 # Si
3577+5.43284542 -10.90291341 -10.90291341 1 # Si
3578+8.17530342 -8.17530342 -10.89255734 1 # Si
3579+8.17530342 -10.89255734 -8.17530342 1 # Si
3580+5.46312108 -8.19345007 -8.19345007 1 # Si
3581+6.82271744 -9.55790920 -9.55790920 1 # Si
3582+9.55790920 -6.82271744 -9.55790920 1 # Si
3583+9.55790920 -9.55790920 -6.82271744 1 # Si
3584+6.82700831 -6.82700831 -6.82700831 1 # Si
3585+5.47569502 -10.94234231 -5.47569502 1 # Si
3586+8.19345007 -8.19345007 -5.46312108 1 # Si
3587+8.19260795 -10.93410102 -2.74437054 1 # Si
3588+5.46221414 -8.19802839 -2.73481248 1 # Si
3589+6.82705083 -9.56489675 -4.10298675 1 # Si
3590+9.56489675 -6.82705083 -4.10298675 1 # Si
3591+9.56265344 -9.56265344 -1.38318968 1 # Si
3592+6.82894278 -6.82894278 -1.37205363 1 # Si
3593+5.46070202 -10.91922202 0.00117430 1 # Si
3594+8.19611273 -8.19611273 -0.01202455 1 # Si
3595+8.18958251 -10.91859845 2.73303284 1 # Si
3596+5.45886981 -8.19145655 2.72890018 1 # Si
3597+6.82852418 -9.55456223 1.36492084 1 # Si
3598+9.55456223 -6.82852418 1.36492084 1 # Si
3599+9.53373688 -9.53373688 4.08178398 1 # Si
3600+6.82463094 -6.82463094 4.09496380 1 # Si
3601+5.45537137 -10.92814020 5.45537137 1 # Si
3602+8.18684496 -8.18684496 5.46061302 1 # Si
3603+5.46061302 -8.18684496 8.18684496 1 # Si
3604+6.80203106 -9.53854106 6.80203106 1 # Si
3605+9.53854106 -6.80203106 6.80203106 1 # Si
3606+6.80203106 -6.80203106 9.53854106 1 # Si
3607+8.34100436 -5.61139268 -13.62454044 1 # Si
3608+5.42510828 -2.69385812 -13.61156377 1 # Si
3609+9.54137501 -4.12509126 -12.28371979 1 # Si
3610+6.82966552 -1.35920843 -12.27960089 1 # Si
3611+5.47569502 -5.47569502 -10.94234231 1 # Si
3612+8.19260795 -2.74437054 -10.93410102 1 # Si
3613+8.19345007 -5.46312108 -8.19345007 1 # Si
3614+5.46221414 -2.73481248 -8.19802839 1 # Si
3615+6.82705083 -4.10298675 -9.56489675 1 # Si
3616+9.56265344 -1.38318968 -9.56265344 1 # Si
3617+9.56489675 -4.10298675 -6.82705083 1 # Si
3618+6.82894278 -1.37205363 -6.82894278 1 # Si
3619+5.46236779 -5.46236779 -5.46236779 1 # Si
3620+8.19802839 -2.73481248 -5.46221414 1 # Si
3621+8.19802839 -5.46221414 -2.73481248 1 # Si
3622+5.46412728 -2.73166810 -2.73166810 1 # Si
3623+6.82965269 -4.09758805 -4.09758805 1 # Si
3624+9.55550648 -1.36300571 -4.09408870 1 # Si
3625+9.55550648 -4.09408870 -1.36300571 1 # Si
3626+6.82823155 -1.36467215 -1.36467215 1 # Si
3627+5.46373779 -5.46373779 -0.00319897 1 # Si
3628+8.19459074 -2.72767707 0.00218763 1 # Si
3629+8.19145655 -5.45886981 2.72890018 1 # Si
3630+5.46104379 -2.72992539 2.72992539 1 # Si
3631+6.82665093 -4.09489509 1.36387577 1 # Si
3632+9.56805296 -1.36256401 1.36256401 1 # Si
3633+9.56309566 -4.09195916 4.09195916 1 # Si
3634+6.82665093 -1.36387577 4.09489509 1 # Si
3635+5.45846237 -5.45846237 5.45846237 1 # Si
3636+8.19145655 -2.72890018 5.45886981 1 # Si
3637+8.18684496 -5.46061302 8.18684496 1 # Si
3638+5.45886981 -2.72890018 8.19145655 1 # Si
3639+6.82463094 -4.09496380 6.82463094 1 # Si
3640+9.55456223 -1.36492084 6.82852418 1 # Si
3641+9.53373688 -4.08178398 9.53373688 1 # Si
3642+6.82852418 -1.36492084 9.55456223 1 # Si
3643+5.45537137 -5.45537137 10.92814020 1 # Si
3644+8.18958251 -2.73303284 10.91859845 1 # Si
3645+5.45900176 -2.72110232 13.66848041 1 # Si
3646+6.80192861 -4.06859976 12.27486761 1 # Si
3647+9.54338724 -1.35303719 12.25680064 1 # Si
3648+6.81988895 -1.33325565 15.00302593 1 # Si
3649+5.63195305 -0.16771494 -16.34134070 1 # Si
3650+8.18242131 0.02610591 -13.61801556 1 # Si
3651+5.45900176 2.72110232 -13.66848041 1 # Si
3652+6.81988895 1.33325565 -15.00302593 1 # Si
3653+9.54338724 1.35303719 -12.25680064 1 # Si
3654+6.80192861 4.06859976 -12.27486761 1 # Si
3655+5.46070202 0.00117430 -10.91922202 1 # Si
3656+8.18958251 2.73303284 -10.91859845 1 # Si
3657+8.19611273 -0.01202455 -8.19611273 1 # Si
3658+5.45886981 2.72890018 -8.19145655 1 # Si
3659+6.82852418 1.36492084 -9.55456223 1 # Si
3660+9.53373688 4.08178398 -9.53373688 1 # Si
3661+9.55456223 1.36492084 -6.82852418 1 # Si
3662+6.82463094 4.09496380 -6.82463094 1 # Si
3663+5.46373779 -0.00319897 -5.46373779 1 # Si
3664+8.19145655 2.72890018 -5.45886981 1 # Si
3665+8.19459074 0.00218763 -2.72767707 1 # Si
3666+5.46104379 2.72992539 -2.72992539 1 # Si
3667+6.82665093 1.36387577 -4.09489509 1 # Si
3668+9.56309566 4.09195916 -4.09195916 1 # Si
3669+9.56805296 1.36256401 -1.36256401 1 # Si
3670+6.82665093 4.09489509 -1.36387577 1 # Si
3671+5.46151579 -0.00000000 0.00000000 1 # Si
3672+8.19459074 2.72767707 -0.00218763 1 # Si
3673+8.19459074 -0.00218763 2.72767707 1 # Si
3674+5.46412728 2.73166810 2.73166810 1 # Si
3675+6.82823155 1.36467215 1.36467215 1 # Si
3676+9.55550648 4.09408870 1.36300571 1 # Si
3677+9.55550648 1.36300571 4.09408870 1 # Si
3678+6.82965269 4.09758805 4.09758805 1 # Si
3679+5.46373779 0.00319897 5.46373779 1 # Si
3680+8.19802839 2.73481248 5.46221414 1 # Si
3681+8.19611273 0.01202455 8.19611273 1 # Si
3682+5.46221414 2.73481248 8.19802839 1 # Si
3683+6.82894278 1.37205363 6.82894278 1 # Si
3684+9.56489675 4.10298675 6.82705083 1 # Si
3685+9.56265344 1.38318968 9.56265344 1 # Si
3686+6.82705083 4.10298675 9.56489675 1 # Si
3687+5.46070202 -0.00117430 10.91922202 1 # Si
3688+8.19260795 2.74437054 10.93410102 1 # Si
3689+8.18242131 -0.02610591 13.61801556 1 # Si
3690+5.42510828 2.69385812 13.61156377 1 # Si
3691+6.82966552 1.35920843 12.27960089 1 # Si
3692+9.54137501 4.12509126 12.28371979 1 # Si
3693+5.63195305 0.16771494 16.34134070 1 # Si
3694+5.45537137 5.45537137 -10.92814020 1 # Si
3695+8.18684496 5.46061302 -8.18684496 1 # Si
3696+5.46061302 8.18684496 -8.18684496 1 # Si
3697+6.80203106 6.80203106 -9.53854106 1 # Si
3698+9.53854106 6.80203106 -6.80203106 1 # Si
3699+6.80203106 9.53854106 -6.80203106 1 # Si
3700+5.45846237 5.45846237 -5.45846237 1 # Si
3701+8.18684496 8.18684496 -5.46061302 1 # Si
3702+8.19145655 5.45886981 -2.72890018 1 # Si
3703+5.45886981 8.19145655 -2.72890018 1 # Si
3704+6.82463094 6.82463094 -4.09496380 1 # Si
3705+9.53373688 9.53373688 -4.08178398 1 # Si
3706+9.55456223 6.82852418 -1.36492084 1 # Si
3707+6.82852418 9.55456223 -1.36492084 1 # Si
3708+5.46373779 5.46373779 0.00319897 1 # Si
3709+8.19611273 8.19611273 0.01202455 1 # Si
3710+8.19802839 5.46221414 2.73481248 1 # Si
3711+5.46221414 8.19802839 2.73481248 1 # Si
3712+6.82894278 6.82894278 1.37205363 1 # Si
3713+9.56265344 9.56265344 1.38318968 1 # Si
3714+9.56489675 6.82705083 4.10298675 1 # Si
3715+6.82705083 9.56489675 4.10298675 1 # Si
3716+5.46236779 5.46236779 5.46236779 1 # Si
3717+8.19345007 8.19345007 5.46312108 1 # Si
3718+8.19345007 5.46312108 8.19345007 1 # Si
3719+5.46312108 8.19345007 8.19345007 1 # Si
3720+6.82700831 6.82700831 6.82700831 1 # Si
3721+9.55790920 9.55790920 6.82271744 1 # Si
3722+9.55790920 6.82271744 9.55790920 1 # Si
3723+6.82271744 9.55790920 9.55790920 1 # Si
3724+5.47569502 5.47569502 10.94234231 1 # Si
3725+8.17530342 8.17530342 10.89255734 1 # Si
3726+8.34100436 5.61139268 13.62454044 1 # Si
3727+5.61139268 8.34100436 13.62454044 1 # Si
3728+6.84800235 6.84800235 12.30649445 1 # Si
3729+5.45537137 10.92814020 -5.45537137 1 # Si
3730+8.18958251 10.91859845 -2.73303284 1 # Si
3731+5.45900176 13.66848041 -2.72110232 1 # Si
3732+6.80192861 12.27486761 -4.06859976 1 # Si
3733+9.54338724 12.25680064 -1.35303719 1 # Si
3734+6.81988895 15.00302593 -1.33325565 1 # Si
3735+5.46070202 10.91922202 -0.00117430 1 # Si
3736+8.18242131 13.61801556 -0.02610591 1 # Si
3737+8.19260795 10.93410102 2.74437054 1 # Si
3738+5.42510828 13.61156377 2.69385812 1 # Si
3739+6.82966552 12.27960089 1.35920843 1 # Si
3740+9.54137501 12.28371979 4.12509126 1 # Si
3741+5.47569502 10.94234231 5.47569502 1 # Si
3742+8.34100436 13.62454044 5.61139268 1 # Si
3743+8.17530342 10.89255734 8.17530342 1 # Si
3744+5.61139268 13.62454044 8.34100436 1 # Si
3745+6.84800235 12.30649445 6.84800235 1 # Si
3746+5.43284542 10.90291341 10.90291341 1 # Si
3747+5.63195305 16.34134070 0.16771494 1 # Si
3748+10.89255734 -8.17530342 -8.17530342 1 # Si
3749+12.30649445 -6.84800235 -6.84800235 1 # Si
3750+10.90291341 -10.90291341 -5.43284542 1 # Si
3751+13.62454044 -8.34100436 -5.61139268 1 # Si
3752+10.93410102 -8.19260795 -2.74437054 1 # Si
3753+12.28371979 -9.54137501 -4.12509126 1 # Si
3754+12.27960089 -6.82966552 -1.35920843 1 # Si
3755+10.90149235 -10.90149235 0.01304838 1 # Si
3756+13.61801556 -8.18242131 0.02610591 1 # Si
3757+10.91859845 -8.18958251 2.73303284 1 # Si
3758+12.25680064 -9.54338724 1.35303719 1 # Si
3759+15.00302593 -6.81988895 1.33325565 1 # Si
3760+12.27486761 -6.80192861 4.06859976 1 # Si
3761+10.90291341 -5.43284542 -10.90291341 1 # Si
3762+13.62454044 -5.61139268 -8.34100436 1 # Si
3763+10.93410102 -2.74437054 -8.19260795 1 # Si
3764+12.28371979 -4.12509126 -9.54137501 1 # Si
3765+12.27960089 -1.35920843 -6.82966552 1 # Si
3766+10.94234231 -5.47569502 -5.47569502 1 # Si
3767+13.61156377 -2.69385812 -5.42510828 1 # Si
3768+13.61156377 -5.42510828 -2.69385812 1 # Si
3769+10.91829649 -2.72813440 -2.72813440 1 # Si
3770+12.27225867 -4.09402380 -4.09402380 1 # Si
3771+15.03413728 -1.37590571 -4.10489826 1 # Si
3772+15.03413728 -4.10489826 -1.37590571 1 # Si
3773+12.30671934 -1.37011664 -1.37011664 1 # Si
3774+10.91922202 -5.46070202 0.00117430 1 # Si
3775+13.68153993 -2.73058729 -0.00300340 1 # Si
3776+13.66848041 -5.45900176 2.72110232 1 # Si
3777+10.93548204 -2.72665137 2.72665137 1 # Si
3778+12.30121531 -4.09496434 1.35797308 1 # Si
3779+15.04397328 -1.36373167 1.36373167 1 # Si
3780+15.01032944 -4.07390228 4.07390228 1 # Si
3781+12.30121531 -1.35797308 4.09496434 1 # Si
3782+10.92814020 -5.45537137 5.45537137 1 # Si
3783+13.66848041 -2.72110232 5.45900176 1 # Si
3784+10.91859845 -2.73303284 8.18958251 1 # Si
3785+12.27486761 -4.06859976 6.80192861 1 # Si
3786+15.00302593 -1.33325565 6.81988895 1 # Si
3787+12.25680064 -1.35303719 9.54338724 1 # Si
3788+10.90149235 0.01304838 -10.90149235 1 # Si
3789+13.61801556 0.02610591 -8.18242131 1 # Si
3790+10.91859845 2.73303284 -8.18958251 1 # Si
3791+12.25680064 1.35303719 -9.54338724 1 # Si
3792+15.00302593 1.33325565 -6.81988895 1 # Si
3793+12.27486761 4.06859976 -6.80192861 1 # Si
3794+10.91922202 0.00117430 -5.46070202 1 # Si
3795+13.66848041 2.72110232 -5.45900176 1 # Si
3796+13.68153993 -0.00300340 -2.73058729 1 # Si
3797+10.93548204 2.72665137 -2.72665137 1 # Si
3798+12.30121531 1.35797308 -4.09496434 1 # Si
3799+15.01032944 4.07390228 -4.07390228 1 # Si
3800+15.04397328 1.36373167 -1.36373167 1 # Si
3801+12.30121531 4.09496434 -1.35797308 1 # Si
3802+10.94349563 0.00000000 -0.00000000 1 # Si
3803+13.68153993 2.73058729 0.00300340 1 # Si
3804+13.68153993 0.00300340 2.73058729 1 # Si
3805+10.91829649 2.72813440 2.72813440 1 # Si
3806+12.30671934 1.37011664 1.37011664 1 # Si
3807+15.03413728 4.10489826 1.37590571 1 # Si
3808+15.03413728 1.37590571 4.10489826 1 # Si
3809+12.27225867 4.09402380 4.09402380 1 # Si
3810+10.91922202 -0.00117430 5.46070202 1 # Si
3811+13.61156377 2.69385812 5.42510828 1 # Si
3812+13.61801556 -0.02610591 8.18242131 1 # Si
3813+10.93410102 2.74437054 8.19260795 1 # Si
3814+12.27960089 1.35920843 6.82966552 1 # Si
3815+12.28371979 4.12509126 9.54137501 1 # Si
3816+10.90149235 -0.01304838 10.90149235 1 # Si
3817+10.92814020 5.45537137 -5.45537137 1 # Si
3818+13.66848041 5.45900176 -2.72110232 1 # Si
3819+10.91859845 8.18958251 -2.73303284 1 # Si
3820+12.27486761 6.80192861 -4.06859976 1 # Si
3821+15.00302593 6.81988895 -1.33325565 1 # Si
3822+12.25680064 9.54338724 -1.35303719 1 # Si
3823+10.91922202 5.46070202 -0.00117430 1 # Si
3824+13.61801556 8.18242131 -0.02610591 1 # Si
3825+13.61156377 5.42510828 2.69385812 1 # Si
3826+10.93410102 8.19260795 2.74437054 1 # Si
3827+12.27960089 6.82966552 1.35920843 1 # Si
3828+12.28371979 9.54137501 4.12509126 1 # Si
3829+10.94234231 5.47569502 5.47569502 1 # Si
3830+13.62454044 8.34100436 5.61139268 1 # Si
3831+13.62454044 5.61139268 8.34100436 1 # Si
3832+10.89255734 8.17530342 8.17530342 1 # Si
3833+12.30649445 6.84800235 6.84800235 1 # Si
3834+10.90291341 5.43284542 10.90291341 1 # Si
3835+10.90149235 10.90149235 -0.01304838 1 # Si
3836+10.90291341 10.90291341 5.43284542 1 # Si
3837+16.34634363 -2.90187977 -2.90187977 1 # Si
3838+16.34134070 -5.63195305 -0.16771494 1 # Si
3839+16.37023166 -2.73136988 2.73136988 1 # Si
3840+16.34134070 -0.16771494 -5.63195305 1 # Si
3841+16.37023166 2.73136988 -2.73136988 1 # Si
3842+16.37821316 -0.00000000 -0.00000000 1 # Si
3843+16.34634363 2.90187977 2.90187977 1 # Si
3844+16.34134070 0.16771494 5.63195305 1 # Si
3845+16.34134070 5.63195305 0.16771494 1 # Si
3846+-13.14107205 -7.67106057 -4.93773883 2 # H
3847+-13.12200813 -10.41265519 -2.22197561 2 # H
3848+-15.87081431 -7.68828624 -2.20095245 2 # H
3849+-14.47909953 -9.05114009 0.84828128 2 # H
3850+-13.14542628 -10.40942624 3.25101117 2 # H
3851+-13.12200813 -2.22197561 -10.41265519 2 # H
3852+-13.14107205 -4.93773883 -7.67106057 2 # H
3853+-15.87081431 -2.20095245 -7.68828624 2 # H
3854+-15.87684776 -4.94293745 -4.94293745 2 # H
3855+-14.48904263 -6.28831840 3.55914512 2 # H
3856+-14.48904263 -3.55914512 6.28831840 2 # H
3857+-13.14542628 -3.25101117 10.40942624 2 # H
3858+-14.47909953 0.84828128 -9.05114009 2 # H
3859+-13.14542628 3.25101117 -10.40942624 2 # H
3860+-14.48904263 3.55914512 -6.28831840 2 # H
3861+-15.87684776 4.94293745 4.94293745 2 # H
3862+-14.47909953 -0.84828128 9.05114009 2 # H
3863+-15.87081431 2.20095245 7.68828624 2 # H
3864+-13.14107205 4.93773883 7.67106057 2 # H
3865+-13.12200813 2.22197561 10.41265519 2 # H
3866+-14.48904263 6.28831840 -3.55914512 2 # H
3867+-13.14542628 10.40942624 -3.25101117 2 # H
3868+-14.47909953 9.05114009 -0.84828128 2 # H
3869+-15.87081431 7.68828624 2.20095245 2 # H
3870+-13.12200813 10.41265519 2.22197561 2 # H
3871+-13.14107205 7.67106057 4.93773883 2 # H
3872+-7.67106057 -13.14107205 -4.93773883 2 # H
3873+-7.68828624 -15.87081431 -2.20095245 2 # H
3874+-10.41265519 -13.12200813 -2.22197561 2 # H
3875+-9.05114009 -14.47909953 0.84828128 2 # H
3876+-10.40942624 -13.14542628 3.25101117 2 # H
3877+-7.67101940 -7.67101940 -10.40498818 2 # H
3878+-7.67101940 -10.40498818 -7.67101940 2 # H
3879+-10.40498818 -7.67101940 -7.67101940 2 # H
3880+-10.40134885 -10.40134885 -4.95086615 2 # H
3881+-11.77396546 -11.77396546 0.84562385 2 # H
3882+-11.76622906 -11.76622906 6.31060507 2 # H
3883+-9.04902753 -11.74957272 9.04902753 2 # H
3884+-11.74957272 -9.04902753 9.04902753 2 # H
3885+-9.04902753 -9.04902753 11.74957272 2 # H
3886+-7.68828624 -2.20095245 -15.87081431 2 # H
3887+-7.67106057 -4.93773883 -13.14107205 2 # H
3888+-10.41265519 -2.22197561 -13.12200813 2 # H
3889+-10.40134885 -4.95086615 -10.40134885 2 # H
3890+-11.76622906 -6.31060507 11.76622906 2 # H
3891+-10.40942624 -3.25101117 13.14542628 2 # H
3892+-9.05114009 0.84828128 -14.47909953 2 # H
3893+-10.40942624 3.25101117 -13.14542628 2 # H
3894+-11.77396546 0.84562385 -11.77396546 2 # H
3895+-10.40134885 4.95086615 10.40134885 2 # H
3896+-11.77396546 -0.84562385 11.77396546 2 # H
3897+-9.05114009 -0.84828128 14.47909953 2 # H
3898+-10.41265519 2.22197561 13.12200813 2 # H
3899+-7.67106057 4.93773883 13.14107205 2 # H
3900+-7.68828624 2.20095245 15.87081431 2 # H
3901+-11.76622906 6.31060507 -11.76622906 2 # H
3902+-9.04902753 9.04902753 -11.74957272 2 # H
3903+-11.74957272 9.04902753 -9.04902753 2 # H
3904+-10.40134885 10.40134885 4.95086615 2 # H
3905+-10.40498818 7.67101940 7.67101940 2 # H
3906+-7.67101940 10.40498818 7.67101940 2 # H
3907+-7.67101940 7.67101940 10.40498818 2 # H
3908+-9.04902753 11.74957272 -9.04902753 2 # H
3909+-11.76622906 11.76622906 -6.31060507 2 # H
3910+-10.40942624 13.14542628 -3.25101117 2 # H
3911+-11.77396546 11.77396546 -0.84562385 2 # H
3912+-9.05114009 14.47909953 -0.84828128 2 # H
3913+-10.41265519 13.12200813 2.22197561 2 # H
3914+-7.68828624 15.87081431 2.20095245 2 # H
3915+-7.67106057 13.14107205 4.93773883 2 # H
3916+-2.22197561 -13.12200813 -10.41265519 2 # H
3917+-2.20095245 -15.87081431 -7.68828624 2 # H
3918+-4.93773883 -13.14107205 -7.67106057 2 # H
3919+-4.94293745 -15.87684776 -4.94293745 2 # H
3920+-6.28831840 -14.48904263 3.55914512 2 # H
3921+-3.55914512 -14.48904263 6.28831840 2 # H
3922+-3.25101117 -13.14542628 10.40942624 2 # H
3923+-2.20095245 -7.68828624 -15.87081431 2 # H
3924+-2.22197561 -10.41265519 -13.12200813 2 # H
3925+-4.93773883 -7.67106057 -13.14107205 2 # H
3926+-4.95086615 -10.40134885 -10.40134885 2 # H
3927+-6.31060507 -11.76622906 11.76622906 2 # H
3928+-3.25101117 -10.40942624 13.14542628 2 # H
3929+-4.94293745 -4.94293745 -15.87684776 2 # H
3930+-3.55914512 -6.28831840 14.48904263 2 # H
3931+-6.28831840 -3.55914512 14.48904263 2 # H
3932+-6.28831840 3.55914512 -14.48904263 2 # H
3933+-4.94293745 4.94293745 15.87684776 2 # H
3934+-3.55914512 6.28831840 -14.48904263 2 # H
3935+-3.25101117 10.40942624 -13.14542628 2 # H
3936+-4.95086615 10.40134885 10.40134885 2 # H
3937+-4.93773883 7.67106057 13.14107205 2 # H
3938+-2.22197561 10.41265519 13.12200813 2 # H
3939+-2.20095245 7.68828624 15.87081431 2 # H
3940+-6.31060507 11.76622906 -11.76622906 2 # H
3941+-3.25101117 13.14542628 -10.40942624 2 # H
3942+-3.55914512 14.48904263 -6.28831840 2 # H
3943+-6.28831840 14.48904263 -3.55914512 2 # H
3944+-4.94293745 15.87684776 4.94293745 2 # H
3945+-4.93773883 13.14107205 7.67106057 2 # H
3946+-2.20095245 15.87081431 7.68828624 2 # H
3947+-2.22197561 13.12200813 10.41265519 2 # H
3948+0.84828128 -14.47909953 -9.05114009 2 # H
3949+3.25101117 -13.14542628 -10.40942624 2 # H
3950+3.55914512 -14.48904263 -6.28831840 2 # H
3951+4.94293745 -15.87684776 4.94293745 2 # H
3952+-0.84828128 -14.47909953 9.05114009 2 # H
3953+2.20095245 -15.87081431 7.68828624 2 # H
3954+4.93773883 -13.14107205 7.67106057 2 # H
3955+2.22197561 -13.12200813 10.41265519 2 # H
3956+0.84828128 -9.05114009 -14.47909953 2 # H
3957+3.25101117 -10.40942624 -13.14542628 2 # H
3958+0.84562385 -11.77396546 -11.77396546 2 # H
3959+4.95086615 -10.40134885 10.40134885 2 # H
3960+-0.84562385 -11.77396546 11.77396546 2 # H
3961+-0.84828128 -9.05114009 14.47909953 2 # H
3962+2.22197561 -10.41265519 13.12200813 2 # H
3963+4.93773883 -7.67106057 13.14107205 2 # H
3964+2.20095245 -7.68828624 15.87081431 2 # H
3965+3.55914512 -6.28831840 -14.48904263 2 # H
3966+4.94293745 -4.94293745 15.87684776 2 # H
3967+4.94293745 4.94293745 -15.87684776 2 # H
3968+-0.84828128 9.05114009 -14.47909953 2 # H
3969+2.20095245 7.68828624 -15.87081431 2 # H
3970+4.93773883 7.67106057 -13.14107205 2 # H
3971+2.22197561 10.41265519 -13.12200813 2 # H
3972+4.95086615 10.40134885 -10.40134885 2 # H
3973+3.55914512 6.28831840 14.48904263 2 # H
3974+0.84828128 9.05114009 14.47909953 2 # H
3975+3.25101117 10.40942624 13.14542628 2 # H
3976+-0.84562385 11.77396546 -11.77396546 2 # H
3977+-0.84828128 14.47909953 -9.05114009 2 # H
3978+2.22197561 13.12200813 -10.41265519 2 # H
3979+4.93773883 13.14107205 -7.67106057 2 # H
3980+2.20095245 15.87081431 -7.68828624 2 # H
3981+4.94293745 15.87684776 -4.94293745 2 # H
3982+3.55914512 14.48904263 6.28831840 2 # H
3983+0.84828128 14.47909953 9.05114009 2 # H
3984+3.25101117 13.14542628 10.40942624 2 # H
3985+0.84562385 11.77396546 11.77396546 2 # H
3986+6.28831840 -14.48904263 -3.55914512 2 # H
3987+10.40942624 -13.14542628 -3.25101117 2 # H
3988+9.05114009 -14.47909953 -0.84828128 2 # H
3989+7.68828624 -15.87081431 2.20095245 2 # H
3990+10.41265519 -13.12200813 2.22197561 2 # H
3991+7.67106057 -13.14107205 4.93773883 2 # H
3992+6.31060507 -11.76622906 -11.76622906 2 # H
3993+9.04902753 -9.04902753 -11.74957272 2 # H
3994+9.04902753 -11.74957272 -9.04902753 2 # H
3995+10.40134885 -10.40134885 4.95086615 2 # H
3996+7.67101940 -10.40498818 7.67101940 2 # H
3997+10.40498818 -7.67101940 7.67101940 2 # H
3998+7.67101940 -7.67101940 10.40498818 2 # H
3999+6.28831840 -3.55914512 -14.48904263 2 # H
4000+10.40942624 -3.25101117 -13.14542628 2 # H
4001+10.40134885 -4.95086615 10.40134885 2 # H
4002+7.67106057 -4.93773883 13.14107205 2 # H
4003+10.41265519 -2.22197561 13.12200813 2 # H
4004+7.68828624 -2.20095245 15.87081431 2 # H
4005+9.05114009 -0.84828128 -14.47909953 2 # H
4006+7.68828624 2.20095245 -15.87081431 2 # H
4007+10.41265519 2.22197561 -13.12200813 2 # H
4008+7.67106057 4.93773883 -13.14107205 2 # H
4009+10.40134885 4.95086615 -10.40134885 2 # H
4010+9.05114009 0.84828128 14.47909953 2 # H
4011+6.28831840 3.55914512 14.48904263 2 # H
4012+10.40942624 3.25101117 13.14542628 2 # H
4013+7.67101940 7.67101940 -10.40498818 2 # H
4014+10.40498818 7.67101940 -7.67101940 2 # H
4015+7.67101940 10.40498818 -7.67101940 2 # H
4016+10.40134885 10.40134885 -4.95086615 2 # H
4017+9.04902753 9.04902753 11.74957272 2 # H
4018+7.67106057 13.14107205 -4.93773883 2 # H
4019+10.41265519 13.12200813 -2.22197561 2 # H
4020+7.68828624 15.87081431 -2.20095245 2 # H
4021+9.05114009 14.47909953 0.84828128 2 # H
4022+6.28831840 14.48904263 3.55914512 2 # H
4023+10.40942624 13.14542628 3.25101117 2 # H
4024+9.04902753 11.74957272 9.04902753 2 # H
4025+6.31060507 11.76622906 11.76622906 2 # H
4026+11.74957272 -9.04902753 -9.04902753 2 # H
4027+11.76622906 -11.76622906 -6.31060507 2 # H
4028+13.14542628 -10.40942624 -3.25101117 2 # H
4029+11.77396546 -11.77396546 -0.84562385 2 # H
4030+14.47909953 -9.05114009 -0.84828128 2 # H
4031+13.12200813 -10.41265519 2.22197561 2 # H
4032+15.87081431 -7.68828624 2.20095245 2 # H
4033+13.14107205 -7.67106057 4.93773883 2 # H
4034+11.76622906 -6.31060507 -11.76622906 2 # H
4035+13.14542628 -3.25101117 -10.40942624 2 # H
4036+14.48904263 -3.55914512 -6.28831840 2 # H
4037+14.48904263 -6.28831840 -3.55914512 2 # H
4038+15.87684776 -4.94293745 4.94293745 2 # H
4039+13.14107205 -4.93773883 7.67106057 2 # H
4040+15.87081431 -2.20095245 7.68828624 2 # H
4041+13.12200813 -2.22197561 10.41265519 2 # H
4042+11.77396546 -0.84562385 -11.77396546 2 # H
4043+14.47909953 -0.84828128 -9.05114009 2 # H
4044+13.12200813 2.22197561 -10.41265519 2 # H
4045+15.87081431 2.20095245 -7.68828624 2 # H
4046+13.14107205 4.93773883 -7.67106057 2 # H
4047+15.87684776 4.94293745 -4.94293745 2 # H
4048+14.48904263 3.55914512 6.28831840 2 # H
4049+14.47909953 0.84828128 9.05114009 2 # H
4050+13.14542628 3.25101117 10.40942624 2 # H
4051+11.77396546 0.84562385 11.77396546 2 # H
4052+13.14107205 7.67106057 -4.93773883 2 # H
4053+15.87081431 7.68828624 -2.20095245 2 # H
4054+13.12200813 10.41265519 -2.22197561 2 # H
4055+14.47909953 9.05114009 0.84828128 2 # H
4056+14.48904263 6.28831840 3.55914512 2 # H
4057+13.14542628 10.40942624 3.25101117 2 # H
4058+11.74957272 9.04902753 9.04902753 2 # H
4059+11.76622906 6.31060507 11.76622906 2 # H
4060+11.77396546 11.77396546 0.84562385 2 # H
4061+11.76622906 11.76622906 6.31060507 2 # H
4062+-14.25049869 -9.30903112 6.56976732 2 # H
4063+-14.73854562 -7.62887682 4.90661761 2 # H
4064+-17.26005319 -3.58592496 -1.88119305 2 # H
4065+-17.26005319 -1.88119305 -3.58592496 2 # H
4066+-16.92458989 -6.60756054 1.14534264 2 # H
4067+-17.48851584 -4.95538862 -0.51858345 2 # H
4068+-16.92565018 -3.87958702 3.87958702 2 # H
4069+-17.49703350 -2.22300603 2.22300603 2 # H
4070+-14.25049869 -6.56976732 9.30903112 2 # H
4071+-14.73854562 -4.90661761 7.62887682 2 # H
4072+-17.48851584 -0.51858345 -4.95538862 2 # H
4073+-16.92458989 1.14534264 -6.60756054 2 # H
4074+-17.49703350 2.22300603 -2.22300603 2 # H
4075+-16.92565018 3.87958702 -3.87958702 2 # H
4076+-17.26821617 -0.85235208 0.85235208 2 # H
4077+-17.26821617 0.85235208 -0.85235208 2 # H
4078+-17.26005319 1.88119305 3.58592496 2 # H
4079+-17.26005319 3.58592496 1.88119305 2 # H
4080+-16.92458989 -1.14534264 6.60756054 2 # H
4081+-17.48851584 0.51858345 4.95538862 2 # H
4082+-14.73854562 4.90661761 -7.62887682 2 # H
4083+-14.25049869 6.56976732 -9.30903112 2 # H
4084+-14.73854562 7.62887682 -4.90661761 2 # H
4085+-14.25049869 9.30903112 -6.56976732 2 # H
4086+-17.48851584 4.95538862 0.51858345 2 # H
4087+-16.92458989 6.60756054 -1.14534264 2 # H
4088+-7.62887682 -14.73854562 4.90661761 2 # H
4089+-9.30903112 -14.25049869 6.56976732 2 # H
4090+-7.62887682 -4.90661761 14.73854562 2 # H
4091+-9.30903112 -6.56976732 14.25049869 2 # H
4092+-9.30903112 6.56976732 -14.25049869 2 # H
4093+-7.62887682 4.90661761 -14.73854562 2 # H
4094+-9.30903112 14.25049869 -6.56976732 2 # H
4095+-7.62887682 14.73854562 -4.90661761 2 # H
4096+-1.88119305 -17.26005319 -3.58592496 2 # H
4097+-3.58592496 -17.26005319 -1.88119305 2 # H
4098+-4.95538862 -17.48851584 -0.51858345 2 # H
4099+-6.60756054 -16.92458989 1.14534264 2 # H
4100+-2.22300603 -17.49703350 2.22300603 2 # H
4101+-3.87958702 -16.92565018 3.87958702 2 # H
4102+-4.90661761 -14.73854562 7.62887682 2 # H
4103+-6.56976732 -14.25049869 9.30903112 2 # H
4104+-6.56976732 -9.30903112 14.25049869 2 # H
4105+-4.90661761 -7.62887682 14.73854562 2 # H
4106+-3.58592496 -1.88119305 -17.26005319 2 # H
4107+-1.88119305 -3.58592496 -17.26005319 2 # H
4108+-2.22300603 -2.22300603 17.49703350 2 # H
4109+-3.87958702 -3.87958702 16.92565018 2 # H
4110+-6.60756054 1.14534264 -16.92458989 2 # H
4111+-4.95538862 -0.51858345 -17.48851584 2 # H
4112+-3.87958702 3.87958702 -16.92565018 2 # H
4113+-2.22300603 2.22300603 -17.49703350 2 # H
4114+-6.60756054 -1.14534264 16.92458989 2 # H
4115+-4.95538862 0.51858345 17.48851584 2 # H
4116+-1.88119305 3.58592496 17.26005319 2 # H
4117+-3.58592496 1.88119305 17.26005319 2 # H
4118+-6.56976732 9.30903112 -14.25049869 2 # H
4119+-4.90661761 7.62887682 -14.73854562 2 # H
4120+-4.90661761 14.73854562 -7.62887682 2 # H
4121+-6.56976732 14.25049869 -9.30903112 2 # H
4122+-3.87958702 16.92565018 -3.87958702 2 # H
4123+-2.22300603 17.49703350 -2.22300603 2 # H
4124+-4.95538862 17.48851584 0.51858345 2 # H
4125+-6.60756054 16.92458989 -1.14534264 2 # H
4126+-3.58592496 17.26005319 1.88119305 2 # H
4127+-1.88119305 17.26005319 3.58592496 2 # H
4128+1.14534264 -16.92458989 -6.60756054 2 # H
4129+-0.51858345 -17.48851584 -4.95538862 2 # H
4130+3.87958702 -16.92565018 -3.87958702 2 # H
4131+2.22300603 -17.49703350 -2.22300603 2 # H
4132+0.85235208 -17.26821617 -0.85235208 2 # H
4133+-0.85235208 -17.26821617 0.85235208 2 # H
4134+3.58592496 -17.26005319 1.88119305 2 # H
4135+1.88119305 -17.26005319 3.58592496 2 # H
4136+0.51858345 -17.48851584 4.95538862 2 # H
4137+-1.14534264 -16.92458989 6.60756054 2 # H
4138+-0.51858345 -4.95538862 -17.48851584 2 # H
4139+1.14534264 -6.60756054 -16.92458989 2 # H
4140+2.22300603 -2.22300603 -17.49703350 2 # H
4141+3.87958702 -3.87958702 -16.92565018 2 # H
4142+-1.14534264 -6.60756054 16.92458989 2 # H
4143+0.51858345 -4.95538862 17.48851584 2 # H
4144+3.58592496 -1.88119305 17.26005319 2 # H
4145+1.88119305 -3.58592496 17.26005319 2 # H
4146+-0.85235208 0.85235208 -17.26821617 2 # H
4147+0.85235208 -0.85235208 -17.26821617 2 # H
4148+1.88119305 3.58592496 -17.26005319 2 # H
4149+3.58592496 1.88119305 -17.26005319 2 # H
4150+-0.85235208 -0.85235208 17.26821617 2 # H
4151+0.85235208 0.85235208 17.26821617 2 # H
4152+3.87958702 3.87958702 16.92565018 2 # H
4153+2.22300603 2.22300603 17.49703350 2 # H
4154+-1.14534264 6.60756054 -16.92458989 2 # H
4155+0.51858345 4.95538862 -17.48851584 2 # H
4156+-0.51858345 4.95538862 17.48851584 2 # H
4157+1.14534264 6.60756054 16.92458989 2 # H
4158+0.51858345 17.48851584 -4.95538862 2 # H
4159+-1.14534264 16.92458989 -6.60756054 2 # H
4160+1.88119305 17.26005319 -3.58592496 2 # H
4161+3.58592496 17.26005319 -1.88119305 2 # H
4162+0.85235208 17.26821617 0.85235208 2 # H
4163+-0.85235208 17.26821617 -0.85235208 2 # H
4164+2.22300603 17.49703350 2.22300603 2 # H
4165+3.87958702 16.92565018 3.87958702 2 # H
4166+1.14534264 16.92458989 6.60756054 2 # H
4167+-0.51858345 17.48851584 4.95538862 2 # H
4168+6.56976732 -14.25049869 -9.30903112 2 # H
4169+4.90661761 -14.73854562 -7.62887682 2 # H
4170+9.30903112 -14.25049869 -6.56976732 2 # H
4171+7.62887682 -14.73854562 -4.90661761 2 # H
4172+6.60756054 -16.92458989 -1.14534264 2 # H
4173+4.95538862 -17.48851584 0.51858345 2 # H
4174+4.90661761 -7.62887682 -14.73854562 2 # H
4175+6.56976732 -9.30903112 -14.25049869 2 # H
4176+7.62887682 -4.90661761 -14.73854562 2 # H
4177+9.30903112 -6.56976732 -14.25049869 2 # H
4178+4.95538862 0.51858345 -17.48851584 2 # H
4179+6.60756054 -1.14534264 -16.92458989 2 # H
4180+4.95538862 -0.51858345 17.48851584 2 # H
4181+6.60756054 1.14534264 16.92458989 2 # H
4182+9.30903112 6.56976732 14.25049869 2 # H
4183+7.62887682 4.90661761 14.73854562 2 # H
4184+4.90661761 7.62887682 14.73854562 2 # H
4185+6.56976732 9.30903112 14.25049869 2 # H
4186+7.62887682 14.73854562 4.90661761 2 # H
4187+9.30903112 14.25049869 6.56976732 2 # H
4188+6.56976732 14.25049869 9.30903112 2 # H
4189+4.90661761 14.73854562 7.62887682 2 # H
4190+6.60756054 16.92458989 1.14534264 2 # H
4191+4.95538862 17.48851584 -0.51858345 2 # H
4192+14.73854562 -7.62887682 -4.90661761 2 # H
4193+14.25049869 -9.30903112 -6.56976732 2 # H
4194+14.25049869 -6.56976732 -9.30903112 2 # H
4195+14.73854562 -4.90661761 -7.62887682 2 # H
4196+14.25049869 9.30903112 6.56976732 2 # H
4197+14.73854562 7.62887682 4.90661761 2 # H
4198+14.73854562 4.90661761 7.62887682 2 # H
4199+14.25049869 6.56976732 9.30903112 2 # H
4200+17.49703350 -2.22300603 -2.22300603 2 # H
4201+16.92565018 -3.87958702 -3.87958702 2 # H
4202+16.92458989 -6.60756054 -1.14534264 2 # H
4203+17.48851584 -4.95538862 0.51858345 2 # H
4204+17.26005319 -1.88119305 3.58592496 2 # H
4205+17.26005319 -3.58592496 1.88119305 2 # H
4206+16.92458989 -1.14534264 -6.60756054 2 # H
4207+17.48851584 0.51858345 -4.95538862 2 # H
4208+17.26005319 3.58592496 -1.88119305 2 # H
4209+17.26005319 1.88119305 -3.58592496 2 # H
4210+17.26821617 -0.85235208 -0.85235208 2 # H
4211+17.26821617 0.85235208 0.85235208 2 # H
4212+16.92565018 3.87958702 3.87958702 2 # H
4213+17.49703350 2.22300603 2.22300603 2 # H
4214+17.48851584 -0.51858345 4.95538862 2 # H
4215+16.92458989 1.14534264 6.60756054 2 # H
4216+17.48851584 4.95538862 -0.51858345 2 # H
4217+16.92458989 6.60756054 1.14534264 2 # H
4218+%endblock atomic-coordinates-and-atomic-species
4219
4220=== added file 'Tests/si-quantum-dot/makefile'
4221--- Tests/si-quantum-dot/makefile 1970-01-01 00:00:00 +0000
4222+++ Tests/si-quantum-dot/makefile 2019-01-16 11:00:58 +0000
4223@@ -0,0 +1,7 @@
4224+#
4225+# Single-test makefile
4226+#
4227+name=si-quantum-dot
4228+EXTRAFILES=coords.Si987H372.fdf
4229+#
4230+include ../test.mk
4231
4232=== added file 'Tests/si-quantum-dot/si-quantum-dot.fdf'
4233--- Tests/si-quantum-dot/si-quantum-dot.fdf 1970-01-01 00:00:00 +0000
4234+++ Tests/si-quantum-dot/si-quantum-dot.fdf 2019-01-16 11:00:58 +0000
4235@@ -0,0 +1,39 @@
4236+use-tree-timer T
4237+SolutionMethod ELSI
4238+elsi-solver elpa
4239+
4240+WriteDMHS.NetCDF F
4241+WriteDM.NetCDF F
4242+WriteDM T
4243+
4244+SystemName Si Quantum Dot
4245+SystemLabel si-quantum-dot
4246+
4247+%block ChemicalSpeciesLabel
4248+ 1 14 Si
4249+ 2 1 H
4250+%endblock ChemicalSpeciesLabel
4251+
4252+# CSIRO quantum dot
4253+%include coords.Si987H372.fdf
4254+
4255+PAO.BasisSize SZ
4256+PAO.EnergyShift 300 meV
4257+
4258+#
4259+LatticeConstant 45.0 Ang
4260+%block LatticeVectors
4261+ 1.000 0.000 0.000
4262+ 0.000 1.000 0.000
4263+ 0.000 0.000 1.000
4264+%endblock LatticeVectors
4265+
4266+MeshCutoff 150.0 Ry
4267+
4268+MaxSCFIterations 40
4269+DM.MixingWeight 0.3
4270+DM.NumberPulay 4
4271+DM.Tolerance 1.d-3
4272+DM.UseSaveDM
4273+
4274+ElectronicTemperature 25 meV
4275
4276=== added file 'Tests/si-quantum-dot/si-quantum-dot.pseudos'
4277--- Tests/si-quantum-dot/si-quantum-dot.pseudos 1970-01-01 00:00:00 +0000
4278+++ Tests/si-quantum-dot/si-quantum-dot.pseudos 2019-01-16 11:00:58 +0000
4279@@ -0,0 +1,1 @@
4280+Si H
4281
4282=== added directory 'Tests/sih-elsi'
4283=== added file 'Tests/sih-elsi/makefile'
4284--- Tests/sih-elsi/makefile 1970-01-01 00:00:00 +0000
4285+++ Tests/sih-elsi/makefile 2019-01-16 11:00:58 +0000
4286@@ -0,0 +1,6 @@
4287+#
4288+# Single-test makefile
4289+#
4290+name=sih-elsi
4291+#
4292+include ../test.mk
4293
4294=== added file 'Tests/sih-elsi/sih-elsi.fdf'
4295--- Tests/sih-elsi/sih-elsi.fdf 1970-01-01 00:00:00 +0000
4296+++ Tests/sih-elsi/sih-elsi.fdf 2019-01-16 11:00:58 +0000
4297@@ -0,0 +1,121 @@
4298+# -----------------------------------------------------------------------------
4299+# FDF for interstitial H in a cubic c-Si supercell with 64 atoms
4300+#
4301+# E. Artacho, April 1999
4302+# -----------------------------------------------------------------------------
4303+
4304+#-------
4305+## compute-forces F
4306+SolutionMethod ELSI
4307+#MPI.Nprocs.SIESTA 1
4308+
4309+WriteDMHS.NetCDF F
4310+WriteDM.NetCDF F
4311+WriteDM T
4312+
4313+SystemName H in 64-atom silicon - ELSI solver
4314+SystemLabel sih-elsi
4315+
4316+NumberOfAtoms 65
4317+NumberOfSpecies 2
4318+
4319+%block ChemicalSpeciesLabel
4320+ 1 14 Si
4321+ 2 1 H
4322+%endblock ChemicalSpeciesLabel
4323+
4324+PAO.BasisSize DZP
4325+PAO.EnergyShift 300 meV
4326+
4327+%block PAO.Basis
4328+ H 1
4329+ 0 1
4330+ 0.
4331+ 0.8
4332+%endblock PAO.Basis
4333+
4334+LatticeConstant 5.430 Ang
4335+%block LatticeVectors
4336+ 2.000 0.000 0.000
4337+ 0.000 2.000 0.000
4338+ 0.000 0.000 2.000
4339+%endblock LatticeVectors
4340+
4341+MeshCutoff 100.0 Ry
4342+
4343+MaxSCFIterations 20
4344+DM.MixingWeight 0.3
4345+DM.NumberPulay 4
4346+DM.Tolerance 1.d-3
4347+DM.UseSaveDM
4348+
4349+ElectronicTemperature 25 meV
4350+
4351+AtomicCoordinatesFormat ScaledCartesian
4352+%block AtomicCoordinatesAndAtomicSpecies
4353+ 0.000 0.000 0.000 1 # Si 1
4354+ 0.250 0.250 0.250 1 # Si 2
4355+ 0.000 0.500 0.500 1 # Si 3
4356+ 0.250 0.750 0.750 1 # Si 4
4357+ 0.500 0.000 0.500 1 # Si 5
4358+ 0.750 0.250 0.750 1 # Si 6
4359+ 0.500 0.500 0.000 1 # Si 7
4360+ 0.750 0.750 0.250 1 # Si 8
4361+ 1.000 0.000 0.000 1 # Si 9
4362+ 1.250 0.250 0.250 1 # Si 10
4363+ 1.000 0.500 0.500 1 # Si 11
4364+ 1.250 0.750 0.750 1 # Si 12
4365+ 1.500 0.000 0.500 1 # Si 13
4366+ 1.750 0.250 0.750 1 # Si 14
4367+ 1.500 0.500 0.000 1 # Si 15
4368+ 1.750 0.750 0.250 1 # Si 16
4369+ 0.000 1.000 0.000 1 # Si 17
4370+ 0.250 1.250 0.250 1 # Si 18
4371+ 0.000 1.500 0.500 1 # Si 19
4372+ 0.250 1.750 0.750 1 # Si 20
4373+ 0.500 1.000 0.500 1 # Si 21
4374+ 0.750 1.250 0.750 1 # Si 22
4375+ 0.500 1.500 0.000 1 # Si 23
4376+ 0.750 1.750 0.250 1 # Si 24
4377+ 0.000 0.000 1.000 1 # Si 25
4378+ 0.250 0.250 1.250 1 # Si 26
4379+ 0.000 0.500 1.500 1 # Si 27
4380+ 0.250 0.750 1.750 1 # Si 28
4381+ 0.500 0.000 1.500 1 # Si 29
4382+ 0.750 0.250 1.750 1 # Si 30
4383+ 0.500 0.500 1.000 1 # Si 31
4384+ 0.750 0.750 1.250 1 # Si 32
4385+ 1.000 1.000 0.000 1 # Si 33
4386+ 1.250 1.250 0.250 1 # Si 34
4387+ 1.000 1.500 0.500 1 # Si 35
4388+ 1.250 1.750 0.750 1 # Si 36
4389+ 1.500 1.000 0.500 1 # Si 37
4390+ 1.750 1.250 0.750 1 # Si 38
4391+ 1.500 1.500 0.000 1 # Si 39
4392+ 1.750 1.750 0.250 1 # Si 40
4393+ 1.000 0.000 1.000 1 # Si 41
4394+ 1.250 0.250 1.250 1 # Si 42
4395+ 1.000 0.500 1.500 1 # Si 43
4396+ 1.250 0.750 1.750 1 # Si 44
4397+ 1.500 0.000 1.500 1 # Si 45
4398+ 1.750 0.250 1.750 1 # Si 46
4399+ 1.500 0.500 1.000 1 # Si 47
4400+ 1.750 0.750 1.250 1 # Si 48
4401+ 0.000 1.000 1.000 1 # Si 49
4402+ 0.250 1.250 1.250 1 # Si 50
4403+ 0.000 1.500 1.500 1 # Si 51
4404+ 0.250 1.750 1.750 1 # Si 52
4405+ 0.500 1.000 1.500 1 # Si 53
4406+ 0.750 1.250 1.750 1 # Si 54
4407+ 0.500 1.500 1.000 1 # Si 55
4408+ 0.750 1.750 1.250 1 # Si 56
4409+ 1.000 1.000 1.000 1 # Si 57
4410+ 1.250 1.250 1.250 1 # Si 58
4411+ 1.000 1.500 1.500 1 # Si 59
4412+ 1.250 1.750 1.750 1 # Si 60
4413+ 1.500 1.000 1.500 1 # Si 61
4414+ 1.750 1.250 1.750 1 # Si 62
4415+ 1.500 1.500 1.000 1 # Si 63
4416+ 1.750 1.750 1.250 1 # Si 64
4417+ 1.125 1.125 1.125 2 # H 65
4418+%endblock AtomicCoordinatesAndAtomicSpecies
4419
4420=== added file 'Tests/sih-elsi/sih-elsi.pseudos'
4421--- Tests/sih-elsi/sih-elsi.pseudos 1970-01-01 00:00:00 +0000
4422+++ Tests/sih-elsi/sih-elsi.pseudos 2019-01-16 11:00:58 +0000
4423@@ -0,0 +1,1 @@
4424+Si H
4425
4426=== modified file 'version.info'
4427--- version.info 2019-01-12 22:35:38 +0000
4428+++ version.info 2019-01-16 11:00:58 +0000
4429@@ -1,2 +1,7 @@
4430+<<<<<<< TREE
4431 trunk-755
4432
4433+=======
4434+trunk-710--elsi-dm-695
4435+
4436+>>>>>>> MERGE-SOURCE

Subscribers

People subscribed via source and target branches