Merge lp:~garsua/siesta/trunk-elsi-dm into lp:siesta
- trunk-elsi-dm
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~garsua/siesta/trunk-elsi-dm |
Merge into: | lp:siesta |
Diff against target: |
4436 lines (+3744/-70) (has conflicts) 32 files modified
Docs/siesta.tex (+119/-2) Src/MPI/mpi_siesta.F90 (+18/-1) Src/Makefile (+26/-18) Src/compute_dm.F (+25/-1) Src/dhscf.F (+10/-0) Src/fold_auxcell.f90 (+85/-0) Src/fsiesta_mpi.F90 (+4/-0) Src/initparallel.F (+1/-1) Src/m_elsi_interface.F90 (+1601/-0) Src/m_ncdf_siesta.F90 (+3/-0) Src/m_pexsi_dos.F90 (+1/-1) Src/m_pexsi_driver.F90 (+1/-1) Src/m_pexsi_local_dos.F90 (+1/-1) Src/m_redist_spmatrix.F90 (+187/-21) Src/parallel.F (+1/-1) Src/post_scf_work.F (+33/-2) Src/read_options.F90 (+11/-0) Src/siesta.F (+9/-9) Src/siesta_forces.F90 (+30/-5) Src/siesta_init.F (+23/-5) Src/siesta_options.F90 (+1/-0) Src/siesta_tddft.F90 (+1/-1) Src/state_init.F (+4/-0) Src/write_subs.F (+1/-0) Tests/si-quantum-dot/coords.Si987H372.fdf (+1368/-0) Tests/si-quantum-dot/makefile (+7/-0) Tests/si-quantum-dot/si-quantum-dot.fdf (+39/-0) Tests/si-quantum-dot/si-quantum-dot.pseudos (+1/-0) Tests/sih-elsi/makefile (+6/-0) Tests/sih-elsi/sih-elsi.fdf (+121/-0) Tests/sih-elsi/sih-elsi.pseudos (+1/-0) version.info (+5/-0) Text conflict in Src/siesta_forces.F90 Text conflict in version.info |
To merge this branch: | bzr merge lp:~garsua/siesta/trunk-elsi-dm |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Alberto Garcia | Needs Resubmitting | ||
Review via email: mp+361820@code.launchpad.net |
Commit message
Cosmetic changes on m_elsi_
- Removed "use fdf, only: fdf_get" from elsi_real_solver and elsi_complex_solver subroutines
- Unindented lines from 401 to 510 (in elsi_real_solver) and from 1351 to 1460 (in elsi_complex_
Description of the change
Victor GS (garsua) wrote : | # |
Gracias Alberto, se me pasó incluirlo. Vuelvo a hacer el merge request, a
ver si ahora sale bien.
On Wed, Jan 16, 2019 at 3:34 PM Alberto Garcia <email address hidden> wrote:
> Review: Resubmit
>
> Victor: As a test of the MR process it is fine, but you should specify the
> target branch as
> ~albertog/
>
> --
> https:/
> You are the owner of lp:~garsua/siesta/trunk-elsi-dm.
>
Unmerged revisions
- 696. By Victor GS
-
Cosmetic changes on m_elsi_
interface. F90
- Removed "use fdf, only: fdf_get" from elsi_real_solver and elsi_complex_solver subroutines
- Unindented lines from 401 to 510 (in elsi_real_solver) and from 1351 to 1460 (in elsi_complex_solver) - 695. By Alberto Garcia
-
Do not get the EDM during the SCF cycle + NTPoly support
* In compute_dm, the call to the ELSI ('get_dm') interface uses a
flag to disable the generation of the EDM matrix. In this case
only the DM is computed.After the scf loop, in 'post_scf_work', a further call to the ELSI
interface requests (only) the EDM.Note that this is achieved through a logical flag 'Get_EDM_Only',
which is not really appropriate. It would be better to have a
more explicit flag, such as:output_ mode="DM| EDM" The ELSI shutdown has been moved towards the end of 'siesta_forces',
and the 'Siesta_Worker' logic preserved in that routine (even though
it is not working elsewhere).* NTpoly support has been added to the interface
(Thanks to Victor M. Garcia-Suarez and Victor Yu)
- 694. By Alberto Garcia
-
Wrap MPI check on number of procs for ELSI solver
- 693. By Alberto Garcia
-
Add Si quantum dot test case for ELSI
- 692. By Alberto Garcia
-
Fix bug in computation of Delta-V for PEXSI mu bracketing
A new array is needed to properly update the value of the
change in potential.(+ added a declaration for 'date_stamp' in the auxiliary
code for version compatibility) - 691. By Alberto Garcia
-
Change the spin-reduction convention for entropy
For ELSI versions after 2018-08-17, there is no need to reduce over spins
when computing the entropy.To use older versions (such as the 2.0.2 release), use the preprocessor symbol
SIESTA_
_ELSI__ OLD_SPIN_ CONVENTION If the version is 'old', but the above symbol is not used, a compilation error will
be triggered. (This works by means of a call to 'elsi_get_datestamp' , which was added just
around the same time as the spin-reduction-convention change: Aug 17 vs Aug 21, 2018.
ELSI versions in between will not work without further intervention. - 690. By Alberto Garcia
-
Update documentation. Check that Nodes is a multiple of nk*nspin
- 689. By Alberto Garcia
-
Fix weight of DM and EDM. Working
- 688. By Alberto Garcia
-
Do not make shadow copies of elsi_global_comm
- 687. By Alberto Garcia
-
Change some ptrs to allocs. Fix communicators
Preview Diff
1 | === modified file 'Docs/siesta.tex' |
2 | --- Docs/siesta.tex 2019-01-04 07:01:47 +0000 |
3 | +++ Docs/siesta.tex 2019-01-16 11:00:58 +0000 |
4 | @@ -952,6 +952,33 @@ |
5 | LIBS += -Wl,--allow-multiple-definition -lpexsi_linux <> |
6 | \end{shellexample} |
7 | |
8 | + \item[\href{http://elsi-interchange.org}{ELSI}]% |
9 | + \index{ELSI}% |
10 | + \index{External library!ELSI}% |
11 | + The ELSI library may be used with \siesta\ for access to a range of |
12 | + solvers, see Sec.~\ref{SolverELSI}. Currently the interface is |
13 | + implemented (tested) as of ELSI version 2.0.2. If |
14 | + newer versions retain the same interface they may also be used. |
15 | + |
16 | + Add these flags to your \file{arch.make} file to enable ELSI |
17 | +\begin{shellexample} |
18 | + INCFLAGS += -I/opt/elsi/include |
19 | + LIBS += -L/opt/elsi/lib <> |
20 | + FPPFLAGS += -DSIESTA__ELSI |
21 | +\end{shellexample} |
22 | + \index{compile!pre-processor!-DSIESTA\_\_ELSI} |
23 | + where \shell{<>} represents the appropriate ELSI libraries (see the |
24 | + ELSI installation instructions). |
25 | + % |
26 | + If one experiences linker failures, one possible solution that may |
27 | + help is |
28 | +\begin{shellexample} |
29 | + LIBS += -lmpi_cxx -lstdc++ |
30 | +\end{shellexample} |
31 | + which is due to ELSI containing some parts written in C++, while the |
32 | + linking is done by the Fortran compiler. The exact library name for |
33 | + your MPI vendor may vary. |
34 | + |
35 | |
36 | \item[CheSS]% |
37 | \index{CheSS}% |
38 | @@ -6021,6 +6048,11 @@ |
39 | of the density matrix via an efficient scheme of selected |
40 | inversion (see Sec~\ref{SolverPEXSI}). |
41 | |
42 | +Additionally, \siesta\ can use the ELSI library (see |
43 | +Sec.~\ref{SolverELSI}) of solvers (which provides versions of OMM |
44 | +and PEXSI, in addition to diagonalization), and the new linear-scaling |
45 | +CheSS library (see Sec.~\ref{SolverCheSS}) |
46 | + |
47 | The calculation of the H and S matrix elements is always done with an |
48 | O(N) method. The actual scaling is not linear for small systems, but |
49 | it becomes O(N) when the system dimensions are larger than the scale |
50 | @@ -6044,8 +6076,8 @@ |
51 | Character string to choose among diagonalization (\fdf*{diagon}), |
52 | cubic-scaling minimization (\fdf*{OMM}), Order-N (\fdf*{OrderN}) |
53 | solution of the Kohn-Sham Hamiltonian, \fdf*{transiesta}, the |
54 | - PEXSI method (\fdf*{PEXSI}) or the \fdf*{CheSS} solver. |
55 | - |
56 | + PEXSI method (\fdf*{PEXSI}), ELSI solver (\fdf*{ELSI}) or the \fdf*{CheSS} solver. |
57 | + |
58 | \end{fdfentry} |
59 | |
60 | |
61 | @@ -6867,6 +6899,91 @@ |
62 | |
63 | \end{fdflogicalF} |
64 | |
65 | +\subsection{The ELSI solver family} |
66 | +\label{SolverELSI} |
67 | +\index{ELSI solvers} |
68 | + |
69 | +The ELSI library provides a unified interface to a range of solvers, |
70 | +including ELPA (diagonalization), PEXSI, OMM, and SIPS (based on spectrum |
71 | +slicing). It can be downloaded and installed freely from |
72 | +\url{http://elsi-interchange.org}. |
73 | + |
74 | +The current interface in \siesta\ is based on the ``DM'' mode in ELSI: |
75 | +sparse H and S matrices are passed to the library, and sparse DM and |
76 | +EDM matrices are returned. Internally, the library performs any extra |
77 | +operations needed. K-points and/or spin-polarization (in collinear |
78 | +form) are also supported, in fully parallel mode. This means that |
79 | +k-points/spin-polarized calculations will be carried out splitting the |
80 | +communicator for orbital distribution into nk$\times$nspin copies, |
81 | +where nk is the number of k-points and nspin the number of spin |
82 | +components. The total number of MPI processors must be a multiple of |
83 | +nk$\times$nspin. |
84 | + |
85 | +See Sec.~\ref{sec:libs} for details on installing \siesta\ with ELSI support. |
86 | + |
87 | +\subsubsection{Input parameters} |
88 | + |
89 | +The most important parameter is the solver to be used. Most other |
90 | +options are given default values by the program which are nearly |
91 | +always appropriate. |
92 | + |
93 | +\begin{fdfentry}{ELSI!Solver}[string]<ELPA> |
94 | + A specification of the solver wanted. Possible values are |
95 | + \begin{itemize} |
96 | + \item ELPA |
97 | + \item OMM |
98 | + \item PEXSI |
99 | + \item SIPS |
100 | + \end{itemize} |
101 | +\end{fdfentry} |
102 | + |
103 | +\begin{fdfentry}{ELSI!Broadening!Method}[string]<''fermi''> |
104 | +The scheme for level broadening. Valid options are ``fermi'', |
105 | +``gauss'', ``mp'' (Methfessel-Paxton), ``cold'' |
106 | +(Marzari-Vanderbilt cold smearing). |
107 | +\end{fdfentry} |
108 | + |
109 | +\begin{fdfentry}{ELSI!Output!Level}[integer]<0> |
110 | + The level of detail provided in the output. Possible values are 0, 1, |
111 | + 2, and 3. |
112 | +\end{fdfentry} |
113 | + |
114 | +\begin{fdfentry}{ELSI!Output!Json}[integer]<1> |
115 | + If not 0, a separate log file in JSON format will be written out. |
116 | +\end{fdfentry} |
117 | + |
118 | +Other fdf options are mapped in a direct way to the options described in the |
119 | +ELSI manual: |
120 | + |
121 | +\begin{fdfentry}{ELSI!Broadening!MPOrder}[integer]<1> |
122 | +\end{fdfentry} |
123 | + |
124 | +\begin{fdfentry}{ELSI!ELPA!Flavor}[integer]<2> |
125 | +\end{fdfentry} |
126 | + |
127 | +\begin{fdfentry}{ELSI!OMM!Flavor}[integer]<0> |
128 | +\end{fdfentry} |
129 | + |
130 | +\begin{fdfentry}{ELSI!OMM!ELPA!Steps}[integer]<3> |
131 | +\end{fdfentry} |
132 | + |
133 | +\begin{fdfentry}{ELSI!OMM!Tolerance}[real]<10e-9> |
134 | +\end{fdfentry} |
135 | + |
136 | +\begin{fdfentry}{ELSI!PEXSI!TasksPerPole}[integer]< {\bf no default}> |
137 | + See the discussion on the three-level PEXSI parallelism in the ELSI manual. |
138 | +\end{fdfentry} |
139 | + |
140 | +\begin{fdfentry}{ELSI!PEXSI!TasksSymbolic}[integer]<1> |
141 | +\end{fdfentry} |
142 | + |
143 | +\begin{fdfentry}{ELSI!SIPS!Slices}[integer]<{\bf no default}> |
144 | + See the discussion in the ELSI manual. |
145 | +\end{fdfentry} |
146 | + |
147 | +\begin{fdfentry}{ELSI!SIPS!ELPA!Steps}[integer]<2> |
148 | +\end{fdfentry} |
149 | + |
150 | |
151 | \subsection{The CheSS solver} |
152 | \label{SolverCheSS} |
153 | |
154 | === modified file 'Src/MPI/mpi_siesta.F90' |
155 | --- Src/MPI/mpi_siesta.F90 2016-09-16 12:25:04 +0000 |
156 | +++ Src/MPI/mpi_siesta.F90 2019-01-16 11:00:58 +0000 |
157 | @@ -37,12 +37,29 @@ |
158 | USE MPI__INCLUDE, true_MPI_Comm_World => MPI_Comm_World |
159 | #endif /* NO_MPI_INTERFACES */ |
160 | |
161 | + ! The reason to define true_MPI_Comm_World as a pointer to the |
162 | + ! (inmutable) MPI_Comm_World parameter in the MPI module is to keep |
163 | + ! it as reference. The MPI_Comm_World *variable* defined below can |
164 | + ! be redefined at will. This practice is needed because most Siesta |
165 | + ! routines have hard-wired references to MPI_Comm_World as their |
166 | + ! communicator. Instead of generalizing the interfaces and pass the |
167 | + ! communicator as an argument (work in progress), we use the kludge |
168 | + ! of re-defining MPI_Comm_World. |
169 | + |
170 | ! The following construction allows to supplant MPI_Comm_World within SIESTA, |
171 | ! and to use it as a subroutine with its own internal MPI communicator. |
172 | |
173 | + ! Siesta-instances should refer to MPI_Comm_DFT |
174 | + ! This is the case, for example, when a driver program splits the |
175 | + ! global communicator to dispatch several simultaneous Siesta instances |
176 | + ! Typically, comm_world = comm_dft, but not for calculations (PEXSI, ELSI) |
177 | + ! for which only a subset of processors carry out the core Siesta operations |
178 | + ! that have hard-wired references to mpi_comm_world |
179 | + |
180 | integer, public :: MPI_Comm_World = true_MPI_Comm_World |
181 | + integer, public :: MPI_Comm_DFT = true_MPI_Comm_World |
182 | |
183 | - public :: true_MPI_Comm_World |
184 | + public :: true_MPI_Comm_World |
185 | |
186 | |
187 | #ifdef GRID_SP |
188 | |
189 | === modified file 'Src/Makefile' |
190 | --- Src/Makefile 2018-12-06 09:30:22 +0000 |
191 | +++ Src/Makefile 2019-01-16 11:00:58 +0000 |
192 | @@ -156,6 +156,8 @@ |
193 | write_orb_indx.o \ |
194 | die.o m_pexsi.o m_pexsi_driver.o m_pexsi_dos.o m_pexsi_local_dos.o\ |
195 | m_chess.o \ |
196 | + m_elsi_interface.o \ |
197 | + fold_auxcell.o \ |
198 | m_redist_spmatrix.o \ |
199 | class_Distribution.o\ |
200 | m_dminim.o m_zminim.o \ |
201 | @@ -293,7 +295,7 @@ |
202 | "FFLAGS=$(FFLAGS:$(IPO_FLAG)=)" module ) |
203 | |
204 | #-------------------------------------------------------------- |
205 | -MS=MatrixSwitch.a |
206 | +MS?=MatrixSwitch.a |
207 | $(MS): $(MPI_INTERFACE) |
208 | (cd MatrixSwitch/src ; \ |
209 | $(MAKE) -j 1 \ |
210 | @@ -641,11 +643,11 @@ |
211 | class_TriMat.o: alloc.o intrinsic_missing.o |
212 | coceri.o: files.o periodic_table.o precision.o units.o |
213 | compute_dm.o: atomlist.o class_SpData1D.o compute_ebs_shift.o diagon.o |
214 | -compute_dm.o: iodmhs_netcdf.o kpoint_scf.o m_chess.o m_dminim.o m_energies.o |
215 | -compute_dm.o: m_eo.o m_hsx.o m_pexsi_driver.o m_rmaxh.o m_spin.o m_steps.o |
216 | -compute_dm.o: m_transiesta.o m_ts_global_vars.o m_zminim.o normalize_dm.o |
217 | -compute_dm.o: ordern.o parallel.o precision.o siesta_geom.o siesta_options.o |
218 | -compute_dm.o: sparse_matrices.o sys.o units.o |
219 | +compute_dm.o: iodmhs_netcdf.o kpoint_scf.o m_chess.o m_dminim.o |
220 | +compute_dm.o: m_elsi_interface.o m_energies.o m_eo.o m_hsx.o m_pexsi_driver.o |
221 | +compute_dm.o: m_rmaxh.o m_spin.o m_steps.o m_transiesta.o m_ts_global_vars.o |
222 | +compute_dm.o: m_zminim.o normalize_dm.o ordern.o parallel.o precision.o |
223 | +compute_dm.o: siesta_geom.o siesta_options.o sparse_matrices.o sys.o units.o |
224 | compute_ebs_shift.o: m_mpi_utils.o parallel.o precision.o |
225 | compute_energies.o: atomlist.o class_SpData1D.o class_SpData2D.o |
226 | compute_energies.o: class_SpData2D.o dhscf.o files.o m_dipol.o m_energies.o |
227 | @@ -683,11 +685,12 @@ |
228 | dfscf.o: meshphi.o parallel.o parallelsubs.o precision.o sys.o |
229 | dhscf.o: alloc.o atmfuncs.o bsc_xcmod.o cellxc_mod.o delk.o dfscf.o |
230 | dhscf.o: doping_uniform.o files.o forhar.o iogrid_netcdf.o m_charge_add.o |
231 | -dhscf.o: m_efield.o m_hartree_add.o m_iorho.o m_iotddft.o m_mesh_node.o |
232 | -dhscf.o: m_ncdf_io.o m_ncdf_siesta.o m_partial_charges.o m_rhog.o m_spin.o |
233 | -dhscf.o: m_ts_global_vars.o m_ts_hartree.o m_ts_options.o m_ts_voltage.o mesh.o |
234 | -dhscf.o: meshdscf.o meshsubs.o moremeshsubs.o parallel.o parsing.o precision.o |
235 | -dhscf.o: rhofft.o rhoofd.o siesta_options.o sys.o units.o vmat.o |
236 | +dhscf.o: m_efield.o m_elsi_interface.o m_hartree_add.o m_iorho.o m_iotddft.o |
237 | +dhscf.o: m_mesh_node.o m_ncdf_io.o m_ncdf_siesta.o m_partial_charges.o m_rhog.o |
238 | +dhscf.o: m_spin.o m_ts_global_vars.o m_ts_hartree.o m_ts_options.o |
239 | +dhscf.o: m_ts_voltage.o mesh.o meshdscf.o meshsubs.o moremeshsubs.o parallel.o |
240 | +dhscf.o: parsing.o precision.o rhofft.o rhoofd.o siesta_options.o sys.o units.o |
241 | +dhscf.o: vmat.o |
242 | diag.o: alloc.o diag_option.o parallel.o precision.o sys.o |
243 | diag2g.o: fermid.o intrinsic_missing.o parallel.o parallelsubs.o precision.o |
244 | diag2g.o: sys.o |
245 | @@ -838,6 +841,8 @@ |
246 | m_dscfcomm.o: alloc.o parallel.o parallelsubs.o precision.o schecomm.o |
247 | m_efield.o: atmfuncs.o m_cite.o m_ts_global_vars.o mesh.o parallel.o |
248 | m_efield.o: precision.o siesta_cml.o siesta_geom.o sys.o units.o |
249 | +m_elsi_interface.o: alloc.o class_Distribution.o fold_auxcell.o m_mpi_utils.o |
250 | +m_elsi_interface.o: m_redist_spmatrix.o parallel.o precision.o units.o |
251 | m_energies.o: precision.o |
252 | m_eo.o: precision.o |
253 | m_evolve.o: atomlist.o cranknic_evolg.o cranknic_evolk.o m_spin.o precision.o |
254 | @@ -1194,8 +1199,9 @@ |
255 | post_scf_work.o: atomlist.o class_Fstack_Pair_Geometry_SpData2D.o |
256 | post_scf_work.o: class_Geometry.o class_Pair_Geometry_SpData2D.o compute_dm.o |
257 | post_scf_work.o: diagon.o final_H_f_stress.o kpoint_scf.o m_dminim.o |
258 | -post_scf_work.o: m_energies.o m_eo.o m_spin.o m_steps.o m_zminim.o mneighb.o |
259 | -post_scf_work.o: parallel.o siesta_geom.o siesta_options.o sparse_matrices.o |
260 | +post_scf_work.o: m_elsi_interface.o m_energies.o m_eo.o m_spin.o m_steps.o |
261 | +post_scf_work.o: m_zminim.o mneighb.o parallel.o siesta_geom.o siesta_options.o |
262 | +post_scf_work.o: sparse_matrices.o |
263 | print_spin.o: atomlist.o m_mpi_utils.o m_spin.o parallel.o precision.o |
264 | print_spin.o: siesta_cml.o sparse_matrices.o |
265 | printmatrix.o: alloc.o |
266 | @@ -1288,11 +1294,12 @@ |
267 | siesta_forces.o: atomlist.o class_Fstack_Data1D.o class_SpData2D.o compute_dm.o |
268 | siesta_forces.o: compute_energies.o compute_max_diff.o densematrix.o |
269 | siesta_forces.o: dm_charge.o files.o final_H_f_stress.o flook_siesta.o |
270 | -siesta_forces.o: kpoint_scf.o m_check_walltime.o m_convergence.o m_energies.o |
271 | -siesta_forces.o: m_forces.o m_initwf.o m_iodm_old.o m_mixing.o m_mixing_scf.o |
272 | -siesta_forces.o: m_mpi_utils.o m_ncdf_siesta.o m_pexsi.o m_pexsi_driver.o |
273 | -siesta_forces.o: m_rhog.o m_spin.o m_steps.o m_stress.o m_transiesta.o |
274 | -siesta_forces.o: m_ts_charge.o m_ts_electype.o m_ts_global_vars.o m_ts_method.o |
275 | +siesta_forces.o: kpoint_scf.o m_check_walltime.o m_convergence.o |
276 | +siesta_forces.o: m_elsi_interface.o m_energies.o m_forces.o m_initwf.o |
277 | +siesta_forces.o: m_iodm_old.o m_mixing.o m_mixing_scf.o m_mpi_utils.o |
278 | +siesta_forces.o: m_ncdf_siesta.o m_pexsi.o m_pexsi_driver.o m_rhog.o m_spin.o |
279 | +siesta_forces.o: m_steps.o m_stress.o m_transiesta.o m_ts_charge.o |
280 | +siesta_forces.o: m_ts_electype.o m_ts_global_vars.o m_ts_method.o |
281 | siesta_forces.o: m_ts_options.o mixer.o parallel.o post_scf_work.o precision.o |
282 | siesta_forces.o: save_density_matrix.o scfconvergence_test.o setup_H0.o |
283 | siesta_forces.o: setup_hamiltonian.o siesta_cml.o siesta_dicts.o siesta_geom.o |
284 | @@ -1493,6 +1500,7 @@ |
285 | m_final_h_f_stress.o: final_H_f_stress.o |
286 | m_find_kgrid.o: find_kgrid.o |
287 | m_fire_optim.o: fire_optim.o |
288 | +m_fold_auxcell.o: fold_auxcell.o |
289 | m_forhar.o: forhar.o |
290 | m_get_kpoints_scale.o: get_kpoints_scale.o |
291 | m_gradient.o: gradient.o |
292 | |
293 | === modified file 'Src/compute_dm.F' |
294 | --- Src/compute_dm.F 2018-11-09 09:36:46 +0000 |
295 | +++ Src/compute_dm.F 2019-01-16 11:00:58 +0000 |
296 | @@ -36,6 +36,9 @@ |
297 | #ifdef SIESTA__PEXSI |
298 | use m_pexsi_solver, only: pexsi_solver |
299 | #endif |
300 | +#ifdef SIESTA__ELSI |
301 | + use m_elsi_interface, only: elsi_getdm |
302 | +#endif |
303 | use m_hsx, only : write_hs_formatted |
304 | #ifdef MPI |
305 | use mpi_siesta |
306 | @@ -75,7 +78,7 @@ |
307 | if (SIESTA_worker) call timer( 'compute_dm', 1 ) |
308 | |
309 | #ifdef MPI |
310 | - call MPI_Bcast(isolve,1,MPI_integer,0,true_MPI_Comm_World,mpierr) |
311 | + call MPI_Bcast(isolve,1,MPI_integer,0,MPI_Comm_DFT,mpierr) |
312 | #endif |
313 | |
314 | if (SIESTA_worker) then |
315 | @@ -137,6 +140,27 @@ |
316 | endif |
317 | if (.not. SIESTA_worker) RETURN |
318 | #endif |
319 | +#ifdef SIESTA__ELSI |
320 | + if (isolve .eq. SOLVE_ELSI) then |
321 | + ! This test done in node 0 since NonCol and SpOrb |
322 | + ! are not set for ELSI-solver-only processes |
323 | + if (ionode) then |
324 | + if (spin%NCol .or. spin%SO) call die( |
325 | + $ "The ELSI solver does not implement "// |
326 | + $ "non-coll spins or Spin-orbit yet") |
327 | + endif |
328 | + call elsi_getdm(iscf, no_s, spin%spinor, |
329 | + & no_l, maxnh, no_u, |
330 | + & numh, listhptr, listh, |
331 | + & H, S, qtot, temp, |
332 | + & xijo, |
333 | + & kpoint_scf%N, kpoint_scf%k, kpoint_scf%w, |
334 | + & eo, qo, Dscf, Escf, ef, Entropy, |
335 | + & occtol, neigwanted, Get_EDM_Only=.false.) |
336 | + Ecorrec = 0.0_dp |
337 | + endif |
338 | + if (.not. SIESTA_worker) RETURN |
339 | +#endif |
340 | ! Here we decide if we want to calculate one or more SCF steps by |
341 | ! diagonalization before proceeding with the OMM routine |
342 | CallDiagon=.false. |
343 | |
344 | === modified file 'Src/dhscf.F' |
345 | --- Src/dhscf.F 2019-01-04 07:01:47 +0000 |
346 | +++ Src/dhscf.F 2019-01-16 11:00:58 +0000 |
347 | @@ -694,6 +694,10 @@ |
348 | use m_ts_voltage, only: ts_voltage |
349 | use m_ts_hartree, only: ts_hartree_fix |
350 | |
351 | +#ifdef SIESTA__ELSI |
352 | + use m_elsi_interface, only: elsi_save_potential |
353 | +#endif |
354 | + |
355 | implicit none |
356 | |
357 | integer |
358 | @@ -1653,6 +1657,12 @@ |
359 | #endif |
360 | endif |
361 | |
362 | +#ifdef SIESTA__ELSI |
363 | + ! This routine, and the associated data, live |
364 | + ! in the ELSI interface module |
365 | + call elsi_save_potential(ntpl,nspin,Vscf,comm=MPI_Comm_World) |
366 | +#endif |
367 | + |
368 | C ---------------------------------------------------------------------- |
369 | C Print vacuum level |
370 | C ---------------------------------------------------------------------- |
371 | |
372 | === added file 'Src/fold_auxcell.f90' |
373 | --- Src/fold_auxcell.f90 1970-01-01 00:00:00 +0000 |
374 | +++ Src/fold_auxcell.f90 2019-01-16 11:00:58 +0000 |
375 | @@ -0,0 +1,85 @@ |
376 | +module m_fold_auxcell |
377 | + public :: fold_sparse_arrays |
378 | + private |
379 | + CONTAINS |
380 | + subroutine fold_sparse_arrays(no_l,no_u,numh,listhptr,nnz,listh, & |
381 | + numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u) |
382 | + |
383 | +! Fold-in a sparse matrix from the auxiliary supercell into the |
384 | +! unit cell (see picture) |
385 | + |
386 | + ! Number of orbitals handled by this node |
387 | + integer, intent(in) :: no_l |
388 | + ! Number of orbitals in unit cell |
389 | + integer, intent(in) :: no_u |
390 | + |
391 | + ! Number of interactions per orbital; pointer to beginning of sparse array data |
392 | + integer, intent(in) :: numh(no_l), listhptr(no_l) |
393 | + ! Total number of interactions |
394 | + integer, intent(in) :: nnz |
395 | + ! Columns of the (rectangular) supercell-based matrix |
396 | + integer, intent(in) :: listh(nnz) |
397 | + |
398 | + ! Output: All interactions are collapsed to the unit cell orbitals |
399 | + |
400 | + ! Number of interactions per orbital; pointer to beginning of sparse array data |
401 | + integer, intent(out) :: numh_u(no_l), listhptr_u(no_l) |
402 | + ! Total number of interactions |
403 | + integer, intent(out) :: nnz_u |
404 | + ! Columns of the (square) unit cell-based matrix |
405 | + integer, allocatable, intent(out) :: listh_u(:) |
406 | + ! Mapper of indexes from supercell-sparse-array to unit-cell sparse array |
407 | + integer, allocatable, intent(out) :: ind2ind_u(:) |
408 | + |
409 | + |
410 | + ! Local variables |
411 | + integer :: iuo, ind, j, jo, j_u, ind_u, juo |
412 | + integer, allocatable :: mask(:) |
413 | + |
414 | + allocate(mask(no_u)) |
415 | + |
416 | + ! Find out how many "folded" interactions there are |
417 | + nnz_u = 0 |
418 | + do iuo = 1,no_l |
419 | + mask = 0 |
420 | + do j = 1,numh(iuo) |
421 | + ind = listhptr(iuo) + j |
422 | + jo = listh(ind) |
423 | + juo = mod(jo-1,no_u) + 1 |
424 | + mask(juo) = 1 ! Found one |
425 | + enddo |
426 | + numh_u(iuo) = sum(mask) |
427 | + nnz_u = nnz_u + numh_u(iuo) |
428 | + enddo |
429 | + |
430 | + allocate(listh_u(nnz_u)) |
431 | + allocate(ind2ind_u(nnz)) |
432 | + |
433 | + ! Generate folded pointer array |
434 | + listhptr_u(1) = 0 |
435 | + do iuo = 2, no_l |
436 | + listhptr_u(iuo) = listhptr_u(iuo-1) + numh_u(iuo-1) |
437 | + enddo |
438 | + |
439 | + ! Complete the mapping |
440 | + do iuo = 1,no_l |
441 | + mask(:) = 0 |
442 | + ind_u = listhptr_u(iuo) |
443 | + ind = listhptr(iuo) |
444 | + do j = 1,numh(iuo) |
445 | + ind = ind + 1 |
446 | + jo = listh(ind) |
447 | + juo = mod(jo-1,no_u) + 1 |
448 | + if (mask(juo) > 0) then |
449 | + ! juo already seen and its place recorded |
450 | + ind2ind_u(ind) = mask(juo) |
451 | + else |
452 | + ind_u = ind_u + 1 |
453 | + listh_u(ind_u) = juo |
454 | + ind2ind_u(ind) = ind_u ! map |
455 | + mask(juo) = ind_u ! record |
456 | + endif |
457 | + enddo |
458 | + enddo |
459 | + end subroutine fold_sparse_arrays |
460 | +end module m_fold_auxcell |
461 | |
462 | === modified file 'Src/fsiesta_mpi.F90' |
463 | --- Src/fsiesta_mpi.F90 2018-12-03 22:23:36 +0000 |
464 | +++ Src/fsiesta_mpi.F90 2019-01-16 11:00:58 +0000 |
465 | @@ -197,6 +197,7 @@ |
466 | ! MPI_COMM_WORLD |
467 | use mpi_siesta, only: MPI_Comm_Siesta => MPI_Comm_World ! What siesta uses |
468 | ! as MPI_Comm_World |
469 | + use mpi_siesta, only: MPI_Comm_DFT ! Siesta-instance (dft-calc) |
470 | use mpi_siesta, only: MPI_Integer ! Integer data type |
471 | use mpi_siesta, only: MPI_Character ! Character data type |
472 | use mpi_siesta, only: MPI_Double_Precision ! Real double precision type |
473 | @@ -372,6 +373,9 @@ |
474 | // trim(label) // ' 2> /dev/null') |
475 | endif |
476 | |
477 | + ! Set the whole-siesta-instance (dft calc) communicator |
478 | + MPI_Comm_DFT = MPI_Comm_Siesta |
479 | + |
480 | #endif |
481 | |
482 | ! Initialize flags and counters |
483 | |
484 | === modified file 'Src/initparallel.F' |
485 | --- Src/initparallel.F 2018-11-23 10:32:56 +0000 |
486 | +++ Src/initparallel.F 2019-01-16 11:00:58 +0000 |
487 | @@ -16,7 +16,7 @@ |
488 | use fdf |
489 | use siesta_options, only: want_domain_decomposition |
490 | use siesta_options, only: want_spatial_decomposition |
491 | - use siesta_options, only: isolve, SOLVE_ORDERN, SOLVE_PEXSI |
492 | + use siesta_options, only: isolve, SOLVE_ORDERN |
493 | use siesta_options, only: rcoor |
494 | use parallel, only : Node, Nodes, BlockSize, ProcessorY |
495 | use parallelsubs, only : WhichNodeOrb, GetNodeOrbs |
496 | |
497 | === added file 'Src/m_elsi_interface.F90' |
498 | --- Src/m_elsi_interface.F90 1970-01-01 00:00:00 +0000 |
499 | +++ Src/m_elsi_interface.F90 2019-01-16 11:00:58 +0000 |
500 | @@ -0,0 +1,1601 @@ |
501 | +! |
502 | +! Alberto Garcia, February-July 2018 |
503 | +! Modified by Victor M. Garcia-Suarez. January 2019 |
504 | +! (Introduction of an extra flag to only get the EDM instead of carrying |
505 | +! out a full computation --- to be refactored) |
506 | +! |
507 | +! ELSI DM-based interface to Siesta. It uses the sparse matrices from Siesta, |
508 | +! and obtains the DM (and optionally the EDM) matrices in sparse form. |
509 | +! |
510 | +! The elsi_getdm routine is in principle able to perform (spin-polarized) |
511 | +! calculations for real matrices (i.e., at the Gamma point), and periodic |
512 | +! calculations for complex matrices (i.e., multiple k-points), including spin. |
513 | +! |
514 | +! The structure of the solver routine is such that it will detect when it is |
515 | +! called for the first scf step, so it can perform any needed initialization. |
516 | +! The same idea can be used for the diagonalization mode, with (more extensive) |
517 | +! appropriate changes. |
518 | +! |
519 | +! The module also exports the "elsi_finalize_scfloop" routine, to be called from |
520 | +! the appropriate place. Some variables are kept at the module level for this. |
521 | +! |
522 | +! Usage: Compile Siesta with -DSIESTA__ELSI |
523 | +! Define |
524 | +! SolutionMethod ELSI |
525 | +! in the fdf file |
526 | +! TODO: |
527 | +! - MPI.Nprocs.SIESTA is not working -- maybe we will remove that feature |
528 | +! - Add more documentation |
529 | +! - Maybe refactor a few things |
530 | +! |
531 | +module m_elsi_interface |
532 | + |
533 | +#if SIESTA__ELSI |
534 | + |
535 | + use precision, only: dp |
536 | + use elsi |
537 | + |
538 | + implicit none |
539 | + |
540 | + private |
541 | + |
542 | + integer, parameter :: ELPA_SOLVER = 1 ! solver |
543 | + integer, parameter :: OMM_SOLVER = 2 ! solver |
544 | + integer, parameter :: PEXSI_SOLVER = 3 ! solver |
545 | + integer, parameter :: SIPS_SOLVER = 5 ! solver |
546 | + integer, parameter :: NTPOLY_SOLVER = 6 ! solver |
547 | + integer, parameter :: MULTI_PROC = 1 ! parallel_mode |
548 | + integer, parameter :: SIESTA_CSC = 2 ! distribution |
549 | + integer, parameter :: GAUSSIAN = 0 ! broadening |
550 | + integer, parameter :: FERMI = 1 ! broadening |
551 | + integer, parameter :: METHFESSEL_PAXTON = 2 ! broadening |
552 | + integer, parameter :: CUBIC = 3 ! broadening |
553 | + integer, parameter :: COLD = 4 ! broadening |
554 | + integer, parameter :: ELSI_NOT_SET = -910910 |
555 | + |
556 | + type(elsi_handle) :: elsi_h |
557 | + |
558 | + integer :: elsi_global_comm ! Used by all routines. Freed at end of scf loop |
559 | + integer :: which_solver |
560 | + integer :: which_broad |
561 | + integer :: out_level |
562 | + integer :: out_json |
563 | + integer :: mp_order |
564 | + integer :: elpa_flavor |
565 | + integer :: omm_flavor |
566 | + integer :: omm_n_elpa |
567 | + integer :: pexsi_tasks_per_pole |
568 | + integer :: pexsi_tasks_symbolic |
569 | + integer :: sips_n_slice |
570 | + integer :: sips_n_elpa |
571 | + integer :: ntpoly_method |
572 | + |
573 | + real(dp) :: omm_tol |
574 | + real(dp) :: ntpoly_filter |
575 | + real(dp) :: ntpoly_tol |
576 | + |
577 | + character(len=6) :: solver_string |
578 | + character(len=5) :: broad_string |
579 | + |
580 | + real(dp), allocatable :: v_old(:,:) ! For mu update in PEXSI solver |
581 | + real(dp), allocatable :: delta_v(:,:) ! For mu update in PEXSI solver |
582 | + |
583 | + public :: elsi_getdm |
584 | + public :: elsi_finalize_scfloop |
585 | + public :: elsi_save_potential |
586 | + |
587 | +CONTAINS |
588 | + |
589 | +subroutine elsi_getdm(iscf, no_s, nspin, no_l, maxnh, no_u, & |
590 | + numh, listhptr, listh, H, S, qtot, temp, & |
591 | + xijo, nkpnt, kpoint, kweight, & |
592 | + eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only) |
593 | + |
594 | + ! |
595 | + ! Analogous to 'diagon', it dispatches ELSI solver routines as needed |
596 | + ! |
597 | + |
598 | + use m_fold_auxcell, only: fold_sparse_arrays ! Could be called in state_init |
599 | + |
600 | +! Missing for now |
601 | +! eo(no_u,nspin,nk) : Eigenvalues |
602 | +! qo(no_u,nspin,nk) : Occupations of eigenstates |
603 | + |
604 | + |
605 | + real(dp), intent(inout) :: H(:,:), S(:) ! Note: we might overwrite these |
606 | + integer, intent(in) :: iscf, maxnh, no_u, no_l, no_s, nkpnt |
607 | + integer, intent(in) :: neigwanted, nspin |
608 | + integer, intent(in) :: listh(maxnh), numh(no_l), listhptr(no_l) |
609 | + real(dp), intent(in) :: kpoint(3,nkpnt), qtot, temp, kweight(nkpnt), occtol,xijo(3,maxnh) |
610 | + |
611 | + real(dp), intent(out) :: Dscf(maxnh,nspin), ef, Escf(maxnh,nspin), Entropy |
612 | + real(dp), intent(out) :: eo(no_u,nspin,nkpnt), qo(no_u,nspin,nkpnt) |
613 | + |
614 | + ! Interim flag to just get the EDM |
615 | + ! Note that, if .true., the DM is NOT obtained |
616 | + ! This needs to be refactored |
617 | + |
618 | + logical, intent(in) :: Get_EDM_Only |
619 | + |
620 | + |
621 | + logical :: gamma, using_aux_cell |
622 | + integer, allocatable :: numh_u(:), listhptr_u(:), listh_u(:) |
623 | + integer, allocatable :: ind2ind_u(:) |
624 | + |
625 | + ! |
626 | + real(dp), allocatable, dimension(:,:) :: Dscf_u, Escf_u, H_u |
627 | + real(dp), allocatable, dimension(:) :: S_u |
628 | + |
629 | + integer :: iuo, ispin, j, ind, ind_u, nnz_u |
630 | + |
631 | + external die |
632 | + |
633 | + gamma = ((nkpnt == 1) .and. (sum(abs(kpoint(:,1))) == 0.0_dp)) |
634 | + using_aux_cell = (no_s /= no_u) |
635 | + |
636 | + if (gamma) then |
637 | + |
638 | + if (.not. using_aux_cell) then |
639 | + |
640 | + call elsi_real_solver(iscf, no_u, no_l, nspin, & |
641 | + maxnh, listhptr, listh, qtot, temp, & |
642 | + H, S, Dscf, Escf, ef, Entropy, Get_EDM_Only) |
643 | + |
644 | + else |
645 | + |
646 | + ! We fold-in the sparse arrays so that they add up all the |
647 | + ! matrix elements whose 'column' supercell index maps to the same |
648 | + ! unit-cell column. Note that they are still sparse. |
649 | + |
650 | + ! First, determine the appropriate index arrays (ending in _u), and |
651 | + ! a mapper ind2ind_u |
652 | + |
653 | + allocate(numh_u(no_l), listhptr_u(no_l)) |
654 | + call fold_sparse_arrays(no_l,no_u, & |
655 | + numh,listhptr,maxnh,listh, & |
656 | + numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u) |
657 | + |
658 | + allocate(H_u(nnz_u,nspin), S_u(nnz_u)) |
659 | + S_u = 0 |
660 | + H_u = 0 |
661 | + do iuo = 1,no_l |
662 | + do j = 1,numh(iuo) |
663 | + ind = listhptr(iuo) + j |
664 | + ind_u = ind2ind_u(ind) |
665 | + S_u(ind_u) = S_u(ind_u) + S(ind) |
666 | + do ispin = 1, nspin |
667 | + H_u(ind_u,ispin) = H_u(ind_u,ispin) + H(ind,ispin) |
668 | + enddo |
669 | + enddo |
670 | + enddo |
671 | + |
672 | + ! We can now call the standard real solver routine |
673 | + allocate(Dscf_u(nnz_u,nspin), Escf_u(nnz_u,nspin)) |
674 | + |
675 | + call elsi_real_solver(iscf, no_u, no_l, nspin, & |
676 | + nnz_u, listhptr_u, listh_u, qtot, temp, & |
677 | + H_u, S_u, Dscf_u, Escf_u, ef, Entropy, Get_EDM_Only) |
678 | + |
679 | + deallocate(H_u, S_u) |
680 | + deallocate(numh_u, listhptr_u, listh_u) |
681 | + |
682 | + |
683 | + ! Unfold. We put the same '_u' DM entry into all image slots. |
684 | + do iuo = 1,no_l |
685 | + do j = 1,numh(iuo) |
686 | + ind = listhptr(iuo) + j |
687 | + ind_u = ind2ind_u(ind) |
688 | + do ispin = 1, nspin |
689 | + Dscf(ind,ispin) = Dscf_u(ind_u,ispin) |
690 | + Escf(ind,ispin) = Escf_u(ind_u,ispin) |
691 | + enddo |
692 | + enddo |
693 | + enddo |
694 | + deallocate(Dscf_u, Escf_u) |
695 | + deallocate(ind2ind_u) |
696 | + |
697 | + endif ! using auxiliary supercell with Gamma sampling |
698 | + |
699 | + else |
700 | + |
701 | + ! We need more preparation |
702 | + call elsi_kpoints_dispatcher(iscf, no_s, nspin, no_l, maxnh, no_u, & |
703 | + numh, listhptr, listh, H, S, qtot, temp, & |
704 | + xijo, nkpnt, kpoint, kweight, & |
705 | + eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only) |
706 | + |
707 | + endif |
708 | + |
709 | +end subroutine elsi_getdm |
710 | + |
711 | +subroutine elsi_get_opts() |
712 | + |
713 | + use fdf, only: fdf_get |
714 | + |
715 | + solver_string = fdf_get("ELSISolver", "elpa") |
716 | + out_level = fdf_get("ELSIOutputLevel", 0) |
717 | + out_json = fdf_get("ELSIOutputJson", 1) |
718 | + broad_string = fdf_get("ELSIBroadeningMethod", "fermi") |
719 | + mp_order = fdf_get("ELSIBroadeningMPOrder", 1) |
720 | + elpa_flavor = fdf_get("ELSIELPAFlavor", 2) |
721 | + omm_flavor = fdf_get("ELSIOMMFlavor", 0) |
722 | + omm_n_elpa = fdf_get("ELSIOMMELPASteps", 3) |
723 | + omm_tol = fdf_get("ELSIOMMTolerance", 1.0e-9_dp) |
724 | + pexsi_tasks_per_pole = fdf_get("ELSIPEXSITasksPerPole", ELSI_NOT_SET) |
725 | + pexsi_tasks_symbolic = fdf_get("ELSIPEXSITasksSymbolic", 1) |
726 | + sips_n_slice = fdf_get("ELSISIPSSlices", ELSI_NOT_SET) |
727 | + sips_n_elpa = fdf_get("ELSISIPSELPASteps", 2) |
728 | + ntpoly_method = fdf_get("ELSINTPOLYMethod", 2) |
729 | + ntpoly_filter = fdf_get("ELSINTPOLYFilter", 1.0e-9_dp) |
730 | + ntpoly_tol = fdf_get("ELSINTPOLYTolerance", 1.0e-6_dp) |
731 | + |
732 | + select case (solver_string) |
733 | + case ("elpa", "ELPA") |
734 | + which_solver = ELPA_SOLVER |
735 | + case ("omm", "OMM") |
736 | + which_solver = OMM_SOLVER |
737 | + case ("pexsi", "PEXSI") |
738 | + which_solver = PEXSI_SOLVER |
739 | + case ("sips", "SIPS", "SIPs") |
740 | + which_solver = SIPS_SOLVER |
741 | + case ("ntpoly", "NTPOLY", "NTPoly") |
742 | + which_solver = NTPOLY_SOLVER |
743 | + case default |
744 | + which_solver = ELPA_SOLVER |
745 | + end select |
746 | + |
747 | + select case (broad_string) |
748 | + case ("gauss") |
749 | + which_broad = GAUSSIAN |
750 | + case ("fermi") |
751 | + which_broad = FERMI |
752 | + case ("mp") |
753 | + which_broad = METHFESSEL_PAXTON |
754 | + case ("cubic") |
755 | + which_broad = CUBIC |
756 | + case ("cold") |
757 | + which_broad = COLD |
758 | + case default |
759 | + which_broad = FERMI |
760 | + end select |
761 | + |
762 | +end subroutine elsi_get_opts |
763 | + |
764 | +!====================================================================================== |
765 | +! ELSI takes 1D block-cyclic distributed CSC/CSR matrices as its |
766 | +! input/output. |
767 | +! |
768 | +! But note taht this version assumes *the same* distributions for Siesta (setup_H et al) and ELSI |
769 | +! operations. |
770 | +! |
771 | +subroutine elsi_real_solver(iscf, n_basis, n_basis_l, n_spin, nnz_l, row_ptr, & |
772 | + col_idx, qtot, temp, ham, ovlp, dm, edm, ef, ets, Get_EDM_Only) |
773 | + |
774 | + use m_mpi_utils, only: globalize_sum |
775 | + use parallel, only: BlockSize |
776 | +#ifdef MPI |
777 | + use mpi_siesta |
778 | +#endif |
779 | + use class_Distribution |
780 | + use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix |
781 | + use alloc |
782 | + |
783 | + implicit none |
784 | + |
785 | + integer, intent(in) :: iscf ! SCF step counter |
786 | + integer, intent(in) :: n_basis ! Global basis |
787 | + integer, intent(in) :: n_basis_l ! Local basis |
788 | + integer, intent(in) :: n_spin |
789 | + integer, intent(in) :: nnz_l ! Local nonzero |
790 | + integer, intent(in) :: row_ptr(n_basis_l) |
791 | + integer, intent(in), target :: col_idx(nnz_l) |
792 | + real(dp), intent(in) :: qtot |
793 | + real(dp), intent(in) :: temp |
794 | + real(dp), intent(inout), target :: ham(nnz_l,n_spin) |
795 | + real(dp), intent(inout), target :: ovlp(nnz_l) |
796 | + real(dp), intent(out) :: dm(nnz_l,n_spin) |
797 | + real(dp), intent(out) :: edm(nnz_l,n_spin) |
798 | + real(dp), intent(out) :: ef ! Fermi energy |
799 | + real(dp), intent(out) :: ets ! Entropy/k, dimensionless |
800 | + |
801 | + integer :: ierr |
802 | + integer :: n_state |
803 | + integer :: nnz_g |
804 | + |
805 | + real(dp) :: energy |
806 | + |
807 | + integer, allocatable, dimension(:) :: row_ptr2 |
808 | + integer, allocatable, dimension(:), target :: numh |
809 | + |
810 | + integer :: elsi_Spatial_comm, elsi_Spin_comm |
811 | + |
812 | + type(distribution) :: dist_global |
813 | + type(distribution) :: dist_spin(2) |
814 | + |
815 | + type(aux_matrix) :: pkg_global, pkg_spin ! Packages for transfer |
816 | + |
817 | + integer :: my_spin |
818 | + |
819 | + integer :: my_no_l |
820 | + integer :: my_nnz_l |
821 | + integer :: my_nnz |
822 | + integer, pointer :: my_row_ptr2(:) => null() |
823 | + integer :: i, ih, ispin, spin_rank |
824 | + real(dp) :: ets_spin |
825 | + |
826 | + integer, pointer :: my_col_idx(:) |
827 | + real(dp), pointer :: my_S(:) |
828 | + real(dp), pointer :: my_H(:) |
829 | + real(dp), pointer :: my_DM(:) => null() |
830 | + real(dp), pointer :: my_EDM(:) => null() |
831 | + |
832 | + integer :: date_stamp |
833 | + |
834 | + logical :: Get_EDM_Only |
835 | + |
836 | + external :: timer |
837 | + |
838 | +#ifndef MPI |
839 | + call die("This ELSI solver interface needs MPI") |
840 | +#endif |
841 | + |
842 | + ! Global communicator is a duplicate of passed communicator |
843 | + call MPI_Comm_Dup(MPI_Comm_DFT, elsi_global_comm, ierr) |
844 | + |
845 | + call timer("elsi", 1) |
846 | + |
847 | + ! Initialization |
848 | + if (iscf == 1) then |
849 | + |
850 | + ! Get ELSI options |
851 | + call elsi_get_opts() |
852 | + |
853 | + ! Number of states to solve when calling an eigensolver |
854 | + n_state = min(n_basis, n_basis/2+5) |
855 | + |
856 | + ! Now we have all ingredients to initialize ELSI |
857 | + call elsi_init(elsi_h, which_solver, MULTI_PROC, SIESTA_CSC, n_basis, & |
858 | + qtot, n_state) |
859 | + |
860 | + ! Output |
861 | + call elsi_set_output(elsi_h, out_level) |
862 | + call elsi_set_output_log(elsi_h, out_json) |
863 | + call elsi_set_write_unit(elsi_h, 6) |
864 | + |
865 | + ! Broadening |
866 | + call elsi_set_mu_broaden_scheme(elsi_h, which_broad) |
867 | + call elsi_set_mu_broaden_width(elsi_h, temp) |
868 | + call elsi_set_mu_mp_order(elsi_h, mp_order) |
869 | + |
870 | + ! Solver settings |
871 | + call elsi_set_elpa_solver(elsi_h, elpa_flavor) |
872 | + |
873 | + call elsi_set_omm_flavor(elsi_h, omm_flavor) |
874 | + call elsi_set_omm_n_elpa(elsi_h, omm_n_elpa) |
875 | + call elsi_set_omm_tol(elsi_h, omm_tol) |
876 | + |
877 | + if (pexsi_tasks_per_pole /= ELSI_NOT_SET) then |
878 | + call elsi_set_pexsi_np_per_pole(elsi_h, pexsi_tasks_per_pole) |
879 | + end if |
880 | + |
881 | + call elsi_set_pexsi_np_symbo(elsi_h, pexsi_tasks_symbolic) |
882 | + call elsi_set_pexsi_temp(elsi_h, temp) |
883 | +! call elsi_set_pexsi_gap(elsi_h, gap) |
884 | +! call elsi_set_pexsi_delta_e(elsi_h, delta_e) |
885 | + call elsi_set_pexsi_mu_min(elsi_h, -10.0_dp) |
886 | + call elsi_set_pexsi_mu_max(elsi_h, 10.0_dp) |
887 | + |
888 | + if (sips_n_slice /= ELSI_NOT_SET) then |
889 | + call elsi_set_sips_n_slice(elsi_h, sips_n_slice) |
890 | + end if |
891 | + |
892 | + call elsi_set_sips_n_elpa(elsi_h, sips_n_elpa) |
893 | + call elsi_set_sips_interval(elsi_h, -10.0_dp, 10.0_dp) |
894 | + |
895 | + call elsi_set_ntpoly_method(elsi_h, ntpoly_method) |
896 | + call elsi_set_ntpoly_filter(elsi_h, ntpoly_filter) |
897 | + call elsi_set_ntpoly_tol(elsi_h, ntpoly_tol) |
898 | + |
899 | + endif ! iscf == 1 |
900 | + |
901 | + if (n_spin == 1) then |
902 | + |
903 | + ! Sparsity pattern |
904 | + call globalize_sum(nnz_l, nnz_g, comm=elsi_global_comm) |
905 | + |
906 | + allocate(row_ptr2(n_basis_l+1)) |
907 | + row_ptr2(1:n_basis_l) = row_ptr(1:n_basis_l)+1 |
908 | + row_ptr2(n_basis_l+1) = nnz_l+1 |
909 | + |
910 | + call elsi_set_csc(elsi_h, nnz_g, nnz_l, n_basis_l, col_idx, row_ptr2) |
911 | + deallocate(row_ptr2) |
912 | + |
913 | + call elsi_set_csc_blk(elsi_h, BlockSize) |
914 | + call elsi_set_mpi(elsi_h, elsi_global_comm) |
915 | + |
916 | + else |
917 | + |
918 | + ! MPI logic for spin polarization |
919 | + |
920 | + ! Re-create numh, as we use it in the transfer |
921 | + allocate(numh(n_basis_l)) |
922 | + numh(1) = row_ptr(2) |
923 | + do i = 2, n_basis_l-1 |
924 | + numh(i) = row_ptr(i+1)-row_ptr(i) |
925 | + enddo |
926 | + numh(n_basis_l) = nnz_l - row_ptr(n_basis_l) |
927 | + |
928 | + ! Split the communicator in spins and get distribution objects |
929 | + ! for the data redistribution needed |
930 | + ! Note that dist_spin is an array |
931 | + call get_spin_comms_and_dists(elsi_global_comm,elsi_global_comm, & |
932 | + blocksize, n_spin, & |
933 | + dist_global,dist_spin, elsi_spatial_comm, elsi_spin_comm) |
934 | + |
935 | + ! Find out which spin team we are in, and tag the spin we work on |
936 | + call mpi_comm_rank( elsi_Spin_Comm, spin_rank, ierr ) |
937 | + my_spin = spin_rank+1 ! {1,2} |
938 | + |
939 | + |
940 | + ! This is done serially, each time filling one spin set |
941 | + ! Note that **all processes** need to have the same pkg_global |
942 | + |
943 | + do ispin = 1, n_spin |
944 | + |
945 | + ! Load pkg_global data package |
946 | + pkg_global%norbs = n_basis |
947 | + pkg_global%no_l = n_basis_l |
948 | + pkg_global%nnzl = nnz_l |
949 | + pkg_global%numcols => numh |
950 | + pkg_global%cols => col_idx |
951 | + |
952 | + allocate(pkg_global%vals(2)) |
953 | + ! Link the vals items to the appropriate arrays (no extra memory here) |
954 | + pkg_global%vals(1)%data => ovlp(:) |
955 | + ! Note that we *cannot* say => ham(:,my_spin) |
956 | + ! and avoid the sequential loop, as then half the processors will send |
957 | + ! the information for 'spin up' and the other half the information for 'spin down', |
958 | + ! which is *not* what we want. |
959 | + pkg_global%vals(2)%data => ham(:,ispin) |
960 | + |
961 | + call timer("redist_orbs_fwd", 1) |
962 | + |
963 | + ! We are doing the transfers sequentially. One spin team is |
964 | + ! 'idle' (in the receiving side) in each pass, as the dist_spin(ispin) distribution |
965 | + ! does not involve them. |
966 | + |
967 | + call redistribute_spmatrix(n_basis,pkg_global,dist_global, & |
968 | + pkg_spin,dist_spin(ispin),elsi_global_Comm) |
969 | + |
970 | + call timer("redist_orbs_fwd", 2) |
971 | + |
972 | + if (my_spin == ispin) then ! Each team gets their own data |
973 | + |
974 | + !nrows = pkg_spin%norbs ! or simply 'norbs' |
975 | + my_no_l = pkg_spin%no_l |
976 | + my_nnz_l = pkg_spin%nnzl |
977 | + call MPI_AllReduce(my_nnz_l,my_nnz,1,MPI_integer,MPI_sum,elsi_Spatial_Comm,ierr) |
978 | + ! generate off-by-one row pointer |
979 | + call re_alloc(my_row_ptr2,1,my_no_l+1,"my_row_ptr2","elsi_solver") |
980 | + my_row_ptr2(1) = 1 |
981 | + do ih = 1,my_no_l |
982 | + my_row_ptr2(ih+1) = my_row_ptr2(ih) + pkg_spin%numcols(ih) |
983 | + enddo |
984 | + |
985 | + my_col_idx => pkg_spin%cols |
986 | + my_S => pkg_spin%vals(1)%data |
987 | + my_H => pkg_spin%vals(2)%data |
988 | + |
989 | + call re_alloc(my_DM,1,my_nnz_l,"my_DM","elsi_solver") |
990 | + call re_alloc(my_EDM,1,my_nnz_l,"my_EDM","elsi_solver") |
991 | + endif |
992 | + |
993 | + ! Clean pkg_global |
994 | + nullify(pkg_global%vals(1)%data) |
995 | + nullify(pkg_global%vals(2)%data) |
996 | + deallocate(pkg_global%vals) |
997 | + nullify(pkg_global%numcols) |
998 | + nullify(pkg_global%cols) |
999 | + |
1000 | + enddo |
1001 | + |
1002 | + call elsi_set_csc(elsi_h, my_nnz, my_nnz_l, my_no_l, my_col_idx, my_row_ptr2) |
1003 | + call de_alloc(my_row_ptr2,"my_row_ptr2","elsi_solver") |
1004 | + |
1005 | + call elsi_set_csc_blk(elsi_h, BlockSize) |
1006 | + call elsi_set_spin(elsi_h, n_spin, my_spin) |
1007 | + call elsi_set_mpi(elsi_h, elsi_Spatial_comm) |
1008 | + call elsi_set_mpi_global(elsi_h, elsi_global_comm) |
1009 | + |
1010 | + endif ! n_spin |
1011 | + |
1012 | + call timer("elsi-solver", 1) |
1013 | + |
1014 | + if (n_spin == 1) then |
1015 | + if (.not.Get_EDM_Only) then |
1016 | + call elsi_dm_real_sparse(elsi_h, ham, ovlp, DM, energy) |
1017 | + call elsi_get_entropy(elsi_h, ets) |
1018 | + else |
1019 | + call elsi_get_edm_real_sparse(elsi_h, EDM) |
1020 | + endif |
1021 | + else |
1022 | + ! Solve DM, and get (at every step for now) EDM, Fermi energy, and entropy |
1023 | + ! Energy is already summed over spins |
1024 | + if (.not.Get_EDM_Only) then |
1025 | + call elsi_dm_real_sparse(elsi_h, my_H, my_S, my_DM, energy) |
1026 | + call elsi_get_entropy(elsi_h, ets_spin) |
1027 | + else |
1028 | + call elsi_get_edm_real_sparse(elsi_h, my_EDM) |
1029 | + endif |
1030 | +#ifdef SIESTA__ELSI__OLD_SPIN_CONVENTION |
1031 | + ! NOTE** |
1032 | + ! For ELSI versions before 2018-08-17 we still need to sum the entropy over spins |
1033 | + ! ... but to figure out the version is not trivial, as the relevant routines changed... |
1034 | + call globalize_sum(ets_spin, ets, comm=elsi_Spin_comm) |
1035 | +#else |
1036 | + ! If this gives an error, then your version of ELSI is old, and you should |
1037 | + ! be using the pre-processor symbol above |
1038 | + ! (This works because 'elsi_get_datestamp' was added just |
1039 | + ! around the same time as the spin-reduction-convention change) (Aug 17 vs Aug 21, 2018...) |
1040 | + ! If you are unlucky enough to have an ELSI version in between, get rid of it. |
1041 | + call elsi_get_datestamp(date_stamp) |
1042 | +#endif |
1043 | + endif |
1044 | + |
1045 | + call elsi_get_mu(elsi_h, ef) |
1046 | + ets = ets/temp |
1047 | + |
1048 | + ! Ef, energy, and ets are known to all nodes |
1049 | + |
1050 | + call timer("elsi-solver", 2) |
1051 | + |
1052 | + if ( n_spin == 2) then |
1053 | + ! Now we need to redistribute back |
1054 | + |
1055 | + do ispin = 1, n_spin |
1056 | + |
1057 | + if (my_spin == ispin) then |
1058 | + ! Prepare pkg_spin to transfer the right spin information |
1059 | + ! The other fields (numcols, cols) are the same and are still there |
1060 | + ! Deallocate my_S and my_H |
1061 | + call de_alloc(pkg_spin%vals(1)%data,"pkg_spin%vals(1)%data","elsi_solver") |
1062 | + call de_alloc(pkg_spin%vals(2)%data,"pkg_spin%vals(2)%data","elsi_solver") |
1063 | + |
1064 | + pkg_spin%vals(1)%data => my_DM(1:my_nnz_l) |
1065 | + pkg_spin%vals(2)%data => my_EDM(1:my_nnz_l) |
1066 | + |
1067 | + endif |
1068 | + |
1069 | + ! pkg_global is clean now |
1070 | + call timer("redist_orbs_bck", 1) |
1071 | + call redistribute_spmatrix(n_basis,pkg_spin,dist_spin(ispin) & |
1072 | + ,pkg_global,dist_global,elsi_global_Comm) |
1073 | + call timer("redist_orbs_bck", 2) |
1074 | + |
1075 | + ! Clean pkg_spin |
1076 | + if (my_spin == ispin) then |
1077 | + ! Each team deallocates during "its" spin cycle |
1078 | + call de_alloc(my_DM, "my_DM", "elsi_solver") |
1079 | + call de_alloc(my_EDM,"my_EDM","elsi_solver") |
1080 | + |
1081 | + nullify(pkg_spin%vals(1)%data) ! formerly pointing to DM |
1082 | + nullify(pkg_spin%vals(2)%data) ! formerly pointing to EDM |
1083 | + deallocate(pkg_spin%vals) |
1084 | + ! allocated in the direct transfer |
1085 | + call de_alloc(pkg_spin%numcols,"pkg_spin%numcols","elsi_solver") |
1086 | + call de_alloc(pkg_spin%cols, "pkg_spin%cols", "elsi_solver") |
1087 | + endif |
1088 | + |
1089 | + |
1090 | + ! In future, pkg_global%vals(1,2) could be pointing to DM and EDM, |
1091 | + ! and the 'redistribute' routine check whether the vals arrays are |
1092 | + ! associated, to use them instead of allocating them. |
1093 | + DM(:,ispin) = pkg_global%vals(1)%data(:) |
1094 | + EDM(:,ispin) = pkg_global%vals(2)%data(:) |
1095 | + ! Check no_l |
1096 | + if (n_basis_l /= pkg_global%no_l) then |
1097 | + call die("Mismatch in no_l") |
1098 | + endif |
1099 | + ! Check listH |
1100 | + if (any(col_idx(:) /= pkg_global%cols(:))) then |
1101 | + call die("Mismatch in listH") |
1102 | + endif |
1103 | + |
1104 | + ! Clean pkg_global |
1105 | + ! allocated by the transfer routine, but we did not actually |
1106 | + ! look at them |
1107 | + call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver") |
1108 | + call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver") |
1109 | + |
1110 | + call de_alloc(pkg_global%vals(1)%data,"pkg_global%vals(1)%data","elsi_solver") |
1111 | + call de_alloc(pkg_global%vals(2)%data,"pkg_global%vals(2)%data","elsi_solver") |
1112 | + deallocate(pkg_global%vals) |
1113 | + |
1114 | + enddo |
1115 | + |
1116 | + call MPI_Comm_Free(elsi_Spatial_comm, ierr) |
1117 | + call MPI_Comm_Free(elsi_Spin_comm, ierr) |
1118 | + |
1119 | + endif |
1120 | + |
1121 | + call timer("elsi", 2) |
1122 | + |
1123 | + |
1124 | +end subroutine elsi_real_solver |
1125 | + |
1126 | +subroutine elsi_kpoints_dispatcher(iscf, no_s, nspin, no_l, maxnh, no_u, & |
1127 | + numh, listhptr, listh, H, S, qtot, temp, & |
1128 | + xijo, nkpnt, kpoint, kweight, & |
1129 | + eo, qo, Dscf, Escf, ef, Entropy, occtol, neigwanted, Get_EDM_Only) |
1130 | + |
1131 | + use mpi_siesta, only: mpi_comm_dft |
1132 | + use mpi |
1133 | + use parallel, only: blocksize |
1134 | + use class_Distribution |
1135 | + use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix |
1136 | + use alloc |
1137 | + |
1138 | + ! |
1139 | + ! K-point redistribution, Hk and Sk building, and call to complex ELSI solver |
1140 | + ! |
1141 | + |
1142 | + use m_fold_auxcell, only: fold_sparse_arrays ! Could be called in state_init |
1143 | + |
1144 | +! Missing for now |
1145 | +! eo(no_u,nspin,nk) : Eigenvalues |
1146 | +! qo(no_u,nspin,nk) : Occupations of eigenstates |
1147 | + |
1148 | + |
1149 | + real(dp), intent(in), target :: H(:,:), S(:) ! Note that now we do not change them |
1150 | + integer, intent(in) :: iscf, maxnh, no_u, no_l, no_s, nkpnt |
1151 | + integer, intent(in) :: neigwanted, nspin |
1152 | + integer, intent(in) :: listhptr(no_l) |
1153 | + integer, intent(in), target :: listh(maxnh), numh(no_l) |
1154 | + |
1155 | + real(dp), intent(out) :: Dscf(maxnh,nspin), ef, Escf(maxnh,nspin), Entropy |
1156 | + real(dp), intent(out) :: eo(no_u,nspin,nkpnt), qo(no_u,nspin,nkpnt) |
1157 | + real(dp), intent(in) :: kpoint(3,nkpnt), qtot, temp, kweight(nkpnt), occtol,xijo(3,maxnh) |
1158 | + |
1159 | + |
1160 | + integer :: mpirank, kcolrank, npGlobal |
1161 | + integer :: npPerK, color, my_kpt_n |
1162 | + |
1163 | + integer :: kpt_comm, kpt_col_comm |
1164 | + integer :: Global_Group, kpt_Group |
1165 | + |
1166 | + integer, allocatable :: ranks_in_world(:), ranks_in_world_AllK(:,:) |
1167 | + type(distribution) :: dist_global |
1168 | + type(distribution), allocatable :: dist_k(:) |
1169 | + |
1170 | + type(aux_matrix) :: pkg_global, pkg_k |
1171 | + integer :: nvals |
1172 | + real(dp), allocatable, target :: xijo_transp(:,:) |
1173 | + real(dp), allocatable :: my_xijo_transp(:,:) |
1174 | + real(dp), allocatable :: my_xij(:,:) |
1175 | + |
1176 | + integer :: my_no_l, my_nnz_l |
1177 | + integer, pointer :: my_numh(:) |
1178 | + integer, pointer :: my_listh(:) |
1179 | + integer, pointer :: my_listhptr(:) => null() |
1180 | + real(dp), allocatable :: my_S(:) |
1181 | + real(dp), allocatable :: my_H(:,:) |
1182 | + real(dp), pointer :: buffer(:) ! for unpacking help |
1183 | + |
1184 | + real(dp), allocatable :: my_Escf(:,:) |
1185 | + real(dp), allocatable :: my_Dscf(:,:) |
1186 | + real(dp), allocatable, target :: my_Escf_reduced(:,:) |
1187 | + real(dp), allocatable, target :: my_Dscf_reduced(:,:) |
1188 | + |
1189 | + complex(dp), allocatable :: DM_k(:,:) |
1190 | + complex(dp), allocatable :: EDM_k(:,:) |
1191 | + complex(dp), allocatable :: Hk(:,:) |
1192 | + complex(dp), allocatable :: Sk(:) |
1193 | + |
1194 | + integer :: iuo, j, ind, ind_u, ispin |
1195 | + real(dp) :: kxij, my_kpoint(3) |
1196 | + complex(dp) :: kphs |
1197 | + |
1198 | + integer :: i, ih, ik, ierr |
1199 | + |
1200 | + integer, allocatable :: numh_u(:), listhptr_u(:), listh_u(:), ind2ind_u(:) |
1201 | + integer :: nnz_u |
1202 | + |
1203 | + logical :: Get_EDM_Only |
1204 | + |
1205 | + external die |
1206 | + |
1207 | + ! Split the global communicator |
1208 | + ! Re-distribute H and S to the k-point (and spin) teams |
1209 | + ! Generate Hk, Sk |
1210 | + ! Call elsi_complex_solver |
1211 | + ! Construct and re-distribute global DM and EDM |
1212 | + |
1213 | + ! Global communicator is a duplicate of passed communicator |
1214 | + call MPI_Comm_Dup(MPI_Comm_DFT, elsi_global_comm, ierr) |
1215 | + call MPI_Comm_Rank(elsi_global_comm, mpirank, ierr) |
1216 | + call MPI_Comm_Size(elsi_global_comm, npGlobal, ierr) |
1217 | + |
1218 | + ! Split elsi_global_comm |
1219 | + |
1220 | + npPerK = npGlobal/nkpnt |
1221 | + |
1222 | + ! Options for split: As many groups as nkpnt, so numbering is trivial |
1223 | + color = mpirank/npPerK ! : color 0: 0,1,2,3 ; color 1: 4,5,6,7 |
1224 | + call MPI_Comm_Split(elsi_global_comm, color, mpirank, kpt_Comm, ierr) |
1225 | + my_kpt_n = 1 + color ! 1 + mpirank/npPerK |
1226 | + ! Column communicator |
1227 | + color = mod(mpirank, npPerK) ! : color 0: 0,4 1: 1,5 2: 2,6 3: 3,7 |
1228 | + call MPI_Comm_Split(elsi_global_comm, color, mpirank, kpt_col_Comm, ierr) ! OK to use mpirank: just ordering |
1229 | + call MPI_Comm_Rank(kpt_col_comm, kcolrank, ierr) |
1230 | + |
1231 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " rank in col:", kcolrank |
1232 | + |
1233 | + ! Create distribution for k-point group |
1234 | + call MPI_Comm_Group(elsi_global_comm, Global_Group, Ierr) |
1235 | + call MPI_Comm_Group(kpt_Comm, kpt_Group, Ierr) |
1236 | + allocate(ranks_in_world(npPerK)) |
1237 | + call MPI_Group_translate_ranks( kpt_Group, npPerK, & |
1238 | + (/ (i,i=0,npPerK-1) /), & |
1239 | + Global_Group, ranks_in_world, ierr ) |
1240 | + |
1241 | + call MPI_Group_Free(kpt_Group, ierr) |
1242 | + |
1243 | + allocate (ranks_in_World_AllK(npPerK,nkpnt)) |
1244 | + ! We need everybody to have this information, as all nodes |
1245 | + ! are part of the global distribution side of the communication |
1246 | + ! (global_comm is the common-address space) |
1247 | + |
1248 | + ! Use the k-column communicator |
1249 | + call MPI_AllGather(ranks_in_world,npPerK,MPI_integer,& |
1250 | + Ranks_in_World_AllK(1,1),npPerK, & |
1251 | + MPI_integer,kpt_col_Comm,ierr) |
1252 | + |
1253 | + allocate(dist_k(nkpnt)) |
1254 | + ! Create distributions known to all nodes (again, those in base_comm) |
1255 | + do ik = 1, nkpnt |
1256 | + call newDistribution(dist_k(ik), elsi_global_Comm, & |
1257 | + Ranks_in_World_AllK(:,ik), & |
1258 | + TYPE_BLOCK_CYCLIC, blockSize, "kpt dist") |
1259 | + enddo |
1260 | + deallocate(ranks_in_world,Ranks_in_World_AllK) |
1261 | + call MPI_Barrier(elsi_global_Comm,ierr) |
1262 | + |
1263 | + call newDistribution(dist_global,elsi_global_Comm, (/ (i, i=0, npGlobal-1) /), & |
1264 | + TYPE_BLOCK_CYCLIC,BlockSize,"global dist") |
1265 | + call MPI_Barrier(elsi_global_Comm,ierr) |
1266 | + |
1267 | + ! Redistribute arrays |
1268 | + |
1269 | + ! No need to go sequentially over k-points |
1270 | + ! Load pkg_global data package |
1271 | + pkg_global%norbs = no_u |
1272 | + pkg_global%no_l = no_l |
1273 | + pkg_global%nnzl = maxnh |
1274 | + pkg_global%numcols => numh |
1275 | + pkg_global%cols => listh |
1276 | + |
1277 | + nvals = 1 + 3 + nspin ! S, xijo (tranposed), H(:,nspin) |
1278 | + allocate(pkg_global%vals(nvals)) |
1279 | + ! Link the vals items to the appropriate arrays (no extra memory here) |
1280 | + pkg_global%vals(1)%data => S(:) |
1281 | + call transpose(xijo,xijo_transp) |
1282 | + do i = 1, 3 |
1283 | + pkg_global%vals(1+i)%data => xijo_transp(:,i) |
1284 | + enddo |
1285 | + do ispin = 1, nspin |
1286 | + pkg_global%vals(4+ispin)%data => H(:,ispin) |
1287 | + enddo |
1288 | + |
1289 | + ! Do sequential |
1290 | + do ik=1, nkpnt |
1291 | + |
1292 | + call timer("redist_orbs_fwd", 1) |
1293 | + call redistribute_spmatrix(no_u,pkg_global,dist_global, & |
1294 | + pkg_k,dist_k(ik),elsi_global_Comm) |
1295 | + call timer("redist_orbs_fwd", 2) |
1296 | + |
1297 | + if (my_kpt_n == ik) then |
1298 | + !------------------------------------------ |
1299 | + ! Unpack info: real S and H (and index arrays) distributed over each kpt_comm |
1300 | + my_no_l = pkg_k%no_l |
1301 | + my_nnz_l = pkg_k%nnzl |
1302 | + my_numh => pkg_k%numcols |
1303 | + |
1304 | + my_listh => pkg_k%cols |
1305 | + |
1306 | + allocate(my_S(my_nnz_l)) |
1307 | + my_S(:) = pkg_k%vals(1)%data(:) |
1308 | + call de_alloc(pkg_k%vals(1)%data) |
1309 | + |
1310 | + allocate(my_xijo_transp(my_nnz_l,3)) |
1311 | + do i = 1, 3 |
1312 | + buffer => pkg_k%vals(1+i)%data |
1313 | + my_xijo_transp(:,i) = buffer(:) |
1314 | + call de_alloc(pkg_k%vals(1+i)%data) |
1315 | + enddo |
1316 | + |
1317 | + allocate(my_H(my_nnz_l,nspin)) |
1318 | + do ispin = 1, nspin |
1319 | + buffer => pkg_k%vals(4+ispin)%data |
1320 | + my_H(:,ispin) = buffer(:) |
1321 | + call de_alloc(pkg_k%vals(4+ispin)%data) |
1322 | + enddo |
1323 | + deallocate(pkg_k%vals) |
1324 | + |
1325 | + ! Now we could clear the rest of pkg_k-- |
1326 | + ! nullify(pkg_k%numcols) |
1327 | + ! nullify(pkg_k%cols) |
1328 | + ! |
1329 | + ! but we need to remember to deallocate the actual arrays after use |
1330 | + ! it is probably safer to keep the pkg references |
1331 | + !--------------------------- |
1332 | + |
1333 | + endif |
1334 | + enddo !ik |
1335 | + |
1336 | + ! Clean pkg_global -- This is safe, as we use pointers to data only |
1337 | + do i = 1, size(pkg_global%vals) |
1338 | + nullify(pkg_global%vals(i)%data) |
1339 | + enddo |
1340 | + deallocate(pkg_global%vals) |
1341 | + nullify(pkg_global%numcols) |
1342 | + nullify(pkg_global%cols) |
1343 | + |
1344 | + deallocate(xijo_transp) ! Auxiliary array used for sending |
1345 | + |
1346 | + ! generate listhptr for folding/unfolding operations |
1347 | + call re_alloc(my_listhptr,1,my_no_l,"my_listhptr","elsi_solver") |
1348 | + my_listhptr(1) = 0 |
1349 | + do ih = 2,my_no_l |
1350 | + my_listhptr(ih) = my_listhptr(ih-1) + my_numh(ih-1) |
1351 | + enddo |
1352 | + |
1353 | + ! The folded variables (*_u) are local to each team |
1354 | + |
1355 | + allocate(numh_u(my_no_l), listhptr_u(my_no_l)) |
1356 | + call fold_sparse_arrays(my_no_l,no_u, & |
1357 | + my_numh,my_listhptr,my_nnz_l,my_listh, & |
1358 | + numh_u,listhptr_u,nnz_u,listh_u,ind2ind_u) |
1359 | + |
1360 | + ! |
1361 | + call transpose(my_xijo_transp,my_xij) |
1362 | + deallocate(my_xijo_transp) |
1363 | + |
1364 | + my_kpoint(:) = kpoint(:,my_kpt_n) |
1365 | + |
1366 | + allocate(Hk(nnz_u,nspin), Sk(nnz_u)) |
1367 | + Sk = 0 |
1368 | + Hk = 0 |
1369 | + do iuo = 1,my_no_l |
1370 | + do j = 1,my_numh(iuo) |
1371 | + ind = my_listhptr(iuo) + j |
1372 | + ind_u = ind2ind_u(ind) |
1373 | + |
1374 | + kxij = my_kpoint(1) * my_xij(1,ind) + & |
1375 | + my_kpoint(2) * my_xij(2,ind) + & |
1376 | + my_kpoint(3) * my_xij(3,ind) |
1377 | + kphs = cdexp(dcmplx(0.0_dp, -1.0_dp)*kxij) |
1378 | + |
1379 | + Sk(ind_u) = Sk(ind_u) + my_S(ind) *kphs |
1380 | + do ispin = 1, nspin |
1381 | + Hk(ind_u,ispin) = Hk(ind_u,ispin) + my_H(ind,ispin) *kphs |
1382 | + enddo |
1383 | + enddo |
1384 | + enddo |
1385 | + |
1386 | + deallocate(my_S,my_H) |
1387 | + ! |
1388 | + |
1389 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done folding" |
1390 | + ! Prepare arrays for holding results |
1391 | + allocate(DM_k(nnz_u,nspin)) |
1392 | + allocate(EDM_k(nnz_u,nspin)) |
1393 | + |
1394 | + call elsi_complex_solver(iscf, no_u, my_no_l, nspin, nnz_u, numh_u, listhptr_u, & |
1395 | + listh_u, qtot, temp, Hk, Sk, DM_k, EDM_k, Ef, Entropy, & |
1396 | + nkpnt, my_kpt_n, kpoint(:,my_kpt_n), kweight(my_kpt_n), & |
1397 | + kpt_Comm, Get_EDM_Only ) |
1398 | + |
1399 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done elsi_complex_solver" |
1400 | + deallocate(listhptr_u, numh_u, listh_u) |
1401 | + deallocate(Hk,Sk) |
1402 | + |
1403 | + ! Re-create DM and EDM: |
1404 | + ! Unfold within a given k |
1405 | + ! Add up all the k contributions with the appropriate phases |
1406 | + |
1407 | + ! Prepare arrays for holding results: Note sizes: these are folded out |
1408 | + allocate(my_Dscf(my_nnz_l,nspin)) |
1409 | + allocate(my_Escf(my_nnz_l,nspin)) |
1410 | + |
1411 | + do iuo = 1, my_no_l |
1412 | + do j = 1, my_numh(iuo) |
1413 | + ind = my_listhptr(iuo) + j |
1414 | + ind_u = ind2ind_u(ind) |
1415 | + |
1416 | + kxij = my_kpoint(1) * my_xij(1,ind) + & |
1417 | + my_kpoint(2) * my_xij(2,ind) + & |
1418 | + my_kpoint(3) * my_xij(3,ind) |
1419 | + kphs = cdexp(dcmplx(0.0_dp, +1.0_dp)*kxij) |
1420 | + |
1421 | + do ispin = 1, nspin |
1422 | + my_Dscf(ind,ispin) = real( DM_k(ind_u,ispin) * kphs, kind=dp ) |
1423 | + my_Escf(ind,ispin) = real( EDM_k(ind_u,ispin) * kphs, kind=dp ) |
1424 | + enddo |
1425 | + enddo |
1426 | + enddo |
1427 | + |
1428 | + ! Apply the k-point weight to the (apparently normalized) DM and EDM that |
1429 | + ! come out of ELSI |
1430 | + |
1431 | + my_Dscf = kweight(my_kpt_n) * my_Dscf |
1432 | + my_Escf = kweight(my_kpt_n) * my_Escf |
1433 | + |
1434 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done generating my_Dscf" |
1435 | + deallocate(my_xij) |
1436 | + deallocate(ind2ind_u) |
1437 | + deallocate(DM_k,EDM_k) |
1438 | + |
1439 | + if (my_kpt_n == 1 ) then |
1440 | + ! Prepare arrays for holding reduced data |
1441 | + allocate(my_Dscf_reduced(my_nnz_l,nspin)) |
1442 | + allocate(my_Escf_reduced(my_nnz_l,nspin)) |
1443 | + if (kcolrank /= 0) call die("Rank 0 in kpt_comm not doing kpt 1") |
1444 | + else |
1445 | + ! These should not be referenced |
1446 | + allocate(my_Dscf_reduced(1,1)) |
1447 | + allocate(my_Escf_reduced(1,1)) |
1448 | + endif |
1449 | + |
1450 | + ! Use k-point column communicator, and reduce to rank 0, |
1451 | + ! which *should* correspond to kpt=1... (checked above) |
1452 | + call MPI_Reduce( my_Dscf, my_Dscf_reduced, nspin*my_nnz_l, MPI_Double_Precision, & |
1453 | + MPI_Sum, 0, kpt_col_comm, ierr ) |
1454 | + call MPI_Reduce( my_Escf, my_Escf_reduced, nspin*my_nnz_l, MPI_Double_Precision, & |
1455 | + MPI_Sum, 0, kpt_col_comm, ierr ) |
1456 | + |
1457 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done reducing my_Dscf" |
1458 | + deallocate(my_Dscf, my_Escf) |
1459 | + |
1460 | + ! redistribute to global distribution, only from the first k-point |
1461 | + |
1462 | + if (my_kpt_n == 1) then |
1463 | + |
1464 | + ! These bits are still there *** update if we ever clean pkgs after use |
1465 | + ! pkg_k%norbs = no_u |
1466 | + ! pkg_k%no_l = my_no_l |
1467 | + ! pkg_k%nnzl = my_nnz_l |
1468 | + ! pkg_k%numcols => my_numh |
1469 | + ! pkg_k%cols => my_listh |
1470 | + |
1471 | + nvals = 2*nspin ! DM, EDM |
1472 | + allocate(pkg_k%vals(nvals)) |
1473 | + do ispin = 1, nspin |
1474 | + pkg_k%vals(ispin)%data => my_Dscf_reduced(:,ispin) |
1475 | + pkg_k%vals(nspin+ispin)%data => my_Escf_reduced(:,ispin) |
1476 | + enddo |
1477 | + |
1478 | + endif |
1479 | + |
1480 | + ! Everybody participates in the transfer in the receiving side, but |
1481 | + ! only kpt=1 in the sending side, because we are using dist_k(1) |
1482 | + call timer("redist_dm-edm_bck", 1) |
1483 | + call redistribute_spmatrix(no_u,pkg_k,dist_k(1), & |
1484 | + pkg_global,dist_global,elsi_global_Comm) |
1485 | + call timer("redist_dm-edm_bck", 2) |
1486 | + |
1487 | + ! Deallocate aux arrays |
1488 | + deallocate(my_Dscf_reduced, my_Escf_reduced) |
1489 | + if (my_kpt_n == 1) then |
1490 | + !Clean pkg_k in sender |
1491 | + deallocate(pkg_k%vals) ! just pointers |
1492 | + endif |
1493 | + |
1494 | + !Clean all pkg_k: These were actually allocated in the forward transfer |
1495 | + call de_alloc(pkg_k%numcols,"pkg_k%numcols") |
1496 | + call de_alloc(pkg_k%cols,"pkg_k%cols") |
1497 | + |
1498 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done cleaning pkg_k" |
1499 | + |
1500 | + ! Unpack data (all processors, original global distribution) |
1501 | + ! In future, pkg_global%vals(:) could be pointing to DM and EDM, |
1502 | + ! and the 'redistribute' routine check whether the vals arrays are |
1503 | + ! associated, to use them instead of allocating them. |
1504 | + do ispin = 1, nspin |
1505 | + Dscf(:,ispin) = pkg_global%vals(ispin)%data(:) |
1506 | + Escf(:,ispin) = pkg_global%vals(nspin+ispin)%data(:) |
1507 | + enddo |
1508 | + ! Check no_l |
1509 | + if (no_l /= pkg_global%no_l) then |
1510 | + call die("Mismatch in no_l at end of kpoints-dispatcher") |
1511 | + endif |
1512 | + ! Check listH |
1513 | + if (any(listh(:) /= pkg_global%cols(:))) then |
1514 | + call die("Mismatch in listH at end of kpoints-dispatcher") |
1515 | + endif |
1516 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done Dscf" |
1517 | + |
1518 | + ! Clean pkg_global |
1519 | + call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver") |
1520 | + call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver") |
1521 | + do i = 1, size(pkg_global%vals) |
1522 | + call de_alloc(pkg_global%vals(i)%data,"pkg_global%vals%data","kpoints_dispatcher") |
1523 | + enddo |
1524 | + deallocate(pkg_global%vals) |
1525 | + !print *, mpirank, "| ", "k-point ", my_kpt_n, " Done cleaning pkg_global" |
1526 | + |
1527 | + ! Reduction of entropy over kpt_col_comm is not necessary |
1528 | + |
1529 | + call MPI_Comm_Free(kpt_comm, ierr) |
1530 | + call MPI_Comm_Free(kpt_col_comm, ierr) |
1531 | + |
1532 | +end subroutine elsi_kpoints_dispatcher |
1533 | + |
1534 | +! Clean up: Finalize ELSI instance and free MPI communicator. |
1535 | +! |
1536 | +subroutine elsi_finalize_scfloop() |
1537 | + |
1538 | + integer :: ierr |
1539 | + |
1540 | + ! Make which_solver invalid |
1541 | + which_solver = ELSI_NOT_SET |
1542 | + |
1543 | + if (allocated(v_old)) then |
1544 | + deallocate(v_old) |
1545 | + deallocate(delta_v) |
1546 | + end if |
1547 | + |
1548 | + call elsi_finalize(elsi_h) |
1549 | + |
1550 | + call MPI_Comm_Free(elsi_global_comm, ierr) |
1551 | + |
1552 | +end subroutine elsi_finalize_scfloop |
1553 | + |
1554 | +! Save Hartree + XC potential and find minimum and maximum change of it between |
1555 | +! two SCF iterations. |
1556 | +! |
1557 | +subroutine elsi_save_potential(n_pts, n_spin, v_scf, comm) |
1558 | + |
1559 | + use m_mpi_utils, only: globalize_min, globalize_max |
1560 | + use parallel, only: ionode, node |
1561 | + use units, only: eV |
1562 | + |
1563 | + integer, intent(in) :: n_pts |
1564 | + integer, intent(in) :: n_spin |
1565 | + real(dp), intent(in) :: v_scf(n_pts,n_spin) |
1566 | + integer , intent(in) :: comm ! The Siesta communicator used for grid operations |
1567 | + ! since this routine is called from dhscf... |
1568 | + |
1569 | + real(dp) :: mu_min |
1570 | + real(dp) :: mu_max |
1571 | + real(dp) :: dv_min |
1572 | + real(dp) :: dv_max |
1573 | + real(dp) :: tmp |
1574 | + |
1575 | + if (which_solver == PEXSI_SOLVER) then |
1576 | + if (.not. allocated(v_old)) then |
1577 | + allocate(v_old(n_pts,n_spin)) |
1578 | + allocate(delta_v(n_pts,n_spin)) |
1579 | + |
1580 | + v_old = v_scf |
1581 | + |
1582 | + mu_min = -10.0_dp |
1583 | + mu_max = 10.0_dp |
1584 | + else |
1585 | + call elsi_get_pexsi_mu_min(elsi_h, mu_min) |
1586 | + call elsi_get_pexsi_mu_max(elsi_h, mu_max) |
1587 | + if (ionode) then |
1588 | + print *, "*-- solver mu_min, mu_max:", mu_min/eV, mu_max/eV |
1589 | + endif |
1590 | + |
1591 | + delta_v = v_scf - v_old |
1592 | + v_old = v_scf |
1593 | + |
1594 | + ! Get minimum and maximum of change of total potential |
1595 | + tmp = minval(delta_v) |
1596 | + |
1597 | + call globalize_min(tmp, dv_min, comm=comm) |
1598 | + |
1599 | + tmp = maxval(delta_v) |
1600 | + |
1601 | + call globalize_max(tmp, dv_max, comm=comm) |
1602 | + if (ionode) then |
1603 | + print *, " *---- min and max Delta-V: ", dv_min/eV, dv_max/eV |
1604 | + endif |
1605 | + |
1606 | + mu_min = mu_min+dv_min |
1607 | + mu_max = mu_max+dv_max |
1608 | + end if |
1609 | + |
1610 | + ! Adjust chemical potential range for PEXSI |
1611 | + call elsi_set_pexsi_mu_min(elsi_h, mu_min) |
1612 | + call elsi_set_pexsi_mu_max(elsi_h, mu_max) |
1613 | + if (ionode) then |
1614 | + print *, "*-- updated mu_min, mu_max:", mu_min/eV, mu_max/eV |
1615 | + endif |
1616 | + end if |
1617 | + |
1618 | +end subroutine elsi_save_potential |
1619 | + |
1620 | +! To-Do: Check the implications of distinguishing between 'global_comm' |
1621 | +! (common space for communications) and 'base_comm' (to be split) |
1622 | +! For the real case they can be the same |
1623 | +! |
1624 | +! This routine should be called by everybody in global_comm |
1625 | + |
1626 | +subroutine get_spin_comms_and_dists(global_comm, base_comm, blocksize, n_spin,& |
1627 | + dist_global, dist_spin, spatial_comm, spin_comm) |
1628 | + |
1629 | + use mpi_siesta |
1630 | + use class_Distribution ! distribution, newDistribution, types |
1631 | + |
1632 | + integer, intent(in) :: global_comm ! Communicator for global distribution |
1633 | + ! Needed to include the actual global ranks |
1634 | + ! in the distribution objects |
1635 | + |
1636 | + integer, intent(in) :: base_comm ! Communicator to be split |
1637 | + integer, intent(out) :: spatial_comm ! "Orbital" comm (one for each spin team) |
1638 | + integer, intent(out) :: spin_comm ! "Inner" spin comm (for reductions) |
1639 | + |
1640 | + integer, intent(in) :: blocksize |
1641 | + integer, intent(in) :: n_spin ! should be 2 in this version |
1642 | + |
1643 | + ! Note inout for bud objects |
1644 | + type(distribution), intent(inout) :: dist_global ! global distribution object |
1645 | + type(distribution), intent(inout) :: dist_spin(2) ! per-spin objects |
1646 | + |
1647 | + ! Local variables |
1648 | + integer :: color, base_rank |
1649 | + integer :: npGlobal, npPerSpin |
1650 | + integer, allocatable, dimension(:) :: global_ranks_in_world |
1651 | + integer, allocatable, dimension(:) :: ranks_in_world ! should be 'spatial_ranks...' |
1652 | + integer, allocatable, dimension(:,:) :: ranks_in_World_Spin |
1653 | + integer :: global_group, Spatial_group |
1654 | + integer :: i, ispin, ierr |
1655 | + |
1656 | + call MPI_Comm_Size(global_comm, npGlobal, ierr) |
1657 | + ! This distribution could be created by the caller... |
1658 | + allocate(global_ranks_in_world(npGlobal)) |
1659 | + global_ranks_in_world = (/ (i, i=0, npGlobal-1) /) |
1660 | + call newDistribution(dist_global,global_Comm,global_ranks_in_world, & |
1661 | + TYPE_BLOCK_CYCLIC,BlockSize,"global dist") |
1662 | + deallocate(global_ranks_in_world) |
1663 | + call MPI_Barrier(global_Comm,ierr) |
1664 | + |
1665 | + ! Split the base communicator |
1666 | + call MPI_Comm_Rank(base_comm, base_rank, ierr) |
1667 | + ! "Row" communicator for independent operations on each spin |
1668 | + ! The name refers to "spatial" degrees of freedom. |
1669 | + color = mod(base_rank,n_spin) ! {0,1} for n_spin = 2, or {0} for n_spin = 1 |
1670 | + call MPI_Comm_Split(base_comm, color, base_rank, Spatial_Comm, ierr) |
1671 | + ! "Column" communicator for spin reductions |
1672 | + color = base_rank/n_spin |
1673 | + call MPI_Comm_Split(base_comm, color, base_rank, Spin_Comm, ierr) |
1674 | + |
1675 | + call MPI_Comm_Size(spatial_comm, npPerSpin, ierr) ! number of procs in each spin team |
1676 | + |
1677 | + call MPI_Comm_Group(Spatial_Comm, Spatial_Group, Ierr) |
1678 | + allocate(ranks_in_world(npPerSpin)) |
1679 | + call MPI_Comm_Group(global_comm, Global_Group, Ierr) |
1680 | + call MPI_Group_translate_ranks( Spatial_Group, npPerSpin, & |
1681 | + (/ (i,i=0,npPerSpin-1) /), & |
1682 | + Global_Group, ranks_in_world, ierr ) |
1683 | + |
1684 | + call MPI_Group_Free(Spatial_Group, ierr) |
1685 | + call MPI_Group_Free(Global_Group, ierr) |
1686 | + |
1687 | + allocate (ranks_in_World_Spin(npPerSpin,n_spin)) |
1688 | + ! We need everybody to have this information, as all nodes |
1689 | + ! are part of the global distribution side of the communication |
1690 | + ! (global_comm is the common-address space) |
1691 | + |
1692 | + ! But note that this will tell only those in base_comm... |
1693 | + ! This routine might not be the end of the story for cascading splits |
1694 | + ! (k-points and spin) |
1695 | + ! We might need an extra 'broadcast' over the k-point 'column' comm. |
1696 | + call MPI_AllGather(ranks_in_world,npPerSpin,MPI_integer,& |
1697 | + Ranks_in_World_Spin(1,1),npPerSpin, & |
1698 | + MPI_integer,Spin_Comm,ierr) |
1699 | + |
1700 | + ! Create distributions known to all nodes (again, those in base_comm) |
1701 | + do ispin = 1, n_spin |
1702 | + call newDistribution(dist_spin(ispin), global_Comm, & |
1703 | + Ranks_in_World_Spin(:,ispin), & |
1704 | + TYPE_BLOCK_CYCLIC, blockSize, "SPATIAL dist") |
1705 | + enddo |
1706 | + deallocate(ranks_in_world,Ranks_in_World_Spin) |
1707 | + call MPI_Barrier(global_Comm,ierr) |
1708 | + |
1709 | +end subroutine get_spin_comms_and_dists |
1710 | + |
1711 | +#endif /* SIESTA__ELSI */ |
1712 | + |
1713 | +#if SIESTA__ELSI || SIESTA__PEXSI |
1714 | + |
1715 | +subroutine elsi_complex_solver(iscf, n_basis, n_basis_l, n_spin, nnz_l, numh, row_ptr, & |
1716 | + col_idx, qtot, temp, ham, ovlp, dm, edm, ef, ets, & |
1717 | + nkpnt, kpt_n, kpt, weight, kpt_comm, Get_EDM_Only) |
1718 | + |
1719 | + use m_mpi_utils, only: globalize_sum |
1720 | + use parallel, only: BlockSize |
1721 | +#ifdef MPI |
1722 | + use mpi_siesta |
1723 | +#endif |
1724 | + use class_Distribution |
1725 | + use m_redist_spmatrix, only: aux_matrix, redistribute_spmatrix |
1726 | + use alloc |
1727 | + use m_mpi_utils, only: globalize_sum |
1728 | + |
1729 | + implicit none |
1730 | + |
1731 | + integer, intent(in) :: iscf ! SCF step counter |
1732 | + integer, intent(in) :: n_basis ! Global basis |
1733 | + integer, intent(in) :: n_basis_l ! Local basis |
1734 | + integer, intent(in) :: n_spin |
1735 | + integer, intent(in) :: nnz_l ! Local nonzero |
1736 | + integer, intent(in), target :: numh(n_basis_l) |
1737 | + integer, intent(in) :: row_ptr(n_basis_l) |
1738 | + integer, intent(in), target :: col_idx(nnz_l) |
1739 | + real(dp), intent(in) :: qtot |
1740 | + real(dp), intent(in) :: temp |
1741 | + complex(dp), intent(inout), target :: ham(nnz_l,n_spin) |
1742 | + complex(dp), intent(inout), target :: ovlp(nnz_l) |
1743 | + complex(dp), intent(out) :: dm(nnz_l,n_spin) |
1744 | + complex(dp), intent(out) :: edm(nnz_l,n_spin) |
1745 | + real(dp), intent(out) :: ef ! Fermi energy |
1746 | + real(dp), intent(out) :: ets ! Entropy/k, dimensionless |
1747 | + integer, intent(in) :: nkpnt ! number of k-points |
1748 | + integer, intent(in) :: kpt_n |
1749 | + real(dp), intent(in) :: kpt(3:) |
1750 | + real(dp), intent(in) :: weight |
1751 | + integer, intent(in) :: kpt_comm |
1752 | + |
1753 | + integer :: ierr |
1754 | + integer :: n_state |
1755 | + integer :: nnz_g |
1756 | + |
1757 | + real(dp) :: energy |
1758 | + |
1759 | + integer, allocatable, dimension(:) :: row_ptr2 |
1760 | + |
1761 | + integer :: elsi_Spatial_comm, elsi_Spin_comm |
1762 | + |
1763 | + type(distribution) :: dist_global |
1764 | + type(distribution) :: dist_spin(2) |
1765 | + |
1766 | + type(aux_matrix) :: pkg_global, pkg_spin ! Packages for transfer |
1767 | + |
1768 | + integer :: my_spin |
1769 | + |
1770 | + integer :: my_no_l |
1771 | + integer :: my_nnz_l |
1772 | + integer :: my_nnz |
1773 | + integer, pointer :: my_row_ptr2(:) => null() |
1774 | + integer :: i, ih, ispin, spin_rank, global_rank |
1775 | + real(dp) :: ets_spin |
1776 | + |
1777 | + integer, pointer :: my_col_idx(:) |
1778 | + complex(dp), pointer :: my_S(:) |
1779 | + complex(dp), pointer :: my_H(:) |
1780 | + complex(dp), pointer :: my_DM(:) => null() |
1781 | + complex(dp), pointer :: my_EDM(:) => null() |
1782 | + |
1783 | + integer :: date_stamp |
1784 | + |
1785 | + logical :: Get_EDM_Only |
1786 | + |
1787 | + external :: timer |
1788 | + |
1789 | +#ifndef MPI |
1790 | + call die("This ELSI solver interface needs MPI") |
1791 | +#endif |
1792 | + |
1793 | + call timer("elsi-complex-solver", 1) |
1794 | + |
1795 | + ! Initialization |
1796 | + if (iscf == 1) then |
1797 | + |
1798 | + ! Get ELSI options |
1799 | + call elsi_get_opts() |
1800 | + |
1801 | + ! Number of states to solve when calling an eigensolver |
1802 | + n_state = min(n_basis, n_basis/2+5) |
1803 | + |
1804 | + ! Now we have all ingredients to initialize ELSI |
1805 | + call elsi_init(elsi_h, which_solver, MULTI_PROC, SIESTA_CSC, n_basis, & |
1806 | + qtot, n_state) |
1807 | + |
1808 | + ! Output |
1809 | + call elsi_set_output(elsi_h, out_level) |
1810 | + call elsi_set_output_log(elsi_h, out_json) |
1811 | + call elsi_set_write_unit(elsi_h, 6) |
1812 | + |
1813 | + ! Broadening |
1814 | + call elsi_set_mu_broaden_scheme(elsi_h, which_broad) |
1815 | + call elsi_set_mu_broaden_width(elsi_h, temp) |
1816 | + call elsi_set_mu_mp_order(elsi_h, mp_order) |
1817 | + |
1818 | + ! Solver settings |
1819 | + call elsi_set_elpa_solver(elsi_h, elpa_flavor) |
1820 | + |
1821 | + call elsi_set_omm_flavor(elsi_h, omm_flavor) |
1822 | + call elsi_set_omm_n_elpa(elsi_h, omm_n_elpa) |
1823 | + call elsi_set_omm_tol(elsi_h, omm_tol) |
1824 | + |
1825 | + if (pexsi_tasks_per_pole /= ELSI_NOT_SET) then |
1826 | + call elsi_set_pexsi_np_per_pole(elsi_h, pexsi_tasks_per_pole) |
1827 | + end if |
1828 | + |
1829 | + call elsi_set_pexsi_np_symbo(elsi_h, pexsi_tasks_symbolic) |
1830 | + call elsi_set_pexsi_temp(elsi_h, temp) |
1831 | +! call elsi_set_pexsi_gap(elsi_h, gap) |
1832 | +! call elsi_set_pexsi_delta_e(elsi_h, delta_e) |
1833 | + call elsi_set_pexsi_mu_min(elsi_h, -10.0_dp) |
1834 | + call elsi_set_pexsi_mu_max(elsi_h, 10.0_dp) |
1835 | + |
1836 | + if (sips_n_slice /= ELSI_NOT_SET) then |
1837 | + call elsi_set_sips_n_slice(elsi_h, sips_n_slice) |
1838 | + end if |
1839 | + |
1840 | + call elsi_set_sips_n_elpa(elsi_h, sips_n_elpa) |
1841 | + call elsi_set_sips_interval(elsi_h, -10.0_dp, 10.0_dp) |
1842 | + |
1843 | + call elsi_set_ntpoly_method(elsi_h, ntpoly_method) |
1844 | + call elsi_set_ntpoly_filter(elsi_h, ntpoly_filter) |
1845 | + call elsi_set_ntpoly_tol(elsi_h, ntpoly_tol) |
1846 | + |
1847 | + endif ! iscf == 1 |
1848 | + |
1849 | + !print *, global_rank, "| ", " Entering elsi_complex_solver" |
1850 | + |
1851 | + if (n_spin == 1) then |
1852 | + |
1853 | + ! Sparsity pattern |
1854 | + call globalize_sum(nnz_l, nnz_g, comm=kpt_comm) |
1855 | + |
1856 | + allocate(row_ptr2(n_basis_l+1)) |
1857 | + row_ptr2(1:n_basis_l) = row_ptr(1:n_basis_l)+1 |
1858 | + row_ptr2(n_basis_l+1) = nnz_l+1 |
1859 | + |
1860 | + call elsi_set_csc(elsi_h, nnz_g, nnz_l, n_basis_l, col_idx, row_ptr2) |
1861 | + deallocate(row_ptr2) |
1862 | + |
1863 | + call elsi_set_csc_blk(elsi_h, BlockSize) |
1864 | + call elsi_set_kpoint(elsi_h, nkpnt, kpt_n, weight) |
1865 | + call elsi_set_mpi(elsi_h, kpt_comm) |
1866 | + call elsi_set_mpi_global(elsi_h, elsi_global_comm) |
1867 | + |
1868 | + else |
1869 | + |
1870 | + call mpi_comm_rank( elsi_global_Comm, global_rank, ierr ) |
1871 | + |
1872 | + ! MPI logic for spin polarization |
1873 | + |
1874 | + ! Split the communicator in spins and get distribution objects |
1875 | + ! for the data redistribution needed |
1876 | + ! Note that dist_spin is an array |
1877 | + call get_spin_comms_and_dists(kpt_comm,kpt_comm, & !! **** kpt_comm as global? |
1878 | + blocksize, n_spin, & |
1879 | + dist_global,dist_spin, elsi_spatial_comm, elsi_spin_comm) |
1880 | + |
1881 | + ! Find out which spin team we are in, and tag the spin we work on |
1882 | + call mpi_comm_rank( elsi_Spin_Comm, spin_rank, ierr ) |
1883 | + my_spin = spin_rank+1 ! {1,2} |
1884 | + |
1885 | + !print *, global_rank, "| ", "spin ", my_spin, " After spin splitting" |
1886 | + |
1887 | + ! This is done serially, each time filling one spin set |
1888 | + ! Note that **all processes** need to have the same pkg_global |
1889 | + |
1890 | + do ispin = 1, n_spin |
1891 | + |
1892 | + ! Load pkg_global data package |
1893 | + pkg_global%norbs = n_basis |
1894 | + pkg_global%no_l = n_basis_l |
1895 | + pkg_global%nnzl = nnz_l |
1896 | + pkg_global%numcols => numh |
1897 | + pkg_global%cols => col_idx |
1898 | + |
1899 | + allocate(pkg_global%complex_vals(2)) |
1900 | + ! Link the vals items to the appropriate arrays (no extra memory here) |
1901 | + pkg_global%complex_vals(1)%data => ovlp(:) |
1902 | + ! Note that we *cannot* say => ham(:,my_spin) |
1903 | + ! and avoid the sequential loop, as then half the processors will send |
1904 | + ! the information for 'spin up' and the other half the information for 'spin down', |
1905 | + ! which is *not* what we want. |
1906 | + pkg_global%complex_vals(2)%data => ham(:,ispin) |
1907 | + |
1908 | + call timer("redist_orbs_fwd", 1) |
1909 | + |
1910 | + ! We are doing the transfers sequentially. One spin team is |
1911 | + ! 'idle' (in the receiving side) in each pass, as the dist_spin(ispin) distribution |
1912 | + ! does not involve them. |
1913 | + |
1914 | + call redistribute_spmatrix(n_basis,pkg_global,dist_global, & |
1915 | + pkg_spin,dist_spin(ispin),kpt_Comm) |
1916 | + |
1917 | + call timer("redist_orbs_fwd", 2) |
1918 | + |
1919 | + if (my_spin == ispin) then ! Each team gets their own data |
1920 | + |
1921 | + !nrows = pkg_spin%norbs ! or simply 'norbs' |
1922 | + my_no_l = pkg_spin%no_l |
1923 | + my_nnz_l = pkg_spin%nnzl |
1924 | + call MPI_AllReduce(my_nnz_l,my_nnz,1,MPI_integer,MPI_sum,elsi_Spatial_Comm,ierr) |
1925 | + ! generate off-by-one row pointer |
1926 | + call re_alloc(my_row_ptr2,1,my_no_l+1,"my_row_ptr2","elsi_solver") |
1927 | + my_row_ptr2(1) = 1 |
1928 | + do ih = 1,my_no_l |
1929 | + my_row_ptr2(ih+1) = my_row_ptr2(ih) + pkg_spin%numcols(ih) |
1930 | + enddo |
1931 | + |
1932 | + my_col_idx => pkg_spin%cols |
1933 | + my_S => pkg_spin%complex_vals(1)%data |
1934 | + my_H => pkg_spin%complex_vals(2)%data |
1935 | + |
1936 | + call re_alloc(my_DM,1,my_nnz_l,"my_DM","elsi_solver") |
1937 | + call re_alloc(my_EDM,1,my_nnz_l,"my_EDM","elsi_solver") |
1938 | + endif |
1939 | + |
1940 | + ! Clean pkg_global |
1941 | + nullify(pkg_global%complex_vals(1)%data) |
1942 | + nullify(pkg_global%complex_vals(2)%data) |
1943 | + deallocate(pkg_global%complex_vals) |
1944 | + nullify(pkg_global%numcols) |
1945 | + nullify(pkg_global%cols) |
1946 | + |
1947 | + enddo |
1948 | + |
1949 | + !print *, global_rank, "| ", "spin ", my_spin, "Done spin transfers" |
1950 | + |
1951 | + call elsi_set_csc(elsi_h, my_nnz, my_nnz_l, my_no_l, my_col_idx, my_row_ptr2) |
1952 | + call de_alloc(my_row_ptr2,"my_row_ptr2","elsi_solver") |
1953 | + |
1954 | + call elsi_set_csc_blk(elsi_h, BlockSize) |
1955 | + call elsi_set_spin(elsi_h, n_spin, my_spin) |
1956 | + call elsi_set_kpoint(elsi_h, nkpnt, kpt_n, weight) |
1957 | + call elsi_set_mpi(elsi_h, elsi_Spatial_comm) |
1958 | + call elsi_set_mpi_global(elsi_h, elsi_global_comm) |
1959 | + |
1960 | + endif ! n_spin |
1961 | + |
1962 | + call timer("elsi-solver", 1) |
1963 | + |
1964 | + !print *, global_rank, "| ", "About to call elsi_dm" |
1965 | + if (n_spin == 1) then |
1966 | + if (.not.Get_EDM_Only) then |
1967 | + call elsi_dm_complex_sparse(elsi_h, ham, ovlp, DM, energy) |
1968 | + call elsi_get_entropy(elsi_h, ets) |
1969 | + else |
1970 | + call elsi_get_edm_complex_sparse(elsi_h, EDM) |
1971 | + endif |
1972 | + else |
1973 | + ! Solve DM, and get (at every step for now) EDM, Fermi energy, and entropy |
1974 | + ! Energy is already summed over spins |
1975 | + if (.not.Get_EDM_Only) then |
1976 | + call elsi_dm_complex_sparse(elsi_h, my_H, my_S, my_DM, energy) |
1977 | + !... but we still need to sum the entropy over spins |
1978 | + call elsi_get_entropy(elsi_h, ets_spin) |
1979 | + else |
1980 | + call elsi_get_edm_complex_sparse(elsi_h, my_EDM) |
1981 | + endif |
1982 | +#ifdef SIESTA__ELSI__OLD_SPIN_CONVENTION |
1983 | + ! NOTE** |
1984 | + ! For ELSI versions before 2018-08-17 we still need to sum the entropy over spins |
1985 | + ! ... but to figure out the version is not trivial, as the relevant routines changed... |
1986 | + call globalize_sum(ets_spin, ets, comm=elsi_Spin_comm) |
1987 | +#else |
1988 | + ! If this gives an error, then your version of ELSI is old, and you should |
1989 | + ! be using the pre-processor symbol above |
1990 | + ! (This works because 'elsi_get_datestamp' was added just |
1991 | + ! around the same time as the spin-reduction-convention change) (Aug 17 vs Aug 21, 2018...) |
1992 | + ! If you are unlucky enough to have an ELSI version in between, get rid of it. |
1993 | + call elsi_get_datestamp(date_stamp) |
1994 | +#endif |
1995 | + ! And over kpoints?? -- not necessary, ever |
1996 | + endif |
1997 | + |
1998 | + call elsi_get_mu(elsi_h, ef) |
1999 | + ets = ets/temp |
2000 | + |
2001 | + ! Ef, energy, and ets are known to all nodes |
2002 | + |
2003 | + call timer("elsi-solver", 2) |
2004 | + !print *, global_rank, "| ", "Done elsi_dm" |
2005 | + |
2006 | + if ( n_spin == 2) then |
2007 | + ! Now we need to redistribute back |
2008 | + |
2009 | + do ispin = 1, n_spin |
2010 | + |
2011 | + if (my_spin == ispin) then |
2012 | + ! Prepare pkg_spin to transfer the right spin information |
2013 | + ! The other fields (numcols, cols) are the same and are still there |
2014 | + ! Deallocate my_S and my_H |
2015 | + call de_alloc(pkg_spin%complex_vals(1)%data,"pkg_spin%vals(1)%data","elsi_solver") |
2016 | + call de_alloc(pkg_spin%complex_vals(2)%data,"pkg_spin%vals(2)%data","elsi_solver") |
2017 | + |
2018 | + pkg_spin%complex_vals(1)%data => my_DM(1:my_nnz_l) |
2019 | + pkg_spin%complex_vals(2)%data => my_EDM(1:my_nnz_l) |
2020 | + |
2021 | + endif |
2022 | + |
2023 | + ! pkg_global is clean now |
2024 | + call timer("redist_orbs_bck", 1) |
2025 | + call redistribute_spmatrix(n_basis,pkg_spin,dist_spin(ispin) & |
2026 | + ,pkg_global,dist_global,kpt_Comm) |
2027 | + call timer("redist_orbs_bck", 2) |
2028 | + |
2029 | + ! Clean pkg_spin |
2030 | + if (my_spin == ispin) then |
2031 | + ! Each team deallocates during "its" spin cycle |
2032 | + call de_alloc(my_DM, "my_DM", "elsi_solver") |
2033 | + call de_alloc(my_EDM,"my_EDM","elsi_solver") |
2034 | + |
2035 | + nullify(pkg_spin%complex_vals(1)%data) ! formerly pointing to DM |
2036 | + nullify(pkg_spin%complex_vals(2)%data) ! formerly pointing to EDM |
2037 | + deallocate(pkg_spin%complex_vals) |
2038 | + ! allocated in the direct transfer |
2039 | + call de_alloc(pkg_spin%numcols,"pkg_spin%numcols","elsi_solver") |
2040 | + call de_alloc(pkg_spin%cols, "pkg_spin%cols", "elsi_solver") |
2041 | + endif |
2042 | + |
2043 | + |
2044 | + ! In future, pkg_global%vals(1,2) could be pointing to DM and EDM, |
2045 | + ! and the 'redistribute' routine check whether the vals arrays are |
2046 | + ! associated, to use them instead of allocating them. |
2047 | + DM(:,ispin) = pkg_global%complex_vals(1)%data(:) |
2048 | + EDM(:,ispin) = pkg_global%complex_vals(2)%data(:) |
2049 | + ! Check no_l |
2050 | + if (n_basis_l /= pkg_global%no_l) then |
2051 | + call die("Mismatch in no_l") |
2052 | + endif |
2053 | + ! Check listH |
2054 | + if (any(col_idx(:) /= pkg_global%cols(:))) then |
2055 | + call die("Mismatch in listH") |
2056 | + endif |
2057 | + |
2058 | + ! Clean pkg_global |
2059 | + ! allocated by the transfer routine, but we did not actually |
2060 | + ! look at them |
2061 | + call de_alloc(pkg_global%numcols,"pkg_global%numcols","elsi_solver") |
2062 | + call de_alloc(pkg_global%cols, "pkg_global%cols", "elsi_solver") |
2063 | + |
2064 | + call de_alloc(pkg_global%complex_vals(1)%data,"pkg_global%vals(1)%data","elsi_solver") |
2065 | + call de_alloc(pkg_global%complex_vals(2)%data,"pkg_global%vals(2)%data","elsi_solver") |
2066 | + deallocate(pkg_global%complex_vals) |
2067 | + |
2068 | + enddo |
2069 | + |
2070 | + call MPI_Comm_Free(elsi_Spatial_comm, ierr) |
2071 | + call MPI_Comm_Free(elsi_Spin_comm, ierr) |
2072 | + |
2073 | + else ! n_spin == 1 |
2074 | + |
2075 | + ! Nothing else to do |
2076 | + endif |
2077 | + |
2078 | + call timer("elsi-complex-solver", 2) |
2079 | + |
2080 | + |
2081 | +end subroutine elsi_complex_solver |
2082 | + |
2083 | +subroutine transpose(a,b) |
2084 | + real(dp), intent(in) :: a(:,:) |
2085 | + real(dp), allocatable, intent(out) :: b(:,:) |
2086 | + |
2087 | + integer n, m, i, j |
2088 | + |
2089 | + n = size(a,1) |
2090 | + m = size(a,2) |
2091 | + allocate(b(m,n)) |
2092 | + do i = 1, n |
2093 | + do j = 1, m |
2094 | + b(j, i) = a(i, j) |
2095 | + enddo |
2096 | + enddo |
2097 | +end subroutine transpose |
2098 | + |
2099 | +# endif |
2100 | + |
2101 | +end module m_elsi_interface |
2102 | |
2103 | === modified file 'Src/m_ncdf_siesta.F90' |
2104 | --- Src/m_ncdf_siesta.F90 2019-01-04 07:01:47 +0000 |
2105 | +++ Src/m_ncdf_siesta.F90 2019-01-16 11:00:58 +0000 |
2106 | @@ -51,6 +51,7 @@ |
2107 | use siesta_options, only: SOLVE_DIAGON, SOLVE_ORDERN |
2108 | use siesta_options, only: SOLVE_MINIM, SOLVE_TRANSI |
2109 | use siesta_options, only: SOLVE_PEXSI |
2110 | + use siesta_options, only: SOLVE_ELSI |
2111 | use siesta_options, only: savehs |
2112 | use siesta_options, only: fixspin, total_spin |
2113 | use siesta_options, only: ia1, ia2, dx ! FC information |
2114 | @@ -361,6 +362,8 @@ |
2115 | dic = dic//('method'.kv.'transiesta') |
2116 | else if ( isolve == SOLVE_PEXSI ) then |
2117 | dic = dic//('method'.kv.'pexsi') |
2118 | + else if ( isolve == SOLVE_ELSI ) then |
2119 | + dic = dic//('method'.kv.'elsi') |
2120 | end if |
2121 | |
2122 | ! Attributes are collective |
2123 | |
2124 | === modified file 'Src/m_pexsi_dos.F90' |
2125 | --- Src/m_pexsi_dos.F90 2016-08-04 09:03:32 +0000 |
2126 | +++ Src/m_pexsi_dos.F90 2019-01-16 11:00:58 +0000 |
2127 | @@ -99,7 +99,7 @@ |
2128 | ! |
2129 | ! Our global communicator is a duplicate of the passed communicator |
2130 | ! |
2131 | -call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr) |
2132 | +call MPI_Comm_Dup(mpi_comm_dft, World_Comm, ierr) |
2133 | call mpi_comm_rank( World_Comm, mpirank, ierr ) |
2134 | |
2135 | call timer("pexsi_dos", 1) |
2136 | |
2137 | === modified file 'Src/m_pexsi_driver.F90' |
2138 | --- Src/m_pexsi_driver.F90 2016-08-04 09:03:32 +0000 |
2139 | +++ Src/m_pexsi_driver.F90 2019-01-16 11:00:58 +0000 |
2140 | @@ -123,7 +123,7 @@ |
2141 | ! |
2142 | ! Our global communicator is a duplicate of the passed communicator |
2143 | ! |
2144 | -call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr) |
2145 | +call MPI_Comm_Dup(MPI_Comm_DFT, World_Comm, ierr) |
2146 | call mpi_comm_rank( World_Comm, mpirank, ierr ) |
2147 | |
2148 | ! NOTE: fdf calls will assign values to the whole processor set, |
2149 | |
2150 | === modified file 'Src/m_pexsi_local_dos.F90' |
2151 | --- Src/m_pexsi_local_dos.F90 2018-04-27 09:35:52 +0000 |
2152 | +++ Src/m_pexsi_local_dos.F90 2019-01-16 11:00:58 +0000 |
2153 | @@ -144,7 +144,7 @@ |
2154 | ! |
2155 | ! Our global communicator is a duplicate of the passed communicator |
2156 | ! |
2157 | - call MPI_Comm_Dup(true_MPI_Comm_World, World_Comm, ierr) |
2158 | + call MPI_Comm_Dup(MPI_Comm_DFT, World_Comm, ierr) |
2159 | call mpi_comm_rank( World_Comm, mpirank, ierr ) |
2160 | |
2161 | call timer("pexsi-ldos", 1) |
2162 | |
2163 | === modified file 'Src/m_redist_spmatrix.F90' |
2164 | --- Src/m_redist_spmatrix.F90 2016-09-14 09:35:16 +0000 |
2165 | +++ Src/m_redist_spmatrix.F90 2019-01-16 11:00:58 +0000 |
2166 | @@ -1,18 +1,44 @@ |
2167 | - |
2168 | -! --- Tangled code |
2169 | +! |
2170 | +! This module implements functionality to change the MPI parallel |
2171 | +! distribution of a sparse matrix stored in the modified |
2172 | +! block-compressed form used in Siesta. |
2173 | +! |
2174 | +! The immediate goal is to exchange matrices between Siesta and the |
2175 | +! PEXSI library. Siesta uses a "block-cyclic" distribution over |
2176 | +! N_Siesta processors, and PEXSI uses a simple "greedy" blocked |
2177 | +! distribution on each pole-group, with npPerPole processors. The |
2178 | +! code works in principle with any distribution (subject to some |
2179 | +! conditions spelled out below). The source and target process groups |
2180 | +! are arbitrary, and can overlap. In fact, in SIESTA-PEXSI they |
2181 | +! *will* overlap, as otherwise we would need extra processors for the |
2182 | +! Siesta operations. This overlap forces the code to re-invent much |
2183 | +! of the logic involved in MPI intercommunicators, which cannot be |
2184 | +! created for non-disjoint groups. |
2185 | + |
2186 | +! Written by Alberto Garcia |
2187 | + |
2188 | +! --- Tangled code (using Org mode in emacs) |
2189 | +! Alas, the Org version is no longer maintained. |
2190 | +! |
2191 | module m_redist_spmatrix |
2192 | -#ifdef SIESTA__PEXSI |
2193 | +#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI) |
2194 | implicit none |
2195 | + integer, parameter, private :: dp = selected_real_kind(10,100) |
2196 | + |
2197 | + ! Some auxiliary derived types |
2198 | + |
2199 | type, public :: comm_t |
2200 | integer :: src, dst, i1, i2, nitems |
2201 | end type comm_t |
2202 | - |
2203 | - integer, parameter, private :: dp = selected_real_kind(10,100) |
2204 | - |
2205 | + |
2206 | type, public :: dp_pointer |
2207 | ! boxed array pointer type |
2208 | real(dp), pointer :: data(:) => null() |
2209 | end type dp_pointer |
2210 | + type, public :: complex_dp_pointer |
2211 | + ! boxed array pointer type |
2212 | + complex(dp), pointer :: data(:) => null() |
2213 | + end type complex_dp_pointer |
2214 | |
2215 | type, public :: aux_matrix |
2216 | integer :: norbs = -1 |
2217 | @@ -20,14 +46,21 @@ |
2218 | integer :: nnzl = -1 |
2219 | integer, pointer :: numcols(:) => null() |
2220 | integer, pointer :: cols(:) => null() |
2221 | - ! array of 1D pointers |
2222 | + ! arrays of 1D pointers |
2223 | type(dp_pointer), dimension(:), pointer :: vals(:) => null() |
2224 | + type(complex_dp_pointer), dimension(:), pointer :: complex_vals(:) => null() |
2225 | end type aux_matrix |
2226 | + |
2227 | + |
2228 | public :: redistribute_spmatrix |
2229 | + |
2230 | CONTAINS |
2231 | + |
2232 | subroutine redistribute_spmatrix(norbs,m1,dist1,m2,dist2,bridge_comm) |
2233 | |
2234 | - use mpi |
2235 | + use mpi ! Note that this is fine, as we do not use any Siesta-specific |
2236 | + ! parameter, such as a modified mpi_comm_world |
2237 | + |
2238 | use class_Distribution |
2239 | use alloc, only: re_alloc, de_alloc |
2240 | |
2241 | @@ -47,10 +80,11 @@ |
2242 | logical :: proc_in_set1, proc_in_set2 |
2243 | integer :: ierr |
2244 | |
2245 | - integer :: i, io, g1, g2, j, nvals |
2246 | + integer :: i, io, g1, g2, j, nvals, nvals_complex |
2247 | integer :: comparison, n1, n2, c1, c2 |
2248 | integer, parameter :: dp = selected_real_kind(10,100) |
2249 | real(dp), dimension(:), pointer :: data1 => null(), data2 => null() |
2250 | + complex(dp), dimension(:), pointer :: cdata1 => null(), cdata2 => null() |
2251 | |
2252 | integer, allocatable :: ranks1(:), ranks2(:) |
2253 | |
2254 | @@ -109,10 +143,10 @@ |
2255 | proc_in_set1 = (myrank1 /= MPI_UNDEFINED) |
2256 | proc_in_set2 = (myrank2 /= MPI_UNDEFINED) |
2257 | |
2258 | - if (proc_in_set1 .or. proc_in_set2) then |
2259 | - print "(a,3i6,2l2)", "world_rank, rank1, rank2, ing1?, ing2?", myid, & |
2260 | - myrank1, myrank2, proc_in_set1, proc_in_set2 |
2261 | - endif |
2262 | +! if (proc_in_set1 .or. proc_in_set2) then |
2263 | +! print "(a,3i6,2l2)", "world_rank, rank1, rank2, ing1?, ing2?", myid, & |
2264 | +! myrank1, myrank2, proc_in_set1, proc_in_set2 |
2265 | +! endif |
2266 | |
2267 | ! Figure out the communication needs |
2268 | call analyze_comms() |
2269 | @@ -130,10 +164,10 @@ |
2270 | call re_alloc(m2%numcols,1,m2%no_l,"m2%numcols","redistribute_spmatrix") |
2271 | endif |
2272 | |
2273 | - if (myid == 0) print *, "About to transfer numcols..." |
2274 | +! if (myid == 0) print *, "About to transfer numcols..." |
2275 | call do_transfers_int(comms,m1%numcols,m2%numcols, & |
2276 | g1,g2,bridge_comm) |
2277 | - if (myid == 0) print *, "Transferred numcols." |
2278 | +! if (myid == 0) print *, "Transferred numcols." |
2279 | |
2280 | ! We need to tell the processes in set 2 how many |
2281 | ! "vals" to expect. |
2282 | @@ -143,12 +177,18 @@ |
2283 | else |
2284 | nvals = 0 |
2285 | endif |
2286 | + if (associated(m1%complex_vals)) then |
2287 | + nvals_complex = size(m1%complex_vals) |
2288 | + else |
2289 | + nvals_complex = 0 |
2290 | + endif |
2291 | endif |
2292 | ! Now do a broadcast within bridge_comm, using as root one |
2293 | ! process in the first set. Let's say the one with rank 0 |
2294 | ! in g1, the first in the set, which will have rank=ranks1(1) |
2295 | ! in bridge_comm |
2296 | call MPI_Bcast(nvals,1,MPI_Integer,ranks1(1),bridge_comm,ierr) |
2297 | + call MPI_Bcast(nvals_complex,1,MPI_Integer,ranks1(1),bridge_comm,ierr) |
2298 | |
2299 | ! Now we can figure out how many non-zeros there are |
2300 | if (proc_in_set2) then |
2301 | @@ -163,6 +203,14 @@ |
2302 | enddo |
2303 | endif |
2304 | |
2305 | + if (nvals_complex > 0) then |
2306 | + allocate(m2%complex_vals(nvals_complex)) |
2307 | + do j=1,nvals_complex |
2308 | + call re_alloc(m2%complex_vals(j)%data,1,m2%nnzl, & |
2309 | + "m2%complex_vals(j)%data","redistribute_spmatrix") |
2310 | + enddo |
2311 | + endif |
2312 | + |
2313 | endif |
2314 | |
2315 | ! Generate a new comms-structure with new start/count indexes |
2316 | @@ -202,12 +250,12 @@ |
2317 | !!$ endif |
2318 | !!$ enddo |
2319 | |
2320 | - if (myid == 0) print *, "About to transfer cols..." |
2321 | +! if (myid == 0) print *, "About to transfer cols..." |
2322 | ! Transfer the cols arrays |
2323 | call do_transfers_int(commsnnz,m1%cols,m2%cols, & |
2324 | g1, g2, bridge_comm) |
2325 | |
2326 | - if (myid == 0) print *, "About to transfer values..." |
2327 | +! if (myid == 0) print *, "About to transfer values..." |
2328 | ! Transfer the values arrays |
2329 | do j=1, nvals |
2330 | if (proc_in_set1) data1 => m1%vals(j)%data |
2331 | @@ -216,7 +264,17 @@ |
2332 | g1,g2,bridge_comm) |
2333 | enddo |
2334 | nullify(data1,data2) |
2335 | - if (myid == 0) print *, "Done transfers." |
2336 | + |
2337 | +! if (myid == 0) print *, "About to transfer complex values..." |
2338 | + ! Transfer the values arrays |
2339 | + do j=1, nvals_complex |
2340 | + if (proc_in_set1) cdata1 => m1%complex_vals(j)%data |
2341 | + if (proc_in_set2) cdata2 => m2%complex_vals(j)%data |
2342 | + call do_transfers_complex_dp(commsnnz,cdata1,cdata2, & |
2343 | + g1,g2,bridge_comm) |
2344 | + enddo |
2345 | + nullify(cdata1,cdata2) |
2346 | +! if (myid == 0) print *, "Done transfers." |
2347 | |
2348 | deallocate(commsnnz) |
2349 | deallocate(comms) |
2350 | @@ -312,9 +370,9 @@ |
2351 | if (myid == 0 .and. comms_not_printed) then |
2352 | do i = 1, ncomms |
2353 | c => comms(i) |
2354 | - write(6,"(a,i5,a,2i5,2i7,i5)") & |
2355 | - "comm: ", i, " src, dst, i1, i2, n:", & |
2356 | - c%src, c%dst, c%i1, c%i2, c%nitems |
2357 | +! write(6,"(a,i5,a,2i5,2i7,i5)") & |
2358 | +! "comm: ", i, " src, dst, i1, i2, n:", & |
2359 | +! c%src, c%dst, c%i1, c%i2, c%nitems |
2360 | enddo |
2361 | comms_not_printed = .false. |
2362 | endif |
2363 | @@ -543,6 +601,114 @@ |
2364 | deallocate(local_reqR, local_reqS, statuses) |
2365 | |
2366 | end subroutine do_transfers_dp |
2367 | + !-------------------------------------------------- |
2368 | + subroutine do_transfers_complex_dp(comms,data1,data2,g1,g2,bridge_comm) |
2369 | + |
2370 | + use mpi |
2371 | + integer, parameter :: dp = selected_real_kind(10,100) |
2372 | + |
2373 | + type(comm_t), intent(in), target :: comms(:) |
2374 | + complex(dp), dimension(:), pointer :: data1 |
2375 | + complex(dp), dimension(:), pointer :: data2 |
2376 | + integer, intent(in) :: g1 |
2377 | + integer, intent(in) :: g2 |
2378 | + integer, intent(in) :: bridge_comm |
2379 | + |
2380 | + integer :: basegroup, nsize1, nsize2, ierr |
2381 | + integer, allocatable :: comm_rank1(:), comm_rank2(:) |
2382 | + |
2383 | + |
2384 | + integer :: ncomms |
2385 | + integer :: i |
2386 | + integer :: nrecvs_local, nsends_local |
2387 | + integer, allocatable :: statuses(:,:), local_reqR(:), local_reqS(:) |
2388 | + integer :: src_in_comm, dst_in_comm |
2389 | + integer :: myrank1, myrank2, myid |
2390 | + type(comm_t), pointer :: c |
2391 | + |
2392 | + call MPI_Comm_Rank( bridge_comm, myid, ierr ) |
2393 | + ! print *, "Entering transfer_complex_dp" |
2394 | + ! print *, "rank, Associated data1: ", myid, associated(data1) |
2395 | + ! print *, "rank, Associated data2: ", myid, associated(data2) |
2396 | + |
2397 | + ! Find the rank correspondences, in case |
2398 | + ! there is implicit renumbering at the time of group creation |
2399 | + |
2400 | + call MPI_Comm_group( bridge_comm, basegroup, ierr ) |
2401 | + call MPI_Group_Size( g1, nsize1, ierr ) |
2402 | + call MPI_Group_Size( g2, nsize2, ierr ) |
2403 | + allocate(comm_rank1(0:nsize1-1)) |
2404 | + call MPI_Group_translate_ranks( g1, nsize1, (/ (i,i=0,nsize1-1) /), & |
2405 | + basegroup, comm_rank1, ierr ) |
2406 | + ! print "(a,10i3)", "Ranks of g1 in base group:", comm_rank1 |
2407 | + allocate(comm_rank2(0:nsize2-1)) |
2408 | + call MPI_Group_translate_ranks( g2, nsize2, (/ (i,i=0,nsize2-1) /), & |
2409 | + basegroup, comm_rank2, ierr ) |
2410 | + ! print "(a,10i3)", "Ranks of g2 in base group:", comm_rank2 |
2411 | + |
2412 | + call mpi_group_rank(g1,myrank1,ierr) |
2413 | + call mpi_group_rank(g2,myrank2,ierr) |
2414 | + |
2415 | + ! Do the actual transfers. |
2416 | + ! This version with non-blocking communications |
2417 | + |
2418 | + ncomms = size(comms) |
2419 | + |
2420 | + ! Some bookkeeping for the requests |
2421 | + nrecvs_local = 0 |
2422 | + nsends_local = 0 |
2423 | + do i=1,ncomms |
2424 | + c => comms(i) |
2425 | + if (myrank2 == c%dst) then |
2426 | + nrecvs_local = nrecvs_local + 1 |
2427 | + endif |
2428 | + if (myrank1 == c%src) then |
2429 | + nsends_local = nsends_local + 1 |
2430 | + endif |
2431 | + enddo |
2432 | + allocate(local_reqR(nrecvs_local)) |
2433 | + allocate(local_reqS(nsends_local)) |
2434 | + allocate(statuses(mpi_status_size,nrecvs_local)) |
2435 | + |
2436 | + ! First, post the receives |
2437 | + nrecvs_local = 0 |
2438 | + do i=1,ncomms |
2439 | + c => comms(i) |
2440 | + if (myrank2 == c%dst) then |
2441 | + nrecvs_local = nrecvs_local + 1 |
2442 | + src_in_comm = comm_rank1(c%src) |
2443 | + call MPI_irecv(data2(c%i2),c%nitems,MPI_Double_Complex,src_in_comm, & |
2444 | + i,bridge_comm,local_reqR(nrecvs_local),ierr) |
2445 | + endif |
2446 | + enddo |
2447 | + |
2448 | + ! Post the sends |
2449 | + nsends_local = 0 |
2450 | + do i=1,ncomms |
2451 | + c => comms(i) |
2452 | + if (myrank1 == c%src) then |
2453 | + nsends_local = nsends_local + 1 |
2454 | + dst_in_comm = comm_rank2(c%dst) |
2455 | + call MPI_isend(data1(c%i1),c%nitems,MPI_Double_Complex,dst_in_comm, & |
2456 | + i,bridge_comm,local_reqS(nsends_local),ierr) |
2457 | + endif |
2458 | + enddo |
2459 | + |
2460 | + ! A former loop of waits can be substituted by a "waitall", |
2461 | + ! with every processor keeping track of the actual number of |
2462 | + ! requests in which it is involved. |
2463 | + |
2464 | + ! Should we wait also on the sends? |
2465 | + |
2466 | + call MPI_waitall(nrecvs_local, local_reqR, statuses, ierr) |
2467 | + |
2468 | + |
2469 | + ! This barrier is needed, I think |
2470 | + call MPI_Barrier(bridge_comm,ierr) |
2471 | + |
2472 | + deallocate(local_reqR, local_reqS, statuses) |
2473 | + |
2474 | + end subroutine do_transfers_complex_dp |
2475 | #endif |
2476 | end module m_redist_spmatrix |
2477 | ! --- End of tangled code |
2478 | |
2479 | === modified file 'Src/parallel.F' |
2480 | --- Src/parallel.F 2017-07-14 12:17:28 +0000 |
2481 | +++ Src/parallel.F 2019-01-16 11:00:58 +0000 |
2482 | @@ -1,5 +1,5 @@ |
2483 | ! --- |
2484 | -! Copyright (C) 1996-2016 The SIESTA group |
2485 | +! Copyright- (C) 1996-2016 The SIESTA group |
2486 | ! This file is distributed under the terms of the |
2487 | ! GNU General Public License: see COPYING in the top directory |
2488 | ! or http://www.gnu.org/copyleft/gpl.txt . |
2489 | |
2490 | === modified file 'Src/post_scf_work.F' |
2491 | --- Src/post_scf_work.F 2018-09-12 13:49:21 +0000 |
2492 | +++ Src/post_scf_work.F 2019-01-16 11:00:58 +0000 |
2493 | @@ -27,12 +27,13 @@ |
2494 | use class_Fstack_Pair_Geometry_dSpData2D, only: new, max_size |
2495 | use class_Fstack_Pair_Geometry_dSpData2D, only: push |
2496 | |
2497 | - use parallel, only: ionode |
2498 | + use parallel, only : IOnode, SIESTA_worker |
2499 | + |
2500 | use atomlist, only: lasto, rmaxo, datm, indxuo, no_s, no_u, |
2501 | & iphorb, no_l, qtot, qtots |
2502 | use m_energies |
2503 | use neighbour, only : maxna=>maxnna ! For plcharge... |
2504 | - use m_spin, only : nspin, h_spin_dim |
2505 | + use m_spin, only : nspin, h_spin_dim, spin |
2506 | use m_spin, only : spinor_dim |
2507 | use m_diagon, only : diagon |
2508 | use m_dminim, only : dminim |
2509 | @@ -41,6 +42,9 @@ |
2510 | use m_compute_dm, only : PreviousCallDiagon |
2511 | use m_eo |
2512 | use kpoint_scf_m, only: kpoint_scf, gamma_scf |
2513 | +#ifdef SIESTA__ELSI |
2514 | + use m_elsi_interface, only: elsi_getdm |
2515 | +#endif |
2516 | implicit none |
2517 | |
2518 | ! MD-step, SCF-step |
2519 | @@ -51,10 +55,37 @@ |
2520 | type(Pair_Geometry_dSpData2D) :: pair |
2521 | type(Geometry) :: geom |
2522 | |
2523 | + real(dp), allocatable :: Dscf_dummy(:,:) |
2524 | + real(dp) :: ef_dummy |
2525 | + real(dp) :: Entropy_dummy |
2526 | + |
2527 | call timer( 'PostSCF', 1 ) |
2528 | |
2529 | ! If converged, make one last iteration to find forces and stress |
2530 | |
2531 | +! If we use the ELSI interface, the energy-density |
2532 | +! matrix is not calculated inside the SCF step, and |
2533 | +! so this must be done now |
2534 | + |
2535 | +#ifdef SIESTA__ELSI |
2536 | + if (isolve .eq. SOLVE_ELSI) then |
2537 | + allocate(Dscf_dummy(maxnh,spin%DM)) |
2538 | + ! Note that we do not care about the content of H and S |
2539 | + ! Only that the appropriate elsi_h handle is still active |
2540 | + ! and allows us to get the appropriate EDM |
2541 | + call elsi_getdm(iscf, no_s, spin%spinor, |
2542 | + & no_l, maxnh, no_u, |
2543 | + & numh, listhptr, listh, |
2544 | + & H, S, qtot, temp, |
2545 | + & xijo, |
2546 | + & kpoint_scf%N, kpoint_scf%k, kpoint_scf%w, |
2547 | + & eo, qo, Dscf_dummy, Escf, ef_dummy, Entropy_dummy, |
2548 | + & occtol, neigwanted, Get_EDM_Only=.true.) |
2549 | + deallocate(Dscf_dummy) |
2550 | + endif |
2551 | + if (.not. SIESTA_worker) RETURN |
2552 | +#endif |
2553 | + |
2554 | ! If we use the minimization routine, the energy-density |
2555 | ! matrix is not calculated inside the SCF step, and |
2556 | ! so this must be done now |
2557 | |
2558 | === modified file 'Src/read_options.F90' |
2559 | --- Src/read_options.F90 2018-12-02 00:15:31 +0000 |
2560 | +++ Src/read_options.F90 2019-01-16 11:00:58 +0000 |
2561 | @@ -76,6 +76,7 @@ |
2562 | ! 4 = PEXSI |
2563 | ! 5 = (Matrix write) |
2564 | ! 6 = CheSS |
2565 | + ! 7 = ELSI |
2566 | ! real*8 temp : Temperature for Fermi smearing (Ry) |
2567 | ! logical fixspin : Fix the spin of the system? |
2568 | ! real*8 total_spin : Total spin of the system |
2569 | @@ -706,6 +707,16 @@ |
2570 | #else |
2571 | call die("PEXSI solver is not compiled in. Use -DSIESTA__PEXSI") |
2572 | #endif |
2573 | + else if (leqi(method,"elsi")) then |
2574 | +#ifdef SIESTA__ELSI |
2575 | + isolve = SOLVE_ELSI |
2576 | + if (ionode) then |
2577 | + call add_citation("ELSI PAPER***") |
2578 | + write(*,3) 'redata: Method of Calculation', 'ELSI' |
2579 | + endif |
2580 | +#else |
2581 | + call die("ELSI solver is not compiled in. Use -DSIESTA__ELSI") |
2582 | +#endif |
2583 | |
2584 | #ifdef SIESTA__CHESS |
2585 | else if (leqi(method,'chess')) then |
2586 | |
2587 | === modified file 'Src/siesta.F' |
2588 | --- Src/siesta.F 2018-05-30 09:15:30 +0000 |
2589 | +++ Src/siesta.F 2019-01-16 11:00:58 +0000 |
2590 | @@ -21,7 +21,7 @@ |
2591 | use parallel, only: SIESTA_worker ! whether part of Siesta's communicator |
2592 | use parallel, only: ionode |
2593 | #ifdef MPI |
2594 | - use mpi_siesta, only: true_mpi_comm_world |
2595 | + use mpi_siesta, only: mpi_comm_dft ! Includes Solver nodes |
2596 | #endif |
2597 | use m_mpi_utils, only: broadcast |
2598 | |
2599 | @@ -35,12 +35,12 @@ |
2600 | integer :: istep |
2601 | logical :: relaxd |
2602 | |
2603 | -! Notes for PEXSI operation |
2604 | +! Notes for PEXSI and ELSI operation (extra procs for Solver) |
2605 | ! |
2606 | -! A subset of nodes carries out non-PEXSI Siesta operations |
2607 | +! A subset of nodes carries out non-Solver Siesta operations |
2608 | ! (i.e., setting up H, moving atoms, diagonalizing...). |
2609 | ! These are tagged as "SIESTA_worker" (admittedly, a bad name) |
2610 | -! All nodes are involved in the PEXSI electronic-structure solver, |
2611 | +! All nodes are involved in the (PEXSI or ELSI) electronic-structure solver, |
2612 | ! and in the new LocalDOS computation based on selected inversion. |
2613 | ! |
2614 | ! 'siesta_init', 'siesta_forces', and 'siesta_analysis' need to |
2615 | @@ -66,11 +66,11 @@ |
2616 | C Begin of coordinate relaxation iteration |
2617 | relaxd = .false. |
2618 | |
2619 | -#ifdef SIESTA__PEXSI |
2620 | +#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI) |
2621 | ! Broadcast relevant things for program logic |
2622 | ! These were set in siesta_options, called only by "SIESTA_workers". |
2623 | - call broadcast(inicoor,comm=true_MPI_Comm_World) |
2624 | - call broadcast(fincoor,comm=true_MPI_Comm_World) |
2625 | + call broadcast(inicoor,comm=mpi_comm_dft) |
2626 | + call broadcast(fincoor,comm=mpi_comm_dft) |
2627 | #endif |
2628 | istep = inicoor |
2629 | DO WHILE ((istep.le.fincoor) .AND. (.not. relaxd)) |
2630 | @@ -82,8 +82,8 @@ |
2631 | end if |
2632 | |
2633 | if (SIESTA_worker) call siesta_move( istep, relaxd ) |
2634 | -#ifdef SIESTA__PEXSI |
2635 | - call broadcast(relaxd,comm=true_MPI_Comm_World) |
2636 | +#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI) |
2637 | + call broadcast(relaxd,comm=mpi_comm_dft) |
2638 | #endif |
2639 | if (.not. relaxd) then |
2640 | istep = istep + 1 |
2641 | |
2642 | === modified file 'Src/siesta_forces.F90' |
2643 | --- Src/siesta_forces.F90 2018-09-24 07:34:04 +0000 |
2644 | +++ Src/siesta_forces.F90 2019-01-16 11:00:58 +0000 |
2645 | @@ -84,6 +84,9 @@ |
2646 | #ifdef SIESTA__PEXSI |
2647 | use m_pexsi, only: pexsi_finalize_scfloop |
2648 | #endif |
2649 | +#ifdef SIESTA__ELSI |
2650 | + use m_elsi_interface, only: elsi_finalize_scfloop |
2651 | +#endif |
2652 | use m_check_walltime |
2653 | |
2654 | use m_energies, only: DE_NEGF |
2655 | @@ -140,10 +143,10 @@ |
2656 | call write_debug( ' PRE siesta_forces' ) |
2657 | #endif |
2658 | |
2659 | -#ifdef SIESTA__PEXSI |
2660 | +#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI) |
2661 | ! Broadcast relevant things for program logic |
2662 | ! These were set in read_options, called only by "SIESTA_workers". |
2663 | - call broadcast(nscf, comm=true_MPI_Comm_World) |
2664 | + call broadcast(nscf, comm=mpi_comm_dft) |
2665 | #endif |
2666 | |
2667 | if ( SIESTA_worker ) then |
2668 | @@ -484,9 +487,9 @@ |
2669 | |
2670 | end if |
2671 | |
2672 | -#ifdef SIESTA__PEXSI |
2673 | - call broadcast(iscf, comm=true_MPI_Comm_World) |
2674 | - call broadcast(SCFconverged, comm=true_MPI_Comm_World) |
2675 | +#if defined (SIESTA__PEXSI) || defined (SIESTA__ELSI) |
2676 | + call broadcast(iscf, comm=mpi_comm_dft) |
2677 | + call broadcast(SCFconverged, comm=mpi_comm_dft) |
2678 | #endif |
2679 | |
2680 | ! Exit if converged |
2681 | @@ -500,9 +503,15 @@ |
2682 | end if |
2683 | #endif |
2684 | |
2685 | +<<<<<<< TREE |
2686 | if ( .not. SIESTA_worker ) return |
2687 | |
2688 | call end_of_cycle_save_operations(SCFconverged) |
2689 | +======= |
2690 | + if ( SIESTA_worker ) then |
2691 | +!=== |
2692 | + call end_of_cycle_save_operations() |
2693 | +>>>>>>> MERGE-SOURCE |
2694 | |
2695 | if ( .not. SCFconverged ) then |
2696 | if ( SCFMustConverge ) then |
2697 | @@ -538,11 +547,18 @@ |
2698 | ! Clean-up here to limit memory usage |
2699 | call mixers_scf_history_init( ) |
2700 | |
2701 | +!=== |
2702 | + endif ! Siesta_Worker |
2703 | + |
2704 | ! End of standard SCF loop. |
2705 | ! Do one more pass to compute forces and stresses |
2706 | |
2707 | ! Note that this call will no longer overwrite H while computing the |
2708 | ! final energies, forces and stresses... |
2709 | + |
2710 | + ! If we want to preserve the "Siesta_Worker" subset implementation for ELSI, |
2711 | + ! this block needs to be executed by everybody |
2712 | + ! as it contains a call to the ELSI interface to get the EDM |
2713 | |
2714 | if ( fdf_get("compute-forces",.true.) ) then |
2715 | call post_scf_work( istep, iscf , SCFconverged ) |
2716 | @@ -550,6 +566,15 @@ |
2717 | if (ionode) call memory_snapshot("after post_scf_work") |
2718 | #endif |
2719 | end if |
2720 | + |
2721 | +#ifdef SIESTA__ELSI |
2722 | + ! Recall that there could be an extra call after the scf loop to get the EDM |
2723 | + if ( isolve == SOLVE_ELSI ) then |
2724 | + call elsi_finalize_scfloop() |
2725 | + end if |
2726 | +#endif |
2727 | + |
2728 | + if (.not. Siesta_Worker) RETURN |
2729 | |
2730 | ! ... so H at this point is the latest generator of the DM, except |
2731 | ! if mixing H beyond self-consistency or terminating the scf loop |
2732 | |
2733 | === modified file 'Src/siesta_init.F' |
2734 | --- Src/siesta_init.F 2018-11-27 14:28:55 +0000 |
2735 | +++ Src/siesta_init.F 2019-01-16 11:00:58 +0000 |
2736 | @@ -267,8 +267,11 @@ |
2737 | call reinit( sname ) |
2738 | |
2739 | #ifdef MPI |
2740 | + |
2741 | +#ifdef SIESTA__PEXSI |
2742 | ! Optionally, restrict the number of processors that SIESTA uses, out |
2743 | - ! of the total pool |
2744 | + ! of the total pool (but only for PEXSI, not yet for other solvers) |
2745 | + |
2746 | npSIESTA = fdf_get("MPI.Nprocs.SIESTA",Nodes) |
2747 | call MPI_Comm_Group(MPI_Comm_World, World_Group, MPIerror) |
2748 | call MPI_Group_incl(World_Group, npSIESTA, |
2749 | @@ -278,8 +281,16 @@ |
2750 | $ SIESTA_Comm, MPIerror) |
2751 | |
2752 | SIESTA_worker = (Node < npSIESTA) |
2753 | - |
2754 | - ! Swap communicator |
2755 | +#else |
2756 | + SIESTA_Comm = MPI_Comm_World |
2757 | + SIESTA_worker = .true. |
2758 | +#endif |
2759 | + ! Swap communicators |
2760 | + ! This is needed in case that the whole program is run with |
2761 | + ! a subset of the global processes. In that case, the DFT instance |
2762 | + ! communicator (a better name than 'true world' communicator) |
2763 | + ! is at this point MPI_COMM_WORLD |
2764 | + MPI_COMM_DFT = MPI_COMM_WORLD |
2765 | ! This is needed since MPI_COMM_WORLD is used implicitly by |
2766 | ! Siesta for all operations. |
2767 | MPI_COMM_WORLD = SIESTA_Comm |
2768 | @@ -287,7 +298,7 @@ |
2769 | call MPI_Comm_Rank( MPI_Comm_World, Node, MPIerror ) |
2770 | call MPI_Comm_Size( MPI_Comm_World, Nodes, MPIerror ) |
2771 | endif |
2772 | -#else |
2773 | +#else /* not MPI */ |
2774 | SIESTA_worker = .true. |
2775 | Node = 0 |
2776 | Nodes = 1 |
2777 | @@ -586,7 +597,14 @@ |
2778 | ! auxiliary supercell for the calculation of matrix elements. |
2779 | call setup_kpoint_scf( ucell ) |
2780 | not_using_auxcell = gamma_scf |
2781 | - |
2782 | +#ifdef SIESTA__ELSI |
2783 | + if (isolve == SOLVE_ELSI) then |
2784 | + if (mod(Nodes,kpoint_scf%N*spin%spinor) /=0) then |
2785 | + call die("The number of MPI processes " // |
2786 | + $ "must be a multiple of nk*nspin") |
2787 | + endif |
2788 | + endif |
2789 | +#endif |
2790 | ! Call initialisation of PDOS here since we need to check if |
2791 | ! the auxiliary supercell is needed for a non-gamma calculation |
2792 | call init_projected_DOS( ucell ) |
2793 | |
2794 | === modified file 'Src/siesta_options.F90' |
2795 | --- Src/siesta_options.F90 2018-09-12 13:49:21 +0000 |
2796 | +++ Src/siesta_options.F90 2019-01-16 11:00:58 +0000 |
2797 | @@ -246,6 +246,7 @@ |
2798 | integer, parameter :: SOLVE_PEXSI = 4 |
2799 | integer, parameter :: MATRIX_WRITE = 5 |
2800 | integer, parameter :: SOLVE_CHESS = 6 |
2801 | + integer, parameter :: SOLVE_ELSI = 7 |
2802 | |
2803 | #ifdef SIESTA__FLOOK |
2804 | ! LUA-handle |
2805 | |
2806 | === modified file 'Src/siesta_tddft.F90' |
2807 | --- Src/siesta_tddft.F90 2018-06-26 17:09:38 +0000 |
2808 | +++ Src/siesta_tddft.F90 2019-01-16 11:00:58 +0000 |
2809 | @@ -61,7 +61,7 @@ |
2810 | #ifdef SIESTA__PEXSI |
2811 | ! Broadcast relevant things for program logic |
2812 | ! These were set in read_options, called only by "SIESTA_workers". |
2813 | - call broadcast(nscf,comm=true_MPI_Comm_World) |
2814 | + call broadcast(nscf,comm=MPI_Comm_DFT) |
2815 | #endif |
2816 | |
2817 | if ( SIESTA_worker ) then |
2818 | |
2819 | === modified file 'Src/state_init.F' |
2820 | --- Src/state_init.F 2018-10-23 09:46:30 +0000 |
2821 | +++ Src/state_init.F 2019-01-16 11:00:58 +0000 |
2822 | @@ -811,6 +811,10 @@ |
2823 | #endif |
2824 | call timer( 'state_init', 2 ) |
2825 | |
2826 | +! Call BLACS setup here if needed (all solvers except PEXSI) |
2827 | +! Setup S in 2d Scalapack distribution if needed (ELSI ELPA solver) |
2828 | +! Initialize ELSI |
2829 | + |
2830 | END subroutine state_init |
2831 | |
2832 | subroutine check_cohp() |
2833 | |
2834 | === modified file 'Src/write_subs.F' |
2835 | --- Src/write_subs.F 2018-06-04 14:08:20 +0000 |
2836 | +++ Src/write_subs.F 2019-01-16 11:00:58 +0000 |
2837 | @@ -441,6 +441,7 @@ |
2838 | write(6,"(/a,f14.6,/)") 'siesta: Eharris(eV) = ', Eharrs/eV |
2839 | ! No need for further cml output |
2840 | elseif ((isolve==SOLVE_DIAGON) .or. (isolve==SOLVE_PEXSI) |
2841 | + . .or. (isolve==SOLVE_ELSI) |
2842 | . .or. (isolve==SOLVE_CHESS) |
2843 | . .or. (isolve==SOLVE_TRANSI)) then |
2844 | if (cml_p) then |
2845 | |
2846 | === added directory 'Tests/si-quantum-dot' |
2847 | === added file 'Tests/si-quantum-dot/coords.Si987H372.fdf' |
2848 | --- Tests/si-quantum-dot/coords.Si987H372.fdf 1970-01-01 00:00:00 +0000 |
2849 | +++ Tests/si-quantum-dot/coords.Si987H372.fdf 2019-01-16 11:00:58 +0000 |
2850 | @@ -0,0 +1,1368 @@ |
2851 | +number-of-atoms 1359 |
2852 | +number-of-species 2 |
2853 | +# |
2854 | +# Si987H372 from CSIRO Silicon Quantum Dot dataset |
2855 | +# https://doi.org/10.4225/08/5721BB609EDB0 |
2856 | +# |
2857 | +atomic-coordinates-format Ang |
2858 | +%block atomic-coordinates-and-atomic-species |
2859 | +-12.27486761 -6.80192861 -4.06859976 1 # Si |
2860 | +-12.25680064 -9.54338724 -1.35303719 1 # Si |
2861 | +-15.00302593 -6.81988895 -1.33325565 1 # Si |
2862 | +-13.61801556 -8.18242131 -0.02610591 1 # Si |
2863 | +-12.27960089 -6.82966552 1.35920843 1 # Si |
2864 | +-12.28371979 -9.54137501 4.12509126 1 # Si |
2865 | +-13.62454044 -8.34100436 5.61139268 1 # Si |
2866 | +-12.30649445 -6.84800235 6.84800235 1 # Si |
2867 | +-12.25680064 -1.35303719 -9.54338724 1 # Si |
2868 | +-12.27486761 -4.06859976 -6.80192861 1 # Si |
2869 | +-15.00302593 -1.33325565 -6.81988895 1 # Si |
2870 | +-13.66848041 -2.72110232 -5.45900176 1 # Si |
2871 | +-13.66848041 -5.45900176 -2.72110232 1 # Si |
2872 | +-16.37023166 -2.73136988 -2.73136988 1 # Si |
2873 | +-15.01032944 -4.07390228 -4.07390228 1 # Si |
2874 | +-12.30121531 -1.35797308 -4.09496434 1 # Si |
2875 | +-12.30121531 -4.09496434 -1.35797308 1 # Si |
2876 | +-15.04397328 -1.36373167 -1.36373167 1 # Si |
2877 | +-16.34134070 -5.63195305 0.16771494 1 # Si |
2878 | +-13.68153993 -2.73058729 0.00300340 1 # Si |
2879 | +-13.61156377 -5.42510828 2.69385812 1 # Si |
2880 | +-16.34634363 -2.90187977 2.90187977 1 # Si |
2881 | +-15.03413728 -4.10489826 1.37590571 1 # Si |
2882 | +-12.30671934 -1.37011664 1.37011664 1 # Si |
2883 | +-12.27225867 -4.09402380 4.09402380 1 # Si |
2884 | +-15.03413728 -1.37590571 4.10489826 1 # Si |
2885 | +-13.61156377 -2.69385812 5.42510828 1 # Si |
2886 | +-13.62454044 -5.61139268 8.34100436 1 # Si |
2887 | +-12.27960089 -1.35920843 6.82966552 1 # Si |
2888 | +-12.28371979 -4.12509126 9.54137501 1 # Si |
2889 | +-13.61801556 -0.02610591 -8.18242131 1 # Si |
2890 | +-12.28371979 4.12509126 -9.54137501 1 # Si |
2891 | +-12.27960089 1.35920843 -6.82966552 1 # Si |
2892 | +-16.34134070 0.16771494 -5.63195305 1 # Si |
2893 | +-13.61156377 2.69385812 -5.42510828 1 # Si |
2894 | +-13.68153993 0.00300340 -2.73058729 1 # Si |
2895 | +-16.34634363 2.90187977 -2.90187977 1 # Si |
2896 | +-15.03413728 1.37590571 -4.10489826 1 # Si |
2897 | +-12.27225867 4.09402380 -4.09402380 1 # Si |
2898 | +-12.30671934 1.37011664 -1.37011664 1 # Si |
2899 | +-15.03413728 4.10489826 -1.37590571 1 # Si |
2900 | +-16.37821316 0.00000000 0.00000000 1 # Si |
2901 | +-13.68153993 2.73058729 -0.00300340 1 # Si |
2902 | +-13.68153993 -0.00300340 2.73058729 1 # Si |
2903 | +-16.37023166 2.73136988 2.73136988 1 # Si |
2904 | +-15.04397328 1.36373167 1.36373167 1 # Si |
2905 | +-12.30121531 4.09496434 1.35797308 1 # Si |
2906 | +-12.30121531 1.35797308 4.09496434 1 # Si |
2907 | +-15.01032944 4.07390228 4.07390228 1 # Si |
2908 | +-16.34134070 -0.16771494 5.63195305 1 # Si |
2909 | +-13.66848041 2.72110232 5.45900176 1 # Si |
2910 | +-13.61801556 0.02610591 8.18242131 1 # Si |
2911 | +-15.00302593 1.33325565 6.81988895 1 # Si |
2912 | +-12.27486761 4.06859976 6.80192861 1 # Si |
2913 | +-12.25680064 1.35303719 9.54338724 1 # Si |
2914 | +-13.62454044 5.61139268 -8.34100436 1 # Si |
2915 | +-12.30649445 6.84800235 -6.84800235 1 # Si |
2916 | +-13.62454044 8.34100436 -5.61139268 1 # Si |
2917 | +-13.61156377 5.42510828 -2.69385812 1 # Si |
2918 | +-12.28371979 9.54137501 -4.12509126 1 # Si |
2919 | +-12.27960089 6.82966552 -1.35920843 1 # Si |
2920 | +-16.34134070 5.63195305 -0.16771494 1 # Si |
2921 | +-13.61801556 8.18242131 0.02610591 1 # Si |
2922 | +-13.66848041 5.45900176 2.72110232 1 # Si |
2923 | +-15.00302593 6.81988895 1.33325565 1 # Si |
2924 | +-12.25680064 9.54338724 1.35303719 1 # Si |
2925 | +-12.27486761 6.80192861 4.06859976 1 # Si |
2926 | +-6.80192861 -12.27486761 -4.06859976 1 # Si |
2927 | +-6.81988895 -15.00302593 -1.33325565 1 # Si |
2928 | +-9.54338724 -12.25680064 -1.35303719 1 # Si |
2929 | +-8.18242131 -13.61801556 -0.02610591 1 # Si |
2930 | +-6.82966552 -12.27960089 1.35920843 1 # Si |
2931 | +-9.54137501 -12.28371979 4.12509126 1 # Si |
2932 | +-8.34100436 -13.62454044 5.61139268 1 # Si |
2933 | +-6.84800235 -12.30649445 6.84800235 1 # Si |
2934 | +-6.80203106 -6.80203106 -9.53854106 1 # Si |
2935 | +-6.80203106 -9.53854106 -6.80203106 1 # Si |
2936 | +-9.53854106 -6.80203106 -6.80203106 1 # Si |
2937 | +-8.18684496 -8.18684496 -5.46061302 1 # Si |
2938 | +-8.18958251 -10.91859845 -2.73303284 1 # Si |
2939 | +-10.91859845 -8.18958251 -2.73303284 1 # Si |
2940 | +-9.53373688 -9.53373688 -4.08178398 1 # Si |
2941 | +-6.82463094 -6.82463094 -4.09496380 1 # Si |
2942 | +-6.82852418 -9.55456223 -1.36492084 1 # Si |
2943 | +-9.55456223 -6.82852418 -1.36492084 1 # Si |
2944 | +-10.90149235 -10.90149235 -0.01304838 1 # Si |
2945 | +-8.19611273 -8.19611273 0.01202455 1 # Si |
2946 | +-8.19260795 -10.93410102 2.74437054 1 # Si |
2947 | +-10.93410102 -8.19260795 2.74437054 1 # Si |
2948 | +-9.56265344 -9.56265344 1.38318968 1 # Si |
2949 | +-6.82894278 -6.82894278 1.37205363 1 # Si |
2950 | +-6.82705083 -9.56489675 4.10298675 1 # Si |
2951 | +-9.56489675 -6.82705083 4.10298675 1 # Si |
2952 | +-10.90291341 -10.90291341 5.43284542 1 # Si |
2953 | +-8.19345007 -8.19345007 5.46312108 1 # Si |
2954 | +-8.17530342 -10.89255734 8.17530342 1 # Si |
2955 | +-10.89255734 -8.17530342 8.17530342 1 # Si |
2956 | +-9.55790920 -9.55790920 6.82271744 1 # Si |
2957 | +-6.82700831 -6.82700831 6.82700831 1 # Si |
2958 | +-6.82271744 -9.55790920 9.55790920 1 # Si |
2959 | +-9.55790920 -6.82271744 9.55790920 1 # Si |
2960 | +-8.17530342 -8.17530342 10.89255734 1 # Si |
2961 | +-6.84800235 -6.84800235 12.30649445 1 # Si |
2962 | +-6.81988895 -1.33325565 -15.00302593 1 # Si |
2963 | +-6.80192861 -4.06859976 -12.27486761 1 # Si |
2964 | +-9.54338724 -1.35303719 -12.25680064 1 # Si |
2965 | +-8.18958251 -2.73303284 -10.91859845 1 # Si |
2966 | +-8.18684496 -5.46061302 -8.18684496 1 # Si |
2967 | +-10.91859845 -2.73303284 -8.18958251 1 # Si |
2968 | +-9.53373688 -4.08178398 -9.53373688 1 # Si |
2969 | +-6.82852418 -1.36492084 -9.55456223 1 # Si |
2970 | +-6.82463094 -4.09496380 -6.82463094 1 # Si |
2971 | +-9.55456223 -1.36492084 -6.82852418 1 # Si |
2972 | +-10.92814020 -5.45537137 -5.45537137 1 # Si |
2973 | +-8.19145655 -2.72890018 -5.45886981 1 # Si |
2974 | +-8.19145655 -5.45886981 -2.72890018 1 # Si |
2975 | +-10.93548204 -2.72665137 -2.72665137 1 # Si |
2976 | +-9.56309566 -4.09195916 -4.09195916 1 # Si |
2977 | +-6.82665093 -1.36387577 -4.09489509 1 # Si |
2978 | +-6.82665093 -4.09489509 -1.36387577 1 # Si |
2979 | +-9.56805296 -1.36256401 -1.36256401 1 # Si |
2980 | +-10.91922202 -5.46070202 -0.00117430 1 # Si |
2981 | +-8.19459074 -2.72767707 -0.00218763 1 # Si |
2982 | +-8.19802839 -5.46221414 2.73481248 1 # Si |
2983 | +-10.91829649 -2.72813440 2.72813440 1 # Si |
2984 | +-9.55550648 -4.09408870 1.36300571 1 # Si |
2985 | +-6.82823155 -1.36467215 1.36467215 1 # Si |
2986 | +-6.82965269 -4.09758805 4.09758805 1 # Si |
2987 | +-9.55550648 -1.36300571 4.09408870 1 # Si |
2988 | +-10.94234231 -5.47569502 5.47569502 1 # Si |
2989 | +-8.19802839 -2.73481248 5.46221414 1 # Si |
2990 | +-8.19345007 -5.46312108 8.19345007 1 # Si |
2991 | +-10.93410102 -2.74437054 8.19260795 1 # Si |
2992 | +-9.56489675 -4.10298675 6.82705083 1 # Si |
2993 | +-6.82894278 -1.37205363 6.82894278 1 # Si |
2994 | +-6.82705083 -4.10298675 9.56489675 1 # Si |
2995 | +-9.56265344 -1.38318968 9.56265344 1 # Si |
2996 | +-10.90291341 -5.43284542 10.90291341 1 # Si |
2997 | +-8.19260795 -2.74437054 10.93410102 1 # Si |
2998 | +-8.34100436 -5.61139268 13.62454044 1 # Si |
2999 | +-9.54137501 -4.12509126 12.28371979 1 # Si |
3000 | +-6.82966552 -1.35920843 12.27960089 1 # Si |
3001 | +-8.18242131 -0.02610591 -13.61801556 1 # Si |
3002 | +-6.82966552 1.35920843 -12.27960089 1 # Si |
3003 | +-9.54137501 4.12509126 -12.28371979 1 # Si |
3004 | +-10.90149235 -0.01304838 -10.90149235 1 # Si |
3005 | +-8.19260795 2.74437054 -10.93410102 1 # Si |
3006 | +-8.19611273 0.01202455 -8.19611273 1 # Si |
3007 | +-10.93410102 2.74437054 -8.19260795 1 # Si |
3008 | +-9.56265344 1.38318968 -9.56265344 1 # Si |
3009 | +-6.82705083 4.10298675 -9.56489675 1 # Si |
3010 | +-6.82894278 1.37205363 -6.82894278 1 # Si |
3011 | +-9.56489675 4.10298675 -6.82705083 1 # Si |
3012 | +-10.91922202 -0.00117430 -5.46070202 1 # Si |
3013 | +-8.19802839 2.73481248 -5.46221414 1 # Si |
3014 | +-8.19459074 -0.00218763 -2.72767707 1 # Si |
3015 | +-10.91829649 2.72813440 -2.72813440 1 # Si |
3016 | +-9.55550648 1.36300571 -4.09408870 1 # Si |
3017 | +-6.82965269 4.09758805 -4.09758805 1 # Si |
3018 | +-6.82823155 1.36467215 -1.36467215 1 # Si |
3019 | +-9.55550648 4.09408870 -1.36300571 1 # Si |
3020 | +-10.94349563 0.00000000 -0.00000000 1 # Si |
3021 | +-8.19459074 2.72767707 0.00218763 1 # Si |
3022 | +-8.19459074 0.00218763 2.72767707 1 # Si |
3023 | +-10.93548204 2.72665137 2.72665137 1 # Si |
3024 | +-9.56805296 1.36256401 1.36256401 1 # Si |
3025 | +-6.82665093 4.09489509 1.36387577 1 # Si |
3026 | +-6.82665093 1.36387577 4.09489509 1 # Si |
3027 | +-9.56309566 4.09195916 4.09195916 1 # Si |
3028 | +-10.91922202 0.00117430 5.46070202 1 # Si |
3029 | +-8.19145655 2.72890018 5.45886981 1 # Si |
3030 | +-8.19611273 -0.01202455 8.19611273 1 # Si |
3031 | +-10.91859845 2.73303284 8.18958251 1 # Si |
3032 | +-9.55456223 1.36492084 6.82852418 1 # Si |
3033 | +-6.82463094 4.09496380 6.82463094 1 # Si |
3034 | +-6.82852418 1.36492084 9.55456223 1 # Si |
3035 | +-9.53373688 4.08178398 9.53373688 1 # Si |
3036 | +-10.90149235 0.01304838 10.90149235 1 # Si |
3037 | +-8.18958251 2.73303284 10.91859845 1 # Si |
3038 | +-8.18242131 0.02610591 13.61801556 1 # Si |
3039 | +-9.54338724 1.35303719 12.25680064 1 # Si |
3040 | +-6.80192861 4.06859976 12.27486761 1 # Si |
3041 | +-6.81988895 1.33325565 15.00302593 1 # Si |
3042 | +-8.34100436 5.61139268 -13.62454044 1 # Si |
3043 | +-6.84800235 6.84800235 -12.30649445 1 # Si |
3044 | +-10.90291341 5.43284542 -10.90291341 1 # Si |
3045 | +-8.17530342 8.17530342 -10.89255734 1 # Si |
3046 | +-8.19345007 5.46312108 -8.19345007 1 # Si |
3047 | +-10.89255734 8.17530342 -8.17530342 1 # Si |
3048 | +-9.55790920 6.82271744 -9.55790920 1 # Si |
3049 | +-6.82271744 9.55790920 -9.55790920 1 # Si |
3050 | +-6.82700831 6.82700831 -6.82700831 1 # Si |
3051 | +-9.55790920 9.55790920 -6.82271744 1 # Si |
3052 | +-10.94234231 5.47569502 -5.47569502 1 # Si |
3053 | +-8.19345007 8.19345007 -5.46312108 1 # Si |
3054 | +-8.19802839 5.46221414 -2.73481248 1 # Si |
3055 | +-10.93410102 8.19260795 -2.74437054 1 # Si |
3056 | +-9.56489675 6.82705083 -4.10298675 1 # Si |
3057 | +-6.82705083 9.56489675 -4.10298675 1 # Si |
3058 | +-6.82894278 6.82894278 -1.37205363 1 # Si |
3059 | +-9.56265344 9.56265344 -1.38318968 1 # Si |
3060 | +-10.91922202 5.46070202 0.00117430 1 # Si |
3061 | +-8.19611273 8.19611273 -0.01202455 1 # Si |
3062 | +-8.19145655 5.45886981 2.72890018 1 # Si |
3063 | +-10.91859845 8.18958251 2.73303284 1 # Si |
3064 | +-9.55456223 6.82852418 1.36492084 1 # Si |
3065 | +-6.82852418 9.55456223 1.36492084 1 # Si |
3066 | +-6.82463094 6.82463094 4.09496380 1 # Si |
3067 | +-9.53373688 9.53373688 4.08178398 1 # Si |
3068 | +-10.92814020 5.45537137 5.45537137 1 # Si |
3069 | +-8.18684496 8.18684496 5.46061302 1 # Si |
3070 | +-8.18684496 5.46061302 8.18684496 1 # Si |
3071 | +-9.53854106 6.80203106 6.80203106 1 # Si |
3072 | +-6.80203106 9.53854106 6.80203106 1 # Si |
3073 | +-6.80203106 6.80203106 9.53854106 1 # Si |
3074 | +-8.17530342 10.89255734 -8.17530342 1 # Si |
3075 | +-6.84800235 12.30649445 -6.84800235 1 # Si |
3076 | +-10.90291341 10.90291341 -5.43284542 1 # Si |
3077 | +-8.34100436 13.62454044 -5.61139268 1 # Si |
3078 | +-8.19260795 10.93410102 -2.74437054 1 # Si |
3079 | +-9.54137501 12.28371979 -4.12509126 1 # Si |
3080 | +-6.82966552 12.27960089 -1.35920843 1 # Si |
3081 | +-10.90149235 10.90149235 0.01304838 1 # Si |
3082 | +-8.18242131 13.61801556 0.02610591 1 # Si |
3083 | +-8.18958251 10.91859845 2.73303284 1 # Si |
3084 | +-9.54338724 12.25680064 1.35303719 1 # Si |
3085 | +-6.81988895 15.00302593 1.33325565 1 # Si |
3086 | +-6.80192861 12.27486761 4.06859976 1 # Si |
3087 | +-1.35303719 -12.25680064 -9.54338724 1 # Si |
3088 | +-1.33325565 -15.00302593 -6.81988895 1 # Si |
3089 | +-4.06859976 -12.27486761 -6.80192861 1 # Si |
3090 | +-2.72110232 -13.66848041 -5.45900176 1 # Si |
3091 | +-2.73136988 -16.37023166 -2.73136988 1 # Si |
3092 | +-5.45900176 -13.66848041 -2.72110232 1 # Si |
3093 | +-4.07390228 -15.01032944 -4.07390228 1 # Si |
3094 | +-1.35797308 -12.30121531 -4.09496434 1 # Si |
3095 | +-1.36373167 -15.04397328 -1.36373167 1 # Si |
3096 | +-4.09496434 -12.30121531 -1.35797308 1 # Si |
3097 | +-5.63195305 -16.34134070 0.16771494 1 # Si |
3098 | +-2.73058729 -13.68153993 0.00300340 1 # Si |
3099 | +-2.90187977 -16.34634363 2.90187977 1 # Si |
3100 | +-5.42510828 -13.61156377 2.69385812 1 # Si |
3101 | +-4.10489826 -15.03413728 1.37590571 1 # Si |
3102 | +-1.37011664 -12.30671934 1.37011664 1 # Si |
3103 | +-1.37590571 -15.03413728 4.10489826 1 # Si |
3104 | +-4.09402380 -12.27225867 4.09402380 1 # Si |
3105 | +-2.69385812 -13.61156377 5.42510828 1 # Si |
3106 | +-5.61139268 -13.62454044 8.34100436 1 # Si |
3107 | +-1.35920843 -12.27960089 6.82966552 1 # Si |
3108 | +-4.12509126 -12.28371979 9.54137501 1 # Si |
3109 | +-1.33325565 -6.81988895 -15.00302593 1 # Si |
3110 | +-1.35303719 -9.54338724 -12.25680064 1 # Si |
3111 | +-4.06859976 -6.80192861 -12.27486761 1 # Si |
3112 | +-2.73303284 -8.18958251 -10.91859845 1 # Si |
3113 | +-2.73303284 -10.91859845 -8.18958251 1 # Si |
3114 | +-5.46061302 -8.18684496 -8.18684496 1 # Si |
3115 | +-4.08178398 -9.53373688 -9.53373688 1 # Si |
3116 | +-1.36492084 -6.82852418 -9.55456223 1 # Si |
3117 | +-1.36492084 -9.55456223 -6.82852418 1 # Si |
3118 | +-4.09496380 -6.82463094 -6.82463094 1 # Si |
3119 | +-5.45537137 -10.92814020 -5.45537137 1 # Si |
3120 | +-2.72890018 -8.19145655 -5.45886981 1 # Si |
3121 | +-2.72665137 -10.93548204 -2.72665137 1 # Si |
3122 | +-5.45886981 -8.19145655 -2.72890018 1 # Si |
3123 | +-4.09195916 -9.56309566 -4.09195916 1 # Si |
3124 | +-1.36387577 -6.82665093 -4.09489509 1 # Si |
3125 | +-1.36256401 -9.56805296 -1.36256401 1 # Si |
3126 | +-4.09489509 -6.82665093 -1.36387577 1 # Si |
3127 | +-5.46070202 -10.91922202 -0.00117430 1 # Si |
3128 | +-2.72767707 -8.19459074 -0.00218763 1 # Si |
3129 | +-2.72813440 -10.91829649 2.72813440 1 # Si |
3130 | +-5.46221414 -8.19802839 2.73481248 1 # Si |
3131 | +-4.09408870 -9.55550648 1.36300571 1 # Si |
3132 | +-1.36467215 -6.82823155 1.36467215 1 # Si |
3133 | +-1.36300571 -9.55550648 4.09408870 1 # Si |
3134 | +-4.09758805 -6.82965269 4.09758805 1 # Si |
3135 | +-5.47569502 -10.94234231 5.47569502 1 # Si |
3136 | +-2.73481248 -8.19802839 5.46221414 1 # Si |
3137 | +-2.74437054 -10.93410102 8.19260795 1 # Si |
3138 | +-5.46312108 -8.19345007 8.19345007 1 # Si |
3139 | +-4.10298675 -9.56489675 6.82705083 1 # Si |
3140 | +-1.37205363 -6.82894278 6.82894278 1 # Si |
3141 | +-1.38318968 -9.56265344 9.56265344 1 # Si |
3142 | +-4.10298675 -6.82705083 9.56489675 1 # Si |
3143 | +-5.43284542 -10.90291341 10.90291341 1 # Si |
3144 | +-2.74437054 -8.19260795 10.93410102 1 # Si |
3145 | +-5.61139268 -8.34100436 13.62454044 1 # Si |
3146 | +-4.12509126 -9.54137501 12.28371979 1 # Si |
3147 | +-1.35920843 -6.82966552 12.27960089 1 # Si |
3148 | +-2.73136988 -2.73136988 -16.37023166 1 # Si |
3149 | +-2.72110232 -5.45900176 -13.66848041 1 # Si |
3150 | +-5.45900176 -2.72110232 -13.66848041 1 # Si |
3151 | +-4.07390228 -4.07390228 -15.01032944 1 # Si |
3152 | +-1.36373167 -1.36373167 -15.04397328 1 # Si |
3153 | +-1.35797308 -4.09496434 -12.30121531 1 # Si |
3154 | +-4.09496434 -1.35797308 -12.30121531 1 # Si |
3155 | +-5.45537137 -5.45537137 -10.92814020 1 # Si |
3156 | +-2.72665137 -2.72665137 -10.93548204 1 # Si |
3157 | +-2.72890018 -5.45886981 -8.19145655 1 # Si |
3158 | +-5.45886981 -2.72890018 -8.19145655 1 # Si |
3159 | +-4.09195916 -4.09195916 -9.56309566 1 # Si |
3160 | +-1.36256401 -1.36256401 -9.56805296 1 # Si |
3161 | +-1.36387577 -4.09489509 -6.82665093 1 # Si |
3162 | +-4.09489509 -1.36387577 -6.82665093 1 # Si |
3163 | +-5.45846237 -5.45846237 -5.45846237 1 # Si |
3164 | +-2.72992539 -2.72992539 -5.46104379 1 # Si |
3165 | +-2.72992539 -5.46104379 -2.72992539 1 # Si |
3166 | +-5.46104379 -2.72992539 -2.72992539 1 # Si |
3167 | +-4.09449230 -4.09449230 -4.09449230 1 # Si |
3168 | +-1.36499559 -1.36499559 -4.09571959 1 # Si |
3169 | +-1.36499559 -4.09571959 -1.36499559 1 # Si |
3170 | +-4.09571959 -1.36499559 -1.36499559 1 # Si |
3171 | +-5.46373779 -5.46373779 0.00319897 1 # Si |
3172 | +-2.73149619 -2.73149619 0.00070150 1 # Si |
3173 | +-2.73166810 -5.46412728 2.73166810 1 # Si |
3174 | +-5.46412728 -2.73166810 2.73166810 1 # Si |
3175 | +-4.09725659 -4.09725659 1.36658639 1 # Si |
3176 | +-1.36585704 -1.36585704 1.36585704 1 # Si |
3177 | +-1.36658639 -4.09725659 4.09725659 1 # Si |
3178 | +-4.09725659 -1.36658639 4.09725659 1 # Si |
3179 | +-5.46236779 -5.46236779 5.46236779 1 # Si |
3180 | +-2.73166810 -2.73166810 5.46412728 1 # Si |
3181 | +-2.73481248 -5.46221414 8.19802839 1 # Si |
3182 | +-5.46221414 -2.73481248 8.19802839 1 # Si |
3183 | +-4.09758805 -4.09758805 6.82965269 1 # Si |
3184 | +-1.36467215 -1.36467215 6.82823155 1 # Si |
3185 | +-1.36300571 -4.09408870 9.55550648 1 # Si |
3186 | +-4.09408870 -1.36300571 9.55550648 1 # Si |
3187 | +-5.47569502 -5.47569502 10.94234231 1 # Si |
3188 | +-2.72813440 -2.72813440 10.91829649 1 # Si |
3189 | +-2.69385812 -5.42510828 13.61156377 1 # Si |
3190 | +-5.42510828 -2.69385812 13.61156377 1 # Si |
3191 | +-4.09402380 -4.09402380 12.27225867 1 # Si |
3192 | +-1.37011664 -1.37011664 12.30671934 1 # Si |
3193 | +-1.37590571 -4.10489826 15.03413728 1 # Si |
3194 | +-4.10489826 -1.37590571 15.03413728 1 # Si |
3195 | +-2.90187977 -2.90187977 16.34634363 1 # Si |
3196 | +-5.63195305 0.16771494 -16.34134070 1 # Si |
3197 | +-2.90187977 2.90187977 -16.34634363 1 # Si |
3198 | +-2.73058729 0.00300340 -13.68153993 1 # Si |
3199 | +-5.42510828 2.69385812 -13.61156377 1 # Si |
3200 | +-4.10489826 1.37590571 -15.03413728 1 # Si |
3201 | +-1.37590571 4.10489826 -15.03413728 1 # Si |
3202 | +-1.37011664 1.37011664 -12.30671934 1 # Si |
3203 | +-4.09402380 4.09402380 -12.27225867 1 # Si |
3204 | +-5.46070202 -0.00117430 -10.91922202 1 # Si |
3205 | +-2.72813440 2.72813440 -10.91829649 1 # Si |
3206 | +-2.72767707 -0.00218763 -8.19459074 1 # Si |
3207 | +-5.46221414 2.73481248 -8.19802839 1 # Si |
3208 | +-4.09408870 1.36300571 -9.55550648 1 # Si |
3209 | +-1.36300571 4.09408870 -9.55550648 1 # Si |
3210 | +-1.36467215 1.36467215 -6.82823155 1 # Si |
3211 | +-4.09758805 4.09758805 -6.82965269 1 # Si |
3212 | +-5.46373779 0.00319897 -5.46373779 1 # Si |
3213 | +-2.73166810 2.73166810 -5.46412728 1 # Si |
3214 | +-2.73149619 0.00070150 -2.73149619 1 # Si |
3215 | +-5.46412728 2.73166810 -2.73166810 1 # Si |
3216 | +-4.09725659 1.36658639 -4.09725659 1 # Si |
3217 | +-1.36658639 4.09725659 -4.09725659 1 # Si |
3218 | +-1.36585704 1.36585704 -1.36585704 1 # Si |
3219 | +-4.09725659 4.09725659 -1.36658639 1 # Si |
3220 | +-5.46151579 -0.00000000 0.00000000 1 # Si |
3221 | +-2.73149619 2.73149619 -0.00070150 1 # Si |
3222 | +-2.73149619 -0.00070150 2.73149619 1 # Si |
3223 | +-5.46104379 2.72992539 2.72992539 1 # Si |
3224 | +-4.09571959 1.36499559 1.36499559 1 # Si |
3225 | +-1.36499559 4.09571959 1.36499559 1 # Si |
3226 | +-1.36499559 1.36499559 4.09571959 1 # Si |
3227 | +-4.09449230 4.09449230 4.09449230 1 # Si |
3228 | +-5.46373779 -0.00319897 5.46373779 1 # Si |
3229 | +-2.72992539 2.72992539 5.46104379 1 # Si |
3230 | +-2.72767707 0.00218763 8.19459074 1 # Si |
3231 | +-5.45886981 2.72890018 8.19145655 1 # Si |
3232 | +-4.09489509 1.36387577 6.82665093 1 # Si |
3233 | +-1.36387577 4.09489509 6.82665093 1 # Si |
3234 | +-1.36256401 1.36256401 9.56805296 1 # Si |
3235 | +-4.09195916 4.09195916 9.56309566 1 # Si |
3236 | +-5.46070202 0.00117430 10.91922202 1 # Si |
3237 | +-2.72665137 2.72665137 10.93548204 1 # Si |
3238 | +-2.73058729 -0.00300340 13.68153993 1 # Si |
3239 | +-5.45900176 2.72110232 13.66848041 1 # Si |
3240 | +-4.09496434 1.35797308 12.30121531 1 # Si |
3241 | +-1.35797308 4.09496434 12.30121531 1 # Si |
3242 | +-1.36373167 1.36373167 15.04397328 1 # Si |
3243 | +-4.07390228 4.07390228 15.01032944 1 # Si |
3244 | +-5.63195305 -0.16771494 16.34134070 1 # Si |
3245 | +-2.73136988 2.73136988 16.37023166 1 # Si |
3246 | +-2.69385812 5.42510828 -13.61156377 1 # Si |
3247 | +-5.61139268 8.34100436 -13.62454044 1 # Si |
3248 | +-1.35920843 6.82966552 -12.27960089 1 # Si |
3249 | +-4.12509126 9.54137501 -12.28371979 1 # Si |
3250 | +-5.47569502 5.47569502 -10.94234231 1 # Si |
3251 | +-2.74437054 8.19260795 -10.93410102 1 # Si |
3252 | +-2.73481248 5.46221414 -8.19802839 1 # Si |
3253 | +-5.46312108 8.19345007 -8.19345007 1 # Si |
3254 | +-4.10298675 6.82705083 -9.56489675 1 # Si |
3255 | +-1.38318968 9.56265344 -9.56265344 1 # Si |
3256 | +-1.37205363 6.82894278 -6.82894278 1 # Si |
3257 | +-4.10298675 9.56489675 -6.82705083 1 # Si |
3258 | +-5.46236779 5.46236779 -5.46236779 1 # Si |
3259 | +-2.73481248 8.19802839 -5.46221414 1 # Si |
3260 | +-2.73166810 5.46412728 -2.73166810 1 # Si |
3261 | +-5.46221414 8.19802839 -2.73481248 1 # Si |
3262 | +-4.09758805 6.82965269 -4.09758805 1 # Si |
3263 | +-1.36300571 9.55550648 -4.09408870 1 # Si |
3264 | +-1.36467215 6.82823155 -1.36467215 1 # Si |
3265 | +-4.09408870 9.55550648 -1.36300571 1 # Si |
3266 | +-5.46373779 5.46373779 -0.00319897 1 # Si |
3267 | +-2.72767707 8.19459074 0.00218763 1 # Si |
3268 | +-2.72992539 5.46104379 2.72992539 1 # Si |
3269 | +-5.45886981 8.19145655 2.72890018 1 # Si |
3270 | +-4.09489509 6.82665093 1.36387577 1 # Si |
3271 | +-1.36256401 9.56805296 1.36256401 1 # Si |
3272 | +-1.36387577 6.82665093 4.09489509 1 # Si |
3273 | +-4.09195916 9.56309566 4.09195916 1 # Si |
3274 | +-5.45846237 5.45846237 5.45846237 1 # Si |
3275 | +-2.72890018 8.19145655 5.45886981 1 # Si |
3276 | +-2.72890018 5.45886981 8.19145655 1 # Si |
3277 | +-5.46061302 8.18684496 8.18684496 1 # Si |
3278 | +-4.09496380 6.82463094 6.82463094 1 # Si |
3279 | +-1.36492084 9.55456223 6.82852418 1 # Si |
3280 | +-1.36492084 6.82852418 9.55456223 1 # Si |
3281 | +-4.08178398 9.53373688 9.53373688 1 # Si |
3282 | +-5.45537137 5.45537137 10.92814020 1 # Si |
3283 | +-2.73303284 8.18958251 10.91859845 1 # Si |
3284 | +-2.72110232 5.45900176 13.66848041 1 # Si |
3285 | +-4.06859976 6.80192861 12.27486761 1 # Si |
3286 | +-1.35303719 9.54338724 12.25680064 1 # Si |
3287 | +-1.33325565 6.81988895 15.00302593 1 # Si |
3288 | +-5.43284542 10.90291341 -10.90291341 1 # Si |
3289 | +-2.74437054 10.93410102 -8.19260795 1 # Si |
3290 | +-5.61139268 13.62454044 -8.34100436 1 # Si |
3291 | +-4.12509126 12.28371979 -9.54137501 1 # Si |
3292 | +-1.35920843 12.27960089 -6.82966552 1 # Si |
3293 | +-5.47569502 10.94234231 -5.47569502 1 # Si |
3294 | +-2.69385812 13.61156377 -5.42510828 1 # Si |
3295 | +-2.72813440 10.91829649 -2.72813440 1 # Si |
3296 | +-5.42510828 13.61156377 -2.69385812 1 # Si |
3297 | +-4.09402380 12.27225867 -4.09402380 1 # Si |
3298 | +-1.37590571 15.03413728 -4.10489826 1 # Si |
3299 | +-1.37011664 12.30671934 -1.37011664 1 # Si |
3300 | +-4.10489826 15.03413728 -1.37590571 1 # Si |
3301 | +-5.46070202 10.91922202 0.00117430 1 # Si |
3302 | +-2.73058729 13.68153993 -0.00300340 1 # Si |
3303 | +-2.72665137 10.93548204 2.72665137 1 # Si |
3304 | +-5.45900176 13.66848041 2.72110232 1 # Si |
3305 | +-4.09496434 12.30121531 1.35797308 1 # Si |
3306 | +-1.36373167 15.04397328 1.36373167 1 # Si |
3307 | +-1.35797308 12.30121531 4.09496434 1 # Si |
3308 | +-4.07390228 15.01032944 4.07390228 1 # Si |
3309 | +-5.45537137 10.92814020 5.45537137 1 # Si |
3310 | +-2.72110232 13.66848041 5.45900176 1 # Si |
3311 | +-2.73303284 10.91859845 8.18958251 1 # Si |
3312 | +-4.06859976 12.27486761 6.80192861 1 # Si |
3313 | +-1.33325565 15.00302593 6.81988895 1 # Si |
3314 | +-1.35303719 12.25680064 9.54338724 1 # Si |
3315 | +-2.90187977 16.34634363 -2.90187977 1 # Si |
3316 | +-5.63195305 16.34134070 -0.16771494 1 # Si |
3317 | +-2.73136988 16.37023166 2.73136988 1 # Si |
3318 | +-0.02610591 -13.61801556 -8.18242131 1 # Si |
3319 | +4.12509126 -12.28371979 -9.54137501 1 # Si |
3320 | +1.35920843 -12.27960089 -6.82966552 1 # Si |
3321 | +0.16771494 -16.34134070 -5.63195305 1 # Si |
3322 | +2.69385812 -13.61156377 -5.42510828 1 # Si |
3323 | +2.90187977 -16.34634363 -2.90187977 1 # Si |
3324 | +0.00300340 -13.68153993 -2.73058729 1 # Si |
3325 | +1.37590571 -15.03413728 -4.10489826 1 # Si |
3326 | +4.09402380 -12.27225867 -4.09402380 1 # Si |
3327 | +4.10489826 -15.03413728 -1.37590571 1 # Si |
3328 | +1.37011664 -12.30671934 -1.37011664 1 # Si |
3329 | +-0.00000000 -16.37821316 -0.00000000 1 # Si |
3330 | +2.73058729 -13.68153993 -0.00300340 1 # Si |
3331 | +2.73136988 -16.37023166 2.73136988 1 # Si |
3332 | +-0.00300340 -13.68153993 2.73058729 1 # Si |
3333 | +1.36373167 -15.04397328 1.36373167 1 # Si |
3334 | +4.09496434 -12.30121531 1.35797308 1 # Si |
3335 | +4.07390228 -15.01032944 4.07390228 1 # Si |
3336 | +1.35797308 -12.30121531 4.09496434 1 # Si |
3337 | +-0.16771494 -16.34134070 5.63195305 1 # Si |
3338 | +2.72110232 -13.66848041 5.45900176 1 # Si |
3339 | +0.02610591 -13.61801556 8.18242131 1 # Si |
3340 | +1.33325565 -15.00302593 6.81988895 1 # Si |
3341 | +4.06859976 -12.27486761 6.80192861 1 # Si |
3342 | +1.35303719 -12.25680064 9.54338724 1 # Si |
3343 | +-0.02610591 -8.18242131 -13.61801556 1 # Si |
3344 | +4.12509126 -9.54137501 -12.28371979 1 # Si |
3345 | +1.35920843 -6.82966552 -12.27960089 1 # Si |
3346 | +-0.01304838 -10.90149235 -10.90149235 1 # Si |
3347 | +2.74437054 -8.19260795 -10.93410102 1 # Si |
3348 | +2.74437054 -10.93410102 -8.19260795 1 # Si |
3349 | +0.01202455 -8.19611273 -8.19611273 1 # Si |
3350 | +1.38318968 -9.56265344 -9.56265344 1 # Si |
3351 | +4.10298675 -6.82705083 -9.56489675 1 # Si |
3352 | +4.10298675 -9.56489675 -6.82705083 1 # Si |
3353 | +1.37205363 -6.82894278 -6.82894278 1 # Si |
3354 | +-0.00117430 -10.91922202 -5.46070202 1 # Si |
3355 | +2.73481248 -8.19802839 -5.46221414 1 # Si |
3356 | +2.72813440 -10.91829649 -2.72813440 1 # Si |
3357 | +-0.00218763 -8.19459074 -2.72767707 1 # Si |
3358 | +1.36300571 -9.55550648 -4.09408870 1 # Si |
3359 | +4.09758805 -6.82965269 -4.09758805 1 # Si |
3360 | +4.09408870 -9.55550648 -1.36300571 1 # Si |
3361 | +1.36467215 -6.82823155 -1.36467215 1 # Si |
3362 | +0.00000000 -10.94349563 0.00000000 1 # Si |
3363 | +2.72767707 -8.19459074 0.00218763 1 # Si |
3364 | +2.72665137 -10.93548204 2.72665137 1 # Si |
3365 | +0.00218763 -8.19459074 2.72767707 1 # Si |
3366 | +1.36256401 -9.56805296 1.36256401 1 # Si |
3367 | +4.09489509 -6.82665093 1.36387577 1 # Si |
3368 | +4.09195916 -9.56309566 4.09195916 1 # Si |
3369 | +1.36387577 -6.82665093 4.09489509 1 # Si |
3370 | +0.00117430 -10.91922202 5.46070202 1 # Si |
3371 | +2.72890018 -8.19145655 5.45886981 1 # Si |
3372 | +2.73303284 -10.91859845 8.18958251 1 # Si |
3373 | +-0.01202455 -8.19611273 8.19611273 1 # Si |
3374 | +1.36492084 -9.55456223 6.82852418 1 # Si |
3375 | +4.09496380 -6.82463094 6.82463094 1 # Si |
3376 | +4.08178398 -9.53373688 9.53373688 1 # Si |
3377 | +1.36492084 -6.82852418 9.55456223 1 # Si |
3378 | +0.01304838 -10.90149235 10.90149235 1 # Si |
3379 | +2.73303284 -8.18958251 10.91859845 1 # Si |
3380 | +0.02610591 -8.18242131 13.61801556 1 # Si |
3381 | +1.35303719 -9.54338724 12.25680064 1 # Si |
3382 | +4.06859976 -6.80192861 12.27486761 1 # Si |
3383 | +1.33325565 -6.81988895 15.00302593 1 # Si |
3384 | +0.16771494 -5.63195305 -16.34134070 1 # Si |
3385 | +2.90187977 -2.90187977 -16.34634363 1 # Si |
3386 | +2.69385812 -5.42510828 -13.61156377 1 # Si |
3387 | +0.00300340 -2.73058729 -13.68153993 1 # Si |
3388 | +1.37590571 -4.10489826 -15.03413728 1 # Si |
3389 | +4.10489826 -1.37590571 -15.03413728 1 # Si |
3390 | +4.09402380 -4.09402380 -12.27225867 1 # Si |
3391 | +1.37011664 -1.37011664 -12.30671934 1 # Si |
3392 | +-0.00117430 -5.46070202 -10.91922202 1 # Si |
3393 | +2.72813440 -2.72813440 -10.91829649 1 # Si |
3394 | +2.73481248 -5.46221414 -8.19802839 1 # Si |
3395 | +-0.00218763 -2.72767707 -8.19459074 1 # Si |
3396 | +1.36300571 -4.09408870 -9.55550648 1 # Si |
3397 | +4.09408870 -1.36300571 -9.55550648 1 # Si |
3398 | +4.09758805 -4.09758805 -6.82965269 1 # Si |
3399 | +1.36467215 -1.36467215 -6.82823155 1 # Si |
3400 | +0.00319897 -5.46373779 -5.46373779 1 # Si |
3401 | +2.73166810 -2.73166810 -5.46412728 1 # Si |
3402 | +2.73166810 -5.46412728 -2.73166810 1 # Si |
3403 | +0.00070150 -2.73149619 -2.73149619 1 # Si |
3404 | +1.36658639 -4.09725659 -4.09725659 1 # Si |
3405 | +4.09725659 -1.36658639 -4.09725659 1 # Si |
3406 | +4.09725659 -4.09725659 -1.36658639 1 # Si |
3407 | +1.36585704 -1.36585704 -1.36585704 1 # Si |
3408 | +-0.00000000 -5.46151579 0.00000000 1 # Si |
3409 | +2.73149619 -2.73149619 -0.00070150 1 # Si |
3410 | +2.72992539 -5.46104379 2.72992539 1 # Si |
3411 | +-0.00070150 -2.73149619 2.73149619 1 # Si |
3412 | +1.36499559 -4.09571959 1.36499559 1 # Si |
3413 | +4.09571959 -1.36499559 1.36499559 1 # Si |
3414 | +4.09449230 -4.09449230 4.09449230 1 # Si |
3415 | +1.36499559 -1.36499559 4.09571959 1 # Si |
3416 | +-0.00319897 -5.46373779 5.46373779 1 # Si |
3417 | +2.72992539 -2.72992539 5.46104379 1 # Si |
3418 | +2.72890018 -5.45886981 8.19145655 1 # Si |
3419 | +0.00218763 -2.72767707 8.19459074 1 # Si |
3420 | +1.36387577 -4.09489509 6.82665093 1 # Si |
3421 | +4.09489509 -1.36387577 6.82665093 1 # Si |
3422 | +4.09195916 -4.09195916 9.56309566 1 # Si |
3423 | +1.36256401 -1.36256401 9.56805296 1 # Si |
3424 | +0.00117430 -5.46070202 10.91922202 1 # Si |
3425 | +2.72665137 -2.72665137 10.93548204 1 # Si |
3426 | +2.72110232 -5.45900176 13.66848041 1 # Si |
3427 | +-0.00300340 -2.73058729 13.68153993 1 # Si |
3428 | +1.35797308 -4.09496434 12.30121531 1 # Si |
3429 | +4.09496434 -1.35797308 12.30121531 1 # Si |
3430 | +4.07390228 -4.07390228 15.01032944 1 # Si |
3431 | +1.36373167 -1.36373167 15.04397328 1 # Si |
3432 | +-0.16771494 -5.63195305 16.34134070 1 # Si |
3433 | +2.73136988 -2.73136988 16.37023166 1 # Si |
3434 | +-0.00000000 -0.00000000 -16.37821316 1 # Si |
3435 | +2.73136988 2.73136988 -16.37023166 1 # Si |
3436 | +2.73058729 -0.00300340 -13.68153993 1 # Si |
3437 | +-0.00300340 2.73058729 -13.68153993 1 # Si |
3438 | +1.36373167 1.36373167 -15.04397328 1 # Si |
3439 | +4.07390228 4.07390228 -15.01032944 1 # Si |
3440 | +4.09496434 1.35797308 -12.30121531 1 # Si |
3441 | +1.35797308 4.09496434 -12.30121531 1 # Si |
3442 | +0.00000000 0.00000000 -10.94349563 1 # Si |
3443 | +2.72665137 2.72665137 -10.93548204 1 # Si |
3444 | +2.72767707 0.00218763 -8.19459074 1 # Si |
3445 | +0.00218763 2.72767707 -8.19459074 1 # Si |
3446 | +1.36256401 1.36256401 -9.56805296 1 # Si |
3447 | +4.09195916 4.09195916 -9.56309566 1 # Si |
3448 | +4.09489509 1.36387577 -6.82665093 1 # Si |
3449 | +1.36387577 4.09489509 -6.82665093 1 # Si |
3450 | +-0.00000000 0.00000000 -5.46151579 1 # Si |
3451 | +2.72992539 2.72992539 -5.46104379 1 # Si |
3452 | +2.73149619 -0.00070150 -2.73149619 1 # Si |
3453 | +-0.00070150 2.73149619 -2.73149619 1 # Si |
3454 | +1.36499559 1.36499559 -4.09571959 1 # Si |
3455 | +4.09449230 4.09449230 -4.09449230 1 # Si |
3456 | +4.09571959 1.36499559 -1.36499559 1 # Si |
3457 | +1.36499559 4.09571959 -1.36499559 1 # Si |
3458 | +-0.00000000 0.00000000 -0.00000000 1 # Si |
3459 | +2.73149619 2.73149619 0.00070150 1 # Si |
3460 | +2.73149619 0.00070150 2.73149619 1 # Si |
3461 | +0.00070150 2.73149619 2.73149619 1 # Si |
3462 | +1.36585704 1.36585704 1.36585704 1 # Si |
3463 | +4.09725659 4.09725659 1.36658639 1 # Si |
3464 | +4.09725659 1.36658639 4.09725659 1 # Si |
3465 | +1.36658639 4.09725659 4.09725659 1 # Si |
3466 | +-0.00000000 -0.00000000 5.46151579 1 # Si |
3467 | +2.73166810 2.73166810 5.46412728 1 # Si |
3468 | +2.72767707 -0.00218763 8.19459074 1 # Si |
3469 | +-0.00218763 2.72767707 8.19459074 1 # Si |
3470 | +1.36467215 1.36467215 6.82823155 1 # Si |
3471 | +4.09758805 4.09758805 6.82965269 1 # Si |
3472 | +4.09408870 1.36300571 9.55550648 1 # Si |
3473 | +1.36300571 4.09408870 9.55550648 1 # Si |
3474 | +-0.00000000 -0.00000000 10.94349563 1 # Si |
3475 | +2.72813440 2.72813440 10.91829649 1 # Si |
3476 | +2.73058729 0.00300340 13.68153993 1 # Si |
3477 | +0.00300340 2.73058729 13.68153993 1 # Si |
3478 | +1.37011664 1.37011664 12.30671934 1 # Si |
3479 | +4.09402380 4.09402380 12.27225867 1 # Si |
3480 | +4.10489826 1.37590571 15.03413728 1 # Si |
3481 | +1.37590571 4.10489826 15.03413728 1 # Si |
3482 | +-0.00000000 -0.00000000 16.37821316 1 # Si |
3483 | +2.90187977 2.90187977 16.34634363 1 # Si |
3484 | +-0.16771494 5.63195305 -16.34134070 1 # Si |
3485 | +2.72110232 5.45900176 -13.66848041 1 # Si |
3486 | +0.02610591 8.18242131 -13.61801556 1 # Si |
3487 | +1.33325565 6.81988895 -15.00302593 1 # Si |
3488 | +4.06859976 6.80192861 -12.27486761 1 # Si |
3489 | +1.35303719 9.54338724 -12.25680064 1 # Si |
3490 | +0.00117430 5.46070202 -10.91922202 1 # Si |
3491 | +2.73303284 8.18958251 -10.91859845 1 # Si |
3492 | +2.72890018 5.45886981 -8.19145655 1 # Si |
3493 | +-0.01202455 8.19611273 -8.19611273 1 # Si |
3494 | +1.36492084 6.82852418 -9.55456223 1 # Si |
3495 | +4.08178398 9.53373688 -9.53373688 1 # Si |
3496 | +4.09496380 6.82463094 -6.82463094 1 # Si |
3497 | +1.36492084 9.55456223 -6.82852418 1 # Si |
3498 | +-0.00319897 5.46373779 -5.46373779 1 # Si |
3499 | +2.72890018 8.19145655 -5.45886981 1 # Si |
3500 | +2.72992539 5.46104379 -2.72992539 1 # Si |
3501 | +0.00218763 8.19459074 -2.72767707 1 # Si |
3502 | +1.36387577 6.82665093 -4.09489509 1 # Si |
3503 | +4.09195916 9.56309566 -4.09195916 1 # Si |
3504 | +4.09489509 6.82665093 -1.36387577 1 # Si |
3505 | +1.36256401 9.56805296 -1.36256401 1 # Si |
3506 | +-0.00000000 5.46151579 0.00000000 1 # Si |
3507 | +2.72767707 8.19459074 -0.00218763 1 # Si |
3508 | +2.73166810 5.46412728 2.73166810 1 # Si |
3509 | +-0.00218763 8.19459074 2.72767707 1 # Si |
3510 | +1.36467215 6.82823155 1.36467215 1 # Si |
3511 | +4.09408870 9.55550648 1.36300571 1 # Si |
3512 | +4.09758805 6.82965269 4.09758805 1 # Si |
3513 | +1.36300571 9.55550648 4.09408870 1 # Si |
3514 | +0.00319897 5.46373779 5.46373779 1 # Si |
3515 | +2.73481248 8.19802839 5.46221414 1 # Si |
3516 | +2.73481248 5.46221414 8.19802839 1 # Si |
3517 | +0.01202455 8.19611273 8.19611273 1 # Si |
3518 | +1.37205363 6.82894278 6.82894278 1 # Si |
3519 | +4.10298675 9.56489675 6.82705083 1 # Si |
3520 | +4.10298675 6.82705083 9.56489675 1 # Si |
3521 | +1.38318968 9.56265344 9.56265344 1 # Si |
3522 | +-0.00117430 5.46070202 10.91922202 1 # Si |
3523 | +2.74437054 8.19260795 10.93410102 1 # Si |
3524 | +2.69385812 5.42510828 13.61156377 1 # Si |
3525 | +-0.02610591 8.18242131 13.61801556 1 # Si |
3526 | +1.35920843 6.82966552 12.27960089 1 # Si |
3527 | +4.12509126 9.54137501 12.28371979 1 # Si |
3528 | +0.16771494 5.63195305 16.34134070 1 # Si |
3529 | +0.01304838 10.90149235 -10.90149235 1 # Si |
3530 | +2.73303284 10.91859845 -8.18958251 1 # Si |
3531 | +0.02610591 13.61801556 -8.18242131 1 # Si |
3532 | +1.35303719 12.25680064 -9.54338724 1 # Si |
3533 | +4.06859976 12.27486761 -6.80192861 1 # Si |
3534 | +1.33325565 15.00302593 -6.81988895 1 # Si |
3535 | +0.00117430 10.91922202 -5.46070202 1 # Si |
3536 | +2.72110232 13.66848041 -5.45900176 1 # Si |
3537 | +2.72665137 10.93548204 -2.72665137 1 # Si |
3538 | +-0.00300340 13.68153993 -2.73058729 1 # Si |
3539 | +1.35797308 12.30121531 -4.09496434 1 # Si |
3540 | +4.07390228 15.01032944 -4.07390228 1 # Si |
3541 | +4.09496434 12.30121531 -1.35797308 1 # Si |
3542 | +1.36373167 15.04397328 -1.36373167 1 # Si |
3543 | +-0.00000000 10.94349563 0.00000000 1 # Si |
3544 | +2.73058729 13.68153993 0.00300340 1 # Si |
3545 | +2.72813440 10.91829649 2.72813440 1 # Si |
3546 | +0.00300340 13.68153993 2.73058729 1 # Si |
3547 | +1.37011664 12.30671934 1.37011664 1 # Si |
3548 | +4.10489826 15.03413728 1.37590571 1 # Si |
3549 | +4.09402380 12.27225867 4.09402380 1 # Si |
3550 | +1.37590571 15.03413728 4.10489826 1 # Si |
3551 | +-0.00117430 10.91922202 5.46070202 1 # Si |
3552 | +2.69385812 13.61156377 5.42510828 1 # Si |
3553 | +2.74437054 10.93410102 8.19260795 1 # Si |
3554 | +-0.02610591 13.61801556 8.18242131 1 # Si |
3555 | +1.35920843 12.27960089 6.82966552 1 # Si |
3556 | +4.12509126 12.28371979 9.54137501 1 # Si |
3557 | +-0.01304838 10.90149235 10.90149235 1 # Si |
3558 | +-0.16771494 16.34134070 -5.63195305 1 # Si |
3559 | +2.73136988 16.37023166 -2.73136988 1 # Si |
3560 | +0.00000000 16.37821316 -0.00000000 1 # Si |
3561 | +2.90187977 16.34634363 2.90187977 1 # Si |
3562 | +0.16771494 16.34134070 5.63195305 1 # Si |
3563 | +5.61139268 -13.62454044 -8.34100436 1 # Si |
3564 | +6.84800235 -12.30649445 -6.84800235 1 # Si |
3565 | +8.34100436 -13.62454044 -5.61139268 1 # Si |
3566 | +5.42510828 -13.61156377 -2.69385812 1 # Si |
3567 | +9.54137501 -12.28371979 -4.12509126 1 # Si |
3568 | +6.82966552 -12.27960089 -1.35920843 1 # Si |
3569 | +5.63195305 -16.34134070 -0.16771494 1 # Si |
3570 | +8.18242131 -13.61801556 0.02610591 1 # Si |
3571 | +5.45900176 -13.66848041 2.72110232 1 # Si |
3572 | +6.81988895 -15.00302593 1.33325565 1 # Si |
3573 | +9.54338724 -12.25680064 1.35303719 1 # Si |
3574 | +6.80192861 -12.27486761 4.06859976 1 # Si |
3575 | +5.61139268 -8.34100436 -13.62454044 1 # Si |
3576 | +6.84800235 -6.84800235 -12.30649445 1 # Si |
3577 | +5.43284542 -10.90291341 -10.90291341 1 # Si |
3578 | +8.17530342 -8.17530342 -10.89255734 1 # Si |
3579 | +8.17530342 -10.89255734 -8.17530342 1 # Si |
3580 | +5.46312108 -8.19345007 -8.19345007 1 # Si |
3581 | +6.82271744 -9.55790920 -9.55790920 1 # Si |
3582 | +9.55790920 -6.82271744 -9.55790920 1 # Si |
3583 | +9.55790920 -9.55790920 -6.82271744 1 # Si |
3584 | +6.82700831 -6.82700831 -6.82700831 1 # Si |
3585 | +5.47569502 -10.94234231 -5.47569502 1 # Si |
3586 | +8.19345007 -8.19345007 -5.46312108 1 # Si |
3587 | +8.19260795 -10.93410102 -2.74437054 1 # Si |
3588 | +5.46221414 -8.19802839 -2.73481248 1 # Si |
3589 | +6.82705083 -9.56489675 -4.10298675 1 # Si |
3590 | +9.56489675 -6.82705083 -4.10298675 1 # Si |
3591 | +9.56265344 -9.56265344 -1.38318968 1 # Si |
3592 | +6.82894278 -6.82894278 -1.37205363 1 # Si |
3593 | +5.46070202 -10.91922202 0.00117430 1 # Si |
3594 | +8.19611273 -8.19611273 -0.01202455 1 # Si |
3595 | +8.18958251 -10.91859845 2.73303284 1 # Si |
3596 | +5.45886981 -8.19145655 2.72890018 1 # Si |
3597 | +6.82852418 -9.55456223 1.36492084 1 # Si |
3598 | +9.55456223 -6.82852418 1.36492084 1 # Si |
3599 | +9.53373688 -9.53373688 4.08178398 1 # Si |
3600 | +6.82463094 -6.82463094 4.09496380 1 # Si |
3601 | +5.45537137 -10.92814020 5.45537137 1 # Si |
3602 | +8.18684496 -8.18684496 5.46061302 1 # Si |
3603 | +5.46061302 -8.18684496 8.18684496 1 # Si |
3604 | +6.80203106 -9.53854106 6.80203106 1 # Si |
3605 | +9.53854106 -6.80203106 6.80203106 1 # Si |
3606 | +6.80203106 -6.80203106 9.53854106 1 # Si |
3607 | +8.34100436 -5.61139268 -13.62454044 1 # Si |
3608 | +5.42510828 -2.69385812 -13.61156377 1 # Si |
3609 | +9.54137501 -4.12509126 -12.28371979 1 # Si |
3610 | +6.82966552 -1.35920843 -12.27960089 1 # Si |
3611 | +5.47569502 -5.47569502 -10.94234231 1 # Si |
3612 | +8.19260795 -2.74437054 -10.93410102 1 # Si |
3613 | +8.19345007 -5.46312108 -8.19345007 1 # Si |
3614 | +5.46221414 -2.73481248 -8.19802839 1 # Si |
3615 | +6.82705083 -4.10298675 -9.56489675 1 # Si |
3616 | +9.56265344 -1.38318968 -9.56265344 1 # Si |
3617 | +9.56489675 -4.10298675 -6.82705083 1 # Si |
3618 | +6.82894278 -1.37205363 -6.82894278 1 # Si |
3619 | +5.46236779 -5.46236779 -5.46236779 1 # Si |
3620 | +8.19802839 -2.73481248 -5.46221414 1 # Si |
3621 | +8.19802839 -5.46221414 -2.73481248 1 # Si |
3622 | +5.46412728 -2.73166810 -2.73166810 1 # Si |
3623 | +6.82965269 -4.09758805 -4.09758805 1 # Si |
3624 | +9.55550648 -1.36300571 -4.09408870 1 # Si |
3625 | +9.55550648 -4.09408870 -1.36300571 1 # Si |
3626 | +6.82823155 -1.36467215 -1.36467215 1 # Si |
3627 | +5.46373779 -5.46373779 -0.00319897 1 # Si |
3628 | +8.19459074 -2.72767707 0.00218763 1 # Si |
3629 | +8.19145655 -5.45886981 2.72890018 1 # Si |
3630 | +5.46104379 -2.72992539 2.72992539 1 # Si |
3631 | +6.82665093 -4.09489509 1.36387577 1 # Si |
3632 | +9.56805296 -1.36256401 1.36256401 1 # Si |
3633 | +9.56309566 -4.09195916 4.09195916 1 # Si |
3634 | +6.82665093 -1.36387577 4.09489509 1 # Si |
3635 | +5.45846237 -5.45846237 5.45846237 1 # Si |
3636 | +8.19145655 -2.72890018 5.45886981 1 # Si |
3637 | +8.18684496 -5.46061302 8.18684496 1 # Si |
3638 | +5.45886981 -2.72890018 8.19145655 1 # Si |
3639 | +6.82463094 -4.09496380 6.82463094 1 # Si |
3640 | +9.55456223 -1.36492084 6.82852418 1 # Si |
3641 | +9.53373688 -4.08178398 9.53373688 1 # Si |
3642 | +6.82852418 -1.36492084 9.55456223 1 # Si |
3643 | +5.45537137 -5.45537137 10.92814020 1 # Si |
3644 | +8.18958251 -2.73303284 10.91859845 1 # Si |
3645 | +5.45900176 -2.72110232 13.66848041 1 # Si |
3646 | +6.80192861 -4.06859976 12.27486761 1 # Si |
3647 | +9.54338724 -1.35303719 12.25680064 1 # Si |
3648 | +6.81988895 -1.33325565 15.00302593 1 # Si |
3649 | +5.63195305 -0.16771494 -16.34134070 1 # Si |
3650 | +8.18242131 0.02610591 -13.61801556 1 # Si |
3651 | +5.45900176 2.72110232 -13.66848041 1 # Si |
3652 | +6.81988895 1.33325565 -15.00302593 1 # Si |
3653 | +9.54338724 1.35303719 -12.25680064 1 # Si |
3654 | +6.80192861 4.06859976 -12.27486761 1 # Si |
3655 | +5.46070202 0.00117430 -10.91922202 1 # Si |
3656 | +8.18958251 2.73303284 -10.91859845 1 # Si |
3657 | +8.19611273 -0.01202455 -8.19611273 1 # Si |
3658 | +5.45886981 2.72890018 -8.19145655 1 # Si |
3659 | +6.82852418 1.36492084 -9.55456223 1 # Si |
3660 | +9.53373688 4.08178398 -9.53373688 1 # Si |
3661 | +9.55456223 1.36492084 -6.82852418 1 # Si |
3662 | +6.82463094 4.09496380 -6.82463094 1 # Si |
3663 | +5.46373779 -0.00319897 -5.46373779 1 # Si |
3664 | +8.19145655 2.72890018 -5.45886981 1 # Si |
3665 | +8.19459074 0.00218763 -2.72767707 1 # Si |
3666 | +5.46104379 2.72992539 -2.72992539 1 # Si |
3667 | +6.82665093 1.36387577 -4.09489509 1 # Si |
3668 | +9.56309566 4.09195916 -4.09195916 1 # Si |
3669 | +9.56805296 1.36256401 -1.36256401 1 # Si |
3670 | +6.82665093 4.09489509 -1.36387577 1 # Si |
3671 | +5.46151579 -0.00000000 0.00000000 1 # Si |
3672 | +8.19459074 2.72767707 -0.00218763 1 # Si |
3673 | +8.19459074 -0.00218763 2.72767707 1 # Si |
3674 | +5.46412728 2.73166810 2.73166810 1 # Si |
3675 | +6.82823155 1.36467215 1.36467215 1 # Si |
3676 | +9.55550648 4.09408870 1.36300571 1 # Si |
3677 | +9.55550648 1.36300571 4.09408870 1 # Si |
3678 | +6.82965269 4.09758805 4.09758805 1 # Si |
3679 | +5.46373779 0.00319897 5.46373779 1 # Si |
3680 | +8.19802839 2.73481248 5.46221414 1 # Si |
3681 | +8.19611273 0.01202455 8.19611273 1 # Si |
3682 | +5.46221414 2.73481248 8.19802839 1 # Si |
3683 | +6.82894278 1.37205363 6.82894278 1 # Si |
3684 | +9.56489675 4.10298675 6.82705083 1 # Si |
3685 | +9.56265344 1.38318968 9.56265344 1 # Si |
3686 | +6.82705083 4.10298675 9.56489675 1 # Si |
3687 | +5.46070202 -0.00117430 10.91922202 1 # Si |
3688 | +8.19260795 2.74437054 10.93410102 1 # Si |
3689 | +8.18242131 -0.02610591 13.61801556 1 # Si |
3690 | +5.42510828 2.69385812 13.61156377 1 # Si |
3691 | +6.82966552 1.35920843 12.27960089 1 # Si |
3692 | +9.54137501 4.12509126 12.28371979 1 # Si |
3693 | +5.63195305 0.16771494 16.34134070 1 # Si |
3694 | +5.45537137 5.45537137 -10.92814020 1 # Si |
3695 | +8.18684496 5.46061302 -8.18684496 1 # Si |
3696 | +5.46061302 8.18684496 -8.18684496 1 # Si |
3697 | +6.80203106 6.80203106 -9.53854106 1 # Si |
3698 | +9.53854106 6.80203106 -6.80203106 1 # Si |
3699 | +6.80203106 9.53854106 -6.80203106 1 # Si |
3700 | +5.45846237 5.45846237 -5.45846237 1 # Si |
3701 | +8.18684496 8.18684496 -5.46061302 1 # Si |
3702 | +8.19145655 5.45886981 -2.72890018 1 # Si |
3703 | +5.45886981 8.19145655 -2.72890018 1 # Si |
3704 | +6.82463094 6.82463094 -4.09496380 1 # Si |
3705 | +9.53373688 9.53373688 -4.08178398 1 # Si |
3706 | +9.55456223 6.82852418 -1.36492084 1 # Si |
3707 | +6.82852418 9.55456223 -1.36492084 1 # Si |
3708 | +5.46373779 5.46373779 0.00319897 1 # Si |
3709 | +8.19611273 8.19611273 0.01202455 1 # Si |
3710 | +8.19802839 5.46221414 2.73481248 1 # Si |
3711 | +5.46221414 8.19802839 2.73481248 1 # Si |
3712 | +6.82894278 6.82894278 1.37205363 1 # Si |
3713 | +9.56265344 9.56265344 1.38318968 1 # Si |
3714 | +9.56489675 6.82705083 4.10298675 1 # Si |
3715 | +6.82705083 9.56489675 4.10298675 1 # Si |
3716 | +5.46236779 5.46236779 5.46236779 1 # Si |
3717 | +8.19345007 8.19345007 5.46312108 1 # Si |
3718 | +8.19345007 5.46312108 8.19345007 1 # Si |
3719 | +5.46312108 8.19345007 8.19345007 1 # Si |
3720 | +6.82700831 6.82700831 6.82700831 1 # Si |
3721 | +9.55790920 9.55790920 6.82271744 1 # Si |
3722 | +9.55790920 6.82271744 9.55790920 1 # Si |
3723 | +6.82271744 9.55790920 9.55790920 1 # Si |
3724 | +5.47569502 5.47569502 10.94234231 1 # Si |
3725 | +8.17530342 8.17530342 10.89255734 1 # Si |
3726 | +8.34100436 5.61139268 13.62454044 1 # Si |
3727 | +5.61139268 8.34100436 13.62454044 1 # Si |
3728 | +6.84800235 6.84800235 12.30649445 1 # Si |
3729 | +5.45537137 10.92814020 -5.45537137 1 # Si |
3730 | +8.18958251 10.91859845 -2.73303284 1 # Si |
3731 | +5.45900176 13.66848041 -2.72110232 1 # Si |
3732 | +6.80192861 12.27486761 -4.06859976 1 # Si |
3733 | +9.54338724 12.25680064 -1.35303719 1 # Si |
3734 | +6.81988895 15.00302593 -1.33325565 1 # Si |
3735 | +5.46070202 10.91922202 -0.00117430 1 # Si |
3736 | +8.18242131 13.61801556 -0.02610591 1 # Si |
3737 | +8.19260795 10.93410102 2.74437054 1 # Si |
3738 | +5.42510828 13.61156377 2.69385812 1 # Si |
3739 | +6.82966552 12.27960089 1.35920843 1 # Si |
3740 | +9.54137501 12.28371979 4.12509126 1 # Si |
3741 | +5.47569502 10.94234231 5.47569502 1 # Si |
3742 | +8.34100436 13.62454044 5.61139268 1 # Si |
3743 | +8.17530342 10.89255734 8.17530342 1 # Si |
3744 | +5.61139268 13.62454044 8.34100436 1 # Si |
3745 | +6.84800235 12.30649445 6.84800235 1 # Si |
3746 | +5.43284542 10.90291341 10.90291341 1 # Si |
3747 | +5.63195305 16.34134070 0.16771494 1 # Si |
3748 | +10.89255734 -8.17530342 -8.17530342 1 # Si |
3749 | +12.30649445 -6.84800235 -6.84800235 1 # Si |
3750 | +10.90291341 -10.90291341 -5.43284542 1 # Si |
3751 | +13.62454044 -8.34100436 -5.61139268 1 # Si |
3752 | +10.93410102 -8.19260795 -2.74437054 1 # Si |
3753 | +12.28371979 -9.54137501 -4.12509126 1 # Si |
3754 | +12.27960089 -6.82966552 -1.35920843 1 # Si |
3755 | +10.90149235 -10.90149235 0.01304838 1 # Si |
3756 | +13.61801556 -8.18242131 0.02610591 1 # Si |
3757 | +10.91859845 -8.18958251 2.73303284 1 # Si |
3758 | +12.25680064 -9.54338724 1.35303719 1 # Si |
3759 | +15.00302593 -6.81988895 1.33325565 1 # Si |
3760 | +12.27486761 -6.80192861 4.06859976 1 # Si |
3761 | +10.90291341 -5.43284542 -10.90291341 1 # Si |
3762 | +13.62454044 -5.61139268 -8.34100436 1 # Si |
3763 | +10.93410102 -2.74437054 -8.19260795 1 # Si |
3764 | +12.28371979 -4.12509126 -9.54137501 1 # Si |
3765 | +12.27960089 -1.35920843 -6.82966552 1 # Si |
3766 | +10.94234231 -5.47569502 -5.47569502 1 # Si |
3767 | +13.61156377 -2.69385812 -5.42510828 1 # Si |
3768 | +13.61156377 -5.42510828 -2.69385812 1 # Si |
3769 | +10.91829649 -2.72813440 -2.72813440 1 # Si |
3770 | +12.27225867 -4.09402380 -4.09402380 1 # Si |
3771 | +15.03413728 -1.37590571 -4.10489826 1 # Si |
3772 | +15.03413728 -4.10489826 -1.37590571 1 # Si |
3773 | +12.30671934 -1.37011664 -1.37011664 1 # Si |
3774 | +10.91922202 -5.46070202 0.00117430 1 # Si |
3775 | +13.68153993 -2.73058729 -0.00300340 1 # Si |
3776 | +13.66848041 -5.45900176 2.72110232 1 # Si |
3777 | +10.93548204 -2.72665137 2.72665137 1 # Si |
3778 | +12.30121531 -4.09496434 1.35797308 1 # Si |
3779 | +15.04397328 -1.36373167 1.36373167 1 # Si |
3780 | +15.01032944 -4.07390228 4.07390228 1 # Si |
3781 | +12.30121531 -1.35797308 4.09496434 1 # Si |
3782 | +10.92814020 -5.45537137 5.45537137 1 # Si |
3783 | +13.66848041 -2.72110232 5.45900176 1 # Si |
3784 | +10.91859845 -2.73303284 8.18958251 1 # Si |
3785 | +12.27486761 -4.06859976 6.80192861 1 # Si |
3786 | +15.00302593 -1.33325565 6.81988895 1 # Si |
3787 | +12.25680064 -1.35303719 9.54338724 1 # Si |
3788 | +10.90149235 0.01304838 -10.90149235 1 # Si |
3789 | +13.61801556 0.02610591 -8.18242131 1 # Si |
3790 | +10.91859845 2.73303284 -8.18958251 1 # Si |
3791 | +12.25680064 1.35303719 -9.54338724 1 # Si |
3792 | +15.00302593 1.33325565 -6.81988895 1 # Si |
3793 | +12.27486761 4.06859976 -6.80192861 1 # Si |
3794 | +10.91922202 0.00117430 -5.46070202 1 # Si |
3795 | +13.66848041 2.72110232 -5.45900176 1 # Si |
3796 | +13.68153993 -0.00300340 -2.73058729 1 # Si |
3797 | +10.93548204 2.72665137 -2.72665137 1 # Si |
3798 | +12.30121531 1.35797308 -4.09496434 1 # Si |
3799 | +15.01032944 4.07390228 -4.07390228 1 # Si |
3800 | +15.04397328 1.36373167 -1.36373167 1 # Si |
3801 | +12.30121531 4.09496434 -1.35797308 1 # Si |
3802 | +10.94349563 0.00000000 -0.00000000 1 # Si |
3803 | +13.68153993 2.73058729 0.00300340 1 # Si |
3804 | +13.68153993 0.00300340 2.73058729 1 # Si |
3805 | +10.91829649 2.72813440 2.72813440 1 # Si |
3806 | +12.30671934 1.37011664 1.37011664 1 # Si |
3807 | +15.03413728 4.10489826 1.37590571 1 # Si |
3808 | +15.03413728 1.37590571 4.10489826 1 # Si |
3809 | +12.27225867 4.09402380 4.09402380 1 # Si |
3810 | +10.91922202 -0.00117430 5.46070202 1 # Si |
3811 | +13.61156377 2.69385812 5.42510828 1 # Si |
3812 | +13.61801556 -0.02610591 8.18242131 1 # Si |
3813 | +10.93410102 2.74437054 8.19260795 1 # Si |
3814 | +12.27960089 1.35920843 6.82966552 1 # Si |
3815 | +12.28371979 4.12509126 9.54137501 1 # Si |
3816 | +10.90149235 -0.01304838 10.90149235 1 # Si |
3817 | +10.92814020 5.45537137 -5.45537137 1 # Si |
3818 | +13.66848041 5.45900176 -2.72110232 1 # Si |
3819 | +10.91859845 8.18958251 -2.73303284 1 # Si |
3820 | +12.27486761 6.80192861 -4.06859976 1 # Si |
3821 | +15.00302593 6.81988895 -1.33325565 1 # Si |
3822 | +12.25680064 9.54338724 -1.35303719 1 # Si |
3823 | +10.91922202 5.46070202 -0.00117430 1 # Si |
3824 | +13.61801556 8.18242131 -0.02610591 1 # Si |
3825 | +13.61156377 5.42510828 2.69385812 1 # Si |
3826 | +10.93410102 8.19260795 2.74437054 1 # Si |
3827 | +12.27960089 6.82966552 1.35920843 1 # Si |
3828 | +12.28371979 9.54137501 4.12509126 1 # Si |
3829 | +10.94234231 5.47569502 5.47569502 1 # Si |
3830 | +13.62454044 8.34100436 5.61139268 1 # Si |
3831 | +13.62454044 5.61139268 8.34100436 1 # Si |
3832 | +10.89255734 8.17530342 8.17530342 1 # Si |
3833 | +12.30649445 6.84800235 6.84800235 1 # Si |
3834 | +10.90291341 5.43284542 10.90291341 1 # Si |
3835 | +10.90149235 10.90149235 -0.01304838 1 # Si |
3836 | +10.90291341 10.90291341 5.43284542 1 # Si |
3837 | +16.34634363 -2.90187977 -2.90187977 1 # Si |
3838 | +16.34134070 -5.63195305 -0.16771494 1 # Si |
3839 | +16.37023166 -2.73136988 2.73136988 1 # Si |
3840 | +16.34134070 -0.16771494 -5.63195305 1 # Si |
3841 | +16.37023166 2.73136988 -2.73136988 1 # Si |
3842 | +16.37821316 -0.00000000 -0.00000000 1 # Si |
3843 | +16.34634363 2.90187977 2.90187977 1 # Si |
3844 | +16.34134070 0.16771494 5.63195305 1 # Si |
3845 | +16.34134070 5.63195305 0.16771494 1 # Si |
3846 | +-13.14107205 -7.67106057 -4.93773883 2 # H |
3847 | +-13.12200813 -10.41265519 -2.22197561 2 # H |
3848 | +-15.87081431 -7.68828624 -2.20095245 2 # H |
3849 | +-14.47909953 -9.05114009 0.84828128 2 # H |
3850 | +-13.14542628 -10.40942624 3.25101117 2 # H |
3851 | +-13.12200813 -2.22197561 -10.41265519 2 # H |
3852 | +-13.14107205 -4.93773883 -7.67106057 2 # H |
3853 | +-15.87081431 -2.20095245 -7.68828624 2 # H |
3854 | +-15.87684776 -4.94293745 -4.94293745 2 # H |
3855 | +-14.48904263 -6.28831840 3.55914512 2 # H |
3856 | +-14.48904263 -3.55914512 6.28831840 2 # H |
3857 | +-13.14542628 -3.25101117 10.40942624 2 # H |
3858 | +-14.47909953 0.84828128 -9.05114009 2 # H |
3859 | +-13.14542628 3.25101117 -10.40942624 2 # H |
3860 | +-14.48904263 3.55914512 -6.28831840 2 # H |
3861 | +-15.87684776 4.94293745 4.94293745 2 # H |
3862 | +-14.47909953 -0.84828128 9.05114009 2 # H |
3863 | +-15.87081431 2.20095245 7.68828624 2 # H |
3864 | +-13.14107205 4.93773883 7.67106057 2 # H |
3865 | +-13.12200813 2.22197561 10.41265519 2 # H |
3866 | +-14.48904263 6.28831840 -3.55914512 2 # H |
3867 | +-13.14542628 10.40942624 -3.25101117 2 # H |
3868 | +-14.47909953 9.05114009 -0.84828128 2 # H |
3869 | +-15.87081431 7.68828624 2.20095245 2 # H |
3870 | +-13.12200813 10.41265519 2.22197561 2 # H |
3871 | +-13.14107205 7.67106057 4.93773883 2 # H |
3872 | +-7.67106057 -13.14107205 -4.93773883 2 # H |
3873 | +-7.68828624 -15.87081431 -2.20095245 2 # H |
3874 | +-10.41265519 -13.12200813 -2.22197561 2 # H |
3875 | +-9.05114009 -14.47909953 0.84828128 2 # H |
3876 | +-10.40942624 -13.14542628 3.25101117 2 # H |
3877 | +-7.67101940 -7.67101940 -10.40498818 2 # H |
3878 | +-7.67101940 -10.40498818 -7.67101940 2 # H |
3879 | +-10.40498818 -7.67101940 -7.67101940 2 # H |
3880 | +-10.40134885 -10.40134885 -4.95086615 2 # H |
3881 | +-11.77396546 -11.77396546 0.84562385 2 # H |
3882 | +-11.76622906 -11.76622906 6.31060507 2 # H |
3883 | +-9.04902753 -11.74957272 9.04902753 2 # H |
3884 | +-11.74957272 -9.04902753 9.04902753 2 # H |
3885 | +-9.04902753 -9.04902753 11.74957272 2 # H |
3886 | +-7.68828624 -2.20095245 -15.87081431 2 # H |
3887 | +-7.67106057 -4.93773883 -13.14107205 2 # H |
3888 | +-10.41265519 -2.22197561 -13.12200813 2 # H |
3889 | +-10.40134885 -4.95086615 -10.40134885 2 # H |
3890 | +-11.76622906 -6.31060507 11.76622906 2 # H |
3891 | +-10.40942624 -3.25101117 13.14542628 2 # H |
3892 | +-9.05114009 0.84828128 -14.47909953 2 # H |
3893 | +-10.40942624 3.25101117 -13.14542628 2 # H |
3894 | +-11.77396546 0.84562385 -11.77396546 2 # H |
3895 | +-10.40134885 4.95086615 10.40134885 2 # H |
3896 | +-11.77396546 -0.84562385 11.77396546 2 # H |
3897 | +-9.05114009 -0.84828128 14.47909953 2 # H |
3898 | +-10.41265519 2.22197561 13.12200813 2 # H |
3899 | +-7.67106057 4.93773883 13.14107205 2 # H |
3900 | +-7.68828624 2.20095245 15.87081431 2 # H |
3901 | +-11.76622906 6.31060507 -11.76622906 2 # H |
3902 | +-9.04902753 9.04902753 -11.74957272 2 # H |
3903 | +-11.74957272 9.04902753 -9.04902753 2 # H |
3904 | +-10.40134885 10.40134885 4.95086615 2 # H |
3905 | +-10.40498818 7.67101940 7.67101940 2 # H |
3906 | +-7.67101940 10.40498818 7.67101940 2 # H |
3907 | +-7.67101940 7.67101940 10.40498818 2 # H |
3908 | +-9.04902753 11.74957272 -9.04902753 2 # H |
3909 | +-11.76622906 11.76622906 -6.31060507 2 # H |
3910 | +-10.40942624 13.14542628 -3.25101117 2 # H |
3911 | +-11.77396546 11.77396546 -0.84562385 2 # H |
3912 | +-9.05114009 14.47909953 -0.84828128 2 # H |
3913 | +-10.41265519 13.12200813 2.22197561 2 # H |
3914 | +-7.68828624 15.87081431 2.20095245 2 # H |
3915 | +-7.67106057 13.14107205 4.93773883 2 # H |
3916 | +-2.22197561 -13.12200813 -10.41265519 2 # H |
3917 | +-2.20095245 -15.87081431 -7.68828624 2 # H |
3918 | +-4.93773883 -13.14107205 -7.67106057 2 # H |
3919 | +-4.94293745 -15.87684776 -4.94293745 2 # H |
3920 | +-6.28831840 -14.48904263 3.55914512 2 # H |
3921 | +-3.55914512 -14.48904263 6.28831840 2 # H |
3922 | +-3.25101117 -13.14542628 10.40942624 2 # H |
3923 | +-2.20095245 -7.68828624 -15.87081431 2 # H |
3924 | +-2.22197561 -10.41265519 -13.12200813 2 # H |
3925 | +-4.93773883 -7.67106057 -13.14107205 2 # H |
3926 | +-4.95086615 -10.40134885 -10.40134885 2 # H |
3927 | +-6.31060507 -11.76622906 11.76622906 2 # H |
3928 | +-3.25101117 -10.40942624 13.14542628 2 # H |
3929 | +-4.94293745 -4.94293745 -15.87684776 2 # H |
3930 | +-3.55914512 -6.28831840 14.48904263 2 # H |
3931 | +-6.28831840 -3.55914512 14.48904263 2 # H |
3932 | +-6.28831840 3.55914512 -14.48904263 2 # H |
3933 | +-4.94293745 4.94293745 15.87684776 2 # H |
3934 | +-3.55914512 6.28831840 -14.48904263 2 # H |
3935 | +-3.25101117 10.40942624 -13.14542628 2 # H |
3936 | +-4.95086615 10.40134885 10.40134885 2 # H |
3937 | +-4.93773883 7.67106057 13.14107205 2 # H |
3938 | +-2.22197561 10.41265519 13.12200813 2 # H |
3939 | +-2.20095245 7.68828624 15.87081431 2 # H |
3940 | +-6.31060507 11.76622906 -11.76622906 2 # H |
3941 | +-3.25101117 13.14542628 -10.40942624 2 # H |
3942 | +-3.55914512 14.48904263 -6.28831840 2 # H |
3943 | +-6.28831840 14.48904263 -3.55914512 2 # H |
3944 | +-4.94293745 15.87684776 4.94293745 2 # H |
3945 | +-4.93773883 13.14107205 7.67106057 2 # H |
3946 | +-2.20095245 15.87081431 7.68828624 2 # H |
3947 | +-2.22197561 13.12200813 10.41265519 2 # H |
3948 | +0.84828128 -14.47909953 -9.05114009 2 # H |
3949 | +3.25101117 -13.14542628 -10.40942624 2 # H |
3950 | +3.55914512 -14.48904263 -6.28831840 2 # H |
3951 | +4.94293745 -15.87684776 4.94293745 2 # H |
3952 | +-0.84828128 -14.47909953 9.05114009 2 # H |
3953 | +2.20095245 -15.87081431 7.68828624 2 # H |
3954 | +4.93773883 -13.14107205 7.67106057 2 # H |
3955 | +2.22197561 -13.12200813 10.41265519 2 # H |
3956 | +0.84828128 -9.05114009 -14.47909953 2 # H |
3957 | +3.25101117 -10.40942624 -13.14542628 2 # H |
3958 | +0.84562385 -11.77396546 -11.77396546 2 # H |
3959 | +4.95086615 -10.40134885 10.40134885 2 # H |
3960 | +-0.84562385 -11.77396546 11.77396546 2 # H |
3961 | +-0.84828128 -9.05114009 14.47909953 2 # H |
3962 | +2.22197561 -10.41265519 13.12200813 2 # H |
3963 | +4.93773883 -7.67106057 13.14107205 2 # H |
3964 | +2.20095245 -7.68828624 15.87081431 2 # H |
3965 | +3.55914512 -6.28831840 -14.48904263 2 # H |
3966 | +4.94293745 -4.94293745 15.87684776 2 # H |
3967 | +4.94293745 4.94293745 -15.87684776 2 # H |
3968 | +-0.84828128 9.05114009 -14.47909953 2 # H |
3969 | +2.20095245 7.68828624 -15.87081431 2 # H |
3970 | +4.93773883 7.67106057 -13.14107205 2 # H |
3971 | +2.22197561 10.41265519 -13.12200813 2 # H |
3972 | +4.95086615 10.40134885 -10.40134885 2 # H |
3973 | +3.55914512 6.28831840 14.48904263 2 # H |
3974 | +0.84828128 9.05114009 14.47909953 2 # H |
3975 | +3.25101117 10.40942624 13.14542628 2 # H |
3976 | +-0.84562385 11.77396546 -11.77396546 2 # H |
3977 | +-0.84828128 14.47909953 -9.05114009 2 # H |
3978 | +2.22197561 13.12200813 -10.41265519 2 # H |
3979 | +4.93773883 13.14107205 -7.67106057 2 # H |
3980 | +2.20095245 15.87081431 -7.68828624 2 # H |
3981 | +4.94293745 15.87684776 -4.94293745 2 # H |
3982 | +3.55914512 14.48904263 6.28831840 2 # H |
3983 | +0.84828128 14.47909953 9.05114009 2 # H |
3984 | +3.25101117 13.14542628 10.40942624 2 # H |
3985 | +0.84562385 11.77396546 11.77396546 2 # H |
3986 | +6.28831840 -14.48904263 -3.55914512 2 # H |
3987 | +10.40942624 -13.14542628 -3.25101117 2 # H |
3988 | +9.05114009 -14.47909953 -0.84828128 2 # H |
3989 | +7.68828624 -15.87081431 2.20095245 2 # H |
3990 | +10.41265519 -13.12200813 2.22197561 2 # H |
3991 | +7.67106057 -13.14107205 4.93773883 2 # H |
3992 | +6.31060507 -11.76622906 -11.76622906 2 # H |
3993 | +9.04902753 -9.04902753 -11.74957272 2 # H |
3994 | +9.04902753 -11.74957272 -9.04902753 2 # H |
3995 | +10.40134885 -10.40134885 4.95086615 2 # H |
3996 | +7.67101940 -10.40498818 7.67101940 2 # H |
3997 | +10.40498818 -7.67101940 7.67101940 2 # H |
3998 | +7.67101940 -7.67101940 10.40498818 2 # H |
3999 | +6.28831840 -3.55914512 -14.48904263 2 # H |
4000 | +10.40942624 -3.25101117 -13.14542628 2 # H |
4001 | +10.40134885 -4.95086615 10.40134885 2 # H |
4002 | +7.67106057 -4.93773883 13.14107205 2 # H |
4003 | +10.41265519 -2.22197561 13.12200813 2 # H |
4004 | +7.68828624 -2.20095245 15.87081431 2 # H |
4005 | +9.05114009 -0.84828128 -14.47909953 2 # H |
4006 | +7.68828624 2.20095245 -15.87081431 2 # H |
4007 | +10.41265519 2.22197561 -13.12200813 2 # H |
4008 | +7.67106057 4.93773883 -13.14107205 2 # H |
4009 | +10.40134885 4.95086615 -10.40134885 2 # H |
4010 | +9.05114009 0.84828128 14.47909953 2 # H |
4011 | +6.28831840 3.55914512 14.48904263 2 # H |
4012 | +10.40942624 3.25101117 13.14542628 2 # H |
4013 | +7.67101940 7.67101940 -10.40498818 2 # H |
4014 | +10.40498818 7.67101940 -7.67101940 2 # H |
4015 | +7.67101940 10.40498818 -7.67101940 2 # H |
4016 | +10.40134885 10.40134885 -4.95086615 2 # H |
4017 | +9.04902753 9.04902753 11.74957272 2 # H |
4018 | +7.67106057 13.14107205 -4.93773883 2 # H |
4019 | +10.41265519 13.12200813 -2.22197561 2 # H |
4020 | +7.68828624 15.87081431 -2.20095245 2 # H |
4021 | +9.05114009 14.47909953 0.84828128 2 # H |
4022 | +6.28831840 14.48904263 3.55914512 2 # H |
4023 | +10.40942624 13.14542628 3.25101117 2 # H |
4024 | +9.04902753 11.74957272 9.04902753 2 # H |
4025 | +6.31060507 11.76622906 11.76622906 2 # H |
4026 | +11.74957272 -9.04902753 -9.04902753 2 # H |
4027 | +11.76622906 -11.76622906 -6.31060507 2 # H |
4028 | +13.14542628 -10.40942624 -3.25101117 2 # H |
4029 | +11.77396546 -11.77396546 -0.84562385 2 # H |
4030 | +14.47909953 -9.05114009 -0.84828128 2 # H |
4031 | +13.12200813 -10.41265519 2.22197561 2 # H |
4032 | +15.87081431 -7.68828624 2.20095245 2 # H |
4033 | +13.14107205 -7.67106057 4.93773883 2 # H |
4034 | +11.76622906 -6.31060507 -11.76622906 2 # H |
4035 | +13.14542628 -3.25101117 -10.40942624 2 # H |
4036 | +14.48904263 -3.55914512 -6.28831840 2 # H |
4037 | +14.48904263 -6.28831840 -3.55914512 2 # H |
4038 | +15.87684776 -4.94293745 4.94293745 2 # H |
4039 | +13.14107205 -4.93773883 7.67106057 2 # H |
4040 | +15.87081431 -2.20095245 7.68828624 2 # H |
4041 | +13.12200813 -2.22197561 10.41265519 2 # H |
4042 | +11.77396546 -0.84562385 -11.77396546 2 # H |
4043 | +14.47909953 -0.84828128 -9.05114009 2 # H |
4044 | +13.12200813 2.22197561 -10.41265519 2 # H |
4045 | +15.87081431 2.20095245 -7.68828624 2 # H |
4046 | +13.14107205 4.93773883 -7.67106057 2 # H |
4047 | +15.87684776 4.94293745 -4.94293745 2 # H |
4048 | +14.48904263 3.55914512 6.28831840 2 # H |
4049 | +14.47909953 0.84828128 9.05114009 2 # H |
4050 | +13.14542628 3.25101117 10.40942624 2 # H |
4051 | +11.77396546 0.84562385 11.77396546 2 # H |
4052 | +13.14107205 7.67106057 -4.93773883 2 # H |
4053 | +15.87081431 7.68828624 -2.20095245 2 # H |
4054 | +13.12200813 10.41265519 -2.22197561 2 # H |
4055 | +14.47909953 9.05114009 0.84828128 2 # H |
4056 | +14.48904263 6.28831840 3.55914512 2 # H |
4057 | +13.14542628 10.40942624 3.25101117 2 # H |
4058 | +11.74957272 9.04902753 9.04902753 2 # H |
4059 | +11.76622906 6.31060507 11.76622906 2 # H |
4060 | +11.77396546 11.77396546 0.84562385 2 # H |
4061 | +11.76622906 11.76622906 6.31060507 2 # H |
4062 | +-14.25049869 -9.30903112 6.56976732 2 # H |
4063 | +-14.73854562 -7.62887682 4.90661761 2 # H |
4064 | +-17.26005319 -3.58592496 -1.88119305 2 # H |
4065 | +-17.26005319 -1.88119305 -3.58592496 2 # H |
4066 | +-16.92458989 -6.60756054 1.14534264 2 # H |
4067 | +-17.48851584 -4.95538862 -0.51858345 2 # H |
4068 | +-16.92565018 -3.87958702 3.87958702 2 # H |
4069 | +-17.49703350 -2.22300603 2.22300603 2 # H |
4070 | +-14.25049869 -6.56976732 9.30903112 2 # H |
4071 | +-14.73854562 -4.90661761 7.62887682 2 # H |
4072 | +-17.48851584 -0.51858345 -4.95538862 2 # H |
4073 | +-16.92458989 1.14534264 -6.60756054 2 # H |
4074 | +-17.49703350 2.22300603 -2.22300603 2 # H |
4075 | +-16.92565018 3.87958702 -3.87958702 2 # H |
4076 | +-17.26821617 -0.85235208 0.85235208 2 # H |
4077 | +-17.26821617 0.85235208 -0.85235208 2 # H |
4078 | +-17.26005319 1.88119305 3.58592496 2 # H |
4079 | +-17.26005319 3.58592496 1.88119305 2 # H |
4080 | +-16.92458989 -1.14534264 6.60756054 2 # H |
4081 | +-17.48851584 0.51858345 4.95538862 2 # H |
4082 | +-14.73854562 4.90661761 -7.62887682 2 # H |
4083 | +-14.25049869 6.56976732 -9.30903112 2 # H |
4084 | +-14.73854562 7.62887682 -4.90661761 2 # H |
4085 | +-14.25049869 9.30903112 -6.56976732 2 # H |
4086 | +-17.48851584 4.95538862 0.51858345 2 # H |
4087 | +-16.92458989 6.60756054 -1.14534264 2 # H |
4088 | +-7.62887682 -14.73854562 4.90661761 2 # H |
4089 | +-9.30903112 -14.25049869 6.56976732 2 # H |
4090 | +-7.62887682 -4.90661761 14.73854562 2 # H |
4091 | +-9.30903112 -6.56976732 14.25049869 2 # H |
4092 | +-9.30903112 6.56976732 -14.25049869 2 # H |
4093 | +-7.62887682 4.90661761 -14.73854562 2 # H |
4094 | +-9.30903112 14.25049869 -6.56976732 2 # H |
4095 | +-7.62887682 14.73854562 -4.90661761 2 # H |
4096 | +-1.88119305 -17.26005319 -3.58592496 2 # H |
4097 | +-3.58592496 -17.26005319 -1.88119305 2 # H |
4098 | +-4.95538862 -17.48851584 -0.51858345 2 # H |
4099 | +-6.60756054 -16.92458989 1.14534264 2 # H |
4100 | +-2.22300603 -17.49703350 2.22300603 2 # H |
4101 | +-3.87958702 -16.92565018 3.87958702 2 # H |
4102 | +-4.90661761 -14.73854562 7.62887682 2 # H |
4103 | +-6.56976732 -14.25049869 9.30903112 2 # H |
4104 | +-6.56976732 -9.30903112 14.25049869 2 # H |
4105 | +-4.90661761 -7.62887682 14.73854562 2 # H |
4106 | +-3.58592496 -1.88119305 -17.26005319 2 # H |
4107 | +-1.88119305 -3.58592496 -17.26005319 2 # H |
4108 | +-2.22300603 -2.22300603 17.49703350 2 # H |
4109 | +-3.87958702 -3.87958702 16.92565018 2 # H |
4110 | +-6.60756054 1.14534264 -16.92458989 2 # H |
4111 | +-4.95538862 -0.51858345 -17.48851584 2 # H |
4112 | +-3.87958702 3.87958702 -16.92565018 2 # H |
4113 | +-2.22300603 2.22300603 -17.49703350 2 # H |
4114 | +-6.60756054 -1.14534264 16.92458989 2 # H |
4115 | +-4.95538862 0.51858345 17.48851584 2 # H |
4116 | +-1.88119305 3.58592496 17.26005319 2 # H |
4117 | +-3.58592496 1.88119305 17.26005319 2 # H |
4118 | +-6.56976732 9.30903112 -14.25049869 2 # H |
4119 | +-4.90661761 7.62887682 -14.73854562 2 # H |
4120 | +-4.90661761 14.73854562 -7.62887682 2 # H |
4121 | +-6.56976732 14.25049869 -9.30903112 2 # H |
4122 | +-3.87958702 16.92565018 -3.87958702 2 # H |
4123 | +-2.22300603 17.49703350 -2.22300603 2 # H |
4124 | +-4.95538862 17.48851584 0.51858345 2 # H |
4125 | +-6.60756054 16.92458989 -1.14534264 2 # H |
4126 | +-3.58592496 17.26005319 1.88119305 2 # H |
4127 | +-1.88119305 17.26005319 3.58592496 2 # H |
4128 | +1.14534264 -16.92458989 -6.60756054 2 # H |
4129 | +-0.51858345 -17.48851584 -4.95538862 2 # H |
4130 | +3.87958702 -16.92565018 -3.87958702 2 # H |
4131 | +2.22300603 -17.49703350 -2.22300603 2 # H |
4132 | +0.85235208 -17.26821617 -0.85235208 2 # H |
4133 | +-0.85235208 -17.26821617 0.85235208 2 # H |
4134 | +3.58592496 -17.26005319 1.88119305 2 # H |
4135 | +1.88119305 -17.26005319 3.58592496 2 # H |
4136 | +0.51858345 -17.48851584 4.95538862 2 # H |
4137 | +-1.14534264 -16.92458989 6.60756054 2 # H |
4138 | +-0.51858345 -4.95538862 -17.48851584 2 # H |
4139 | +1.14534264 -6.60756054 -16.92458989 2 # H |
4140 | +2.22300603 -2.22300603 -17.49703350 2 # H |
4141 | +3.87958702 -3.87958702 -16.92565018 2 # H |
4142 | +-1.14534264 -6.60756054 16.92458989 2 # H |
4143 | +0.51858345 -4.95538862 17.48851584 2 # H |
4144 | +3.58592496 -1.88119305 17.26005319 2 # H |
4145 | +1.88119305 -3.58592496 17.26005319 2 # H |
4146 | +-0.85235208 0.85235208 -17.26821617 2 # H |
4147 | +0.85235208 -0.85235208 -17.26821617 2 # H |
4148 | +1.88119305 3.58592496 -17.26005319 2 # H |
4149 | +3.58592496 1.88119305 -17.26005319 2 # H |
4150 | +-0.85235208 -0.85235208 17.26821617 2 # H |
4151 | +0.85235208 0.85235208 17.26821617 2 # H |
4152 | +3.87958702 3.87958702 16.92565018 2 # H |
4153 | +2.22300603 2.22300603 17.49703350 2 # H |
4154 | +-1.14534264 6.60756054 -16.92458989 2 # H |
4155 | +0.51858345 4.95538862 -17.48851584 2 # H |
4156 | +-0.51858345 4.95538862 17.48851584 2 # H |
4157 | +1.14534264 6.60756054 16.92458989 2 # H |
4158 | +0.51858345 17.48851584 -4.95538862 2 # H |
4159 | +-1.14534264 16.92458989 -6.60756054 2 # H |
4160 | +1.88119305 17.26005319 -3.58592496 2 # H |
4161 | +3.58592496 17.26005319 -1.88119305 2 # H |
4162 | +0.85235208 17.26821617 0.85235208 2 # H |
4163 | +-0.85235208 17.26821617 -0.85235208 2 # H |
4164 | +2.22300603 17.49703350 2.22300603 2 # H |
4165 | +3.87958702 16.92565018 3.87958702 2 # H |
4166 | +1.14534264 16.92458989 6.60756054 2 # H |
4167 | +-0.51858345 17.48851584 4.95538862 2 # H |
4168 | +6.56976732 -14.25049869 -9.30903112 2 # H |
4169 | +4.90661761 -14.73854562 -7.62887682 2 # H |
4170 | +9.30903112 -14.25049869 -6.56976732 2 # H |
4171 | +7.62887682 -14.73854562 -4.90661761 2 # H |
4172 | +6.60756054 -16.92458989 -1.14534264 2 # H |
4173 | +4.95538862 -17.48851584 0.51858345 2 # H |
4174 | +4.90661761 -7.62887682 -14.73854562 2 # H |
4175 | +6.56976732 -9.30903112 -14.25049869 2 # H |
4176 | +7.62887682 -4.90661761 -14.73854562 2 # H |
4177 | +9.30903112 -6.56976732 -14.25049869 2 # H |
4178 | +4.95538862 0.51858345 -17.48851584 2 # H |
4179 | +6.60756054 -1.14534264 -16.92458989 2 # H |
4180 | +4.95538862 -0.51858345 17.48851584 2 # H |
4181 | +6.60756054 1.14534264 16.92458989 2 # H |
4182 | +9.30903112 6.56976732 14.25049869 2 # H |
4183 | +7.62887682 4.90661761 14.73854562 2 # H |
4184 | +4.90661761 7.62887682 14.73854562 2 # H |
4185 | +6.56976732 9.30903112 14.25049869 2 # H |
4186 | +7.62887682 14.73854562 4.90661761 2 # H |
4187 | +9.30903112 14.25049869 6.56976732 2 # H |
4188 | +6.56976732 14.25049869 9.30903112 2 # H |
4189 | +4.90661761 14.73854562 7.62887682 2 # H |
4190 | +6.60756054 16.92458989 1.14534264 2 # H |
4191 | +4.95538862 17.48851584 -0.51858345 2 # H |
4192 | +14.73854562 -7.62887682 -4.90661761 2 # H |
4193 | +14.25049869 -9.30903112 -6.56976732 2 # H |
4194 | +14.25049869 -6.56976732 -9.30903112 2 # H |
4195 | +14.73854562 -4.90661761 -7.62887682 2 # H |
4196 | +14.25049869 9.30903112 6.56976732 2 # H |
4197 | +14.73854562 7.62887682 4.90661761 2 # H |
4198 | +14.73854562 4.90661761 7.62887682 2 # H |
4199 | +14.25049869 6.56976732 9.30903112 2 # H |
4200 | +17.49703350 -2.22300603 -2.22300603 2 # H |
4201 | +16.92565018 -3.87958702 -3.87958702 2 # H |
4202 | +16.92458989 -6.60756054 -1.14534264 2 # H |
4203 | +17.48851584 -4.95538862 0.51858345 2 # H |
4204 | +17.26005319 -1.88119305 3.58592496 2 # H |
4205 | +17.26005319 -3.58592496 1.88119305 2 # H |
4206 | +16.92458989 -1.14534264 -6.60756054 2 # H |
4207 | +17.48851584 0.51858345 -4.95538862 2 # H |
4208 | +17.26005319 3.58592496 -1.88119305 2 # H |
4209 | +17.26005319 1.88119305 -3.58592496 2 # H |
4210 | +17.26821617 -0.85235208 -0.85235208 2 # H |
4211 | +17.26821617 0.85235208 0.85235208 2 # H |
4212 | +16.92565018 3.87958702 3.87958702 2 # H |
4213 | +17.49703350 2.22300603 2.22300603 2 # H |
4214 | +17.48851584 -0.51858345 4.95538862 2 # H |
4215 | +16.92458989 1.14534264 6.60756054 2 # H |
4216 | +17.48851584 4.95538862 -0.51858345 2 # H |
4217 | +16.92458989 6.60756054 1.14534264 2 # H |
4218 | +%endblock atomic-coordinates-and-atomic-species |
4219 | |
4220 | === added file 'Tests/si-quantum-dot/makefile' |
4221 | --- Tests/si-quantum-dot/makefile 1970-01-01 00:00:00 +0000 |
4222 | +++ Tests/si-quantum-dot/makefile 2019-01-16 11:00:58 +0000 |
4223 | @@ -0,0 +1,7 @@ |
4224 | +# |
4225 | +# Single-test makefile |
4226 | +# |
4227 | +name=si-quantum-dot |
4228 | +EXTRAFILES=coords.Si987H372.fdf |
4229 | +# |
4230 | +include ../test.mk |
4231 | |
4232 | === added file 'Tests/si-quantum-dot/si-quantum-dot.fdf' |
4233 | --- Tests/si-quantum-dot/si-quantum-dot.fdf 1970-01-01 00:00:00 +0000 |
4234 | +++ Tests/si-quantum-dot/si-quantum-dot.fdf 2019-01-16 11:00:58 +0000 |
4235 | @@ -0,0 +1,39 @@ |
4236 | +use-tree-timer T |
4237 | +SolutionMethod ELSI |
4238 | +elsi-solver elpa |
4239 | + |
4240 | +WriteDMHS.NetCDF F |
4241 | +WriteDM.NetCDF F |
4242 | +WriteDM T |
4243 | + |
4244 | +SystemName Si Quantum Dot |
4245 | +SystemLabel si-quantum-dot |
4246 | + |
4247 | +%block ChemicalSpeciesLabel |
4248 | + 1 14 Si |
4249 | + 2 1 H |
4250 | +%endblock ChemicalSpeciesLabel |
4251 | + |
4252 | +# CSIRO quantum dot |
4253 | +%include coords.Si987H372.fdf |
4254 | + |
4255 | +PAO.BasisSize SZ |
4256 | +PAO.EnergyShift 300 meV |
4257 | + |
4258 | +# |
4259 | +LatticeConstant 45.0 Ang |
4260 | +%block LatticeVectors |
4261 | + 1.000 0.000 0.000 |
4262 | + 0.000 1.000 0.000 |
4263 | + 0.000 0.000 1.000 |
4264 | +%endblock LatticeVectors |
4265 | + |
4266 | +MeshCutoff 150.0 Ry |
4267 | + |
4268 | +MaxSCFIterations 40 |
4269 | +DM.MixingWeight 0.3 |
4270 | +DM.NumberPulay 4 |
4271 | +DM.Tolerance 1.d-3 |
4272 | +DM.UseSaveDM |
4273 | + |
4274 | +ElectronicTemperature 25 meV |
4275 | |
4276 | === added file 'Tests/si-quantum-dot/si-quantum-dot.pseudos' |
4277 | --- Tests/si-quantum-dot/si-quantum-dot.pseudos 1970-01-01 00:00:00 +0000 |
4278 | +++ Tests/si-quantum-dot/si-quantum-dot.pseudos 2019-01-16 11:00:58 +0000 |
4279 | @@ -0,0 +1,1 @@ |
4280 | +Si H |
4281 | |
4282 | === added directory 'Tests/sih-elsi' |
4283 | === added file 'Tests/sih-elsi/makefile' |
4284 | --- Tests/sih-elsi/makefile 1970-01-01 00:00:00 +0000 |
4285 | +++ Tests/sih-elsi/makefile 2019-01-16 11:00:58 +0000 |
4286 | @@ -0,0 +1,6 @@ |
4287 | +# |
4288 | +# Single-test makefile |
4289 | +# |
4290 | +name=sih-elsi |
4291 | +# |
4292 | +include ../test.mk |
4293 | |
4294 | === added file 'Tests/sih-elsi/sih-elsi.fdf' |
4295 | --- Tests/sih-elsi/sih-elsi.fdf 1970-01-01 00:00:00 +0000 |
4296 | +++ Tests/sih-elsi/sih-elsi.fdf 2019-01-16 11:00:58 +0000 |
4297 | @@ -0,0 +1,121 @@ |
4298 | +# ----------------------------------------------------------------------------- |
4299 | +# FDF for interstitial H in a cubic c-Si supercell with 64 atoms |
4300 | +# |
4301 | +# E. Artacho, April 1999 |
4302 | +# ----------------------------------------------------------------------------- |
4303 | + |
4304 | +#------- |
4305 | +## compute-forces F |
4306 | +SolutionMethod ELSI |
4307 | +#MPI.Nprocs.SIESTA 1 |
4308 | + |
4309 | +WriteDMHS.NetCDF F |
4310 | +WriteDM.NetCDF F |
4311 | +WriteDM T |
4312 | + |
4313 | +SystemName H in 64-atom silicon - ELSI solver |
4314 | +SystemLabel sih-elsi |
4315 | + |
4316 | +NumberOfAtoms 65 |
4317 | +NumberOfSpecies 2 |
4318 | + |
4319 | +%block ChemicalSpeciesLabel |
4320 | + 1 14 Si |
4321 | + 2 1 H |
4322 | +%endblock ChemicalSpeciesLabel |
4323 | + |
4324 | +PAO.BasisSize DZP |
4325 | +PAO.EnergyShift 300 meV |
4326 | + |
4327 | +%block PAO.Basis |
4328 | + H 1 |
4329 | + 0 1 |
4330 | + 0. |
4331 | + 0.8 |
4332 | +%endblock PAO.Basis |
4333 | + |
4334 | +LatticeConstant 5.430 Ang |
4335 | +%block LatticeVectors |
4336 | + 2.000 0.000 0.000 |
4337 | + 0.000 2.000 0.000 |
4338 | + 0.000 0.000 2.000 |
4339 | +%endblock LatticeVectors |
4340 | + |
4341 | +MeshCutoff 100.0 Ry |
4342 | + |
4343 | +MaxSCFIterations 20 |
4344 | +DM.MixingWeight 0.3 |
4345 | +DM.NumberPulay 4 |
4346 | +DM.Tolerance 1.d-3 |
4347 | +DM.UseSaveDM |
4348 | + |
4349 | +ElectronicTemperature 25 meV |
4350 | + |
4351 | +AtomicCoordinatesFormat ScaledCartesian |
4352 | +%block AtomicCoordinatesAndAtomicSpecies |
4353 | + 0.000 0.000 0.000 1 # Si 1 |
4354 | + 0.250 0.250 0.250 1 # Si 2 |
4355 | + 0.000 0.500 0.500 1 # Si 3 |
4356 | + 0.250 0.750 0.750 1 # Si 4 |
4357 | + 0.500 0.000 0.500 1 # Si 5 |
4358 | + 0.750 0.250 0.750 1 # Si 6 |
4359 | + 0.500 0.500 0.000 1 # Si 7 |
4360 | + 0.750 0.750 0.250 1 # Si 8 |
4361 | + 1.000 0.000 0.000 1 # Si 9 |
4362 | + 1.250 0.250 0.250 1 # Si 10 |
4363 | + 1.000 0.500 0.500 1 # Si 11 |
4364 | + 1.250 0.750 0.750 1 # Si 12 |
4365 | + 1.500 0.000 0.500 1 # Si 13 |
4366 | + 1.750 0.250 0.750 1 # Si 14 |
4367 | + 1.500 0.500 0.000 1 # Si 15 |
4368 | + 1.750 0.750 0.250 1 # Si 16 |
4369 | + 0.000 1.000 0.000 1 # Si 17 |
4370 | + 0.250 1.250 0.250 1 # Si 18 |
4371 | + 0.000 1.500 0.500 1 # Si 19 |
4372 | + 0.250 1.750 0.750 1 # Si 20 |
4373 | + 0.500 1.000 0.500 1 # Si 21 |
4374 | + 0.750 1.250 0.750 1 # Si 22 |
4375 | + 0.500 1.500 0.000 1 # Si 23 |
4376 | + 0.750 1.750 0.250 1 # Si 24 |
4377 | + 0.000 0.000 1.000 1 # Si 25 |
4378 | + 0.250 0.250 1.250 1 # Si 26 |
4379 | + 0.000 0.500 1.500 1 # Si 27 |
4380 | + 0.250 0.750 1.750 1 # Si 28 |
4381 | + 0.500 0.000 1.500 1 # Si 29 |
4382 | + 0.750 0.250 1.750 1 # Si 30 |
4383 | + 0.500 0.500 1.000 1 # Si 31 |
4384 | + 0.750 0.750 1.250 1 # Si 32 |
4385 | + 1.000 1.000 0.000 1 # Si 33 |
4386 | + 1.250 1.250 0.250 1 # Si 34 |
4387 | + 1.000 1.500 0.500 1 # Si 35 |
4388 | + 1.250 1.750 0.750 1 # Si 36 |
4389 | + 1.500 1.000 0.500 1 # Si 37 |
4390 | + 1.750 1.250 0.750 1 # Si 38 |
4391 | + 1.500 1.500 0.000 1 # Si 39 |
4392 | + 1.750 1.750 0.250 1 # Si 40 |
4393 | + 1.000 0.000 1.000 1 # Si 41 |
4394 | + 1.250 0.250 1.250 1 # Si 42 |
4395 | + 1.000 0.500 1.500 1 # Si 43 |
4396 | + 1.250 0.750 1.750 1 # Si 44 |
4397 | + 1.500 0.000 1.500 1 # Si 45 |
4398 | + 1.750 0.250 1.750 1 # Si 46 |
4399 | + 1.500 0.500 1.000 1 # Si 47 |
4400 | + 1.750 0.750 1.250 1 # Si 48 |
4401 | + 0.000 1.000 1.000 1 # Si 49 |
4402 | + 0.250 1.250 1.250 1 # Si 50 |
4403 | + 0.000 1.500 1.500 1 # Si 51 |
4404 | + 0.250 1.750 1.750 1 # Si 52 |
4405 | + 0.500 1.000 1.500 1 # Si 53 |
4406 | + 0.750 1.250 1.750 1 # Si 54 |
4407 | + 0.500 1.500 1.000 1 # Si 55 |
4408 | + 0.750 1.750 1.250 1 # Si 56 |
4409 | + 1.000 1.000 1.000 1 # Si 57 |
4410 | + 1.250 1.250 1.250 1 # Si 58 |
4411 | + 1.000 1.500 1.500 1 # Si 59 |
4412 | + 1.250 1.750 1.750 1 # Si 60 |
4413 | + 1.500 1.000 1.500 1 # Si 61 |
4414 | + 1.750 1.250 1.750 1 # Si 62 |
4415 | + 1.500 1.500 1.000 1 # Si 63 |
4416 | + 1.750 1.750 1.250 1 # Si 64 |
4417 | + 1.125 1.125 1.125 2 # H 65 |
4418 | +%endblock AtomicCoordinatesAndAtomicSpecies |
4419 | |
4420 | === added file 'Tests/sih-elsi/sih-elsi.pseudos' |
4421 | --- Tests/sih-elsi/sih-elsi.pseudos 1970-01-01 00:00:00 +0000 |
4422 | +++ Tests/sih-elsi/sih-elsi.pseudos 2019-01-16 11:00:58 +0000 |
4423 | @@ -0,0 +1,1 @@ |
4424 | +Si H |
4425 | |
4426 | === modified file 'version.info' |
4427 | --- version.info 2019-01-12 22:35:38 +0000 |
4428 | +++ version.info 2019-01-16 11:00:58 +0000 |
4429 | @@ -1,2 +1,7 @@ |
4430 | +<<<<<<< TREE |
4431 | trunk-755 |
4432 | |
4433 | +======= |
4434 | +trunk-710--elsi-dm-695 |
4435 | + |
4436 | +>>>>>>> MERGE-SOURCE |
Victor: As a test of the MR process it is fine, but you should specify the target branch as siesta/ trunk-elsi- dm
~albertog/