=========== Quick Start =========== ------------- cosmology.par ------------- You can find the file `cosmology.par` in the `examples` directory and it is as follows. .. index:: single: cosmology.par .. literalinclude:: ../examples/cosmology.par To run a simulation you simply provide pkdgrav3 with a file with the appropriate parameter set. The code is a Hybrid MPI/thread code so it is expected that you will run a single MPI process per node (or per NUMA node), and the number of threads equal to the number of cores. .. index:: single: SLURM If you use the SLURM scheduler, then pkdgrav3 will automatically detect the configuration that you requested via SLURM and configure itself appropriately. For example, if you have nodes with 16 cores, and you want to run on 10 nodes, then your SLURM script would be:: #!/bin/bash -l #SBATCH --nodes=10 #SBATCH --ntasks-per-node=1 --cpus-per-task=16 --ntasks-per-core=1 srun pkdgrav3 cosmology.par If you have a node with 128 cores and 8 NUMA nodes then you might prefer:: #!/bin/bash -l #SBATCH --nodes=10 #SBATCH --ntasks-per-node=8 --cpus-per-task=16 --ntasks-per-core=1 srun pkdgrav3 cosmology.par Now let's talk about the parameters used in the example file. Output Name =========== .. index:: single: parameters ; achOutName achOutName This parameter controls the filename part of output names. For example, if this is set to "sim1" then the output for step one will be "sim1.00001". Cosmological Simulation ======================= Periodic and Comoving --------------------- .. index:: single: cosmology; enabling To perform a cosmologic simulation (one with expansion), three additional features must be enabled. .. index:: single: parameters ; bComove bComove = True Turns on comoving coordinates which allows the expansion to be accounted for. .. index:: single: parameters ; bPeriodic bPeriodic = True Enables periodic boundary conditions. This is also required when using the initial condition generator. .. index:: single: parameters ; bEwald bEwald = True Turns on Ewald force correction to account for the infinite extent of the Universe. Accuracy -------- .. index:: pair: accuracy; cosmology You need to choose the timestepping scheme, as well as tune certain accuracy parameters .. index:: single: parameters ; bEpsAccStep bEpsAccStep = True This chooses the classic :math:`\epsilon / a` timestep criteria which is commonly used. .. index:: single: parameters ; dTheta dTheta = classic_theta_switch() Selects the dynamic :math:`\theta` switch which gives better accuracy for large scales at high redshift. .. index:: single: parameters ; nReplicas nReplicas = classic_replicas_switch() Switches the number of replicas between 2 and 1 depending on :math:`\theta`. .. index:: pair: initial conditions; cosmology Initial Conditions ================== The example parameter file is setup to generate an initial condition. Resolution ---------- .. index:: single: resolution .. index:: single: parameters ; dBoxSize dBoxSize The size of one side of the simulation volume (the length of the box) in :math:`h^{-1}` Mpc. .. index:: single: parameters ; nGrid nGrid The grid length. There will be :math:`nGrid^3` particles in the simulation. These two parameters combined set the mass resolution of the simulation. .. math:: M_{\text{particle}} &= \frac{\text{dBoxSize}^3 \times \rho_c}{\text{nGrid}^3}\\ &= \frac{\text{dBoxSize}^3}{\text{nGrid}^3}\times 2.77536627208\times 10^{11}~h^{-1}M_\odot Realization ----------- .. index:: single: parameters ; iLPT iLPT Sets the order of Lagrangian perturbation theory to be used when generating the IC. A value of 1 is 1LPT (Zeldovich), 2 is 2LPT, 3 is 3LPT. .. index:: single: parameters ; iSeed iSeed The random seed to use. A different seed gives a different random realization. Start Redshift -------------- .. index:: single: parameters ; dRedFrom dRedFrom The starting redshift. .. index:: single: cosmology Cosmology --------- To generate the initial condition you also need to specify the cosmology. In this example we use an input file that contains a transfer function generated by a Boltzmann code such as CLASS. .. index:: single: parameters ; achTfFile achTfFile The name of the file containing the tabulated transfer function. The first column is the wave number :math:`k` and the second column is the transfer function :math:`T(k)`. .. index:: single: parameters ; dSigma8 dSigma8 The cosmological parameter :math:`\sigma_8` which is defined as the r.m.s. density variation when smoothed over a sphere of radius :math:`8 h^{-1}` Mpc using a tophat filter. This parameter is used to calculate :math:`A_s` which is used to normalize the power spectrum. .. index:: single: parameters ; dSpectral dSpectral The spectral index :math:`n_s` used to normalize the power spectrum. .. index:: single: parameters ; h h The dimensionless Hubble constant :math:`h`. .. index:: single: parameters ; dOmega0 dOmega0 The matter density of the Universe :math:`\Omega_m`. .. index:: single: parameters ; dLambda dLambda = 0.68 The energy density of the Universe :math:`\Omega_\Lambda`. Simulation Duration =================== .. index:: single: parameters ; iStartStep iStartStep The starting step number. Simulations ordinarily start at step zero. .. index:: single: parameters ; nSteps nSteps The total number of steps. Outputs can only be produced at the end of each step, so this needs to be set at least as large as the number of outputs you want. The code will multistep (take smaller substeps) when the dynamics require it. .. index:: single: parameters ; dRedTo dRedTo What redshift to evolve the simulation to. Often zero (today) is appropriate. This parameter, combined with dRedFrom and nSteps sets :math:`\Delta T`. Output and Logging ================== .. index:: single: parameters ; iOutInterval iOutInterval Indicates how frequently an output is written, in steps. A setting of 10 would write an output at step 10, 20, 30, and so on. .. index:: single: parameters ; iCheckInterval iCheckInterval Similar to ``iOutInterval``, this controls how often a checkpoint is written. Simulations can be restarted from a checkpoint without a loss of precision. .. index:: single: parameters ; bDoDensity bDoDensity = False Causes the SPH densities to be calculated and output as a text file whenever a regular output is performed. .. index:: single: parameters ; bVDetails bVDetails = True Enables additional progress infomation to be displayed as the simulation runs. .. index:: single: parameters ; bOverwrite bOverwrite If the code exits abnormally it leaves a special file behind. If this file exists at startup, then the code exits. This flag disables this check. .. index:: single: parameters ; bParaRead bParaRead = True Read files in parallel. This is faster. .. index:: single: parameters ; bParaWrite bParaWrite = False Write files in parallel. .. index:: single: parameters ; nParaRead nParaRead To avoid overwhelming the file system, the number of simultanious readers can be limited by setting the parameter. .. index:: single: parameters ; nParaWrite nParaWrite This limits the number of simultanious writers during parallel write. Memory and Performance ====================== .. index:: single: parameters ; bMemUnordered bMemUnordered If false then particle IDs are not tracked. Particles are output in a random order and there is no way to track them between steps. This saves some memory. .. index:: single: parameters ; bNewKDK bNewKDK = True Switches to a new timestepping algorithm that does not require that accelerations be saved internally, instead velocity adjustments are applied directly as the forces are calculated. This saves memory.