Quick Start¶

cosmology.par¶

You can find the file cosmology.par in the examples directory and it is as follows.

from accuracy import classic_theta_switch,classic_replicas_switch

achOutName      = "example"

# Initial Condition
dBoxSize        = 2000          # Mpc/h
nGrid           = 128           # Simulation has nGrid^3 particles
iLPT            = 2             # LPT order for IC
iSeed           = 314159265     # Random seed
dRedFrom        = 49            # Starting redshift

# Cosmology
achTfFile       = "euclid_z0_transfer_combined.dat"
h               = 0.67
dOmega0         = 0.32
dLambda         = 0.68
dSigma8         = 0.83
dSpectral       = 0.96

iStartStep      = 0
nSteps          = 100
dRedTo          = 0.0

# Cosmological Simulation
bComove         = True          # Use comoving coordinates
bPeriodic       = True          # with a periodic box
bEwald          = True          # enable Ewald periodic boundaries

# Logging/Output
iOutInterval    = 10
#iCheckInterval = 10
bDoDensity      = False
bVDetails       = True

bOverwrite      = True
bParaRead       = True          # Read in parallel
bParaWrite      = False         # Write in parallel (does not work on all file systems)
#nParaRead      = 8             # Limit number of simultaneous readers to this
#nParaWrite     = 8             # Limit number of simultaneous writers to this

# Accuracy Parameters
bEpsAccStep     = True          # Choose eps/a timestep criteria
dTheta          = classic_theta_switch()        # 0.40, 0.55, 0.70 switch
nReplicas       = classic_replicas_switch()     # 1 if theta > 0.52 otherwise 2

# Memory and performance
bMemUnordered   = True          # iOrder replaced by potential and group id
bNewKDK         = True          # No accelerations in the particle, dual tree possible

To run a simulation you simply provide pkdgrav3 with a file with the appropriate parameter set. The code is a Hybrid MPI/thread code so it is expected that you will run a single MPI process per node (or per NUMA node), and the number of threads equal to the number of cores.

If you use the SLURM scheduler, then pkdgrav3 will automatically detect the configuration that you requested via SLURM and configure itself appropriately. For example, if you have nodes with 16 cores, and you want to run on 10 nodes, then your SLURM script would be:

#!/bin/bash -l
#SBATCH --nodes=10
#SBATCH --ntasks-per-node=1 --cpus-per-task=16 --ntasks-per-core=1
srun pkdgrav3 cosmology.par

If you have a node with 128 cores and 8 NUMA nodes then you might prefer:

#!/bin/bash -l
#SBATCH --nodes=10
#SBATCH --ntasks-per-node=8 --cpus-per-task=16 --ntasks-per-core=1
srun pkdgrav3 cosmology.par

Now let’s talk about the parameters used in the example file.

Output Name¶

achOutName: This parameter controls the filename part of output names. For example, if this is set to “sim1” then the output for step one will be “sim1.00001”.

Cosmological Simulation¶

Periodic and Comoving¶

To perform a cosmologic simulation (one with expansion), three additional features must be enabled.

bComove = True: Turns on comoving coordinates which allows the expansion to be accounted for.

bPeriodic = True: Enables periodic boundary conditions. This is also required when using the initial condition generator.

bEwald = True: Turns on Ewald force correction to account for the infinite extent of the Universe.

Accuracy¶

You need to choose the timestepping scheme, as well as tune certain accuracy parameters

bEpsAccStep = True: This chooses the classic \(\epsilon / a\) timestep criteria which is commonly used.

dTheta = classic_theta_switch(): Selects the dynamic \(\theta\) switch which gives better accuracy for large scales at high redshift.

nReplicas = classic_replicas_switch(): Switches the number of replicas between 2 and 1 depending on \(\theta\).

Initial Conditions¶

The example parameter file is setup to generate an initial condition.

Resolution¶

dBoxSize: The size of one side of the simulation volume (the length of the box) in \(h^{-1}\) Mpc.

nGrid: The grid length. There will be \(nGrid^3\) particles in the simulation.

These two parameters combined set the mass resolution of the simulation.

\[\begin{split}M_{\text{particle}} &= \frac{\text{dBoxSize}^3 \times \rho_c}{\text{nGrid}^3}\\ &= \frac{\text{dBoxSize}^3}{\text{nGrid}^3}\times 2.77536627208\times 10^{11}~h^{-1}M_\odot\end{split}\]

Realization¶

iLPT: Sets the order of Lagrangian perturbation theory to be used when generating the IC. A value of 1 is 1LPT (Zeldovich), 2 is 2LPT, 3 is 3LPT.

iSeed: The random seed to use. A different seed gives a different random realization.

Start Redshift¶

dRedFrom: The starting redshift.

Cosmology¶

To generate the initial condition you also need to specify the cosmology. In this example we use an input file that contains a transfer function generated by a Boltzmann code such as CLASS.

achTfFile: The name of the file containing the tabulated transfer function. The first column is the wave number \(k\) and the second column is the transfer function \(T(k)\).

dSigma8: The cosmological parameter \(\sigma_8\) which is defined as the r.m.s. density variation when smoothed over a sphere of radius \(8 h^{-1}\) Mpc using a tophat filter. This parameter is used to calculate \(A_s\) which is used to normalize the power spectrum.

dSpectral: The spectral index \(n_s\) used to normalize the power spectrum.

h: The dimensionless Hubble constant \(h\).

dOmega0: The matter density of the Universe \(\Omega_m\).

dLambda = 0.68: The energy density of the Universe \(\Omega_\Lambda\).

Simulation Duration¶

iStartStep: The starting step number. Simulations ordinarily start at step zero.

nSteps: The total number of steps. Outputs can only be produced at the end of each step, so this needs to be set at least as large as the number of outputs you want. The code will multistep (take smaller substeps) when the dynamics require it.

dRedTo: What redshift to evolve the simulation to. Often zero (today) is appropriate. This parameter, combined with dRedFrom and nSteps sets \(\Delta T\).

Output and Logging¶

iOutInterval: Indicates how frequently an output is written, in steps. A setting of 10 would write an output at step 10, 20, 30, and so on.

iCheckInterval: Similar to iOutInterval, this controls how often a checkpoint is written. Simulations can be restarted from a checkpoint without a loss of precision.

bDoDensity = False: Causes the SPH densities to be calculated and output as a text file whenever a regular output is performed.

bVDetails = True: Enables additional progress infomation to be displayed as the simulation runs.

bOverwrite: If the code exits abnormally it leaves a special file behind. If this file exists at startup, then the code exits. This flag disables this check.

bParaRead = True: Read files in parallel. This is faster.

bParaWrite = False: Write files in parallel.

nParaRead: To avoid overwhelming the file system, the number of simultanious readers can be limited by setting the parameter.

nParaWrite: This limits the number of simultanious writers during parallel write.

Memory and Performance¶

bMemUnordered: If false then particle IDs are not tracked. Particles are output in a random order and there is no way to track them between steps. This saves some memory.

bNewKDK = True: Switches to a new timestepping algorithm that does not require that accelerations be saved internally, instead velocity adjustments are applied directly as the forces are calculated. This saves memory.