Describing Experimental/Simulation Setup

Research

Often, the results of a performance analysis depend on the computers used and the specific features of software or libraries used. Therefore, it is essential to provide a comprehensive description of the experimental/simulation setup. This not only allows others to replicate the experiments but also facilitates the scrutiny of results for rigor, statistical validity, and impartiality. Ironically, the “experimental setup” section often ends up being the briefest in many research papers and theses, as authors attempt to economise space by minimising details.

Here are some insights and tips on what to incorporate, drawn from my research and supervisory experience, when elaborating on the experimental or simulation setup.

Type of Simulation

Specify whether the results stem from experimentation, emulation, or simulation. In the case of simulation, delve into additional particulars such as whether it is a Discrete Event, Monte Carlo, Stochastic, or Deterministic simulation.

Indicate the timing of result measurements, whether during steady-state, dynamic, starting/ramp-up, or terminating/ramp-down state(s).

Number of Experiments/Simulations

  • Discuss the sampling methods employed — whether random, systematic/deterministic, stratified, snowballing, etc.
  • State the number of samples taken or experiments conducted. While 10 or 20 samples is sometimes acceptable, ideally the number of samples should be determined by the desired confidence interval.
  • Include information about confidence intervals or accuracy levels. The commonly used 95% confidence interval implies a 95% probability that the population mean value falls within ∓1.96 standard deviations from the sample mean.
  • Provide insights into random number generation, including whether a distinct seed is used for each run and the seeds employed — especially when using widely available simulators/emulators.

Computer(s) Used

  • CPU details — Clock speed, number of cores, cache sizes, specific CPU model (state the model number rather than general terms like “Intel i7”, “Xeon”, or “AMD Opteron”), and any special instructions or compiler options used (e.g., MMX, SSE, AVX).
  • Memory specifications — Capacity and speed.
  • Accelerators — Mention GPUs, TPUs, or Intel Xeon Phi when their impact is significant.
  • Sensors and actuators — When relevant, describe those attached and their potential impact on results.
  • Operating system — Type, version, any special features, additional services running, node load, and any performance enhancements or optimisations applied.

The choice of which details to include depends on various factors: the context, type of publication, and the venue. It is advisable to explore how related work describes experimental setups in your area. As a rule of thumb, a thesis typically provides more extensive details than a paper — making the author’s thesis a valuable resource for readers seeking in-depth information.