Palisade Knowledge Base

HomeTechniques and Tips@RISK Simulation: Numerical ResultsConvergence by Testing Percentiles

6.3. Convergence by Testing Percentiles

Why do different percentiles take different numbers of iterations to converge? And why do percentiles sometimes converge more quickly than the mean, even though the mean should be more stable?

There can definitely be some surprises when you use percentiles as your criterion for convergence, and you can also get very different behavior from different distributions.

First, an explanation of how @RISK tests for convergence.  In Simulation Settings, on the Convergence tab, you can specify a convergence tolerance and a confidence level or use the default settings of 3% tolerance and 95% confidence.  Setting 3% tolerance at 95% confidence means that @RISK keeps iterating until there is better than a 95% probability that true percentile of the distribution is within ±3% of the corresponding percentile of the simulation data accumulated so far.  (See also: Convergence Monitoring in @RISK.)

Example: You're testing convergence on P99 (the 99th percentile).  N iterations into the simulation, the 99th percentile of those N iterations is 3872,  A 3% tolerance is 3% of 3872 = about 116.  @RISK computes the chance that the true P99 of the population is within 3872±116.  If that chance is better than 95%, @RISK considers that P99 has converged.  If that chance is less than 95%, @RISK uses the sample P99 (from the N iterations so far) to estimate how many iterations will be needed to get that chance above 95%.  In the Status column of the Results Summary window, @RISK displays the percentage of the necessary iterations that @RISK has performed so far.

Technical details: @RISK computes the probabilities by using the theory in Distribution-Free Confidence Intervals for Percentiles (accessed 2020-07-28).  The article gives an exact computation using the binomial distribution and an approximate calculation using the normal distribution; @RISK uses the binomial calculation.

Now, an explanation of anomalies, including those mentioned above.

Advice: In @RISK's simulation settings you have to set convergence tolerance as a percentage of the tested statistic (mean, standard deviation, or percentile), but the appropriate percentage is not always obvious.  To help you make the decision, run a simulation with a few iterations, say 100, just to get a sense of what the output distribution looks like. Then, if you expect the percentile value to be close to zero, specify a higher tolerance or choose a different statistic. Also check your tolerance against the expected range of the output, and if necessary specify a smaller tolerance.

Last edited: 2020-07-28

This page was: Helpful | Not Helpful