About the Benchmarks
These times are informal and anecdotal, and are provided strictly as an example of the JUMBL's performance. Currently, only model analysis times are presented.
Current versions of the JUMBL 4 provide the following analytical engines for the model analysis (these can be selected via the -M switch):
- The Simple model analysis engine generates results via direct computation, and reports mean first passage times in terms of events (it is the only engine which does this). It requires the most time and memory to run. Its runtime is roughly O(n^3), where n is the number of states in the model.
- The Quick model analysis engine generates results via direct computation, but saves considerable time and memory by using a different computation for the mean first passage times, which are reported in terms of test cases. It is the default engine for model analyses. Its runtime is also roughly O(n^3), where n is the number of states in the model.
- The Fast model analysis engine generates results via successive approximation, and only computes a small subset of the results. It is typically the fastest engine, with a runtime of roughly O(n^2), where n is the number of states in the model.
- The Simulation model analysis engine generates results via a simulation. This engine is not present in all releases of the JUMBL. For larger models (tens of thousands of states) this is the only engine which can currently be used, and it is also the only engine which computes the probability of occurrence for each stimulus. It has low memory requirements, and its runtime is dependent on the average length of a test case in the model.
These runs use randomly-generated models.
- The graphs for Simple, Quick, and Fast are based on two runs for each model size.
- The graphs for Simulation are based on five runs for each model size because of the large variances.
Some quick reference numbers. One hour is 3600 seconds. One day is 86,400 seconds.
Intel
These results were generated with --mem=1024 on an IBM X22 laptop with Intel® Pentium® III Mobile 800MHz processor, 32Kb of cache, and 600Mb of memory. All engines were run until failure (insufficient memory) except the simulation engine.
Sparc
These results were generated with --mem=1024 on a Sun Ultra 5/10 with an UltraSPARC-IIi 440MHz processor, 2Mb of cache, and 512Mb of memory. All engines were run until failure (insufficient memory) except the simulation engine.