bench-all
criterion performance measurements
overview
want to understand this report?
fromList (insert)/Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.063320944697221e-3 | 9.16803295881966e-3 | 9.285300070119086e-3 |
| Standard deviation | 2.3692471035260918e-4 | 3.018990415178708e-4 | 3.799167548250127e-4 |
Outlying measurements have moderate (0.12129327688392422%) effect on estimated standard deviation.
fromList (insert)/IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2783321907086207e-2 | 2.2971008378993733e-2 | 2.313630072328893e-2 |
| Standard deviation | 2.642225020491681e-4 | 3.8676877436829004e-4 | 5.428709821105256e-4 |
Outlying measurements have slight (4.7499999999999994e-2%) effect on estimated standard deviation.
fromAscList/Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8505572879003202e-3 | 1.8657613280517724e-3 | 1.8830158128896601e-3 |
| Standard deviation | 4.488450006021879e-5 | 5.625192016671321e-5 | 7.792347461265182e-5 |
Outlying measurements have moderate (0.1611219878790516%) effect on estimated standard deviation.
fromAscList/IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7527086971013037e-3 | 2.780152943181528e-3 | 2.8090489084223857e-3 |
| Standard deviation | 7.668830265368317e-5 | 9.214404447895278e-5 | 1.1558185321198094e-4 |
Outlying measurements have moderate (0.16778503770273756%) effect on estimated standard deviation.
search/lookup Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8196332618054968e-3 | 2.8247226827683575e-3 | 2.832663984819619e-3 |
| Standard deviation | 1.6887521508627586e-5 | 2.1822120015332523e-5 | 3.076802719713916e-5 |
Outlying measurements have slight (1.922337562475971e-2%) effect on estimated standard deviation.
search/lookup
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.0226562402743e-3 | 4.037131884733791e-3 | 4.06055406042815e-3 |
| Standard deviation | 4.1058937530737084e-5 | 5.90218921532702e-5 | 8.27826212463335e-5 |
Outlying measurements have slight (2.172839506172851e-2%) effect on estimated standard deviation.
search/containing
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.104219045855268e-2 | 2.1094104965802735e-2 | 2.120826296108886e-2 |
| Standard deviation | 8.82658926829415e-5 | 1.6324908563399135e-4 | 2.826329827117625e-4 |
Outlying measurements have slight (4.535147392290238e-2%) effect on estimated standard deviation.
search/intersecting
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.821020122740191e-2 | 3.83314796000944e-2 | 3.8449866439928586e-2 |
| Standard deviation | 1.7995950678907532e-4 | 2.565044731248561e-4 | 3.55317970006939e-4 |
Outlying measurements have slight (6.2222222222222234e-2%) effect on estimated standard deviation.
search/within
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2363697722933626e-2 | 3.24109293906533e-2 | 3.2513834683078474e-2 |
| Standard deviation | 7.202003576796586e-5 | 1.4506744900758296e-4 | 2.5305513723776566e-4 |
Outlying measurements have slight (5.5363321799307905e-2%) effect on estimated standard deviation.
mapKeys/Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.248529532273578e-3 | 5.279030118564385e-3 | 5.315119987148931e-3 |
| Standard deviation | 7.456593469400614e-5 | 1.011227806731691e-4 | 1.3436099789473368e-4 |
Outlying measurements have slight (2.4375000000000212e-2%) effect on estimated standard deviation.
mapKeys/IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1652003964000494e-2 | 2.198104862537561e-2 | 2.221577861883704e-2 |
| Standard deviation | 3.77485378954552e-4 | 6.528617704973618e-4 | 1.0712617370281603e-3 |
Outlying measurements have slight (9.114386216023777e-2%) effect on estimated standard deviation.
mapKeys/Data.Map monotonic
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0432410256196534e-3 | 2.0676456868300333e-3 | 2.101186157402692e-3 |
| Standard deviation | 7.170233938522881e-5 | 9.97489995515263e-5 | 1.5843154901292928e-4 |
Outlying measurements have moderate (0.33637643872015494%) effect on estimated standard deviation.
mapKeys/IntervalMap monotonic
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.685832893967636e-3 | 3.7245570664384223e-3 | 3.7687659311038627e-3 |
| Standard deviation | 9.889373356853597e-5 | 1.3252894811145716e-4 | 1.8152012331679006e-4 |
Outlying measurements have moderate (0.18538243710981187%) effect on estimated standard deviation.
map/Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.5796303089739801e-3 | 1.5934809513189388e-3 | 1.6059517811035573e-3 |
| Standard deviation | 3.432871017470686e-5 | 4.5122152959151e-5 | 6.77750366937583e-5 |
Outlying measurements have moderate (0.15311397993431955%) effect on estimated standard deviation.
map/IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6202318996515251e-3 | 1.639422960678449e-3 | 1.652330117456414e-3 |
| Standard deviation | 3.838679331458805e-5 | 5.134793703943737e-5 | 8.015622389886685e-5 |
Outlying measurements have moderate (0.18661648082364127%) effect on estimated standard deviation.
union/Data.Map Large/Small
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.117694147850352e-3 | 2.1412033966580015e-3 | 2.169480229366322e-3 |
| Standard deviation | 7.073876328983315e-5 | 8.719984693590616e-5 | 1.0055443769122983e-4 |
Outlying measurements have moderate (0.25578670942733056%) effect on estimated standard deviation.
union/Data.Map Small/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.6036292032980734e-3 | 3.642622864485545e-3 | 3.6688292659400558e-3 |
| Standard deviation | 7.493117810705843e-5 | 1.0192151492687482e-4 | 1.5410480747209495e-4 |
Outlying measurements have moderate (0.12305326415451937%) effect on estimated standard deviation.
union/IntervalMap Large/Small
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.665332557245488e-3 | 5.791999330090412e-3 | 5.916885381547013e-3 |
| Standard deviation | 3.314305895800142e-4 | 3.8708385295919094e-4 | 4.5289039807194445e-4 |
Outlying measurements have moderate (0.39515863119802763%) effect on estimated standard deviation.
union/IntervalMap Small/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.699942458751554e-3 | 5.8181145944534605e-3 | 6.004074598484305e-3 |
| Standard deviation | 3.3546144281876983e-4 | 4.445040236151414e-4 | 6.390236070721093e-4 |
Outlying measurements have moderate (0.4708885317443037%) effect on estimated standard deviation.
union/Data.Map Large/Empty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.102417823433325e-4 | 3.1037772632701625e-4 | 3.108481815825123e-4 |
| Standard deviation | 2.468963469030395e-7 | 7.378902265072289e-7 | 1.587794217940366e-6 |
Outlying measurements have slight (1.0525124490719783e-2%) effect on estimated standard deviation.
union/Data.Map Empty/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1308226808316196e-4 | 3.153090288089953e-4 | 3.2024238532690465e-4 |
| Standard deviation | 6.022103167932989e-6 | 1.100218061539847e-5 | 1.6953065766036766e-5 |
Outlying measurements have moderate (0.29551649253028783%) effect on estimated standard deviation.
union/IntervalMap Large/Empty
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.163262772712024e-3 | 3.1909364221613867e-3 | 3.21545440853837e-3 |
| Standard deviation | 6.46337466978707e-5 | 8.567185897643466e-5 | 1.1526977413384337e-4 |
Outlying measurements have moderate (0.11581622074263741%) effect on estimated standard deviation.
union/IntervalMap Empty/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.159267399435712e-3 | 3.188970208330845e-3 | 3.214565579434133e-3 |
| Standard deviation | 6.67358803707312e-5 | 8.662029193448214e-5 | 1.1771986444578077e-4 |
Outlying measurements have moderate (0.11596866746115464%) effect on estimated standard deviation.
union/Data.Map self
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.0735179826524737e-3 | 3.1103162486848346e-3 | 3.143266828867572e-3 |
| Standard deviation | 9.864187781149025e-5 | 1.2104802479466638e-4 | 1.7531181512449686e-4 |
Outlying measurements have moderate (0.21387990692318062%) effect on estimated standard deviation.
union/IntervalMap self
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.290531136693149e-3 | 6.377153718550439e-3 | 6.477887063625975e-3 |
| Standard deviation | 2.282132872298472e-4 | 2.7533818118837016e-4 | 3.435242956107699e-4 |
Outlying measurements have moderate (0.20562573436079062%) effect on estimated standard deviation.
intersection/Data.Map Large/Small
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1186902064434217e-3 | 3.1526176044269568e-3 | 3.202959718368414e-3 |
| Standard deviation | 9.821480120879368e-5 | 1.3131204330889837e-4 | 1.8941150063316592e-4 |
Outlying measurements have moderate (0.25236857473113%) effect on estimated standard deviation.
intersection/Data.Map Small/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.946476090608326e-3 | 2.975688574773587e-3 | 3.0178724659731824e-3 |
| Standard deviation | 8.946032332035171e-5 | 1.1491105079747922e-4 | 1.4866253547680285e-4 |
Outlying measurements have moderate (0.22792953869865878%) effect on estimated standard deviation.
intersection/IntervalMap Large/Small
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.5223860703069636e-3 | 3.5578212699752844e-3 | 3.5917180135520326e-3 |
| Standard deviation | 8.212780576799582e-5 | 1.0686102641820441e-4 | 1.5248573720598948e-4 |
Outlying measurements have moderate (0.14076184138794415%) effect on estimated standard deviation.
intersection/IntervalMap Small/Large
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.3819682210860127e-3 | 3.4239031803649234e-3 | 3.4626203105004505e-3 |
| Standard deviation | 9.519149261547472e-5 | 1.2292684923296206e-4 | 1.6807542534320383e-4 |
Outlying measurements have moderate (0.1973724447260385%) effect on estimated standard deviation.
difference/Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4846420195275055e-3 | 3.5191221099298217e-3 | 3.5559943996470236e-3 |
| Standard deviation | 8.290879598319643e-5 | 1.1174472628011323e-4 | 1.4933143298572046e-4 |
Outlying measurements have moderate (0.1606601783912034%) effect on estimated standard deviation.
difference/IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.43068193523353e-3 | 2.4527651866899313e-3 | 2.479878390068481e-3 |
| Standard deviation | 6.344771843466089e-5 | 8.112902025877605e-5 | 1.2420407099245114e-4 |
Outlying measurements have moderate (0.1761153743203978%) effect on estimated standard deviation.
delete/deleteMin Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.447974249631926e-3 | 1.4526301432430512e-3 | 1.4570875105310009e-3 |
| Standard deviation | 1.4020264575185223e-5 | 1.5480099188911578e-5 | 1.8468519042923117e-5 |
Outlying measurements have slight (1.562106324011086e-2%) effect on estimated standard deviation.
delete/deleteMax Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4139980801697988e-3 | 1.4157623473795668e-3 | 1.420289466837299e-3 |
| Standard deviation | 4.056530732130309e-6 | 8.303489484053621e-6 | 1.569420835639274e-5 |
Outlying measurements have slight (1.5380859374999922e-2%) effect on estimated standard deviation.
delete/deleteMin IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.364911229341958e-3 | 7.417240411892149e-3 | 7.479766055468985e-3 |
| Standard deviation | 1.3330104860448706e-4 | 1.6544993516164058e-4 | 2.078983641795166e-4 |
Outlying measurements have slight (7.912239632308224e-2%) effect on estimated standard deviation.
delete/deleteMax IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.5985835184826924e-3 | 7.657451900163132e-3 | 7.7332393495933025e-3 |
| Standard deviation | 1.5639111885833984e-4 | 1.919086243790127e-4 | 2.4605686150213155e-4 |
Outlying measurements have slight (8.253883015119558e-2%) effect on estimated standard deviation.
delete/minView Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3413345735319195e-3 | 4.3595038815754964e-3 | 4.3770343993675374e-3 |
| Standard deviation | 3.989796245613484e-5 | 5.2958866497237836e-5 | 7.220202272341409e-5 |
Outlying measurements have slight (2.2714981070849107e-2%) effect on estimated standard deviation.
delete/maxView Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.310122957323132e-3 | 4.324609019322243e-3 | 4.342402574012762e-3 |
| Standard deviation | 3.8335553341626396e-5 | 4.7306844424206866e-5 | 7.419124395941667e-5 |
Outlying measurements have slight (2.271498107084892e-2%) effect on estimated standard deviation.
delete/minView IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.694661623480239e-3 | 7.745630077229875e-3 | 7.807463433600263e-3 |
| Standard deviation | 1.085873066312743e-4 | 1.605458411799199e-4 | 2.6866495407757427e-4 |
Outlying measurements have slight (2.8546712802768413e-2%) effect on estimated standard deviation.
delete/maxView IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.437576008648183e-3 | 8.470804518889399e-3 | 8.518201100548356e-3 |
| Standard deviation | 7.070224819775639e-5 | 1.0896617325373903e-4 | 1.8058357065047634e-4 |
Outlying measurements have slight (2.9384756657483933e-2%) effect on estimated standard deviation.
delete/delete Data.Map
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.38639543186776e-3 | 7.412422767654832e-3 | 7.4447684620541745e-3 |
| Standard deviation | 6.668970915886652e-5 | 9.021056348483187e-5 | 1.1995974688041536e-4 |
Outlying measurements have slight (2.7755102040816326e-2%) effect on estimated standard deviation.
delete/delete IntervalMap
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.9355923661029516e-2 | 1.9429843603680812e-2 | 1.9598308425654985e-2 |
| Standard deviation | 1.2787094789535662e-4 | 2.559924950443226e-4 | 4.6955600187636814e-4 |
Outlying measurements have slight (4.338842975206612e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.