mirror of https://github.com/pulumi/pulumi.git
659a027eae
- Add a utility for generating snapshots for diff testing
- Add a framework for testing and benchmarking the deployment differ
Baseline benchmark results:
httpstate ❯ go test -count=10 -run none -benchmem -bench . | tee base.txt
```
│ base.txt │
│ sec/op │
DiffStack/1_x_2_B-10 46.30µ ± 1%
DiffStack/2_x_2_B-10 82.68µ ± 1%
DiffStack/4_x_2_B-10 191.0µ ± 1%
DiffStack/8_x_2_B-10 499.7µ ± 0%
DiffStack/16_x_2_B-10 1.489m ± 0%
DiffStack/32_x_2_B-10 4.895m ± 1%
DiffStack/48_x_2_B-10 9.651m ± 1%
DiffStack/64_x_2_B-10 15.11m ± 0%
DiffStack/1_x_8.2_kB-10 103.3µ ± 1%
DiffStack/2_x_8.2_kB-10 308.8µ ± 0%
DiffStack/4_x_8.2_kB-10 899.2µ ± 0%
DiffStack/8_x_8.2_kB-10 2.989m ± 1%
DiffStack/16_x_8.2_kB-10 10.53m ± 0%
DiffStack/32_x_8.2_kB-10 36.27m ± 1%
DiffStack/48_x_8.2_kB-10 72.53m ± 0%
DiffStack/64_x_8.2_kB-10 125.5m ± 0%
DiffStack/1_x_33_kB-10 317.4µ ± 1%
DiffStack/2_x_33_kB-10 756.8µ ± 1%
DiffStack/4_x_33_kB-10 2.605m ± 0%
DiffStack/8_x_33_kB-10 9.241m ± 2%
DiffStack/16_x_33_kB-10 32.86m ± 1%
DiffStack/32_x_33_kB-10 110.2m ± 2%
DiffStack/48_x_33_kB-10 231.3m ± 0%
DiffStack/64_x_33_kB-10 383.8m ± 1%
DiffStack/2_x_131_kB-10 2.748m ± 1%
DiffStack/4_x_131_kB-10 8.609m ± 0%
DiffStack/8_x_131_kB-10 28.44m ± 0%
DiffStack/16_x_131_kB-10 96.98m ± 1%
DiffStack/32_x_131_kB-10 349.4m ± 1%
DiffStack/48_x_131_kB-10 761.7m ± 0%
DiffStack/64_x_131_kB-10 1.312 ± 1%
DiffStack/1_x_524_kB-10 3.238m ± 1%
DiffStack/2_x_524_kB-10 9.867m ± 0%
DiffStack/4_x_524_kB-10 29.65m ± 0%
DiffStack/8_x_524_kB-10 98.91m ± 1%
DiffStack/16_x_524_kB-10 348.1m ± 1%
DiffStackRecorded/two-large-checkpoints.json-10 46.62m ± 0%
DiffStackRecorded/checkpoints.json-10 835.8m ± 0%
geomean 11.15m
│ base.txt │
│ ratio │
DiffStack/1_x_2_B-10 789.5m ± 0%
DiffStack/2_x_2_B-10 1.034 ± 0%
DiffStack/4_x_2_B-10 1.676 ± 0%
DiffStack/8_x_2_B-10 2.883 ± 0%
DiffStack/16_x_2_B-10 5.270 ± 0%
DiffStack/32_x_2_B-10 10.20 ± 0%
DiffStack/48_x_2_B-10 14.24 ± 0%
DiffStack/64_x_2_B-10 17.51 ± 0%
DiffStack/1_x_8.2_kB-10 950.9m ± 0%
DiffStack/2_x_8.2_kB-10 1.716 ± 0%
DiffStack/4_x_8.2_kB-10 2.792 ± 0%
DiffStack/8_x_8.2_kB-10 5.350 ± 0%
DiffStack/16_x_8.2_kB-10 10.62 ± 0%
DiffStack/32_x_8.2_kB-10 20.01 ± 0%
DiffStack/48_x_8.2_kB-10 27.48 ± 0%
DiffStack/64_x_8.2_kB-10 36.97 ± 0%
DiffStack/1_x_33_kB-10 1.467 ± 0%
DiffStack/2_x_33_kB-10 1.581 ± 0%
DiffStack/4_x_33_kB-10 2.967 ± 0%
DiffStack/8_x_33_kB-10 5.766 ± 0%
DiffStack/16_x_33_kB-10 10.87 ± 0%
DiffStack/32_x_33_kB-10 19.00 ± 0%
DiffStack/48_x_33_kB-10 27.14 ± 0%
DiffStack/64_x_33_kB-10 34.39 ± 0%
DiffStack/2_x_131_kB-10 1.993 ± 0%
DiffStack/4_x_131_kB-10 3.173 ± 0%
DiffStack/8_x_131_kB-10 5.334 ± 0%
DiffStack/16_x_131_kB-10 9.296 ± 0%
DiffStack/32_x_131_kB-10 17.10 ± 0%
DiffStack/48_x_131_kB-10 25.25 ± 0%
DiffStack/64_x_131_kB-10 33.13 ± 0%
DiffStack/1_x_524_kB-10 1.498 ± 0%
DiffStack/2_x_524_kB-10 1.998 ± 0%
DiffStack/4_x_524_kB-10 2.998 ± 0%
DiffStack/8_x_524_kB-10 5.084 ± 0%
DiffStack/16_x_524_kB-10 9.079 ± 0%
DiffStackRecorded/two-large-checkpoints.json-10 1.997 ± 0%
DiffStackRecorded/checkpoints.json-10 36.60 ± 0%
geomean 5.961
│ base.txt │
│ B/op │
DiffStack/1_x_2_B-10 35.17Ki ± 0%
DiffStack/2_x_2_B-10 70.05Ki ± 0%
DiffStack/4_x_2_B-10 186.8Ki ± 0%
DiffStack/8_x_2_B-10 529.0Ki ± 0%
DiffStack/16_x_2_B-10 1.768Mi ± 0%
DiffStack/32_x_2_B-10 6.513Mi ± 0%
DiffStack/48_x_2_B-10 14.05Mi ± 0%
DiffStack/64_x_2_B-10 22.39Mi ± 0%
DiffStack/1_x_8.2_kB-10 154.7Ki ± 0%
DiffStack/2_x_8.2_kB-10 571.9Ki ± 0%
DiffStack/4_x_8.2_kB-10 1.846Mi ± 0%
DiffStack/8_x_8.2_kB-10 6.920Mi ± 0%
DiffStack/16_x_8.2_kB-10 26.50Mi ± 0%
DiffStack/32_x_8.2_kB-10 98.33Mi ± 0%
DiffStack/48_x_8.2_kB-10 209.4Mi ± 0%
DiffStack/64_x_8.2_kB-10 365.6Mi ± 0%
DiffStack/1_x_33_kB-10 808.7Ki ± 0%
DiffStack/2_x_33_kB-10 2.013Mi ± 0%
DiffStack/4_x_33_kB-10 7.551Mi ± 0%
DiffStack/8_x_33_kB-10 28.58Mi ± 0%
DiffStack/16_x_33_kB-10 104.4Mi ± 0%
DiffStack/32_x_33_kB-10 365.5Mi ± 0%
DiffStack/48_x_33_kB-10 801.1Mi ± 0%
DiffStack/64_x_33_kB-10 1.283Gi ± 0%
DiffStack/2_x_131_kB-10 9.179Mi ± 0%
DiffStack/4_x_131_kB-10 29.14Mi ± 0%
DiffStack/8_x_131_kB-10 98.77Mi ± 0%
DiffStack/16_x_131_kB-10 337.1Mi ± 0%
DiffStack/32_x_131_kB-10 1.206Gi ± 0%
DiffStack/48_x_131_kB-10 2.771Gi ± 0%
DiffStack/64_x_131_kB-10 4.606Gi ± 0%
DiffStack/1_x_524_kB-10 12.41Mi ± 1%
DiffStack/2_x_524_kB-10 37.01Mi ± 1%
DiffStack/4_x_524_kB-10 110.6Mi ± 0%
DiffStack/8_x_524_kB-10 367.3Mi ± 1%
DiffStack/16_x_524_kB-10 1.261Gi ± 0%
DiffStackRecorded/two-large-checkpoints.json-10 69.25Mi ± 0%
DiffStackRecorded/checkpoints.json-10 1.198Gi ± 0%
geomean 26.14Mi
│ base.txt │
│ allocs/op │
DiffStack/1_x_2_B-10 361.0 ± 0%
DiffStack/2_x_2_B-10 649.0 ± 0%
DiffStack/4_x_2_B-10 1.512k ± 0%
DiffStack/8_x_2_B-10 4.141k ± 0%
DiffStack/16_x_2_B-10 13.09k ± 0%
DiffStack/32_x_2_B-10 46.38k ± 0%
DiffStack/48_x_2_B-10 94.38k ± 0%
DiffStack/64_x_2_B-10 152.7k ± 0%
DiffStack/1_x_8.2_kB-10 323.0 ± 0%
DiffStack/2_x_8.2_kB-10 630.0 ± 0%
DiffStack/4_x_8.2_kB-10 1.354k ± 0%
DiffStack/8_x_8.2_kB-10 3.606k ± 0%
DiffStack/16_x_8.2_kB-10 11.16k ± 0%
DiffStack/32_x_8.2_kB-10 36.45k ± 0%
DiffStack/48_x_8.2_kB-10 71.55k ± 0%
DiffStack/64_x_8.2_kB-10 123.9k ± 0%
DiffStack/1_x_33_kB-10 346.0 ± 0%
DiffStack/2_x_33_kB-10 607.0 ± 0%
DiffStack/4_x_33_kB-10 1.396k ± 0%
DiffStack/8_x_33_kB-10 3.764k ± 0%
DiffStack/16_x_33_kB-10 11.26k ± 0%
DiffStack/32_x_33_kB-10 34.55k ± 0%
DiffStack/48_x_33_kB-10 69.72k ± 0%
DiffStack/64_x_33_kB-10 114.1k ± 0%
DiffStack/2_x_131_kB-10 678.0 ± 0%
DiffStack/4_x_131_kB-10 1.494k ± 0%
DiffStack/8_x_131_kB-10 3.737k ± 0%
DiffStack/16_x_131_kB-10 10.40k ± 0%
DiffStack/32_x_131_kB-10 32.24k ± 0%
DiffStack/48_x_131_kB-10 66.12k ± 0%
DiffStack/64_x_131_kB-10 110.9k ± 0%
DiffStack/1_x_524_kB-10 374.0 ± 1%
DiffStack/2_x_524_kB-10 707.0 ± 0%
DiffStack/4_x_524_kB-10 1.528k ± 0%
DiffStack/8_x_524_kB-10 3.797k ± 1%
DiffStack/16_x_524_kB-10 10.56k ± 1%
DiffStackRecorded/two-large-checkpoints.json-10 512.1k ± 0%
DiffStackRecorded/checkpoints.json-10 7.240M ± 0%
geomean 8.309k
```
|
||
|---|---|---|
| .. | ||
| snapshot_test.json | ||
| two-large-checkpoints.json | ||