mirrors
/
pulumi
mirror of https://github.com/pulumi/pulumi.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
pulumi/pkg/engine
History
Pat Gavlin 659a027eae [snapshot] Add diff benchmarks and more tests 
- Add a utility for generating snapshots for diff testing
- Add a framework for testing and benchmarking the deployment differ

Baseline benchmark results:

    httpstate ❯ go test -count=10 -run none -benchmem -bench . | tee base.txt

```
                                                │  base.txt   │
                                                │   sec/op    │
DiffStack/1_x_2_B-10                              46.30µ ± 1%
DiffStack/2_x_2_B-10                              82.68µ ± 1%
DiffStack/4_x_2_B-10                              191.0µ ± 1%
DiffStack/8_x_2_B-10                              499.7µ ± 0%
DiffStack/16_x_2_B-10                             1.489m ± 0%
DiffStack/32_x_2_B-10                             4.895m ± 1%
DiffStack/48_x_2_B-10                             9.651m ± 1%
DiffStack/64_x_2_B-10                             15.11m ± 0%
DiffStack/1_x_8.2_kB-10                           103.3µ ± 1%
DiffStack/2_x_8.2_kB-10                           308.8µ ± 0%
DiffStack/4_x_8.2_kB-10                           899.2µ ± 0%
DiffStack/8_x_8.2_kB-10                           2.989m ± 1%
DiffStack/16_x_8.2_kB-10                          10.53m ± 0%
DiffStack/32_x_8.2_kB-10                          36.27m ± 1%
DiffStack/48_x_8.2_kB-10                          72.53m ± 0%
DiffStack/64_x_8.2_kB-10                          125.5m ± 0%
DiffStack/1_x_33_kB-10                            317.4µ ± 1%
DiffStack/2_x_33_kB-10                            756.8µ ± 1%
DiffStack/4_x_33_kB-10                            2.605m ± 0%
DiffStack/8_x_33_kB-10                            9.241m ± 2%
DiffStack/16_x_33_kB-10                           32.86m ± 1%
DiffStack/32_x_33_kB-10                           110.2m ± 2%
DiffStack/48_x_33_kB-10                           231.3m ± 0%
DiffStack/64_x_33_kB-10                           383.8m ± 1%
DiffStack/2_x_131_kB-10                           2.748m ± 1%
DiffStack/4_x_131_kB-10                           8.609m ± 0%
DiffStack/8_x_131_kB-10                           28.44m ± 0%
DiffStack/16_x_131_kB-10                          96.98m ± 1%
DiffStack/32_x_131_kB-10                          349.4m ± 1%
DiffStack/48_x_131_kB-10                          761.7m ± 0%
DiffStack/64_x_131_kB-10                           1.312 ± 1%
DiffStack/1_x_524_kB-10                           3.238m ± 1%
DiffStack/2_x_524_kB-10                           9.867m ± 0%
DiffStack/4_x_524_kB-10                           29.65m ± 0%
DiffStack/8_x_524_kB-10                           98.91m ± 1%
DiffStack/16_x_524_kB-10                          348.1m ± 1%
DiffStackRecorded/two-large-checkpoints.json-10   46.62m ± 0%
DiffStackRecorded/checkpoints.json-10             835.8m ± 0%
geomean                                           11.15m

                                                │  base.txt   │
                                                │    ratio    │
DiffStack/1_x_2_B-10                              789.5m ± 0%
DiffStack/2_x_2_B-10                               1.034 ± 0%
DiffStack/4_x_2_B-10                               1.676 ± 0%
DiffStack/8_x_2_B-10                               2.883 ± 0%
DiffStack/16_x_2_B-10                              5.270 ± 0%
DiffStack/32_x_2_B-10                              10.20 ± 0%
DiffStack/48_x_2_B-10                              14.24 ± 0%
DiffStack/64_x_2_B-10                              17.51 ± 0%
DiffStack/1_x_8.2_kB-10                           950.9m ± 0%
DiffStack/2_x_8.2_kB-10                            1.716 ± 0%
DiffStack/4_x_8.2_kB-10                            2.792 ± 0%
DiffStack/8_x_8.2_kB-10                            5.350 ± 0%
DiffStack/16_x_8.2_kB-10                           10.62 ± 0%
DiffStack/32_x_8.2_kB-10                           20.01 ± 0%
DiffStack/48_x_8.2_kB-10                           27.48 ± 0%
DiffStack/64_x_8.2_kB-10                           36.97 ± 0%
DiffStack/1_x_33_kB-10                             1.467 ± 0%
DiffStack/2_x_33_kB-10                             1.581 ± 0%
DiffStack/4_x_33_kB-10                             2.967 ± 0%
DiffStack/8_x_33_kB-10                             5.766 ± 0%
DiffStack/16_x_33_kB-10                            10.87 ± 0%
DiffStack/32_x_33_kB-10                            19.00 ± 0%
DiffStack/48_x_33_kB-10                            27.14 ± 0%
DiffStack/64_x_33_kB-10                            34.39 ± 0%
DiffStack/2_x_131_kB-10                            1.993 ± 0%
DiffStack/4_x_131_kB-10                            3.173 ± 0%
DiffStack/8_x_131_kB-10                            5.334 ± 0%
DiffStack/16_x_131_kB-10                           9.296 ± 0%
DiffStack/32_x_131_kB-10                           17.10 ± 0%
DiffStack/48_x_131_kB-10                           25.25 ± 0%
DiffStack/64_x_131_kB-10                           33.13 ± 0%
DiffStack/1_x_524_kB-10                            1.498 ± 0%
DiffStack/2_x_524_kB-10                            1.998 ± 0%
DiffStack/4_x_524_kB-10                            2.998 ± 0%
DiffStack/8_x_524_kB-10                            5.084 ± 0%
DiffStack/16_x_524_kB-10                           9.079 ± 0%
DiffStackRecorded/two-large-checkpoints.json-10    1.997 ± 0%
DiffStackRecorded/checkpoints.json-10              36.60 ± 0%
geomean                                            5.961

                                                │   base.txt   │
                                                │     B/op     │
DiffStack/1_x_2_B-10                              35.17Ki ± 0%
DiffStack/2_x_2_B-10                              70.05Ki ± 0%
DiffStack/4_x_2_B-10                              186.8Ki ± 0%
DiffStack/8_x_2_B-10                              529.0Ki ± 0%
DiffStack/16_x_2_B-10                             1.768Mi ± 0%
DiffStack/32_x_2_B-10                             6.513Mi ± 0%
DiffStack/48_x_2_B-10                             14.05Mi ± 0%
DiffStack/64_x_2_B-10                             22.39Mi ± 0%
DiffStack/1_x_8.2_kB-10                           154.7Ki ± 0%
DiffStack/2_x_8.2_kB-10                           571.9Ki ± 0%
DiffStack/4_x_8.2_kB-10                           1.846Mi ± 0%
DiffStack/8_x_8.2_kB-10                           6.920Mi ± 0%
DiffStack/16_x_8.2_kB-10                          26.50Mi ± 0%
DiffStack/32_x_8.2_kB-10                          98.33Mi ± 0%
DiffStack/48_x_8.2_kB-10                          209.4Mi ± 0%
DiffStack/64_x_8.2_kB-10                          365.6Mi ± 0%
DiffStack/1_x_33_kB-10                            808.7Ki ± 0%
DiffStack/2_x_33_kB-10                            2.013Mi ± 0%
DiffStack/4_x_33_kB-10                            7.551Mi ± 0%
DiffStack/8_x_33_kB-10                            28.58Mi ± 0%
DiffStack/16_x_33_kB-10                           104.4Mi ± 0%
DiffStack/32_x_33_kB-10                           365.5Mi ± 0%
DiffStack/48_x_33_kB-10                           801.1Mi ± 0%
DiffStack/64_x_33_kB-10                           1.283Gi ± 0%
DiffStack/2_x_131_kB-10                           9.179Mi ± 0%
DiffStack/4_x_131_kB-10                           29.14Mi ± 0%
DiffStack/8_x_131_kB-10                           98.77Mi ± 0%
DiffStack/16_x_131_kB-10                          337.1Mi ± 0%
DiffStack/32_x_131_kB-10                          1.206Gi ± 0%
DiffStack/48_x_131_kB-10                          2.771Gi ± 0%
DiffStack/64_x_131_kB-10                          4.606Gi ± 0%
DiffStack/1_x_524_kB-10                           12.41Mi ± 1%
DiffStack/2_x_524_kB-10                           37.01Mi ± 1%
DiffStack/4_x_524_kB-10                           110.6Mi ± 0%
DiffStack/8_x_524_kB-10                           367.3Mi ± 1%
DiffStack/16_x_524_kB-10                          1.261Gi ± 0%
DiffStackRecorded/two-large-checkpoints.json-10   69.25Mi ± 0%
DiffStackRecorded/checkpoints.json-10             1.198Gi ± 0%
geomean                                           26.14Mi

                                                │  base.txt   │
                                                │  allocs/op  │
DiffStack/1_x_2_B-10                               361.0 ± 0%
DiffStack/2_x_2_B-10                               649.0 ± 0%
DiffStack/4_x_2_B-10                              1.512k ± 0%
DiffStack/8_x_2_B-10                              4.141k ± 0%
DiffStack/16_x_2_B-10                             13.09k ± 0%
DiffStack/32_x_2_B-10                             46.38k ± 0%
DiffStack/48_x_2_B-10                             94.38k ± 0%
DiffStack/64_x_2_B-10                             152.7k ± 0%
DiffStack/1_x_8.2_kB-10                            323.0 ± 0%
DiffStack/2_x_8.2_kB-10                            630.0 ± 0%
DiffStack/4_x_8.2_kB-10                           1.354k ± 0%
DiffStack/8_x_8.2_kB-10                           3.606k ± 0%
DiffStack/16_x_8.2_kB-10                          11.16k ± 0%
DiffStack/32_x_8.2_kB-10                          36.45k ± 0%
DiffStack/48_x_8.2_kB-10                          71.55k ± 0%
DiffStack/64_x_8.2_kB-10                          123.9k ± 0%
DiffStack/1_x_33_kB-10                             346.0 ± 0%
DiffStack/2_x_33_kB-10                             607.0 ± 0%
DiffStack/4_x_33_kB-10                            1.396k ± 0%
DiffStack/8_x_33_kB-10                            3.764k ± 0%
DiffStack/16_x_33_kB-10                           11.26k ± 0%
DiffStack/32_x_33_kB-10                           34.55k ± 0%
DiffStack/48_x_33_kB-10                           69.72k ± 0%
DiffStack/64_x_33_kB-10                           114.1k ± 0%
DiffStack/2_x_131_kB-10                            678.0 ± 0%
DiffStack/4_x_131_kB-10                           1.494k ± 0%
DiffStack/8_x_131_kB-10                           3.737k ± 0%
DiffStack/16_x_131_kB-10                          10.40k ± 0%
DiffStack/32_x_131_kB-10                          32.24k ± 0%
DiffStack/48_x_131_kB-10                          66.12k ± 0%
DiffStack/64_x_131_kB-10                          110.9k ± 0%
DiffStack/1_x_524_kB-10                            374.0 ± 1%
DiffStack/2_x_524_kB-10                            707.0 ± 0%
DiffStack/4_x_524_kB-10                           1.528k ± 0%
DiffStack/8_x_524_kB-10                           3.797k ± 1%
DiffStack/16_x_524_kB-10                          10.56k ± 1%
DiffStackRecorded/two-large-checkpoints.json-10   512.1k ± 0%
DiffStackRecorded/checkpoints.json-10             7.240M ± 0%
geomean                                           8.309k
```
 		2023-05-10 17:13:20 -07:00
..
lifecycletest [snapshot] Add diff benchmarks and more tests 2023-05-10 17:13:20 -07:00
deployment.go Pass PULUMI_CONFIG through to provider plugins 2023-04-05 10:17:18 +01:00
destroy.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
detailedDiff.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
detailedDiff_test.go [display] Refactor diff translation and display. (#9363) 2022-04-07 12:05:33 -07:00
doc.go Document Go packages (#6009) 2021-01-11 11:07:59 -07:00
engine.go [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
errors.go [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
events.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
events_test.go Do not panic when sending events to closed emitter (#10489) 2022-08-25 12:43:10 -04:00
eventsink.go chore: WriteString(Sprintf(..)) => Fprintf(..) 2023-03-01 13:22:33 -08:00
import.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
journal.go sdk/go: Remove 'nolint' directives from package docs 2023-01-06 09:06:47 -08:00
plugin_host.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
plugins.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
plugins_test.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
project.go Update error handling (#8406) 2021-11-12 18:37:17 -08:00
query.go Pass PULUMI_CONFIG through to provider plugins 2023-04-05 10:17:18 +01:00
refresh.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
snapshot.go [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
update.go all: Reformat with gofumpt 2023-03-03 09:00:24 -08:00
update_test.go ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00