mirror of https://github.com/pulumi/pulumi.git
51a62a9265
In #17623 through #17627 and some follow-up PRs, we built out a framework for fuzzing lifecycle tests in order to help track down snapshot integrity violations in the Pulumi engine. All that remains now is to actually provide ways to trigger a fuzzing run in useful ways. This commit kicks this off by introducing two Go test functions that can be run with `go test` or our `Makefile`: * `TestFuzz` -- this runs the fuzzer and generates a brand new set of scenarios (1,000 by default) and checks whether any of them result in a snapshot integrity error. This test is skipped unless an environment variable is set (which the `Makefile` handles if one runs `make test_lifecycle_fuzz`). The intended purpose of this test is to back one or more CI workflows that will run periodically in order to slowly explore the state space. * `TestFuzzFromStateFile` -- this accepts a path to a JSON state file (such as that produced by a `pulumi stack export`) and uses that state to seed the fuzzer, subsequently trying to find provider and operation configurations that lead to a snapshot integrity error. This test is skipped unless a state file path is set using the relevant environment variable. The intended purpose of this test is to make it possible to find root causes for user issues when all we have is a state and we'd like to guess the program/provider configurations that led to an issue. Alongside introducing these two tests, we bulk out the fuzzing documentation a bit to help engineers run them, and link to the new sections from the existing docs on snapshot integrity issues. |
||
|---|---|---|
| .. | ||
| framework | ||
| fuzzing | ||
| testdata | ||
| README.md | ||
| alias_test.go | ||
| analyzer_test.go | ||
| continue_on_error_test.go | ||
| delete_before_replace_test.go | ||
| fuzz_test.go | ||
| golang_sdk_test.go | ||
| import_test.go | ||
| loader_test.go | ||
| parameterized_test.go | ||
| pending_delete_test.go | ||
| pending_replace_test.go | ||
| provider_test.go | ||
| pulumi_test.go | ||
| refresh_legacy_diff_test.go | ||
| refresh_test.go | ||
| resource_reference_test.go | ||
| retain_on_delete_test.go | ||
| source_query_test.go | ||
| step_generator_test.go | ||
| target_test.go | ||
| transformation_test.go | ||
| update_plan_test.go | ||
README.md
(lifecycle-tests)=
Lifecycle tests
Lifecycle tests exercise the Pulumi engine and serve as a specification for the behaviours and interactions of the various features that define the lifecycle of a Pulumi program. This includes, but is not limited to:
- The operation(s) being executed (
up,preview, etc.) and the options passed to that operation (--target,--target-dependents, etc.). - The programs being executed -- their resources, invocations, and the various
options that might be associated with them (
parent,retainOnDelete, etc.). - The state of the program before and after operations are executed.
How and when to use