pulumi/changelog/pending
Mikhail Shilkov 55edc30d68
Add RequireEmptyPreviewAfterRefresh option to ProgramTest (#15309)
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

While working on [this issue in Azure
Native](https://github.com/pulumi/pulumi-azure-native/issues/2798), I
realized that I don't see how to test the scenario that I care about. I
need the following workflow:

1. Run `pulumi up` for the initial deployment.
2. Run `pulumi refresh` to pull changes from API.
3. Run `pulumi preview --expect-no-changes` to guarantee that there is
no loop of `refresh/up` stepping over each other.

The existing `ExpectRefreshChanges` option is different. If it's set to
false, the test would require no changes on step (2) above, which is a
stronger requirement. Unfortunately, it's not always true in Azure
Native, e.g. when a subresource changes properties of its parent
resource. I want to enable "ExpectRefreshChanges: false" where possible,
but I also want to require step (3) to succeed for all tests in Azure
Native (and potentially everywhere, eventually). I've already found
three issues while drafting that.

Therefore, this PR proposes a new option for ProgramTest:
`RequireEmptyPreviewAfterRefresh`. It defaults to false, so no behavior
changes unless users opt in. If set to true, it runs an extra preview
after refreshing, i.e. the step (3).

Let me know if I missed a better way of achieving the same.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-31 09:17:37 +00:00
..
20230920--programgen--use-importlib-metadata-instead-of-pkg_resources.yaml [sdkgen/python] Use `importlib.metadata` instead of `pkg_resources` (#15266) 2024-01-27 02:13:37 +00:00
20231228--backend-filestate--upgrade-gocloud-dev-to-the-latest-version.yaml upgrade gocloud.dev take 2 (#15202) 2024-01-26 13:14:17 +00:00
20240104--auto-go--add-new-api-to-install-the-pulumi-cli-from-the-automation-api.yaml [auto/go] Add new API to install the Pulumi CLI (#15049) 2024-01-26 16:01:17 +00:00
20240122--auto-python--add-new-api-to-install-the-pulumi-cli.yaml [auto/python] Add new API to install the Pulumi CLI (#15204) 2024-01-26 14:30:19 +00:00
20240122--engine--fix-a-hang-in-preview-import-file.yaml Fix hang in --preview-file (#15203) 2024-01-25 18:03:38 +00:00
20240124--auto-nodejs--allow-disabling-version-validation-when-using-a-custom-pulumicomamnd-instance.yaml Allow disabling version validation when using a custom PulumiCommand isntance (#15242) 2024-01-26 13:41:10 +00:00
20240124--engine--engine-will-now-error-if-aliases-request-a-parent-with-no-urn.yaml Test that an empty parent URN in an alias is treated as the default parent (#15232) 2024-01-26 18:29:22 +00:00
20240124--engine--fix-stack-name-validation-when-pulumi_disable_validation-isnt-set.yaml Fix stack name validation check (#15245) 2024-01-27 10:35:20 +00:00
20240125--programgen-nodejs--fixes-generated-expression-for-filebase64-function-to-use-fs-readfilesync-directly-with-base64-encoding-option.yaml [program-gen/ts] Fixes generated expression for filebase64 function to use fs.readFileSync directly (#15253) 2024-01-25 18:14:38 +00:00
20240127--auto-go--remove-obsolete-note-from-fullyqualifiedstackname-comment.yaml auto.FullyQualifiedStackName doc fix (#15281) 2024-01-29 20:53:01 +00:00
20240127--auto-python--inline-programs-can-now-be-defined-as-async-functions.yaml Support async inline programs (#15278) 2024-01-29 16:10:13 +00:00
20240127--sdk-go--include-error-message-in-local-workspace-validation.yaml local workspace project validation should include error message (#15279) 2024-01-28 15:29:14 +00:00
20240127--sdk-python--remove-compatability-code-for-python-3-7-and-below.yaml Clean up legacy Py 3.7 support (#15276) 2024-01-30 10:23:57 +00:00
20240127--sdkgen-go--remove-pulumix-references-from-generated-enums.yaml [sdk-gen/go] Remove pulumix references from generated enums in non-generic SDKs (#15280) 2024-01-27 18:57:16 +00:00
20240129--engine--fix-a-performance-issue-doing-unneeded-secret-encryption.yaml Don't encrypt secrets to get ciphertext that isn't actually used (#15306) 2024-01-30 09:51:51 +00:00
20240130--backend--allows-the-pulumi_skip_checkpoints-environment-variable-to-be-set-without-requiring-the-pulumi_experimental-flag.yaml Allows SKIP_CHECKPOINTS without EXPERIMENTAL (#15318) 2024-01-30 18:31:49 +00:00
20240130--pkg-testing--add-requireemptypreviewafterrefresh-option-to-programtest.yaml Add RequireEmptyPreviewAfterRefresh option to ProgramTest (#15309) 2024-01-31 09:17:37 +00:00