Commit Graph

1055 Commits

Author SHA1 Message Date
Fraser Waters bb6b9c675a
Add asset/archive to conformance tests and fix engine working dir issues ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/16092.

Two main changes in this PR. Firstly we add a test for assets/archives
to the conformance tests, and had to fix up some small issues in the
test runner to enable this (copying test data files, correcting project
directory layout, handling multi-word package names).

The other change is to fix the engine to respect the programs working
directory when serialising/deserialising assets and archives. It does
this by passing the programs working directory as a MarshalOption (n.b
this option is only really needed in the engine, program SDK doesn't
really need it because it doesn't change directories).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-02 11:32:54 +00:00
Paul C. Roberts b72399bd28
The `--expect-no-changes` flag checks for output diffs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds a pseudo op `OpOutputChange` to the set of changes that are
recorded in `display.ResourceChanges` to count the number of output
changes, this is then included in the check used to evaluate the
`--expect-no-changes` flag

When resource outputs are registered, they are checked against their
previous value using existing functionality, the total count of changes
is then added

The internal capability is validated with an engine test, the cli is
validated using an integration test

This will break user workflows that depend on the previous behavior

Fixes 

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [X] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Paul Roberts <proberts@pulumi.com>
2024-05-01 18:30:49 +00:00
Thomas Gummerer 8219c922a2
display: only hide replacement steps in diff ()
When displaying diff events, we currently try to hide non-logical
replacement steps unless we specifically enable showing them. However we
currently do that for all non-logical operations, regardless whether
they are replacement steps or not.

In particular a RefreshStep is non-logical, but it's also not a
replacement step. We still want to show them during the diff because
their output can be important. Especially if the user just requested a
diff it doesn't make sense to hide the diff from them at the same time.

The intention here is to only hide replacement steps, so do that.

The full diff with the display tests is here:
https://gist.github.com/tgummerer/fcd012f13669a9cdc39530cde7770260 It's
unedited, so it includes some flakyness which isn't interesting.

I looked it over, and I think it looks like what we want, but I'm
curious to hear what others think. E.g.
https://gist.github.com/tgummerer/fcd012f13669a9cdc39530cde7770260#file-testdata-diff-L558
looks more correct now, as it shows the two delete operation that
actually happened, that it didn't show before, and it still shows the
same operation (Calling this one out in particular, since it took me a
bit to understand that we still have the same operation in the diff)

Fixes https://github.com/pulumi/pulumi/issues/7665
2024-04-26 14:54:21 +00:00
Germán Lena d7f24dfcfb
Refactor: move plugin kind to apitype ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR moves PluginKind to apitype to prevent circular dependencies
when adding apitype as a dependency of the workspace module.
It also re-exports PluginKind to keep backward compatibility

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-25 17:30:30 +00:00
Thomas Gummerer 4169755866
Implement up --continue-on-error ()
Similar to destroy --continue-on-error, this flag allows `pulumi up`
to continue if any errors are encountered.

Currently when we encounter an error while creating/updating a
resource, we cancel the context of the deployment executor, and thus
the deployment stops once the resources that are being processed in
parallel with the failed one finish being updated.

For --continue-on-error, we ignore these errors, and let the
deployment executor continue.  In order for the deployment executor to
exit eventually we also have to mark these steps as done, as the
deployment executor will otherwise just hang, and callers with open
channels waiting for it to finish/report back will hang indefinitely.

The errors in the step will still be reported back to the user by the
OnResourceStepPost callback.

Fixes https://github.com/pulumi/pulumi/issues/14515

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-04-22 11:12:45 +00:00
Fraser Waters 75340dd942
Revert "Run integration tests and dev builds with race detection" ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This reverts the sdk and pkg changes from commit
655b76d030.

It also disables race detection from all builds and tests.

Fixes https://github.com/pulumi/pulumi/issues/15991.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-19 16:57:16 +00:00
Thomas Gummerer 477a54b3de
deploytest/RegisterResource: return struct instead of values ()
This method already returns 4 different values, and we want to add more.
Refactor it so it returns a struct, to make adding additional return
values easier in the future.
2024-04-19 11:08:56 +00:00
Thomas Gummerer ae8134f5ad
upgrade to latest version of golangci-lint ()
The version we currently have doesn't support Go 1.22 properly, so it
throws a bunch of warnings locally when trying to run it with the latest
Go version installed. Just running the latest version locally also
doesn't quite work, since it throws a bunch of errors from the
perfsprint linter, which seems to have gotten stricter.

Upgrade to the latest version of golangci-lint, and fix all the errors
we're getting from it. Mostly done via `perfsprint -fix`, with some
manual changes that `perfsprint -fix` wouldn't touch.
2024-04-19 06:20:33 +00:00
Mikhail Shilkov 60adf1823d
Don't bail at preview when a protected resource needs replacement ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Don't bail at preview when a protected resource needs replacement, just
error. This makes sure that users can see the actual diff that causes
the replacement.

Fixes 

Before:

```
$ pulumi preview
Type                                     Name                  Plan     Info
     pulumi:pulumi:Stack                      azure-native-ts-dev2           
     └─ azure-native:resources:ResourceGroup  rg                             1 error

Diagnostics:
  azure-native:resources:ResourceGroup (rg):
    error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`


$ pulumi preview --diff
pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev2::azure-native-ts::pulumi:pulumi:Stack::azure-native-ts-dev2]
error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
Resources:
    1 unchanged
```

After:

```
$ pulumi preview
Type                                     Name                  Plan        Info
     pulumi:pulumi:Stack                      azure-native-ts-dev2              1 error; 2 warnings
 +-  └─ azure-native:resources:ResourceGroup  rg                    replace     [diff: ~location]; 1 error

Diagnostics:
  azure-native:resources:ResourceGroup (rg):
    error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`

  pulumi:pulumi:Stack (azure-native-ts-dev2):
    error: preview failed


$ pulumi preview --diff
pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev2::azure-native-ts::pulumi:pulumi:Stack::azure-native-ts-dev2]
error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
    +-azure-native:resources:ResourceGroup: (replace) 🔒
        [id=/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/rg5f8e30e4]
        [urn=urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg]
        [provider=urn:pulumi:dev2::azure-native-ts::pulumi:providers:azure-native::default_2_30_0::3c957b2a-4852-439c-b211-22a115bbe89a]
      ~ location: "westeurope" => "westus2"
error: preview failed
Resources:
    +-1 to replace
    1 unchanged
```

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-18 16:45:52 +00:00
Fraser Waters ce04d43792
Save component resources as parents ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15843.

This tracks parent information separate from Goal information (as that
was only generated and saved for custom, not component, resources).

We also need to keep track of the transform list between the first
RegisterResource starting a remote component construction and the remote
component _actually_ registering itself. This means we save the
transform list correctly once the component is registered inside the
remote process allowing other resource registrations in the remote
process to see that saved transform list.
If we just wait till the Construct call is done then none of the inner
children pick up the transform list.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-15 12:07:45 +00:00
Fraser Waters 8588aa365c
Lift context parameter to SerializeDeployment/Resource/Operations/Properties ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

SerializePropertyValue needed a `context.Context` object to pass to the
`config.Encrypter`. It was using `context.TODO()`, this change instead
accepts a context on the parameters and lifts that up to
SerializeProperties, SerializeResource, SerializeOperation, and
SerializeDeployment.

There were a few call sites for those methods that already had a context
on hand, and they now pass that context. The other calls sites now use
`context.TODO()`, we should continue to iterate in this area to ensure
everywhere that needs a context has one passed in.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-15 07:45:46 +00:00
Fraser Waters d28fa18eec
Test SnapshotManager and Journal in engine tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This tests the production snapshot manager against the journalling
snapshot system in engine tests. This should ensure that both systems
produce similar (we ignore exact resource ordering, as long as both are
valid) results.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-11 22:54:08 +00:00
Fraser Waters 959fc6c8b4
Run integration tests with race detection ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Combining  and  and including fixes to allow tests to pass.

Enable race detection in the binary we're using for integration tests.
This will allow us to catch more data races before they get into any
release. This does mean the binary we're using for integration tests is
slightly different from the binary we're releasing, however that's
already the case as we're running a binary with coverage enabled for
them. Later we rebuild the binary we're actually releasing.

This requires us to fix the race between snapshot code, display code,
and the step executor. I've done that by adding a lock to the State
struct. This does not feel great, but it's a quick way to fix this and
get race detection running (and unblocks  which was also hitting
the race detector because it started pulling snapshot code into unit
tests as well). There's probably a more principled overhaul that doesn't
require locking at this level.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-11 15:58:42 +00:00
Thomas Gummerer eff61838ac
Make sure TestCall test doesn't flake ()
The ordering of this array isn't guaranteed, so use ElementsMatch
instead of Equal to compare to avoid the flakyness.

I noticed this test being flaky while working on bazel, but also noticed
it in CI logs, here:
https://github.com/pulumi/pulumi/actions/runs/8646652290/job/23706586759?pr=15906
2024-04-11 13:47:29 +00:00
Fraser Waters 9cd05b41b1
Add array conformance test ()
Add an array output test to the conformance suite. Similar to the other
basic output tests.
2024-04-10 16:00:24 +00:00
Fraser Waters 616e911c0f
Use a generic wrapper around `sync.Map` ()
This removes the type casts necessary when using `sync.Map` and ensures
correctness without needing to check other use sites.
2024-04-09 10:56:25 +00:00
Fraser Waters c0f69a1040
Use the local artifacts, not released artifacts in conformance tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This threads the "local_dependencies" property through to
GeneratePackage, following exactly the same semantics as for
"GenerateProgram".

Fixes https://github.com/pulumi/pulumi/issues/15074.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-26 13:10:34 +00:00
Justin Van Patten afd2561fd2
Fix: Don't delete stack outputs on failed deployments ()
If an update fails, update any stack outputs that were updated, but
otherwise leave existing stack outputs as-is. In other words, don't
delete stack outputs if the stack didn't successfully run to completion.

Implementation note: when we receive a RegisterResourceOutputsEvent for
the stack resource, we defer processing it until the end of the
deployment, to know whether the deployment ran to completion without
errors.

Fixes 

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-25 22:37:46 +00:00
Fraser Waters cdd14d21f6
Check for protect in replace chains ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15763.

When doing a replace chain we need to check if the resources implicated
for deletion in the chain are actually ok to delete (i.e. they don't
have `protect` set). This change fixes the step generator to check that
when building the delete chain, and adds a test for it as well.

Sadly because of the nature of the system you have to `state unprotect`
these resources if you actually want them replaced, you can't just
change `protect` in the resource options in the program. This is because
at the point we're processing the "to be replaced" resource we've only
got the old state for the implicated resource because we won't have got
it's new goal state yet because it depends on the resource we're
currently processing.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 20:07:36 +00:00
Fraser Waters c27a2837ba
Always use the snapshot secret manager ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

There were a number of places where we passed a `Snapshot` and a
`secret.Manager` as arguments to a method, where if the `Manger` was nil
we'd fall back to the `Snapshot.SecretManager` (which could also be
nil).

Turns out in all but one place this was always passed as nil or just as
directly the snapshot's `SecretManager` field.
The one place it differed was in
`pkg/cmd/pulumi/stack_change_secrets_provider.go` where we're changing
the secret manager, but it's fine to just set the snapshot's
`SecretManager` field to the new manager.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 10:30:14 +00:00
Fraser Waters a759993581
Add `context.Context` to `cachingCrypter.encryptSecret` ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Was looking at secret code and saw this was an easy fix up of a
`context.TODO()` call.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 10:27:47 +00:00
Fraser Waters 3043adeeb3
Add SupportPack to schemas to write out in the new style ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->


This adds a new flag to the schema metadata to tell codegen to use the
new proposed style of SDKs where we fill in versions and write go.mods
etc.

I've reworked pack to operate on packages assuming they're in this new
style. That is pack no longer has the responsibility to fill in any
version information.

This updates python and node codegen to write out SDKs in this new
style, and fixes their core libraries to still be buildable via pack.
There are two approaches to fixing those, I've chosen option 1 below but
could pretty easily rework for option 2.

1) Write the version information directly to the SDKs at the same time
as we edit the .version file. To simplify this I've added a new
'set-version.py' script that takes a version string an writes it to all
the relevant places (.version, package.json, etc).

2) Write "pack" in the language host to search up the directory tree for
the ".version" file and then fill in the version information as we we're
doing before with envvar tricks and copying and editing package.json.

I think 1 is simpler long term, but does force some amount of cleanup in
unrelated bits of the system right now (release makefiles need a small
edit). 2 is much more localised but keeps this complexity that
sdk/nodejs sdk/python aren't actually valid source modules.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-22 09:25:46 +00:00
Thomas Gummerer 8099d314a7
destroy: implement --continue-on-error ()
Implement a `--continue-on-error` flag for `pulumi destroy`. This makes
sure we continue processing destroys even if errors occur. We will only
continue for resources which do not depend on the failed resource, to
make sure we always have a valid snapshot available, and to not leave
any orphaned resources behind.

Resources that fail to destroy will still continue to be managed by
pulumi, and the process ends in an error to indicate to the user that
there were problems and the destroy didn't succeed cleanly.

The output will continue to be the same as for failed destroys, except
we now may succeed in destroying more resources than before.

/cc https://github.com/pulumi/pulumi/issues/3304
2024-03-22 09:22:40 +00:00
Thomas Gummerer bc70928f67
Remove trustDependencies option ()
This option is always true these days, and we don't expect to set it
false for anything. Remove the flag for a bit of code cleanup.
2024-03-21 10:14:07 +00:00
Fraser Waters f749c2cb24
Send output values to transforms for dependency tracking ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This engine change is necessary for correct dependency tracking of
properties through transform functions.

Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.

This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.

After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.

There are a couple of things to keep in mind with this change. 

Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.

Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
Fraser Waters 08aad5997c
Fix panic when changing untargetted provider versions ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15704.

When doing a targeted run the source evaluator isn't aware of targets
but it is responsible for registering default providers. As such on
getting a resource event with a new provider version (e.g 1.0 -> 2.0) it
will send of a registration for the new version it's seen, which as a
default provider the step generator will accept and add to state (this
is probably ok).
However when the step generator runs for the resource using this
provider it will see it's not targetted and ignore its new goal state
just reusing its old state. This old state will be referring to the old
version of the provider (e.g "default_aws_1_0_0" rather than
"default_aws_2_0_0"), which was causing a panic in the step generator
when trying to build the overall stack state for StackAnalyze as the old
provider had never been registered.

We now catch this situation when generating the same step for a
non-targeted resource and error out that this isn't supported.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:44:49 +00:00
Fraser Waters 84b2dd76a1
Use sets for dependencies inside the engine ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

We're doing more and more manipulation of these URN slices. We should
just make these sets to make it simpler to manipulate them without
worrying about order or duplicates.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-15 13:01:45 +00:00
Fraser Waters a02975203b
Ensure PropertyDependencies are filled in ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Similar to https://github.com/pulumi/pulumi/pull/15450 but for
RegisterResource.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-15 09:05:18 +00:00
Justin Van Patten 7ebcc42455
Downgrade remediate policy violations to mandatory ()
Policy violations should not have a remediate enforcement level. The
Policy SDK currently downgrades the level from remediate to mandatory
for resource policy violations, but isn't currently doing that for stack
policies. A change to the Policy SDK is in-progress to do that.

This change applies the same behavior to the engine. If a resource
policy still has a violation after running remediations and the level is
remediate, "downgrade" the level to mandatory. Similarly, if a stack
policy has a violation with a remediate level, downgrade it to
mandatory.

This avoids a panic when getting a policy violation from a stack policy
and the enforcement level is remediate.

Related: https://github.com/pulumi/pulumi-policy/pull/339

Fixes https://github.com/pulumi/pulumi-policy/issues/332
2024-03-07 15:10:36 +00:00
Thomas Gummerer 43322f276a
fix panic on deserializing deployment ()
When a serialized deployment doesn't include a secrets provider
configuration, but does include ciphertexts, currently we end up with a
panic. Error out earlier if this is the case to avoid the panic.

This fixes the panic seen in
https://github.com/pulumi/pulumi/issues/15547 and
https://github.com/pulumi/pulumi/issues/14761, but it doesn't quite
explain why this is happening in the first place. I asked for some more
info from the users in these issues for that.

Putting this up as PR anyway in case anyone has any idea of why this
could be happening in the first place. I've tried spelunking through the
code, but nothing obvious stood out. It is possible that these were
still v2 snapshots that included no secrets manager, but I would be
somewhat surprised if those still existed in the wild.

Another potential solution here would be to try to pass the secret
manager from the config in to the DeserializeDeployment function, so we
could use that in these cases. It might not always be correct though, so
I'm not sure it's the right thing to do.

Thoughts?

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-03-07 09:31:57 +00:00
Thomas Gummerer 898a682ef6
Make sure non-targeted resources are not updated ()
When the `--target` option is used, resources that already exist in the
snapshot, but aren't directly targeted should not be updated at all.
Internally in the engine, this is done by turning them into a
`SameStep`, meaning no updates will actually be preformed, and we will
make it look like the resource stayed the same.

However, we currently still write the "new" state of the resource (e.g.
updated dependencies, inputs, etc.) into the snapshot. This is mostly
fine as long as the new dependencies already exist. If a dependency on a
resource is that doesn't already exist is added however this breaks.
Since the resource that's being depended on doesn't exist in the
snapshot and isn't targeted, we won't create it. At the same time we're
adding a dependency on that virtually non-existing resource, which makes
the snapshot invalid.

Since we're in `--target` mode, we should do what we promised the user,
and only update the targeted resources, nothing else. Introduce a new
`NonTargetedSameStep` here, which does exactly that. It's essentially
the same as a `SameStep`, but we always use the *old* state instead of
the new one when writing it out. Since the resource is not targeted,
this leaves it in the same state as before.

Fixes 
Fixes 
2024-03-05 07:49:11 +00:00
Fraser Waters 2f01e691a4
Use slices.Reverse ()
We have generics and the slices package now, use it to save having to
think if a reversing for loop is correct or not.
2024-02-22 11:43:26 +00:00
Fraser Waters e42cfbb349
Ensure project plugins are absolute paths ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15467.

This tightens the restriction on paths passed to `NewProgramInfo`.
Previously it allowed relative paths like "./providers/my_provider".
That is now an error. This is correct behaviour. The fields of this
structure are passed via protobuf and the descriptions for them in the
proto spec are that they should always be absolute paths.

Where we build plugin paths we ensure that if they're relative we
resolve them to what they are relative to. That is generally _not_ the
current working directory so `filepath.Abs` doesn't do the right thing
here.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-22 11:43:18 +00:00
Fraser Waters 7422c44ca4
Engine support for remote transforms ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support to the engine for "remote transformations".

A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.

The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.

A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.

The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
Fraser Waters ea74b8aa26
Ensure ArgsDependencies are filled in ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Adds argsDependencies support to the deploytest resource monitor and add
some tests that the engine fills that map in with output values from
args if the SDK hasn't filled it in.

We also need to do this the other way (if the SDK fills in
argsDependencies but doesn't send output<T> values with those deps to
args).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 09:15:38 +00:00
Fraser Waters 556cd9b6cd
Allow component providers to return outputs to the engine ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new flag to provider Construct and Call requests to indicate
that the engine will accept output values in the response.

If a component provider sees that flag set they can return Output
PropertyValues and optionally skip filling in the dependency map in the
response.

The engine will use the combined information of the dependency map and
any dependencies found on any output values in the property to build the
final dependency map that is returned to programs.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-14 08:15:24 +00:00
Fraser Waters 65b8f5c841
Split CallRequest into ResourceCallRequest ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Similar to what we did to the `InvokeRequest` a while ago. We're
currently using the same protobuf structure for `Provider.Call` and
`ResourceMonitor.Call` despite different field sets being filled in for
each of them.

This splits the structure into `CallRequest` for providers and
`ResourceCallRequest` for the resource monitor. A number of fields in
each are removed and marked reserved with a comment explaining why.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-08 13:16:23 +00:00
Fraser Waters 6c52b24cc8
Add an engine test for Construct and Call dependency maps ()
The only real change here is a new test in
pkg/engine/lifecycletest/pulumi_test.go and the small fix up to the
`deploytest.ResourceMonitor` to return the information needed for that
test.
2024-02-08 13:01:47 +00:00
Fraser Waters bee784bc8a
Set KeepOutputValues in deploytest.ResourceMonitor ()
While working on some new tests I noticed that ResourceMonitor in the
tests didn't set KeepOutputValues for remote resources like it should.
This fixes that.
2024-02-07 20:07:02 +00:00
Fraser Waters 1138fdb63c
Upgrade output values in source_eval ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Stage 1 of updating how we deal with OutputValues in source_eval for
transforms.

This makes two changes that should result in basically a no-op for
standard resources, but update all Computed/Secret values to
OutputValues for Call and Construct.

This is a no-op for standard resources because previously we set
"KeepOutputs" to false when we unmarshalled them. That replaced all
OutputValues with Computed/Secret. Now we do "UpdgradeOutputs" at the
start of source_eval but then call "DowngradeOutputs" before passing the
properties on to the rest of the system. So this is a complete no-op.

But for Call/Construct we used to use "KeepOutputs" but now we do
"UpgradeOutputs". So values that previously got sent as Computed/Secret
will now get sent as OutputValue. This _should_ be fine, all users of
Construct/Call already had to handle OutputValue on their interface, so
this just means a few more cases of seeing those values.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-06 16:46:31 +00:00
Thomas Gummerer 6f41d715d9
State: fix panic when deleting non-unique Provider ()
# Description

In
30f59eb30a,
we made sure that we
don't delete multiple resources, when a user is prompted to choose
between multiple resources with the same URN.

If there are non-unique URNs, we currently assume that we can just go
ahead and delete it, as there's still another resource that will
fulfill the requirements.

However that is not true if the dependent is a provider.  Providers
are identified by a {URN, ID} tuple.  Therefore even though after the
deletion of a provider with a specific URN there's still another one
with the same URN left, this is not good enough to fulfill the
dependency requirements.

If we don't have a unique URN, check again to make sure there's no
provider dependency on the condemned resource.  If there is one, we
either error out, or delete the dependencies depending on the options
passed in.

Note that I haven't managed to reproduce this by just issuing pulumi
commands, but I did by editing the state in a way to reproduce the same
issue.

One unanswered question for me is how the deleted cluster in the
original issue ended up lingering in the state. While this PR fixes the
panic, and will make it easier for the user to actually delete the
cluster from the state (it's going to suggest `--target-dependents`
which will do the right thing), it doesn't address that bit.

Fixes https://github.com/pulumi/pulumi/issues/15166

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-02-05 16:21:38 +00:00
Fraser Waters 721d61115b
Add `plugin run` command ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Mostly for providers to experiment with, currently hidden behind
"PULUMI_DEV".
2024-02-05 08:35:48 +00:00
Fraser Waters 0113f27d6c
Don't mutate RegisterResourceRequest ()
Noticed this while trying to prepare a PR for the transforms support.
These tests were checking that we mutated a field on the request object
itself. We no longer do that in the transform branch because everything
has to go through the transform callbacks as well but it caused these
tests to fail.

I've deleted these tests and written a new engine test to validate the
system output is actually as expected.
2024-01-30 16:45:10 +00:00
Fraser Waters 4c1471ec06
Don't encrypt secrets to get ciphertext that isn't actually used ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

While investigating into https://github.com/pulumi/pulumi/issues/15293 I
noticed that `SerializePropertyValue` was always calling into
`Encrypter.EncryptValue` (or `cachingCrypter.encryptSecret`) even when
`showSecrets` was true and the cipher text wasn't being returned.

This is a simple move of that logic into the `!showSecrets` branch so it
gets skipped entirely when the cipher text isn't needed.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-30 09:51:51 +00:00
Fraser Waters dd5fef7091
Fix stack name validation check ()
The disable validation check for this was wrong (it did the assert if
validation was disabled).
2024-01-27 10:35:20 +00:00
Fraser Waters 7ca32437c6
Test that an empty parent URN in an alias is treated as the default parent ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Title sums it up. Some SDKs are sending `parent=""` rather than no
parent at all to specify to the engine to use the default parent.
Technically this isn't the intention of the protocol but given the SDKs
are out we now need to support it. I've added a test to check this works
and left a comment that it would be a nice tidy up to make in V4.

I've also changed the deploy tests to directly use the rpc protocol
rather than `resource.Alias` this lets us write tests like "send an
empty parent URN" which `resource.Alias` wasn't capable of expressing.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-26 18:29:22 +00:00
Zaid Ajaj bd4e50efdd
[conformance tests] Fix run root and use program info everywhere ()
# Description

This PR introduces `ProgramInfo` to replace the old `ProgInfo` and
consistently use it where we require plugin, install dependencies and
initialize language runtimes.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-25 23:28:58 +00:00
Ian Wahbe 9a06694676
Move assets and archives to their own package ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR is motivated by https://github.com/pulumi/pulumi/pull/15145.

`resource.*` should be built on top of `property.Value`,[^1] which means
that `resource`
needs to be able to import `property.Value`, and so `property` cannot
import
`resource`. Since Assets and Archives are both types of properties, they
must be moved out
of `resource`.

[^1]: For example:
a1d686227c/sdk/go/common/resource/resource_state.go (L35-L36)

## Open Question

This PR moves them to their own sub-folders in `resource`. Should
`asset` and `archive`
live somewhere more high level, like `sdk/go/property/{asset,archive}`?

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-25 20:39:31 +00:00
Fraser Waters 5e8945a23a
Fix NodeJS alias objects at the protocol level. ()
Precursor to moving transformations to run for remote components. We
need to re-send alias options to the transform function but we'll want
to do the NodeJS fixup before sending them onwards.
2024-01-23 17:44:53 +00:00
Thomas Gummerer baecc85eaf
turn on the golangci-lint exhaustive linter ()
Turn on the golangci-lint exhaustive linter.  This is the first step
towards catching more missing cases during development rather than
in tests, or in production.

This might be best reviewed commit-by-commit, as the first commit turns
on the linter with the `default-signifies-exhaustive: true` option set,
which requires a lot less changes in the current codebase.

I think it's probably worth doing the second commit as well, as that
will get us the real benefits, even though we end up with a little bit
more churn. However it means all the `switch` statements are covered,
which isn't the case after the first commit, since we do have a lot of
`default` statements that just call `assert.Fail`.
 
Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-17 16:50:41 +00:00