Commit Graph

1085 Commits

Author SHA1 Message Date
Thomas Gummerer 235f21c32c import: use newly introduced helper function
This helper function was introduced in
https://github.com/pulumi/pulumi/pull/16493.  Let's use it here as well.
2024-06-28 10:40:27 +02:00
Thomas Gummerer f4b775847a more fixes 2024-06-28 10:28:04 +02:00
Thomas Gummerer f5b291e6d4 fix test 2024-06-28 10:20:23 +02:00
Thomas Gummerer ac76df38f8 address code review comments 2024-06-28 10:20:23 +02:00
Thomas Gummerer 83501431a9 implement skeleton command for `pulumi state move`
Implement a skeleton command for `pulumi state move`.  This can so far
only more a single resource from one stack to another, including
copying the provider it needs, and leave valid state files behind.

These state files are not written to the backend yet, nor is there any
kind of UI.  Further PRs will build on top of this.

The command is intentionally commented out so this can be merged
independently.
2024-06-28 10:20:23 +02:00
Fraser Waters 4a12902bde
Keep package refs separate from providers ()
A few weeks ago we added the concept of package references, mostly for
parametrised providers so that we wouldn't have to send the parameter
value over and over for each `RegisterResourceRequest`. This was done by
sending the package reference through the `provider` field.

Nothing yet uses this feature (it hasn't yet been added to any language
SDKs), and thinking over this the last week I realised there was an
issue using this for extension packages.

When using an extension provider it's valid to also specify an explicit
provider, which will be using the `provider` field. At this point
there's no where for the SDK to send extra information to the engine
about the parameterisation.

This is a small reworking of this idea to use a separate `package`
field, so that even for an extension type we can specify the
parameterised package it comes from and an explicit provider for it.
2024-06-24 14:59:18 +00:00
Thomas Gummerer 8c86831bc9
resolve providers in the engine before running transforms ()
We're considering whether it makes sense to document setting "default
providers" through transforms. For that we want to make sure the
transforms behave exactly the way we expect before trying to document
anything, and going ahead with that plan.

Currently it is the SDKs responsibility to resolve the providers map and
to resolve the parent relationship and send the provider in a resolved
manner to the engine. For example in the NodeJS SDK this currently
happens here:
91568bc03b/sdk/nodejs/resource.ts (L445-L469)

To make sure this works in the same way in every SDK, and that we can
add an engine test to make sure this works consistently, we add the same
resolution of the provider before we pass these options to the
transform. We'll keep the resolution in the SDK, which will be preferred
(we'll choose the Provider resource option first in any case), but now
also have a fallback to the same extend in the engine.

Also add a test to show the behaviour is correct to the engine.
2024-06-21 08:55:17 +00:00
Julien P a59b694515
Query language runtime for options during “pulumi new” ()
# Description

Fixes https://github.com/pulumi/pulumi/issues/16309

During `pulumi new` we query the language runtime using the new
`RuntimeOptionsPrompts` RPC call to get additional prompts to ask the
user.

<img width="900" alt="Screenshot 2024-06-07 at 14 28 58"
src="https://github.com/pulumi/pulumi/assets/387068/e68ef702-978b-47f7-9d4b-afdf10409ed8">

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"master","parentHead":"","trunk":"master"}
```
-->

---------

Co-authored-by: Will Jones <will@sacharissa.co.uk>
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-17 17:10:55 +00:00
Will Jones 9150a2f275
Remove `Step.Apply`'s `preview` parameter ()
With the clean up of deployment options (), it is no longer
necessary for `Step.Apply` methods to accept a `preview bool` parameter.
This information is already present on the receiver's `deployment`
property and removing the argument a. allows us to consolidate the
surface area of the API, hopefully preventing bugs and b. further
improve consistency around the use of "preview" (referring to the
`preview` action) and "dry run" (referring to any part of an action that
does not perform modifications, such as showing what will happen during
a `refresh` or `destroy`).
2024-06-14 10:19:13 +00:00
Ramon Quitales ba76f29997
Fix --continue-on-error running indefinitely when a resource fails to be created or updated ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR ensures that we only run the `StepCompleteFunc` if the step is
successful. If not, we should use the step's `Fail` method to register
the result. Previously, when a user runs a pulumi up operation with
`--continue-on-error`, the engine would register the result twice, once
in
ee6ec150d8/pkg/resource/deploy/step_executor.go (L342)
and if a failure occurs, it is re-registered in
ee6ec150d8/pkg/resource/deploy/step_executor.go (L373)
I suspect that the double registration is the cause of the user facing
error. Since this would only occur for create and update steps, it also
explains why the error doesn't occur during deletions.

Testing:

- Manually ran the repro identified in  and
https://github.com/pulumi/pulumi-command/issues/435#issuecomment-2114250614
against a locally built CLI with this fix.
- Extended existing lifecycle tests to account for the different code
branches of `CreateStep.Apply()` and `UpdateStep.Apply()`

Fixes 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-13 15:42:57 +00:00
Luke Hoban f1e4b4ff94
Change `pulumi refresh` to report diff relative to desired state instead of relative to only output changes ()
Presently, the behaviour of diffing during refresh steps is incomplete,
returning only an "output diff" that presents the changes in outputs.
This commit changes refresh steps so that:

* they compute a diff similar to the one that would be computed if a
`preview` were run immediately after the refresh, which is more
typically what users expect and want; and
* `IgnoreChanges` resource options are respected when performing the new
desired-state diffs, so that property additions or changes reported by a
refresh can be ignored.

In particular, `IgnoreChanges` can now be used to acknowledge that part
or all of a resource may change in the provider, but the user is OK with
this and doesn't want to be notified about it during a refresh.
Importantly, this means that the diff won't be reported, but also that
the changes won't be applied to state.

The implementation covers the following:

* A diff is computed using the inputs from the program and then
inverting the result, since in the case of a refresh the diff is being
driven by the provider side and not the program. This doesn't change
what is stored back into the state, but it does produce a diff that is
more aligned with the "true changes to the desired state".
* `IgnoreChanges` resource options are now stored in state, so that this
information can be used in refresh operations that do not have access
to/run the program.
* In the context of a refresh operation, `IgnoreChanges` applies to
*both* input and output properties. This differs from the behaviour of a
normal update operation, where `IgnoreChanges` only considers input
properties.
* The special `"*"` value for `IgnoreChanges` can be used to ignore all
properties. It _also_ ignores the case where the resource cannot be
found in the provider, and instead keeps the resource intact in state
with its existing input and output properties.

Because the program is not run for refresh operations, `IgnoreChanges`
options must be applied separately before a refresh takes place. This
can be accomplished using e.g. a `pulumi up` that applies the options
prior to a refresh. We should investigate perhaps providing a `pulumi
state set ...`-like CLI to make these sorts of changes directly to a
state.

For use cases relying on the legacy refresh diff provider, the
`PULUMI_USE_LEGACY_REFRESH_DIFF` environment variable can be set, which
will disable desired-state diff computation. We only need to perform
checks in `RefreshStep.{ResultOp,Apply}`, since downstream code will
work correctly based on the presence or absence of a `DetailedDiff` in
the step.

### Notes

- https://github.com/pulumi/pulumi/issues/16144 affects some of these
cases - though its technically orthogonal
- https://github.com/pulumi/pulumi/issues/11279 is another technically
orthogonal issue that many providers (at least TFBridge ones) - do not
report back changes to input properties on Read when the input property
(or property path) was missing on the inputs. This is again technically
orthogonal - but leads to cases that appear "wrong" in terms of what is
stored back into the state still - though the same as before this
change.
- Azure Native doesn't seem to handle `ignoreChanges` passed to Diff, so
the ability to ignore changes on refresh doesn't currently work for
Azure Native.

### Fixes

* Fixes 
* Fixes 
* Fixes  
* Not quite , but likely replaces the need for that

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-06-12 16:17:05 +00:00
Will Jones f71c764a4c
Clean up deployment options ()
# Description

There are a number of parts of the deployment process that require
context about and configuration for the operation being executed. For
instance:

* Source evaluation -- evaluating programs in order to emit resource
registrations
* Step generation -- processing resource registrations in order to
generate steps (create this, update that, delete the other, etc.)
* Step execution -- executing steps in order to action a deployment.

Presently, these pieces all take some form of `Options` struct or pass
explicit arguments. This is problematic for a couple of reasons:

* It could be possible for different parts of the codebase to end up
operating in different contexts/with different configurations, whether
due to different values being passed explicitly or due to missed
copying/instantiation.
* Some parts need less context/configuration than others, but still
accept full `Options`, making it hard to discern what information is
actually necessary in any given part of the process.

This commit attempts to clean things up by moving deployment options
directly into the `Deployment` itself. Since step generation and
execution already refer to a `Deployment`, they get a consistent view of
the options for free. For source evaluation, we introduce an
`EvalSourceOptions` struct for configuring just the options necessary
there. At the top level, the engine configures a single set of options
to flow through the deployment steps later on.

As part of this work, a few other things have been changed:

* Preview/dry-run parameters have been incorporated into options. This
lets up lop off another argument and mitigate a bit of "boolean
blindness". We don't appear to flip this flag within a deployment
process (indeed, all options seem to be immutable) and so having it as a
separate flag doesn't seem to buy us anything.
* Several methods representing parts of the deployment process have lost
arguments in favour of state that is already being carried on (or can be
carried on) their receiver. For instance, `deployment.run` no longer
takes actions or preview configuration. While doing so means that a
`deployment` could be run multiple times with different actions/preview
arguments, we don't currently exploit this fact anywhere, so moving this
state to the point of construction both simplifies things considerably
and reduces the possibility for error (e.g. passing different values of
`preview` when instantiating a `deployment` and subsequently calling
`run`).
* Event handlers have been split out of the options object and attached
to `Deployment` separately. This means we can talk about options at a
higher level without having to `nil` out/worry about this field and
mutate it correctly later on.
* Options are no longer mutated during deployment. Presently there
appears to be only one case of this -- when handling `ContinueOnError`
in the presence of `IgnoreChanges` (e.g. when performing a refresh).
This case has been refactored so that the mutation is no longer
necessary.

# Notes

* This change is in preparation for , where we'd like to add an
environment variable to control behaviour and having a single unified
`Options` struct would make it easier to pass this configuration down
with introducing (more) global state into deployments. Indeed, this
change should make it easier to factor global state into `Options` so
that it can be controlled and tested more easily/is less susceptible to
bugs, race conditions, etc.
* I've tweaked/extended some comments while I'm here and have learned
things the hard way (e.g. `Refresh` vs `isRefresh`). Feedback welcome on
this if we'd rather not conflate.
* This change does mean that if in future we wanted e.g. to be able to
run a `Deployment` in multiple different ways with multiple sets of
actions, we'd have to refactor. Pushing state to the point of object
construction reduces the flexibility of the code. However, since we are
not presently using that flexibility (nor is there an obvious [to my
mind] use case in the near future), this seems like a good trade-off to
guard against bugs/make it simpler to move that state around.
* I've left some other review comments in the code around
questions/changes that might be a bad idea; happy to receive feedback on
it all though!
2024-06-11 13:37:57 +00:00
Fraser Waters 5a83cfe6e4
Initial work for parameterized providers ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the first part of paramaterized providers in the engine. This
only supports "replacement" packages, that is where we fully replace a
providers package with a new package (think how dynamic tfbridge will
work, vs how crd2pulumi will work).

I've made the decision to _not_ support using parameterised providers by
sending the parameter in the RegisterResource request. This will
necessitate some different work in how we send the parameter for
explicit providers compared to version and pluginDownloadURL, but I
think it's worth it going forward.

No changelog as this is still basically unusable without codegen support
done, and should still be considered primarily for internal experimental
use for now.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-10 17:28:47 +00:00
Ian Wahbe 78c48204e0
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) ()
Normalize methods on plugin.Provider to the form:

```go
Method(context.Context, MethodRequest) (MethodResponse, error)
```

This provides a more consistent and forwards compatible interface for
each of our methods.

---

I'm motivated to work on this because the bridge maintains a copy of
this interface: `ProviderWithContext`. This doubles the pain of dealing
with any breaking change and this PR would allow me to remove the extra
interface. I'm willing to fix consumers of `plugin.Provider` in
`pulumi/pulumi`, but I wanted to make sure that we would be willing to
merge this PR if I get it green.

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes # (issue)

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-07 19:47:49 +00:00
Julien P 8476c3f7f5
Pass ProgramInfo through to LanguageRuntime.About ()
# Description

To correctly determine which python executable we are using, we need the
ProgramInfo so we can determine which virtual environment is in use.

This PR updates the `About` rpc call to take `ProgramInfo` as argument.

Fixes https://github.com/pulumi/pulumi/issues/16299
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-06 08:21:46 +00:00
Ian Wahbe eaeaa3fab7
Plugin interface cleanup ()
This PR is best reviewed commit by commit:
- bbe1fc6b2f exchanges `ParseTolerant` for
`Parse` in `Parameterize`.
- 76df18be56 updates the `Parameterize`
interface for forward compatibility.
- 746c057668 requires implementors of
`plugin.Provider` to make a forward compatibility choice explicitly.
This is similar to what we require for gRPC already. Since this will
release with bbe1fc6b2f, consumers will
already need to update their `plugin.Provider` implementor, minimizing
the disturbance.
2024-05-31 00:28:48 +00:00
Thomas Gummerer 20acf0d344
fix panic with --continue-on-error on delete after failed create ()
For deleting resources when `--continue-on-error` is given, we look up a
failed resources transitive dependencies in the dependency graph. This
is used to skip those resources from being destroyed to keep the
dependency chains correctly intact.

The errors for looking these dependencies up are recorded in the step
executor, whenever a resource execution fails. This includes errors from
resources that are supposed to be created.

However when a resource fails to create that resource does not exist in
the dependency graph at all, so we cannot look it up, and we get a panic
because we try to access a nil pointer.

When deleting resources we do not need to care about resources that
failed earlier, so we can just clear the errored steps before going
ahead with the deletes.

Fixes https://github.com/pulumi/pulumi/issues/16258

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-30 17:18:34 +00:00
Ian Wahbe e220f574d5
Abstract plugin.Parameterize away from the gRPC interface ()
`plugin.Provider` exists to provide a low-level abstraction *on top of*
the gRPC interface. It should not expose the gRPC types directly. The
key diff in this commit is:

```patch
-	Parameterize(
-		ctx context.Context, req *pulumirpc.ParameterizeRequest,
-	) (*pulumirpc.ParameterizeResponse, error)
+	Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error)
```

```patch
+type ParameterizeParameters interface {
+	isParameterizeParameters()
+}
+
+type (
+	ParameterizeArgs struct {
+		Args []string
+	}
+
+	ParameterizeValue struct {
+		Name    string
+		Version *semver.Version
+		// Value must be one of:
+		// - nil
+		// - bool
+		// - int, int32, int64
+		// - uint, uint32, uint64
+		// - float32, float64
+		// - string
+		// - []byte
+		// - map[string]interface{}
+		// - []interface{}
+		Value any
+	}
+)
+
+func (ParameterizeArgs) isParameterizeParameters()  {}
+func (ParameterizeValue) isParameterizeParameters() {}
```

This is the new interface exposed in `plugin`. The rest of the PR is
simply complying with the new interface.

While this change is technically breaking (since it was released in
v3.116.0), its a heavily experimental feature and the providers team (my
team) is the biggest consumer of `plugin`. Before this PR,
`Parameterize` was the *only* method that dirrectly exposed the gRPC
interface it was "abstracting".

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-30 03:17:49 +00:00
Will Jones 6b4805672c
Propagate deleted dependencies of untargeted resources ()
When using `--target` to target specific resources during an update, we
use the list of targets to decide which steps to generate given a set of
resource registrations. Specifically:

* If the registration event names a resource that is targeted, we
process it as usual.
* If the registration event names a resource that _is not_ targeted, we
emit a `SameStep` for it.

In the latter case, the emission of a `SameStep` means that the old
state for the resource will be copied across to the new state. This is
the desired behaviour -- the resource was not targeted and so the new
state should contain the resource exactly as it was prior to the update.
However, this presents a problem if the old state has references to
resources that either will not appear in the new state, or will appear
in the wrong place. Consider the following program in TypeScript-esque
pseudocode:

```typescript
const a = new Resource("a")
const b = new Resource("b", { dependency: a })
const c = new Resource("c")
```

Here, `b` depends on `a`, while `a` and `c` have no dependencies. We run
this program without specifying targets and obtain a state containing
`a`, `b` and `c`, with `a` appearing before `b` due to `b`'s dependency
on `a`. We now modify the program as follows:

```typescript
const b = new Resource("b")
const c = new Resource("c")
```

`a` has been removed from the program and consequently `b` no longer
depends on it. We once more run the program, this time with a `--target`
of `c`. That is to say, neither `a` nor `b` is targeted. The execution
proceeds as follows:

* `a` is not in the program, so no `RegisterResourceEvent` will be
emitted and processed for it.
* `b` is in the program, but it is not targeted. Its
`RegisterResourceEvent` will be turned into a `SameStep` and `b`'s _old
state will be copied as-is to the new state_.
* `c` is in the program and is targeted. It will be processed as normal.

At the end of execution when we come to write the snapshot, we take the
following actions:

* We first write the processed resources: `b`'s old state and `c`'s new
state.
* We then copy over any unprocessed resources from the base (previous)
snapshot. This includes `a` (which is again desirable since its deletion
should not be processed due to it not being targeted).

Our snapshot is now not topologically sorted and thus invalid: `b` has a
dependency on `a`, but `a` appears after it. Presently this bug will
manifest irrespective of the nature of the dependency: `.Dependencies`,
`.PropertyDependencies` and `.DeletedWith` are all affected.

This commit fixes this issue by traversing all untargeted resource
dependency relationships and ensuring that `SameStep`s (or better if
they have been targeted) are emitted before emitting the depending
resource's `SameStep`.

* Fixes  
* Fixes 
2024-05-23 12:31:03 +00:00
Fraser Waters 9eb49a2818
RegisterProvider engine work ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support for a `RegisterProvider` method to the engine. This
allows an SDK process to send the information for a package (name,
version, url, etc, and parameter in the future) and get back a UUID for
that run of the engine that can be used to re-lookup that information.

That allows the SDK to just send the `provider` field in
`RegisterResourceRequest` instead of filling in `version`,
`pluginDownloadURL` etc (and importantly not having to fill in
`parameter` for parameterised providers, which could be a large amount
of data).

This doesn't update any of the SDKs to yet use this method. We can do
that piecemeal, but it will require core sdk and codegen changes for
each language.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-23 06:16:59 +00:00
Fraser Waters 5de4c2aa5f
Lock state access in `builtinProvider.getResource` ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Found by data race checker.
2024-05-18 17:43:22 +00:00
Will Jones 6dfd57d554
Check property dependencies and `DeletedWith` for target dependents ()
The `--target-dependents` flag instructs Pulumi to include dependents of
specified targets. Presently we check dependencies, parents and
providers, but not property dependencies and `DeletedWith`
relationships. This commit fixes this so that these relationships are
correctly checked.
2024-05-17 15:07:13 +00:00
Will Jones 41b967d451
Rewrite `DeletedWith` properties when renaming stacks ()
When renaming a stack, we need to rewrite URN references to use the new
stack name instead of the old one. Presently such references can occur
as:

* `Parent` references
* `Provider` references
* `Dependencies`
* `PropertyDependencies`
* `DeletedWith` references

The current renaming implementation fails to rewrite `DeletedWith`
references. Previously this went unnoticed since such references were
also omitted when checking snapshot integrity. Now that integrity checks
correctly inspect `DeletedWith` properties, the omission during rename
has been surfaced. This commit fixes renaming to handle `DeletedWith`
correctly. A follow-up should also ensure that renaming checks integrity
itself prior to operation completion.

Fixes 
2024-05-16 15:09:32 +00:00
Fraser Waters ab7acdb602
Add Paramaterize to provider interface ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the bare bones changes required to update the provider interface
for parametrization. Nothing in the engine, sdks, cli, or codegen makes
use of these new methods yet.

But providers team can start building on top of this interface. Note
that this is subject to change, while this is _likely_ the right design
for parametrised providers there is a chance that we need to edit this
interface before GA release.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-15 16:22:39 +00:00
Thomas Gummerer 57541d178a
correct spelling of diagnostics () 2024-05-15 07:32:46 +00:00
Thomas Gummerer dacabae85d
fix a panic when refresh is used with --continue-on-error ()
Refreshes have their own way of dealing with errors, which does not
match what we want to do with ContinueOnError (it's further reaching, we
still want to refresh dependents of failed resources). Therefore we made
them mutually exclusive.

The original change however missed that it's also possible for refreshes
to happen during `pulumi up` using the `--refresh` flag. This currently
results in an assert/panic.

Fix this by not passing the `ContinueOnError` option through when we
have a refresh. This still gives us the right behaviour since refresh
ignores the errors anyway.

Fixes https://github.com/pulumi/pulumi/issues/16177

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-13 11:45:00 +00:00
Fraser Waters dcf0a1024b
Revert "Revert "Run integration tests and dev builds with race detection" ()" ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This reverts commit 75340dd942.

Fixes https://github.com/pulumi/pulumi/issues/16018.

This re-enables the locking and race detection. The locking is more
finely scoped to not be held over provider methods like Read/Update.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-09 16:15:41 +00:00
Paul C. Roberts 95f06deccd
Revert "The `--expect-no-changes` flag checks for output diffs" ()
Reverts pulumi/pulumi#15903

This is implicated in Issue  so reverting to allow time to provide
a proper solution

Fixes 
2024-05-06 17:34:24 +00:00
Eron Wright 89d3467b0f
Canonicalize provider version during default provider lookup ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR addresses a problem that leads to two default providers for a
given package+version. The problem occurs when given a combination of
unversioned and versioned provider requests, as can occur in
multi-language programs.

The fix is as follows: given an unversioned provider request, apply the
default version before looking into the provider cache.

Fixes  

## Example
Given the example program from  plus this fix, the stack state has
one provider as expected:

```jsonl
{
    "urn": "urn:pulumi:dev::issue-1939-nodejs::pulumi:providers:kubernetes::default_4_10_0",
    "custom": true,
    "id": "c19b65b0-dd5a-432f-8c75-d7544de782ec",
    "type": "pulumi:providers:kubernetes",
    "inputs": {
        "version": "4.10.0"
    },
    "outputs": {
        "version": "4.10.0"
    },
    "created": "2024-05-02T21:22:09.72166Z",
    "modified": "2024-05-02T21:22:09.72166Z"
}
{
    "urn": "urn:pulumi:dev::issue-1939-nodejs::kubernetes:yaml/v2:ConfigGroup$kubernetes:core/v1:ConfigMap::example:eron/example",
    "custom": true,
    "id": "eron/example",
    "type": "kubernetes:core/v1:ConfigMap",
    "parent": "urn:pulumi:dev::issue-1939-nodejs::kubernetes:yaml/v2:ConfigGroup::example",
    "provider": "urn:pulumi:dev::issue-1939-nodejs::pulumi:providers:kubernetes::default_4_10_0::c19b65b0-dd5a-432f-8c75-d7544de782ec",
    "propertyDependencies": {
        "apiVersion": [],
        "kind": [],
        "metadata": []
    },
}
```

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-06 08:22:28 +00:00
Will Jones bad2d9f6c2
Better handle property dependencies and `deletedWith` ()
A resource `A` depends on a resource `B` if:

1. `B` is `A`'s `Provider`.
2. `B` is `A`'s `Parent`.
3. `B` appears in `A`'s `Dependencies`.
4. `B` appears in one or more of `A`'s `PropertyDependencies`.
5. `B` is referenced by `A`'s `DeletedWith` field.

While cases 1, 2, and 3 (providers, parents, and dependencies) are
handled fairly consistently, there have been a number of cases where the
newer features of `PropertyDependencies` (case 4) and `DeletedWith`
(case 5) have been neglected. This commit addresses some of these
omissions. Specifically:

* When refreshing state, it's important that we remove URNs that point
to resources that we've identified as deleted. Presently we check
pointers to parents and dependencies, but not property dependencies or
`deletedWith`. This commit fixes these gaps where dangling URNs could
lurk.

* Linked to the above, this commit extends snapshot integrity checks to
include property dependencies and `deletedWith`. Some tests that
previously used now invalid states have to be repaired or removed as a
  result of this.

* Fixing snapshot integrity checking reveals that dependency graph
checks also fail to consider property dependencies and `deletedWith`.
This probably hasn't bitten us since property dependencies are currently
rolled up into dependencies (for temporary backwards compatibility) and
`deletedWith` is typically an optimisation (moreover, one that only
matters during deletes), so operations being parallelised due a
perceived lack of dependency still succeed. However, tests that
previously passed now fail as we can spot these races with our better
integrity checks. This commit thus fixes up dependency graph
construction and bulks out its test suite to cover the new cases.

These bugs were discovered as part of the investigation into ,
though they may not be directly responsible for it (though the issues
with dependency graphs are certainly a candidate).
2024-05-03 17:08:06 +00:00
Fraser Waters bb6b9c675a
Add asset/archive to conformance tests and fix engine working dir issues ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/16092.

Two main changes in this PR. Firstly we add a test for assets/archives
to the conformance tests, and had to fix up some small issues in the
test runner to enable this (copying test data files, correcting project
directory layout, handling multi-word package names).

The other change is to fix the engine to respect the programs working
directory when serialising/deserialising assets and archives. It does
this by passing the programs working directory as a MarshalOption (n.b
this option is only really needed in the engine, program SDK doesn't
really need it because it doesn't change directories).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-02 11:32:54 +00:00
Paul C. Roberts b72399bd28
The `--expect-no-changes` flag checks for output diffs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds a pseudo op `OpOutputChange` to the set of changes that are
recorded in `display.ResourceChanges` to count the number of output
changes, this is then included in the check used to evaluate the
`--expect-no-changes` flag

When resource outputs are registered, they are checked against their
previous value using existing functionality, the total count of changes
is then added

The internal capability is validated with an engine test, the cli is
validated using an integration test

This will break user workflows that depend on the previous behavior

Fixes 

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [X] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Paul Roberts <proberts@pulumi.com>
2024-05-01 18:30:49 +00:00
Thomas Gummerer 8219c922a2
display: only hide replacement steps in diff ()
When displaying diff events, we currently try to hide non-logical
replacement steps unless we specifically enable showing them. However we
currently do that for all non-logical operations, regardless whether
they are replacement steps or not.

In particular a RefreshStep is non-logical, but it's also not a
replacement step. We still want to show them during the diff because
their output can be important. Especially if the user just requested a
diff it doesn't make sense to hide the diff from them at the same time.

The intention here is to only hide replacement steps, so do that.

The full diff with the display tests is here:
https://gist.github.com/tgummerer/fcd012f13669a9cdc39530cde7770260 It's
unedited, so it includes some flakyness which isn't interesting.

I looked it over, and I think it looks like what we want, but I'm
curious to hear what others think. E.g.
https://gist.github.com/tgummerer/fcd012f13669a9cdc39530cde7770260#file-testdata-diff-L558
looks more correct now, as it shows the two delete operation that
actually happened, that it didn't show before, and it still shows the
same operation (Calling this one out in particular, since it took me a
bit to understand that we still have the same operation in the diff)

Fixes https://github.com/pulumi/pulumi/issues/7665
2024-04-26 14:54:21 +00:00
Germán Lena d7f24dfcfb
Refactor: move plugin kind to apitype ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR moves PluginKind to apitype to prevent circular dependencies
when adding apitype as a dependency of the workspace module.
It also re-exports PluginKind to keep backward compatibility

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-25 17:30:30 +00:00
Thomas Gummerer 4169755866
Implement up --continue-on-error ()
Similar to destroy --continue-on-error, this flag allows `pulumi up`
to continue if any errors are encountered.

Currently when we encounter an error while creating/updating a
resource, we cancel the context of the deployment executor, and thus
the deployment stops once the resources that are being processed in
parallel with the failed one finish being updated.

For --continue-on-error, we ignore these errors, and let the
deployment executor continue.  In order for the deployment executor to
exit eventually we also have to mark these steps as done, as the
deployment executor will otherwise just hang, and callers with open
channels waiting for it to finish/report back will hang indefinitely.

The errors in the step will still be reported back to the user by the
OnResourceStepPost callback.

Fixes https://github.com/pulumi/pulumi/issues/14515

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-04-22 11:12:45 +00:00
Fraser Waters 75340dd942
Revert "Run integration tests and dev builds with race detection" ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This reverts the sdk and pkg changes from commit
655b76d030.

It also disables race detection from all builds and tests.

Fixes https://github.com/pulumi/pulumi/issues/15991.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-19 16:57:16 +00:00
Thomas Gummerer 477a54b3de
deploytest/RegisterResource: return struct instead of values ()
This method already returns 4 different values, and we want to add more.
Refactor it so it returns a struct, to make adding additional return
values easier in the future.
2024-04-19 11:08:56 +00:00
Thomas Gummerer ae8134f5ad
upgrade to latest version of golangci-lint ()
The version we currently have doesn't support Go 1.22 properly, so it
throws a bunch of warnings locally when trying to run it with the latest
Go version installed. Just running the latest version locally also
doesn't quite work, since it throws a bunch of errors from the
perfsprint linter, which seems to have gotten stricter.

Upgrade to the latest version of golangci-lint, and fix all the errors
we're getting from it. Mostly done via `perfsprint -fix`, with some
manual changes that `perfsprint -fix` wouldn't touch.
2024-04-19 06:20:33 +00:00
Mikhail Shilkov 60adf1823d
Don't bail at preview when a protected resource needs replacement ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Don't bail at preview when a protected resource needs replacement, just
error. This makes sure that users can see the actual diff that causes
the replacement.

Fixes 

Before:

```
$ pulumi preview
Type                                     Name                  Plan     Info
     pulumi:pulumi:Stack                      azure-native-ts-dev2           
     └─ azure-native:resources:ResourceGroup  rg                             1 error

Diagnostics:
  azure-native:resources:ResourceGroup (rg):
    error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`


$ pulumi preview --diff
pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev2::azure-native-ts::pulumi:pulumi:Stack::azure-native-ts-dev2]
error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
Resources:
    1 unchanged
```

After:

```
$ pulumi preview
Type                                     Name                  Plan        Info
     pulumi:pulumi:Stack                      azure-native-ts-dev2              1 error; 2 warnings
 +-  └─ azure-native:resources:ResourceGroup  rg                    replace     [diff: ~location]; 1 error

Diagnostics:
  azure-native:resources:ResourceGroup (rg):
    error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
    as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`

  pulumi:pulumi:Stack (azure-native-ts-dev2):
    error: preview failed


$ pulumi preview --diff
pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev2::azure-native-ts::pulumi:pulumi:Stack::azure-native-ts-dev2]
error: unable to replace resource "urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg"
as it is currently marked for protection. To unprotect the resource, remove the `protect` flag from the resource in your Pulumi program and run `pulumi up`
    +-azure-native:resources:ResourceGroup: (replace) 🔒
        [id=/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/rg5f8e30e4]
        [urn=urn:pulumi:dev2::azure-native-ts::azure-native:resources:ResourceGroup::rg]
        [provider=urn:pulumi:dev2::azure-native-ts::pulumi:providers:azure-native::default_2_30_0::3c957b2a-4852-439c-b211-22a115bbe89a]
      ~ location: "westeurope" => "westus2"
error: preview failed
Resources:
    +-1 to replace
    1 unchanged
```

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-18 16:45:52 +00:00
Fraser Waters ce04d43792
Save component resources as parents ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15843.

This tracks parent information separate from Goal information (as that
was only generated and saved for custom, not component, resources).

We also need to keep track of the transform list between the first
RegisterResource starting a remote component construction and the remote
component _actually_ registering itself. This means we save the
transform list correctly once the component is registered inside the
remote process allowing other resource registrations in the remote
process to see that saved transform list.
If we just wait till the Construct call is done then none of the inner
children pick up the transform list.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-15 12:07:45 +00:00
Fraser Waters 8588aa365c
Lift context parameter to SerializeDeployment/Resource/Operations/Properties ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

SerializePropertyValue needed a `context.Context` object to pass to the
`config.Encrypter`. It was using `context.TODO()`, this change instead
accepts a context on the parameters and lifts that up to
SerializeProperties, SerializeResource, SerializeOperation, and
SerializeDeployment.

There were a few call sites for those methods that already had a context
on hand, and they now pass that context. The other calls sites now use
`context.TODO()`, we should continue to iterate in this area to ensure
everywhere that needs a context has one passed in.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-15 07:45:46 +00:00
Fraser Waters d28fa18eec
Test SnapshotManager and Journal in engine tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This tests the production snapshot manager against the journalling
snapshot system in engine tests. This should ensure that both systems
produce similar (we ignore exact resource ordering, as long as both are
valid) results.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-11 22:54:08 +00:00
Fraser Waters 959fc6c8b4
Run integration tests with race detection ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Combining  and  and including fixes to allow tests to pass.

Enable race detection in the binary we're using for integration tests.
This will allow us to catch more data races before they get into any
release. This does mean the binary we're using for integration tests is
slightly different from the binary we're releasing, however that's
already the case as we're running a binary with coverage enabled for
them. Later we rebuild the binary we're actually releasing.

This requires us to fix the race between snapshot code, display code,
and the step executor. I've done that by adding a lock to the State
struct. This does not feel great, but it's a quick way to fix this and
get race detection running (and unblocks  which was also hitting
the race detector because it started pulling snapshot code into unit
tests as well). There's probably a more principled overhaul that doesn't
require locking at this level.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-11 15:58:42 +00:00
Thomas Gummerer eff61838ac
Make sure TestCall test doesn't flake ()
The ordering of this array isn't guaranteed, so use ElementsMatch
instead of Equal to compare to avoid the flakyness.

I noticed this test being flaky while working on bazel, but also noticed
it in CI logs, here:
https://github.com/pulumi/pulumi/actions/runs/8646652290/job/23706586759?pr=15906
2024-04-11 13:47:29 +00:00
Fraser Waters 9cd05b41b1
Add array conformance test ()
Add an array output test to the conformance suite. Similar to the other
basic output tests.
2024-04-10 16:00:24 +00:00
Fraser Waters 616e911c0f
Use a generic wrapper around `sync.Map` ()
This removes the type casts necessary when using `sync.Map` and ensures
correctness without needing to check other use sites.
2024-04-09 10:56:25 +00:00
Fraser Waters c0f69a1040
Use the local artifacts, not released artifacts in conformance tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This threads the "local_dependencies" property through to
GeneratePackage, following exactly the same semantics as for
"GenerateProgram".

Fixes https://github.com/pulumi/pulumi/issues/15074.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-26 13:10:34 +00:00
Justin Van Patten afd2561fd2
Fix: Don't delete stack outputs on failed deployments ()
If an update fails, update any stack outputs that were updated, but
otherwise leave existing stack outputs as-is. In other words, don't
delete stack outputs if the stack didn't successfully run to completion.

Implementation note: when we receive a RegisterResourceOutputsEvent for
the stack resource, we defer processing it until the end of the
deployment, to know whether the deployment ran to completion without
errors.

Fixes 

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-25 22:37:46 +00:00
Fraser Waters cdd14d21f6
Check for protect in replace chains ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15763.

When doing a replace chain we need to check if the resources implicated
for deletion in the chain are actually ok to delete (i.e. they don't
have `protect` set). This change fixes the step generator to check that
when building the delete chain, and adds a test for it as well.

Sadly because of the nature of the system you have to `state unprotect`
these resources if you actually want them replaced, you can't just
change `protect` in the resource options in the program. This is because
at the point we're processing the "to be replaced" resource we've only
got the old state for the implicated resource because we won't have got
it's new goal state yet because it depends on the resource we're
currently processing.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 20:07:36 +00:00
Fraser Waters c27a2837ba
Always use the snapshot secret manager ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

There were a number of places where we passed a `Snapshot` and a
`secret.Manager` as arguments to a method, where if the `Manger` was nil
we'd fall back to the `Snapshot.SecretManager` (which could also be
nil).

Turns out in all but one place this was always passed as nil or just as
directly the snapshot's `SecretManager` field.
The one place it differed was in
`pkg/cmd/pulumi/stack_change_secrets_provider.go` where we're changing
the secret manager, but it's fine to just set the snapshot's
`SecretManager` field to the new manager.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 10:30:14 +00:00