Commit Graph

167 Commits

Author SHA1 Message Date
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Fraser Waters c39756e6de
Tests and fix for --target-dependents with explicit providers ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13591.

This changes the logic for providers to always be targeted, this means
they can be skipped from --targets lists most of the time.

Because they don't need to be in the --targets list it makes the
behaviour of --target-dependents much more useful. If you want to update
a resource and it's children but it has an explicit provider you can
just --targets the resource.

If you want to use --target-dependents to target _all_ the resources
managed by an explicit provider that will work if the provider is in
--targets.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-18 13:10:22 +00:00
Fraser Waters 47ecd5a11d
Mark diff as an input diff when auto-diffing in the step generator ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/14040

When a provider returns `DiffUnknown` the step generator calculates a
simple diff based on the old and new inputs.

We were not correctly marking that this is an input diff, and so when
reconstructing objects from the detailed diff later in
`TranslateDetailedDiff` we we're looking at the old output properties
rather than the old input properties.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2023-10-18 10:33:04 +00:00
Joe Duffy 96a9a77167
Policy remediations feature ()
This PR implements the new policy transforms feature, which allows
policy packs to not only issue warnings and errors in response to policy
violations, but actually fix them by rewriting resource property state.
This can be used, for instance, to auto-tag resources, remove Internet
access on the fly, or apply encryption to storage, among other use
cases.
2023-10-09 18:31:17 +00:00
Fraser Waters fffc52ce08
Cleanup use of result.Result in step_generator.go ()
Continued result cleanup.
2023-10-06 15:57:27 +00:00
Fraser Waters e263e15363
Replace use of result.Result in step generator ()
More cleanup of result.Result, this time a pass through the step
generator.
2023-09-25 12:25:23 +00:00
Justin Van Patten b0048b35eb
[engine] Add extra comment on old resource lookup ()
Add a comment closer to where we actually lookup an old resource, noting
that we should check the URN first before aliases and why.

Follow-up from postmortem discussion on
https://github.com/pulumi/pulumi/issues/13848
2023-09-11 06:54:02 +00:00
Justin Van Patten 4c0b08d8ae
[engine] Check for old resources first by URN and then aliases ()
This change fixes a regression in the step generator when looking for
old resources. When generating steps for a register resource event, we
previously looked for old resources first by the resource's URN and then
by aliases.

This regressed with :

```diff
-	for _, urnOrAlias := range append([]resource.URN{urn}, goal.Aliases...) {
+	aliases[urn] = struct{}{}
+	for urnOrAlias := range aliases {
```

Previously, aliases were in a slice and we always looked for the URN
first, then aliases.

With , aliases changed to being stored in a map (a set). The URN
was added to the map before iterating over it, but there's no guarantee
it will be looked at first (iteration order for maps is unspecified),
and with the current behavior when there are aliases in the map, the URN
very likely won't come first.

This can lead to duplicate resources in the state (stack corruption)
when the wrong old resource is chosen.

The fix is to move back to always checking for old resources using the
URN first. We also move back to maintaining aliases in a slice for
consistent ordering.

Fixes 
2023-09-06 12:17:02 +00:00
Fraser Waters 7dbb5a68a2 Test and fix --target-dependents 2023-07-24 22:02:06 +01:00
bors[bot] 8e83ce445d
Merge
13480: Fix resoloution of aliased parent aliases. r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/13324.

When collapsing an alias that specifies a parent we need to use the parents alias if it had one. 

This is so that for example given a state with three resources:
```
urn:pulumi:test::test::prog:index:myStandardType::resA
urn:pulumi:test::test::prog:index:myType::resB
urn:pulumi:test::test::prog:index:myType$pkgAⓂ️typA::resC
```
Which is then updated such that resB and resC are re-parented to resA we end up with the following alias specifications:
```
resB: { NoParent = True}
resC: { Parent = "urn:pulumi:test::test::prog:index:myStandardType$prog:index:myType::resB
```
Note that resC's alias uses the _new_ URN for resB. If we do a simple collapse of that alias we'd end up trying to find an old URN of `urn:pulumi:test::test::prog:index:myStandardType$prog:index:myType$pkgAⓂ️typA::resC` but actually it's old URN would have been `urn:pulumi:test::test::prog:index:myType$pkgAⓂ️typA::resC`.

This change fixes alias to URN collapse to take into account if the parent was aliased so we get the correct old URN.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-07-17 16:28:39 +00:00
Fraser Waters 0b00a089c1 Fix resoloution of aliased parent aliases.
Fixes https://github.com/pulumi/pulumi/issues/13324.
2023-07-13 16:27:17 +01:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
bors[bot] db579129f4
Merge
13139: Send old inputs to diff and update r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/5973

Both Azure and Kubernetes set an extra field on created resource outputs (called "__inputs") that doesn't exist in the inputs to the corresponding Pulumi resources. Their Diff logic needs the subset of inputs that are managed by Pulumi to compare these values to live state in subsequent updates, while not showing diffs for fields that we don't manage directly. This change adds that data to the RPC protocol directly so these providers no longer need to manage these extra "__input" fields.

This adds two new properties to the RPC interfaces with providers. Firstly we pass a new bool flag to Configure to tell the provider if it will get sent old inputs as part of Diff. Secondly we now pass the old inputs and the old outputs to Diff and Update as opposed to just the old outputs.
    
The flag passed to Configure isn't strictly needed. Providers should be able to tell that the old inputs aren't being sent to Diff and Updae by virtue of them being `null/nil/None` rather than an empty map. For DiffConfig this is the only way to detect this case because it's called before Configure. But its still useful to send this explictly to Configure to allow providers to return a configure error that they need to be ran against a newer engine version.

There is one fairly major failure case that can happen with this feature and providers who update to make use of this feature should ensure they call this out in their changelog.
That is if a provider has been saving old inputs in outputs as a way to get access to them in diff, and then updates to use these new protocol properties instead and stops saving the inputs in outputs _and then_ the user downgrades back to an old provider version things will probably get _very_ confused.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-06-23 07:14:31 +00:00
Fraser Waters 841c99a9dd Send old inputs to diff and update
Fixes https://github.com/pulumi/pulumi/issues/5973

This adds two new properties to the RPC interfaces with providers.
Firstly we pass a new bool flag to Configure to tell the provider if it
will get sent old inputs as part of Diff.
Secondly we now pass the old inputs and the old outputs to Diff and
Update as opposed to just the old outputs.

The flag passed to Configure isn't strictly needed. Providers should be
able to tell that the old inputs aren't being sent to Diff and Updae by
virtue of them being `null/nil/None` rather than an empty map. For
DiffConfig this is the only way to detect this case because it's called
before Configure. But its still useful to send this explictly to
Configure to allow providers to return a configure error that they need
to be ran against a newer engine version.

There is one fairly major failure case that can happen with this feature
and providers who update to make use of this feature should ensure they
call this out in their changelog.
That is if a provider has been saving old inputs in outputs as a way to
get access to them in diff, and then updates to use these new protocol
properties instead and stops saving the inputs in outputs _and then_ the
user downgrades back to an old provider version things will probably get
_very_ confused.
2023-06-21 21:04:50 +01:00
Fraser Waters c881441890 Fix wildcards and indexers in IgnoreChanges
Fixes https://github.com/pulumi/pulumi/issues/12581

Also fixes ignoreChanges[idx] resting values to zero.

Before this change PropertiesPath.Delete would "delete" locations from
an array by just writing null to them but not changing the size of the
array.

This could result in a deployment that looked like it should have been a
same ending up as an update. For example given a resource `X` with a
property `foo` set to `[1]` if you changed the program to set `foo` to
`[1, 2]` and set `ignoreChanges` to `foo[1]` the deployment would do an
update with `foo` set to `[1, 0]`.

For PropertiesPath.Reset this now errors that the path is invalid, this
is similar to how we error on invalid paths if intermediate objects are
missing.
2023-06-20 09:45:02 +01:00
Justin Van Patten 64a2288883 [engine] Fix aliasing children
There is an issue with how the engine computes the aliases when the
resource is a child and doesn't have `Parent` set on the alias spec
(and the parent doesn't have any aliases).

```python
class FooResource(pulumi.ComponentResource):
    def __init__(self, name, opts=None):
        super().__init__("my:module:FooResource", name, None, opts)

class ComponentResource(pulumi.ComponentResource):
    def __init__(self, name, opts=None):
        super().__init__("my:module:ComponentResource", name, None, opts)
        FooResource("childrenamed", pulumi.ResourceOptions(
            parent=self,
            aliases=[pulumi.Alias(name="child")]
        ))
```

In the example above, `ComponentResource` has a child `FooResource`
which was renamed from `child` to `childrenamed`.

The engine does not compute the correct alias:

```
expected: urn:pulumi:stack::project::my:module:ComponentResource$my:module:FooResource::child
  actual: urn:pulumi:stack::project::my:module:FooResource::child
```

The problem is due to:

117955ce14/pkg/resource/deploy/step_generator.go (L370-L382)

... and:

117955ce14/sdk/go/common/resource/alias.go (L24-L26)

Because the alias spec doesn't have `parent` specified, the parent type
is not being included the computed alias URN.

Existing tests such as https://github.com/pulumi/pulumi/tree/master/tests/integration/aliases/python/rename_component_and_child
didn't catch the problem because the alias specifies both the `name` and
`parent`:

117955ce14/tests/integration/aliases/python/rename_component_and_child/step2/__main__.py (L15)

In this case, specifying `parent` on the alias shouldn't be necessary.
However, even after removing `parent` from the alias spec, the test
still succeeds because the parent itself has an alias:

117955ce14/tests/integration/aliases/python/rename_component_and_child/step2/__main__.py (L18)

... and parent aliases are inherited as part of a child's aliases, so we
still get an alias that works from the inheritance.

If we change the test to make no changes to the parent such that it
doesn't have any aliases, then we get the failure as we'd expect.

A similar problem will happen when retyping a child.

**Fix**

The fix involves using the child's parent in the calculated alias URN
when `Parent` isn't specified for the alias.

As part of this, we need to properly handled `NoParent` because right
now the engine is not correctly using it. The struct representing an
alias in the engine does not have a `NoParent` field:

117955ce14/sdk/go/common/resource/alias.go (L8-L15)

And therefore does not copy it over in the gRPC request:

117955ce14/pkg/resource/deploy/source_eval.go (L1082-L1088)

Instead, the `Alias` struct has an incorrect `NoParent` method which
returns `true` if the `Parent` field has a value of `""`:

117955ce14/sdk/go/common/resource/alias.go (L24-L26)
2023-06-14 05:19:17 -07:00
Justin Van Patten e77f9362aa Refactor: Split out `generateAliases` method
Split out the "generate aliases" functionality from `generateSteps` to
its own method, to make it possible to test just `generateAliases`.
2023-06-14 05:19:17 -07:00
Kyle Dixler 2b803b4ab8
Consolidated Target parameters
Consolidated `Target` parameters to a single variable. The deployment
executor is not well aware of the overall update that is going on and
runs individual resource operations. The consolidation minimizes
leakage.

Moved `--target` validation for `destroy` and `refresh` into `pkg/engine`.

Fixed  where `checkTargets()` would check the `prev` Snapshot for
resources after `rebuildBaseState()` was called. This would mutate prev
by removing resources from the snapshot. This caused deleted resources
on targeted updates not to be found and be reported as an error due to
having an unknown target.
2023-06-08 08:42:03 -07:00
Fraser Waters 72381ba976 Fix inconsistency in calling DiffConfig
Diff/DiffConfig is currently always called with the old outputs and new inputs,
except for this one case in step_generator where we called DiffConfig
with the old inputs instead of old outputs.

For providers inputs and outputs are always the same (Configure doesn't
allow providers to return any properties, so the engine enforces that
outputs=inputs) so using inputs or outputs here doesn't have any effect,
but less confusing to not have this inconsistency.
2023-06-01 11:45:35 +01:00
Kyle Dixler 23d9c20730
Fix --replace behavior regression
This fixes a regression where `--replace` was incorrectly considered a
targeted update. It removes a helper function indicating such, but the
incorrect logic is retained at the remaining usage site to avoid changes
in behavior.
2023-05-26 12:29:01 -07:00
bors[bot] c25b8483f9
Merge
12657: Don't load providers at startup r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This changes the provider registry to no longer load all the providers
from the old state on startup (in `NewRegistry`) instead the load logic
has been moved to the `Same` method. The step_executor and
step_generator have been fixed up to ensure that for cases where a
resource might not have had it's provider created yet (i.e. for DBR'ing
the old version of a resource, for refreshes or deletes) they ask the
`Deployment` to look up the provider in the old state and `Same` it in the
registry.

All of the above means we only load providers we're going to use (even
taking --targets into account).

One fix mot done in this change is to auto-update providers for deletes.
That is given a program state with two resources both using V1 of a
provider, if you run the program to update one of those resource to use
V2 of the provider but to delete the other resource currently we'll
still load V1 to do that delete. It _might_ be possible (although this
is definitly questionable) to see that another resource changed it's
provider from V1 to V2 and to just assume the same change should have
happened to the deleted resource.
This could be helpful for not loading old provider versions at all, but
can be done in two passes now pretty easily. Just run `up` without any
program changes except for the SDK version bump to update all the
provider references to V2 of the provider, then do another `up` that
deletes the second resource.
    
Fixes https://github.com/pulumi/pulumi/issues/12177.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-05-22 10:57:34 +00:00
Kyle Dixler ce7e74a4c2
Revert change in targeted update behavior.
Fixes 

This commit uses an untargeted resource's old inputs directly and does
not send the untargeted resource's inputs to the provider for
alterations in `provider.Check()`.

This commit fixes a bug in behavior where `provider.Check()` was
no longer be called on untargeted resources. `provider.Check()` would
alter the engine's inputs for use in resource.Goals and skipping this
would cause `__defaults` to not be configured properly in same steps by
`provider.Check()` and cause replaces in some cases.
2023-05-18 15:25:17 -07:00
Kyle Dixler 6cf2b6e5ac
Untargeted resources now have update plans.
Fixes 

What:
This change modifies the step_generator to include untargeted resources
in plans.

Why:
This prevents the step_executor from erroring from an untargeted
resource being registered.

The `step_executor` would error due to an untargeted resource being
registered while not being included in the update plan.

This appeared to be applicable only to users using update plans, but
update plans are enabled by default internally.

There are some rare cases where they are not such as if `--skip-preview`
is set, or if using `pulumi up` with URLs or Templates without
PULUMI_EXPERIMENTAL being truthy.

Previous Context:
This error previously appeared in  and was believed to be due to
the root stack resource not being targeted in liue of informative error
messages.

A fix was merged in:

https://github.com/pulumi/pulumi/pull/12834

This also contained an enhanced error message containing the offending
URN of the resource.

The fix was shipped in 3.67.0, but  was opened with the new error
message indicating that the problem was still outstanding and it was
applicable to resources that weren't the root stack resource.
2023-05-15 07:10:44 -07:00
Fraser Waters 209d279353 Don't load providers at startup
This changes the provider registry to no longer load all the providers
from the old state on startup (in `NewRegistry`) instead the load logic
has been moved to the `Same` method. The step_executor and
step_generator have been fixed up to ensure that for cases where a
resource might not have had it's provider created yet (i.e. for DBR'ing
the old version of a resource, for refreshes or deletes) they ask the
`Deployment` to look up the provider in the old state and `Same` it in the
registry.

All of the above means we only load providers we're going to use (even
taking --targets into account).

One fix mot done in this change is to auto-update providers for deletes.
That is given a program state with two resources both using V1 of a
provider, if you run the program to update one of those resource to use
V2 of the provider but to delete the other resource currently we'll
still load V1 to do that delete. It _might_ be possible (although this
is definitly questionable) to see that another resource changed it's
provider from V1 to V2 and to just assume the same change should have
happened to the deleted resource.
This could be helpful for not loading old provider versions at all, but
can be done in two passes now pretty easily. Just run `up` without any
program changes except for the SDK version bump to update all the
provider references to V2 of the provider, then do another `up` that
deletes the second resource.

Fixes https://github.com/pulumi/pulumi/issues/12177.
2023-05-12 10:04:21 +01:00
Kyle Dixler 56280ed759
Fix targeted-replace and update plans.
Resources marked --target-replace are now correctly targeted. The Pulumi
root stack resource is now also marked targeted similarly to default
providers.

Users were running into an error `error: this should already have a plan
from when we called register resources when previewing update` due to a
bug where `--target` being left empty would assume it was an
unconstrained update targeting everything despite `--target-replace`
being set.
2023-05-09 07:48:53 -07:00
Kyle Dixler c98e944bff
mark default providers as always targeted 2023-05-02 11:37:30 -07:00
Kyle Dixler 433dc8811f
`--target` now only creates Providers if targeted in step generation
Providers must now be specified in `--target` if they are to be created
during an update. This fixes a bug where a provider that was not
specified in `--target` was being created with a parent that did not
exist because the parent was also not specified in `--target` this would
break snapshot validation.
2023-04-28 11:25:17 -07:00
Kyle Dixler 044fecb000
Revert PR moving deletedWith inheritance logic to the engine as `get` resources and MLCs are incompatible. 2023-03-30 10:18:27 -07:00
Fraser Waters a031490c5a Test and fix component->custom->component lifecycle
Fixes 

If we're updating a resource from a component to custom resource (or
vice versa) we always mark it as a provider diff and so trigger a
replace.
2023-03-30 15:18:05 +01:00
bors[bot] 523b3f3290
Merge
12082: Track resource created and updated times r=dixler a=blampe

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This PR adds the `Created` and `Modified` timestamps that are optional.

`Created`: Created tracks when the resource was first added to state by pulumi. Checkpoints prior to early 2023 do not include this. (Create, Import)
`Modified`: Modified tracks when the resource state was last altered. Checkpoints prior to early 2023 do not include this. (Create, Import, Read, Refresh, Update)

When serialized they will follow RFC3339 with nanoseconds captured by a test case.
https://pkg.go.dev/time#RFC3339

Note: Older versions of pulumi may strip these fields when modifying the state.

For future expansion, when we inevitably need to track other timestamps, we'll add a new "operationTimestamps" field (or something similarly named that clarified these are timestamps of the actual Pulumi operations).

	operationTimestamps: {
		created: ...,
		updated: ...,
		imported: ...,
	}


Fixes https://github.com/pulumi/pulumi/issues/12022

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Bryce Lampe <bryce@pulumi.com>
2023-03-27 16:37:03 +00:00
Bryce Lampe 7e0dd3f3c2
This commit adds the `Created` and `Modified` timestamps to pulumi state that are optional.
`Created`: Created tracks when the remote resource was first added to state by pulumi. Checkpoints prior to early 2023 do not include this. (Create, Import)
`Modified`: Modified tracks when the resource state was last altered. Checkpoints prior to early 2023 do not include this. (Create, Import, Read, Refresh, Update)

When serialized they will follow RFC3339 with nanoseconds captured by a test case.
https://pkg.go.dev/time#RFC3339

Note: Older versions of pulumi may strip these fields when modifying the state.

For future expansion, when we inevitably need to track other timestamps, we'll add a new "operationTimestamps" field (or something similarly named that clarified these are timestamps of the actual Pulumi operations).

	operationTimestamps: {
		created: ...,
		updated: ...,
		imported: ...,
	}

Fixes https://github.com/pulumi/pulumi/issues/12022
2023-03-27 09:28:26 -07:00
Kyle Dixler a27432c461
Resources now inherit the `DeletedWith` Resource Option across SDKs.
This is implemented in the engine and interprets the empty string `""`
to inherit the value from the resource's parent if it exists signifying
that it was unspecified by the user's program. There is currently no way
to override this in a child to unset it when set by the parent, but can
be addressed by not parenting the resource to a resource with
`deletedWith` set.
2023-03-27 08:43:24 -07:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta d8089b0c53
pkg/resource/deploy: Prefer Assertf
pkg/resource/deploy has a bunch of Assert usages
without messages.

This migrates all of them to `*f` variants,
preferring `Requiref` for paramters.

Refs 
2023-02-21 15:14:42 -08:00
Abhinav Gupta d8c8c74c5d
golangci-lint: Enable staticcheck
Remove staticcheck from the list of disabled linters.
It's enabled by default in golangci-lint.

This also fixes minor remaining staticcheck issues
that don't merit their own pull requests,
or opts out of those that cannot be fixed yet.

Notably, we're opting out of:

- Resource.Name is deprecated ()
- github.com/golang/protobuf is deprecated ()
- strings.Title has been deprecated ()

Besides that, other issues addressed in this change are:

```
// all issues are in pkg
codegen/schema/docs_parser.go:103:4: SA4006: this value of `text` is never used (staticcheck)
codegen/schema/loader.go:253:3: SA9003: empty branch (staticcheck)
resource/deploy/step_executor.go:328:12: SA9003: empty branch (staticcheck)
resource/deploy/step_generator.go:141:10: SA9003: empty branch (staticcheck)
codegen/pcl/invoke.go:97:10: SA9003: empty branch (staticcheck)
codegen/hcl2/model/type_const.go:57:2: SA9003: empty branch (staticcheck)
codegen/hcl2/model/type_enum.go:99:9: SA4001: &*x will be simplified to x. It will not copy x. (staticcheck)
codegen/go/gen_test.go:399:19: SA4017: HasPrefix is a pure function but its return value is ignored (staticcheck)
```

Depends on , , , , , , , , 

Resolves 
2023-01-14 16:59:46 -08:00
Robbie McKinstry 1f78baae71
Preallocate slices with a known capacity.
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
2023-01-11 12:52:51 -08:00
Kyle Pitzen 92cb70996c feat(sdk/nodejs): delegates alias computation to the engine 2022-12-15 17:23:31 -05:00
Fraser Waters 162f1e1116 Handle replacements of resources marked for deletion 2022-12-07 19:59:38 +00:00
Fraser Waters 64eb33cb0c Disable auto stack parenting 2022-11-07 16:19:16 +00:00
Fraser Waters 7279aa03b4 Merge remote-tracking branch 'origin/master' into same-id/master 2022-11-04 15:28:01 +00:00
Fraser Waters 40c0b800c7
Merge branch 'master' into master 2022-11-01 13:42:59 +00:00
Fraser Waters 25bd778d02 Check for duplicate aliases and plain URNs
We need to check that for example that if a resource X is created that
we don't allow another resource Y to be alias against X. Likewise if our
old state has X and we then create Y aliased against X we should not
then allow X to be created later in the deployment.

Fixes https://github.com/pulumi/pulumi/issues/11173
2022-11-01 13:40:05 +00:00
bors[bot] 1ccfa9fdc4
Merge
11009: Fix update plans with dependent replacements r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

We weren't correctly handling the case where a resource was marked for deletion due to one of it's dependencies being deleted. We would add an entry to it's "Ops" list, but then overwrite that "Ops" list when we came to generate the recreation step.

Fixes https://github.com/pulumi/pulumi/issues/10924

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11027: Do not execute pending deletes at the start of deployment r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This removes all the handling of pending deletes from the start of deployments. Instead we allow resources to just be deleted as they usually would at the end of the deployment.

There's a big comment in TestPendingDeleteOrder that explains the order of operations in a succesful run and how that order differs if we try and do pending deletes up-front.

Fixes https://github.com/pulumi/pulumi/issues/2948

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2022-11-01 10:17:07 +00:00
Sam Eiderman 4bbe365f15 Add DeletedWith resource option
In many cases there is no need to delete resources if the container
resource is going to be deleted as well.

A few examples:
 * Database object (roles, tables) when database is being deleted
 * Cloud IAM bindings when user itself is being deleted

This helps with:
 * Speeding the deletion process
 * Removing unnecessary calls to providers
 * Avoiding failed deletions when the pulumi user running the
   plan has access to the container resource but not the contained
   ones

To avoid deleting contained resources, set the `DeletedWith` resource
option to the container resource.

TODO:
 Should we support DeletedWith with PendingDeletes?
 Special case might be when the contained resource is marked as pending
 deletion but we now want to delete the container resource, so
 ultimately there is no need to delete the contained anymore
2022-10-31 12:03:18 +02:00
Ian Wahbe 856bded330 test # This is a combination of 3 commits.
Support future glob patterns

Improve name for unspecified targets

Add a test in target_test
2022-10-25 16:54:58 -07:00
Fraser Waters a3128e50be Do not execute pending deletes at the start of deployment
This removes all the handling of pending deletes from the start of
deployments. Instead we allow resources to just be deleted as they
usually would at the end of the deployment.

There's a big comment in TestPendingDeleteOrder that explains the order
of operations in a succesful run and how that order differs if we try
and do pending deletes up-front.

Fixes https://github.com/pulumi/pulumi/issues/2948
2022-10-14 15:52:13 +01:00
Fraser Waters 1482df9321 Fix update plans with dependent replacements
We weren't correctly handling the case where a resource was marked for
deletion due to one of it's dependencies being deleted. We would add an
entry to it's "Ops" list, but then overwrite that "Ops" list when we
came to generate the recreation step.

Fixes https://github.com/pulumi/pulumi/issues/10924
2022-10-13 13:33:05 +01:00
Kyle Pitzen 95e2505de5 feat(engine): Adds structured alias support to the engine
feat(engine): Adds alias lineage calculation to the engine
2022-10-11 17:56:32 -04:00
Kyle Pitzen 0ce3821d97 chore: Renames existing aliases field to AliasURNs
feat: exposes new aliases field as actual Alias structs
2022-10-11 17:56:32 -04:00