Commit Graph

72 Commits

Author SHA1 Message Date
Will Jones 5c0ea39b24
Update lifecycle test APIs to match provider APIs ()
In , provider methods were normalised to the form
`Method(context.Context, MethodRequest) (MethodResponse, error)`. This
commit makes the same changes to the lifecycle tests, both for
consistency and discoverability when writing stubs for provider methods.
2024-07-26 12:14:45 +00:00
Fraser Waters 9f7134039d
Add parameterized read support to the engine ()
This enables the engine to receive ReadResource for parameterized
providers.
2024-07-26 10:03:18 +00:00
Fraser Waters 0758303dc9
Add parameterized invoke support to the engine ()
This enables the engine to receive Invokes for parameterized providers.
2024-07-24 17:23:13 +00:00
Fraser Waters 5cbf4cf728
Extend the TestReplacementParameterizedProvider test ()
The test now checks that destroy works, and also that the parameterized
provider state is what we expect.

This required flipping around some of the state managment around reading
and writing provider inputs.
2024-07-15 08:33:36 +00:00
Luke Hoban f1e4b4ff94
Change `pulumi refresh` to report diff relative to desired state instead of relative to only output changes ()
Presently, the behaviour of diffing during refresh steps is incomplete,
returning only an "output diff" that presents the changes in outputs.
This commit changes refresh steps so that:

* they compute a diff similar to the one that would be computed if a
`preview` were run immediately after the refresh, which is more
typically what users expect and want; and
* `IgnoreChanges` resource options are respected when performing the new
desired-state diffs, so that property additions or changes reported by a
refresh can be ignored.

In particular, `IgnoreChanges` can now be used to acknowledge that part
or all of a resource may change in the provider, but the user is OK with
this and doesn't want to be notified about it during a refresh.
Importantly, this means that the diff won't be reported, but also that
the changes won't be applied to state.

The implementation covers the following:

* A diff is computed using the inputs from the program and then
inverting the result, since in the case of a refresh the diff is being
driven by the provider side and not the program. This doesn't change
what is stored back into the state, but it does produce a diff that is
more aligned with the "true changes to the desired state".
* `IgnoreChanges` resource options are now stored in state, so that this
information can be used in refresh operations that do not have access
to/run the program.
* In the context of a refresh operation, `IgnoreChanges` applies to
*both* input and output properties. This differs from the behaviour of a
normal update operation, where `IgnoreChanges` only considers input
properties.
* The special `"*"` value for `IgnoreChanges` can be used to ignore all
properties. It _also_ ignores the case where the resource cannot be
found in the provider, and instead keeps the resource intact in state
with its existing input and output properties.

Because the program is not run for refresh operations, `IgnoreChanges`
options must be applied separately before a refresh takes place. This
can be accomplished using e.g. a `pulumi up` that applies the options
prior to a refresh. We should investigate perhaps providing a `pulumi
state set ...`-like CLI to make these sorts of changes directly to a
state.

For use cases relying on the legacy refresh diff provider, the
`PULUMI_USE_LEGACY_REFRESH_DIFF` environment variable can be set, which
will disable desired-state diff computation. We only need to perform
checks in `RefreshStep.{ResultOp,Apply}`, since downstream code will
work correctly based on the presence or absence of a `DetailedDiff` in
the step.

### Notes

- https://github.com/pulumi/pulumi/issues/16144 affects some of these
cases - though its technically orthogonal
- https://github.com/pulumi/pulumi/issues/11279 is another technically
orthogonal issue that many providers (at least TFBridge ones) - do not
report back changes to input properties on Read when the input property
(or property path) was missing on the inputs. This is again technically
orthogonal - but leads to cases that appear "wrong" in terms of what is
stored back into the state still - though the same as before this
change.
- Azure Native doesn't seem to handle `ignoreChanges` passed to Diff, so
the ability to ignore changes on refresh doesn't currently work for
Azure Native.

### Fixes

* Fixes 
* Fixes 
* Fixes  
* Not quite , but likely replaces the need for that

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-06-12 16:17:05 +00:00
Will Jones f71c764a4c
Clean up deployment options ()
# Description

There are a number of parts of the deployment process that require
context about and configuration for the operation being executed. For
instance:

* Source evaluation -- evaluating programs in order to emit resource
registrations
* Step generation -- processing resource registrations in order to
generate steps (create this, update that, delete the other, etc.)
* Step execution -- executing steps in order to action a deployment.

Presently, these pieces all take some form of `Options` struct or pass
explicit arguments. This is problematic for a couple of reasons:

* It could be possible for different parts of the codebase to end up
operating in different contexts/with different configurations, whether
due to different values being passed explicitly or due to missed
copying/instantiation.
* Some parts need less context/configuration than others, but still
accept full `Options`, making it hard to discern what information is
actually necessary in any given part of the process.

This commit attempts to clean things up by moving deployment options
directly into the `Deployment` itself. Since step generation and
execution already refer to a `Deployment`, they get a consistent view of
the options for free. For source evaluation, we introduce an
`EvalSourceOptions` struct for configuring just the options necessary
there. At the top level, the engine configures a single set of options
to flow through the deployment steps later on.

As part of this work, a few other things have been changed:

* Preview/dry-run parameters have been incorporated into options. This
lets up lop off another argument and mitigate a bit of "boolean
blindness". We don't appear to flip this flag within a deployment
process (indeed, all options seem to be immutable) and so having it as a
separate flag doesn't seem to buy us anything.
* Several methods representing parts of the deployment process have lost
arguments in favour of state that is already being carried on (or can be
carried on) their receiver. For instance, `deployment.run` no longer
takes actions or preview configuration. While doing so means that a
`deployment` could be run multiple times with different actions/preview
arguments, we don't currently exploit this fact anywhere, so moving this
state to the point of construction both simplifies things considerably
and reduces the possibility for error (e.g. passing different values of
`preview` when instantiating a `deployment` and subsequently calling
`run`).
* Event handlers have been split out of the options object and attached
to `Deployment` separately. This means we can talk about options at a
higher level without having to `nil` out/worry about this field and
mutate it correctly later on.
* Options are no longer mutated during deployment. Presently there
appears to be only one case of this -- when handling `ContinueOnError`
in the presence of `IgnoreChanges` (e.g. when performing a refresh).
This case has been refactored so that the mutation is no longer
necessary.

# Notes

* This change is in preparation for , where we'd like to add an
environment variable to control behaviour and having a single unified
`Options` struct would make it easier to pass this configuration down
with introducing (more) global state into deployments. Indeed, this
change should make it easier to factor global state into `Options` so
that it can be controlled and tested more easily/is less susceptible to
bugs, race conditions, etc.
* I've tweaked/extended some comments while I'm here and have learned
things the hard way (e.g. `Refresh` vs `isRefresh`). Feedback welcome on
this if we'd rather not conflate.
* This change does mean that if in future we wanted e.g. to be able to
run a `Deployment` in multiple different ways with multiple sets of
actions, we'd have to refactor. Pushing state to the point of object
construction reduces the flexibility of the code. However, since we are
not presently using that flexibility (nor is there an obvious [to my
mind] use case in the near future), this seems like a good trade-off to
guard against bugs/make it simpler to move that state around.
* I've left some other review comments in the code around
questions/changes that might be a bad idea; happy to receive feedback on
it all though!
2024-06-11 13:37:57 +00:00
Fraser Waters 5a83cfe6e4
Initial work for parameterized providers ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the first part of paramaterized providers in the engine. This
only supports "replacement" packages, that is where we fully replace a
providers package with a new package (think how dynamic tfbridge will
work, vs how crd2pulumi will work).

I've made the decision to _not_ support using parameterised providers by
sending the parameter in the RegisterResource request. This will
necessitate some different work in how we send the parameter for
explicit providers compared to version and pluginDownloadURL, but I
think it's worth it going forward.

No changelog as this is still basically unusable without codegen support
done, and should still be considered primarily for internal experimental
use for now.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-10 17:28:47 +00:00
Fraser Waters 9eb49a2818
RegisterProvider engine work ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support for a `RegisterProvider` method to the engine. This
allows an SDK process to send the information for a package (name,
version, url, etc, and parameter in the future) and get back a UUID for
that run of the engine that can be used to re-lookup that information.

That allows the SDK to just send the `provider` field in
`RegisterResourceRequest` instead of filling in `version`,
`pluginDownloadURL` etc (and importantly not having to fill in
`parameter` for parameterised providers, which could be a large amount
of data).

This doesn't update any of the SDKs to yet use this method. We can do
that piecemeal, but it will require core sdk and codegen changes for
each language.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-23 06:16:59 +00:00
Eron Wright 89d3467b0f
Canonicalize provider version during default provider lookup ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR addresses a problem that leads to two default providers for a
given package+version. The problem occurs when given a combination of
unversioned and versioned provider requests, as can occur in
multi-language programs.

The fix is as follows: given an unversioned provider request, apply the
default version before looking into the provider cache.

Fixes  

## Example
Given the example program from  plus this fix, the stack state has
one provider as expected:

```jsonl
{
    "urn": "urn:pulumi:dev::issue-1939-nodejs::pulumi:providers:kubernetes::default_4_10_0",
    "custom": true,
    "id": "c19b65b0-dd5a-432f-8c75-d7544de782ec",
    "type": "pulumi:providers:kubernetes",
    "inputs": {
        "version": "4.10.0"
    },
    "outputs": {
        "version": "4.10.0"
    },
    "created": "2024-05-02T21:22:09.72166Z",
    "modified": "2024-05-02T21:22:09.72166Z"
}
{
    "urn": "urn:pulumi:dev::issue-1939-nodejs::kubernetes:yaml/v2:ConfigGroup$kubernetes:core/v1:ConfigMap::example:eron/example",
    "custom": true,
    "id": "eron/example",
    "type": "kubernetes:core/v1:ConfigMap",
    "parent": "urn:pulumi:dev::issue-1939-nodejs::kubernetes:yaml/v2:ConfigGroup::example",
    "provider": "urn:pulumi:dev::issue-1939-nodejs::pulumi:providers:kubernetes::default_4_10_0::c19b65b0-dd5a-432f-8c75-d7544de782ec",
    "propertyDependencies": {
        "apiVersion": [],
        "kind": [],
        "metadata": []
    },
}
```

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-06 08:22:28 +00:00
Thomas Gummerer 477a54b3de
deploytest/RegisterResource: return struct instead of values ()
This method already returns 4 different values, and we want to add more.
Refactor it so it returns a struct, to make adding additional return
values easier in the future.
2024-04-19 11:08:56 +00:00
Thomas Gummerer eff61838ac
Make sure TestCall test doesn't flake ()
The ordering of this array isn't guaranteed, so use ElementsMatch
instead of Equal to compare to avoid the flakyness.

I noticed this test being flaky while working on bazel, but also noticed
it in CI logs, here:
https://github.com/pulumi/pulumi/actions/runs/8646652290/job/23706586759?pr=15906
2024-04-11 13:47:29 +00:00
Fraser Waters 7422c44ca4
Engine support for remote transforms ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support to the engine for "remote transformations".

A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.

The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.

A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.

The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
Fraser Waters 65b8f5c841
Split CallRequest into ResourceCallRequest ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Similar to what we did to the `InvokeRequest` a while ago. We're
currently using the same protobuf structure for `Provider.Call` and
`ResourceMonitor.Call` despite different field sets being filled in for
each of them.

This splits the structure into `CallRequest` for providers and
`ResourceCallRequest` for the resource monitor. A number of fields in
each are removed and marked reserved with a comment explaining why.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-08 13:16:23 +00:00
Fraser Waters 6c52b24cc8
Add an engine test for Construct and Call dependency maps ()
The only real change here is a new test in
pkg/engine/lifecycletest/pulumi_test.go and the small fix up to the
`deploytest.ResourceMonitor` to return the information needed for that
test.
2024-02-08 13:01:47 +00:00
Fraser Waters 1138fdb63c
Upgrade output values in source_eval ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Stage 1 of updating how we deal with OutputValues in source_eval for
transforms.

This makes two changes that should result in basically a no-op for
standard resources, but update all Computed/Secret values to
OutputValues for Call and Construct.

This is a no-op for standard resources because previously we set
"KeepOutputs" to false when we unmarshalled them. That replaced all
OutputValues with Computed/Secret. Now we do "UpdgradeOutputs" at the
start of source_eval but then call "DowngradeOutputs" before passing the
properties on to the rest of the system. So this is a complete no-op.

But for Call/Construct we used to use "KeepOutputs" but now we do
"UpgradeOutputs". So values that previously got sent as Computed/Secret
will now get sent as OutputValue. This _should_ be fine, all users of
Construct/Call already had to handle OutputValue on their interface, so
this just means a few more cases of seeing those values.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-06 16:46:31 +00:00
Fraser Waters 0113f27d6c
Don't mutate RegisterResourceRequest ()
Noticed this while trying to prepare a PR for the transforms support.
These tests were checking that we mutated a field on the request object
itself. We no longer do that in the transform branch because everything
has to go through the transform callbacks as well but it caused these
tests to fail.

I've deleted these tests and written a new engine test to validate the
system output is actually as expected.
2024-01-30 16:45:10 +00:00
Fraser Waters dd5fef7091
Fix stack name validation check ()
The disable validation check for this was wrong (it did the assert if
validation was disabled).
2024-01-27 10:35:20 +00:00
Zaid Ajaj bd4e50efdd
[conformance tests] Fix run root and use program info everywhere ()
# Description

This PR introduces `ProgramInfo` to replace the old `ProgInfo` and
consistently use it where we require plugin, install dependencies and
initialize language runtimes.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-25 23:28:58 +00:00
Fraser Waters 5e8945a23a
Fix NodeJS alias objects at the protocol level. ()
Precursor to moving transformations to run for remote components. We
need to re-send alias options to the transform function but we'll want
to do the NodeJS fixup before sending them onwards.
2024-01-23 17:44:53 +00:00
Kyle Dixler 57d4d7a46e
[ci] `pkg/resource/deploy/source_eval.go` coverage ()
Adds to `pkg/resource/deploy/source_eval_test.go` which adds missing
coverage to `pkg/resource/deploy/source_eval.go`
2023-12-22 21:14:04 +00:00
Kyle Dixler 8ecdaa2bc0
[ci] `pkg/resource/deploy` coverage ()
code coverage `pkg/resource/deploy` from 76% -> 83%

Will remove CI changes and squash commits prior to merging PR.
2023-12-19 16:14:40 +00:00
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Fraser Waters d771acf707
Add tokens.StackName ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new type `tokens.StackName` which is a relatively strongly
typed container for a stack name. The only weakly typed aspect of it is
Go will always allow the "zero" value to be created for a struct, which
for a stack name is the empty string which is invalid. To prevent
introducing unexpected empty strings when working with stack names the
`String()` method will panic for zero initialized stack names.
 
Apart from the zero value, all other instances of `StackName` are via
`ParseStackName` which returns a descriptive error if the string is not
valid.

This PR only updates "pkg/" to use this type. There are a number of
places in "sdk/" which could do with this type as well, but there's no
harm in doing a staggered roll out, and some parts of "sdk/" are user
facing and will probably have to stay on the current `tokens.Name` and
`tokens.QName` types.

There are two places in the system where we panic on invalid stack
names, both in the http backend. This _should_ be fine as we've had long
standing validation that stacks created in the service are valid stack
names.

Just in case people have managed to introduce invalid stack names, there
is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn
off the validation _and_ panicing for stack names. Users can use that to
temporarily disable the validation and continue working, but it should
only be seen as a temporary measure. If they have invalid names they
should rename them, or if they think they should be valid raise an issue
with us to change the validation code.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-15 07:44:54 +00:00
Fraser Waters cf5b4a2790
Use `assert.NoError` rather than `assert.Nil` ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Likewise `require.NoError` instead of `require.Nil`, and `assert.Error`
rather than `assert.NotNil`.

The error variants of these functions print the errors nicer for test
failures using `Error()` rather than `GoString()`.

For bail errors this is _much_ better than the `result.Result` days
where we now get errors like:
```
Error:      	Received unexpected error:
            	BAIL: inner error
```
instead of:
```
Error:      	Expected nil, but got: &simpleResult{}
```

Also print the bail error in `TestPlan.Run` so we can see the
description of it.
2023-10-13 09:46:07 +00:00
Fraser Waters f557548e6d
Reuse provider instances where possible ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/13987.

This reworks the registry to better track provider instances such that
we can reuse unconfigured instances between Creates, Updates, and Sames.

When we allocate a provider instance in the registry for a Check call we
save it with the special id "unconfigured". This value should never make
its way back to program SDKs, it's purely an internal value for the
engine.

When we do a Create, Update or Same we look to see if there's an
unconfigured provider to use and if so configures that one, else it
starts up a fresh one. (N.B. Update we can assume there will always be
an unconfigured one from the Check call before).

This has also fixed registry Create to use the ID `UnknownID` rather
than `""`, have added some contract assertions to check that and fixed
up some test fallout because of that (the tests had been getting away
with leaving ID blank before).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-12 20:46:01 +00:00
Fraser Waters 83e38c2ac9 Add ConstructInfo to deploytest
This is so we can access the other parts of ConstructInfo inside
ConstructF functions in tests, same as CallInfo and CallF.
2023-07-25 09:03:48 +01:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
Justin Van Patten f5b117505c Maintain alias compat for older Node.js SDKs on new CLIs
This change updates the engine to detect if a `RegisterResource` request
is coming from an older Node.js SDK that is using incorrect alias specs
and, if so, transforms the aliases to be correct. This allows us to
maintain compatibility for users who have upgraded their CLI but are
still using an older version of the Node.js SDK with incorrect alias
specs.

We detect if the request is from a Node.js SDK by looking at the gRPC
request's metadata headers, specifically looking at the "pulumi-runtime"
and "user-agent" headers.

First, if the request has a "pulumi-runtime" header with a value of
"nodejs", we know it's coming from the Node.js plugin. The Node.js
language plugin proxies gRPC calls from the Node.js SDK to the resource
monitor and the proxy now sets the "pulumi-runtime" header to "nodejs"
for `RegisterResource` calls.

Second, if the request has a "user-agent" header that starts with
"grpc-node-js/", we know it's coming from the Node.js SDK. This is the
case for inline programs in the automation API, which connects directly
to the resource monitor, rather than going through the language plugin's
proxy.

We can't just look at "user-agent", because in the proxy case it will
have a Go-specific "user-agent".

Updated Node.js SDKs set a new `aliasSpecs` field on the
`RegisterResource` request, which indicates that the alias specs are
correct, and no transforms are needed.
2023-06-14 08:34:32 -07:00
Kyle Dixler 653adbe621
Resources now inherit the `DeletedWith` Resource Option across SDKs.
This is implemented in the engine and interprets the empty string `""`
to inherit the value from the resource's parent if it exists signifying
that it was unspecified by the user's program. There is currently no way
to override this in a child to unset it when set by the parent, but can
be addressed by not parenting the resource to a resource with
`deletedWith` set.
2023-06-02 09:20:26 -07:00
Abhinav Gupta 633efb823b
engine: Propagate options to MLCs
In the engine, propagate the following options to provider.Construct:

- additionalSecretOutputs
- customTimeouts
- deleteBeforeReplace
- deletedWith
- ignoreChanges
- replaceOnChanges
- retainOnDelete

Note that in RegisterResource, there's a bit that goes:

    if remote {
        provider.Construct(...)
    } else {
        resource.NewGoal(...)
    }

With this change, all information passed to resource.NewGoal
is also touched for provider.Construct in some form.
2023-04-21 10:58:57 -07:00
Bryce Lampe 7e0dd3f3c2
This commit adds the `Created` and `Modified` timestamps to pulumi state that are optional.
`Created`: Created tracks when the remote resource was first added to state by pulumi. Checkpoints prior to early 2023 do not include this. (Create, Import)
`Modified`: Modified tracks when the resource state was last altered. Checkpoints prior to early 2023 do not include this. (Create, Import, Read, Refresh, Update)

When serialized they will follow RFC3339 with nanoseconds captured by a test case.
https://pkg.go.dev/time#RFC3339

Note: Older versions of pulumi may strip these fields when modifying the state.

For future expansion, when we inevitably need to track other timestamps, we'll add a new "operationTimestamps" field (or something similarly named that clarified these are timestamps of the actual Pulumi operations).

	operationTimestamps: {
		created: ...,
		updated: ...,
		imported: ...,
	}

Fixes https://github.com/pulumi/pulumi/issues/12022
2023-03-27 09:28:26 -07:00
Abhinav Gupta 1bdf2a8667
all: Don't use cmdutil.Diag in tests
Update all tests that use cmdutil.Diag directly
(because these write to the test process' stdout/stderr)
and instead use diagtest.Sink to have them write
to the test's logger.
2023-03-13 12:54:04 -07:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Sam Eiderman 4bbe365f15 Add DeletedWith resource option
In many cases there is no need to delete resources if the container
resource is going to be deleted as well.

A few examples:
 * Database object (roles, tables) when database is being deleted
 * Cloud IAM bindings when user itself is being deleted

This helps with:
 * Speeding the deletion process
 * Removing unnecessary calls to providers
 * Avoiding failed deletions when the pulumi user running the
   plan has access to the container resource but not the contained
   ones

To avoid deleting contained resources, set the `DeletedWith` resource
option to the container resource.

TODO:
 Should we support DeletedWith with PendingDeletes?
 Special case might be when the contained resource is marked as pending
 deletion but we now want to delete the container resource, so
 ultimately there is no need to delete the contained anymore
2022-10-31 12:03:18 +02:00
Fraser Waters 9dfbee3b2d
Remove sequence numbers ()
* Remove sequenceNumber from protobufs

* Regenerate protobufs

* Remove setting and reading of sequence number in Check

* Remove sequence numbers from state

* Replace sequenceNumber with randomSeed in Check

* Fix tests

* Add to CHANGELOG
2022-07-25 12:08:03 +01:00
Fraser Waters 859052d6d9
Revert "Strip Aliases from state ()" ()
This reverts commit 17068e9b49.

Turns out NormalizeURNReferences needs this in the state to fix up URNs while the deployment is running. It feels like we should be able to either thread this information through to the snapshot manager another way but it's not obvious how. It's also tricky to test because snapshot code differs massively in unit tests compared to proper runs.
2022-03-24 20:08:18 +01:00
Fraser Waters 17068e9b49
Strip Aliases from state ()
* Strip Aliases from state

* chore: note fix, reason for change in changelog

* remove redundant test

see: ed2923653c/pkg/engine/lifeycletest/step_generator_test.go (L16)

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-03-23 17:55:06 -07:00
Fraser Waters 86f3f712aa
Readd "Make StackReference.Name a tokens.Name ()" ()
* Readd "Make StackReference.Name a tokens.Name ()"

This reverts commit f0aa4df149.

This also removes the AsName asserting casts for stack names. We do want
to add them in at some point to be sure that bad names don't slip in
somehow but they don't need adding with this.

* Update sdk/go/common/util/fsutil/qname.go

Co-authored-by: Ian Wahbe <ian@wahbe.com>

Co-authored-by: Ian Wahbe <ian@wahbe.com>
2022-03-17 21:37:11 +00:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Ian Wahbe 999da67dcd
Fix (Read,Invoke) denied default provider handling ()
* Fix (Read,Invoke) denied default provider handling

When denying default providers was added, we had no special handling for
Reads and Invokes. This lead to confusing error messages. The fix ()
involved checking on invokes. This check didn't apply to several types
of calls (Read) as well as blocking invokes with providers applied.

This PR fixes the logic to only deny providers when they are default
providers.

It also pushes the change into `getProviderFromSource`, which ensures
that this behavior is handled the same way (and correctly) for both
Invokes and Reads.

* Append to changelog

* Add testing

* Fix lint

* Fix spelling + nit
2022-02-28 15:33:45 -08:00
Fraser Waters 5d91f8f322
Add RetainOnDelete resource option ()
* Plumb in basics of retainOnDelete

* Add test

* Make test pass

* Add to changelog

* Add to API list

* lint

* Add semicolon

* Fix Infof call

* Fix method call

* new delete mode work

* cleanup

* protectTest

* Fix up test

* Fix replace

* Fix up test

* Warn on drop

* lint

* Change to just a bool flag

* Regenerate proto

* Rework to just a bool flag with no error

* Remove old comment

* Fix C# typo

* rm extra space

* Add missing semicolon

* Reformat python

* False typo

* Fix typo in js function name

* Reword docs

* lint

* Read doesn't need retainOnDelete
2022-02-16 22:11:12 +00:00
Fraser Waters ec3ef44841
Make resource autonames determinstic ()
* Start adding SequenceNumber

* Start adding sequence number to state

* New generate functions

* notes

* Don't increment if unknown

* Deterministic name test

* Check replace

* typo

* lint

* Increment on targetted replace

* Some comments and external fixes

* Add test for resetting sequence number after replace

* Reset sequence numbers after replace

* assert check we never pass -1 to check

* Add to dynamic providers

* lint

* Add to changelog
2022-01-20 11:18:54 +00:00
Luke Hoban eb32039013
Add `replaceOnChanges` resource option ()
Adds a new resource option to force replacement when certain properties report changes, even if the resource provider itself does not require a replacement.

Fixes .

Co-authored-by: Levi Blackstone <levi@pulumi.com>
2021-07-01 13:32:08 -06:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Pat Gavlin 249140242e
Add support for provider-side preview. ()
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.

If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.

If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.

Fixes .
2020-10-09 13:13:55 -07:00
Justin Van Patten 7f27618e2d
Avoid replace on second update with import applied ()
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.

It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.

Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.

To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
2020-04-15 18:52:40 -07:00
CyrusNajmabadi 66bd3f4aa8
Breaking changes due to Feature 2.0 work
* Make `async:true` the default for `invoke` calls ()

* Switch away from native grpc impl. ()

* Remove usage of the 'deasync' library from @pulumi/pulumi. ()

* Only retry as long as we get unavailable back.  Anything else continues. ()

* Handle all errors for now. ()


* Do not assume --yes was present when using pulumi in non-interactive mode ()

* Upgrade all paths for sdk and pkg to v2

* Backport C# invoke classes and other recent gen changes ()

Adjust C# generation

* Replace IDeployment with a sealed class ()

Replace IDeployment with a sealed class

* .NET: default to args subtype rather than Args.Empty ()

* Adding system namespace for Dotnet code gen

This is required for using Obsolute attributes for deprecations

```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```

* Fix the nullability of config type properties in C# codegen ()
2020-04-14 09:30:25 +01:00
evanboyle d3f5bbce48 go fmt 2020-03-18 17:27:02 -07:00
evanboyle c1d3a8524b move pkg/util/cmdutil -> sdk/go/common/util/cmdutil 2020-03-18 15:39:00 -07:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00