Commit Graph

54 Commits

Author SHA1 Message Date
Fraser Waters 5a83cfe6e4
Initial work for parameterized providers ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the first part of paramaterized providers in the engine. This
only supports "replacement" packages, that is where we fully replace a
providers package with a new package (think how dynamic tfbridge will
work, vs how crd2pulumi will work).

I've made the decision to _not_ support using parameterised providers by
sending the parameter in the RegisterResource request. This will
necessitate some different work in how we send the parameter for
explicit providers compared to version and pluginDownloadURL, but I
think it's worth it going forward.

No changelog as this is still basically unusable without codegen support
done, and should still be considered primarily for internal experimental
use for now.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-10 17:28:47 +00:00
Fraser Waters 9eb49a2818
RegisterProvider engine work ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support for a `RegisterProvider` method to the engine. This
allows an SDK process to send the information for a package (name,
version, url, etc, and parameter in the future) and get back a UUID for
that run of the engine that can be used to re-lookup that information.

That allows the SDK to just send the `provider` field in
`RegisterResourceRequest` instead of filling in `version`,
`pluginDownloadURL` etc (and importantly not having to fill in
`parameter` for parameterised providers, which could be a large amount
of data).

This doesn't update any of the SDKs to yet use this method. We can do
that piecemeal, but it will require core sdk and codegen changes for
each language.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-23 06:16:59 +00:00
Thomas Gummerer 4169755866
Implement up --continue-on-error ()
Similar to destroy --continue-on-error, this flag allows `pulumi up`
to continue if any errors are encountered.

Currently when we encounter an error while creating/updating a
resource, we cancel the context of the deployment executor, and thus
the deployment stops once the resources that are being processed in
parallel with the failed one finish being updated.

For --continue-on-error, we ignore these errors, and let the
deployment executor continue.  In order for the deployment executor to
exit eventually we also have to mark these steps as done, as the
deployment executor will otherwise just hang, and callers with open
channels waiting for it to finish/report back will hang indefinitely.

The errors in the step will still be reported back to the user by the
OnResourceStepPost callback.

Fixes https://github.com/pulumi/pulumi/issues/14515

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-04-22 11:12:45 +00:00
Thomas Gummerer 477a54b3de
deploytest/RegisterResource: return struct instead of values ()
This method already returns 4 different values, and we want to add more.
Refactor it so it returns a struct, to make adding additional return
values easier in the future.
2024-04-19 11:08:56 +00:00
Fraser Waters 7422c44ca4
Engine support for remote transforms ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds support to the engine for "remote transformations".

A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.

The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.

A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.

The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
Fraser Waters ea74b8aa26
Ensure ArgsDependencies are filled in ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Adds argsDependencies support to the deploytest resource monitor and add
some tests that the engine fills that map in with output values from
args if the SDK hasn't filled it in.

We also need to do this the other way (if the SDK fills in
argsDependencies but doesn't send output<T> values with those deps to
args).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 09:15:38 +00:00
Fraser Waters 65b8f5c841
Split CallRequest into ResourceCallRequest ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Similar to what we did to the `InvokeRequest` a while ago. We're
currently using the same protobuf structure for `Provider.Call` and
`ResourceMonitor.Call` despite different field sets being filled in for
each of them.

This splits the structure into `CallRequest` for providers and
`ResourceCallRequest` for the resource monitor. A number of fields in
each are removed and marked reserved with a comment explaining why.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-08 13:16:23 +00:00
Fraser Waters 6c52b24cc8
Add an engine test for Construct and Call dependency maps ()
The only real change here is a new test in
pkg/engine/lifecycletest/pulumi_test.go and the small fix up to the
`deploytest.ResourceMonitor` to return the information needed for that
test.
2024-02-08 13:01:47 +00:00
Fraser Waters bee784bc8a
Set KeepOutputValues in deploytest.ResourceMonitor ()
While working on some new tests I noticed that ResourceMonitor in the
tests didn't set KeepOutputValues for remote resources like it should.
This fixes that.
2024-02-07 20:07:02 +00:00
Fraser Waters 1138fdb63c
Upgrade output values in source_eval ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Stage 1 of updating how we deal with OutputValues in source_eval for
transforms.

This makes two changes that should result in basically a no-op for
standard resources, but update all Computed/Secret values to
OutputValues for Call and Construct.

This is a no-op for standard resources because previously we set
"KeepOutputs" to false when we unmarshalled them. That replaced all
OutputValues with Computed/Secret. Now we do "UpdgradeOutputs" at the
start of source_eval but then call "DowngradeOutputs" before passing the
properties on to the rest of the system. So this is a complete no-op.

But for Call/Construct we used to use "KeepOutputs" but now we do
"UpgradeOutputs". So values that previously got sent as Computed/Secret
will now get sent as OutputValue. This _should_ be fine, all users of
Construct/Call already had to handle OutputValue on their interface, so
this just means a few more cases of seeing those values.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-06 16:46:31 +00:00
Fraser Waters 7ca32437c6
Test that an empty parent URN in an alias is treated as the default parent ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Title sums it up. Some SDKs are sending `parent=""` rather than no
parent at all to specify to the engine to use the default parent.
Technically this isn't the intention of the protocol but given the SDKs
are out we now need to support it. I've added a test to check this works
and left a comment that it would be a nice tidy up to make in V4.

I've also changed the deploy tests to directly use the rpc protocol
rather than `resource.Alias` this lets us write tests like "send an
empty parent URN" which `resource.Alias` wasn't capable of expressing.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-26 18:29:22 +00:00
Fraser Waters a254a53206
Pass provider checksums in requests and save to state ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This extends the resource monitor interface with fields for plugin
checksums (on top of the existing plugin version and download url
fields). These fields are threaded through the engine and are persisted
in resource state. The sent or saved data is then used when installing
plugins to ensure that the checksums match what was recorded at the time
the SDK was built.

Similar to https://github.com/pulumi/pulumi/pull/13776 nothing is using
this yet, but this lays the engine side plumbing for them.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-11 15:54:07 +00:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
Justin Van Patten f5b117505c Maintain alias compat for older Node.js SDKs on new CLIs
This change updates the engine to detect if a `RegisterResource` request
is coming from an older Node.js SDK that is using incorrect alias specs
and, if so, transforms the aliases to be correct. This allows us to
maintain compatibility for users who have upgraded their CLI but are
still using an older version of the Node.js SDK with incorrect alias
specs.

We detect if the request is from a Node.js SDK by looking at the gRPC
request's metadata headers, specifically looking at the "pulumi-runtime"
and "user-agent" headers.

First, if the request has a "pulumi-runtime" header with a value of
"nodejs", we know it's coming from the Node.js plugin. The Node.js
language plugin proxies gRPC calls from the Node.js SDK to the resource
monitor and the proxy now sets the "pulumi-runtime" header to "nodejs"
for `RegisterResource` calls.

Second, if the request has a "user-agent" header that starts with
"grpc-node-js/", we know it's coming from the Node.js SDK. This is the
case for inline programs in the automation API, which connects directly
to the resource monitor, rather than going through the language plugin's
proxy.

We can't just look at "user-agent", because in the proxy case it will
have a Go-specific "user-agent".

Updated Node.js SDKs set a new `aliasSpecs` field on the
`RegisterResource` request, which indicates that the alias specs are
correct, and no transforms are needed.
2023-06-14 08:34:32 -07:00
Justin Van Patten 64a2288883 [engine] Fix aliasing children
There is an issue with how the engine computes the aliases when the
resource is a child and doesn't have `Parent` set on the alias spec
(and the parent doesn't have any aliases).

```python
class FooResource(pulumi.ComponentResource):
    def __init__(self, name, opts=None):
        super().__init__("my:module:FooResource", name, None, opts)

class ComponentResource(pulumi.ComponentResource):
    def __init__(self, name, opts=None):
        super().__init__("my:module:ComponentResource", name, None, opts)
        FooResource("childrenamed", pulumi.ResourceOptions(
            parent=self,
            aliases=[pulumi.Alias(name="child")]
        ))
```

In the example above, `ComponentResource` has a child `FooResource`
which was renamed from `child` to `childrenamed`.

The engine does not compute the correct alias:

```
expected: urn:pulumi:stack::project::my:module:ComponentResource$my:module:FooResource::child
  actual: urn:pulumi:stack::project::my:module:FooResource::child
```

The problem is due to:

117955ce14/pkg/resource/deploy/step_generator.go (L370-L382)

... and:

117955ce14/sdk/go/common/resource/alias.go (L24-L26)

Because the alias spec doesn't have `parent` specified, the parent type
is not being included the computed alias URN.

Existing tests such as https://github.com/pulumi/pulumi/tree/master/tests/integration/aliases/python/rename_component_and_child
didn't catch the problem because the alias specifies both the `name` and
`parent`:

117955ce14/tests/integration/aliases/python/rename_component_and_child/step2/__main__.py (L15)

In this case, specifying `parent` on the alias shouldn't be necessary.
However, even after removing `parent` from the alias spec, the test
still succeeds because the parent itself has an alias:

117955ce14/tests/integration/aliases/python/rename_component_and_child/step2/__main__.py (L18)

... and parent aliases are inherited as part of a child's aliases, so we
still get an alias that works from the inheritance.

If we change the test to make no changes to the parent such that it
doesn't have any aliases, then we get the failure as we'd expect.

A similar problem will happen when retyping a child.

**Fix**

The fix involves using the child's parent in the calculated alias URN
when `Parent` isn't specified for the alias.

As part of this, we need to properly handled `NoParent` because right
now the engine is not correctly using it. The struct representing an
alias in the engine does not have a `NoParent` field:

117955ce14/sdk/go/common/resource/alias.go (L8-L15)

And therefore does not copy it over in the gRPC request:

117955ce14/pkg/resource/deploy/source_eval.go (L1082-L1088)

Instead, the `Alias` struct has an incorrect `NoParent` method which
returns `true` if the `Parent` field has a value of `""`:

117955ce14/sdk/go/common/resource/alias.go (L24-L26)
2023-06-14 05:19:17 -07:00
Abhinav Gupta e6cc299d86
pkg/deploytest: Don't zero CustomTimeouts < 1 minute
deploytest.ResourceMonitor currently zeroes out any custom timeout value
smaller than 60 seconds because it does integer division on it.
Further, it alawys uses a non-nil CustomTimeouts
even when the input was nil.
This causes undesirable breaks in unit tests that use this type.

This change fixes both issues in CustomTimeouts. Now:

- The RegisterResourceRequest.CustomTimeouts field is non-nil
  only if the input CustomTimeouts was non-nil
- time.Duration and float divison is used to accurately format
  the specified duration as a string
2023-04-14 15:05:51 -07:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta c747513ed9
deploytest/ResourceMonitor: Don't drop deps
Another step towards .

ResourceMonitor.Call calls ResourceMonitorClient.Call
and converts the output into a few different results.

During the conversion, it builds a `map[PropertyKey][]URN`,
but it never fills that map with values,
and instead always returns the empty map.

This bug was caught by staticcheck.

```
resource/deploy/deploytest/resourcemonitor.go:390:11: SA4010: this result of append is never used, except maybe in other appends (staticcheck)
```
2023-01-12 14:08:34 -08:00
Abhinav Gupta 69762acedf
grpc: WithInsecure is deprecated
grpc.WithInsecure is deprecated [1].
Instead, we're expected to use:

    grpc.WithTransportCredentials(insecure.NewCredentials())

This updates all instances across pkg/, sdk/ and tests/.

  [1]: https://pkg.go.dev/google.golang.org/grpc#WithInsecure

In service to 
2023-01-11 12:15:40 -08:00
Sam Eiderman 4bbe365f15 Add DeletedWith resource option
In many cases there is no need to delete resources if the container
resource is going to be deleted as well.

A few examples:
 * Database object (roles, tables) when database is being deleted
 * Cloud IAM bindings when user itself is being deleted

This helps with:
 * Speeding the deletion process
 * Removing unnecessary calls to providers
 * Avoiding failed deletions when the pulumi user running the
   plan has access to the container resource but not the contained
   ones

To avoid deleting contained resources, set the `DeletedWith` resource
option to the container resource.

TODO:
 Should we support DeletedWith with PendingDeletes?
 Special case might be when the contained resource is marked as pending
 deletion but we now want to delete the container resource, so
 ultimately there is no need to delete the contained anymore
2022-10-31 12:03:18 +02:00
Kyle Pitzen 95e2505de5 feat(engine): Adds structured alias support to the engine
feat(engine): Adds alias lineage calculation to the engine
2022-10-11 17:56:32 -04:00
Kyle Pitzen 0ce3821d97 chore: Renames existing aliases field to AliasURNs
feat: exposes new aliases field as actual Alias structs
2022-10-11 17:56:32 -04:00
Fraser Waters 09d67511dd
Revert "Send smart aliases via gRPC to engine ()" ()
* Revert "Send smart aliases via gRPC to engine ()"

This reverts commit cba1880e21.

* Just fully revert the protobuf changes

* Add to CHANGELOG
2022-06-29 13:18:14 +01:00
Fraser Waters cba1880e21
Send smart aliases via gRPC to engine ()
* Send smart aliases via gRPC to engine

* Add to SupportsFeature

* Restore old logic when the engine doesn't support smartAliases

* Add to deploytest ResourceOptions

* Add tests

* Add to CHANGELOG

* Fix test

* Rename proto fields

* Regenerate protobufs

* Fix up SDKs after field rename

* Rename deploytest aliases

* Rename internal fields

* Fix typo in c# code

* Fix typescript

* Rename feature to aliasSpecs

* Rename type to Spec
2022-06-16 22:07:28 +01:00
Fraser Waters 2b8e3ae94b
Warn about missing AdditionalSecretOutputs ()
* Warn about missing AdditionalSecretOutputs

* Add to CHANGELOG

* Move id check to StepExecutor
2022-06-06 17:34:04 +01:00
Fraser Waters 32dcaa58d6
Monitor.Invoke and Provider.Invoke take different arguments () 2022-04-14 10:59:46 +01:00
Fraser Waters 5d91f8f322
Add RetainOnDelete resource option ()
* Plumb in basics of retainOnDelete

* Add test

* Make test pass

* Add to changelog

* Add to API list

* lint

* Add semicolon

* Fix Infof call

* Fix method call

* new delete mode work

* cleanup

* protectTest

* Fix up test

* Fix replace

* Fix up test

* Warn on drop

* lint

* Change to just a bool flag

* Regenerate proto

* Rework to just a bool flag with no error

* Remove old comment

* Fix C# typo

* rm extra space

* Add missing semicolon

* Reformat python

* False typo

* Fix typo in js function name

* Reword docs

* lint

* Read doesn't need retainOnDelete
2022-02-16 22:11:12 +00:00
Ian Wahbe 8e5e4caf6e
Pipe serverURL through register resource ()
* [engine] Pipe serverURL through register resource

* Fix lint

* Thread serverURL through default provider calls

* Change tag from "serverURL" to "pluginDownloadURL"

* Update CHANGELOG_PENDING.md

* Allow provider to be null

* Fix tests

* Include server url passthrough in test

* Fix parseProviderRequest

* Add test for url pass through

* Fix lint

* Correct small nits from @justinp

* Add test for default providers

* Move special helpers to providers

* Partial conversion serverURL -> pluginDownloadURL

* Remove serverURL

* Remove more serverURL instances

* const correctness

* Add url to ProviderRequest.Name()

I also canonicalize the url by removing any trailing '/'

* Fix typo + lint

* Add test for url canonicalization

* Fix ProviderRequest.Name for version=nil
2021-12-17 14:52:01 -08:00
Ian Wahbe 272c4643b2
Update error handling ()
This is the result of a change applied via `go-rewrap-errors`.
2021-11-12 18:37:17 -08:00
Pat Gavlin e6d95a8aa9
[engine/test] Add a repro for . ()
This test is derived from a user-reported issue rooted in dependencies
on remote comoponents.
2021-09-29 16:05:45 -07:00
Justin Van Patten 343bc4c778
Add methods lifecycle tests () 2021-07-12 15:51:11 -07:00
Luke Hoban eb32039013
Add `replaceOnChanges` resource option ()
Adds a new resource option to force replacement when certain properties report changes, even if the resource provider itself does not require a replacement.

Fixes .

Co-authored-by: Levi Blackstone <levi@pulumi.com>
2021-07-01 13:32:08 -06:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Pat Gavlin 9b6a7a4397
Improve resource ref unit tests. ()
- Add component ref coverage to the existing test
- Add coverage for a downlevel SDK communicating with an engine that
  supports resource refs
- Add coverage for a downlevel engine communicating with an SDK that
  supports resource refs

As part of improving coverage, these changes add a knob to explicitly
disable resource refs in the engine without the use of the environment
variable. The environment variable is now only read by the CLI, and has
been restored to its prior polarity (i.e. `PULUMI_ENABLE_RESOURCE_REFERENCES`).
2020-12-16 12:38:20 -08:00
Pat Gavlin 3d2e31289a
Add support for serialized resource references. ()
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.

This is the foundation of cross-process resources.

Related to .

Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
2020-10-27 10:12:12 -07:00
Pat Gavlin 855f1fd1cd
Revise host mode. ()
* Revise host mode.

The current implementation of host mode uses a `pulumi host` command and
an ad-hoc communication protocol between the engine and client to
connect a language host after the host has begun listening. The most
significant disadvantages of this approach are the communication
protocol (which currently requires the use of stdout), the host-specific
command, and the difficulty of accommodating the typical program-bound
lifetime for an update.

These changes reimplement host mode by adding engine support for
connecting to an existing language runtime service rather than launching
a plugin. This capability is provided via an engine-specific language
runtime, `client`, which accepts the address of the existing languge
runtime service as a runtime option. The CLI exposes this runtime via
the `--client` flag to the `up` and `preview` commands, which similarly
accepts the address of an existing language runtime service as an
argument. These changes also adjust the automation API to consume the
new host mode implementation.
2020-09-14 17:40:17 -07:00
Pat Gavlin 2585b86aa4
Initial support for remote component construction. ()
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.

Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.

The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.

These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).

An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.

This is the core of .
2020-09-07 19:33:55 -07:00
CyrusNajmabadi 66bd3f4aa8
Breaking changes due to Feature 2.0 work
* Make `async:true` the default for `invoke` calls ()

* Switch away from native grpc impl. ()

* Remove usage of the 'deasync' library from @pulumi/pulumi. ()

* Only retry as long as we get unavailable back.  Anything else continues. ()

* Handle all errors for now. ()


* Do not assume --yes was present when using pulumi in non-interactive mode ()

* Upgrade all paths for sdk and pkg to v2

* Backport C# invoke classes and other recent gen changes ()

Adjust C# generation

* Replace IDeployment with a sealed class ()

Replace IDeployment with a sealed class

* .NET: default to args subtype rather than Args.Empty ()

* Adding system namespace for Dotnet code gen

This is required for using Obsolute attributes for deprecations

```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```

* Fix the nullability of config type properties in C# codegen ()
2020-04-14 09:30:25 +01:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00
evanboyle dfab571aac move pkg/resource/plugin -> sdk/go/common/resource/plugin 2020-03-18 14:26:24 -07:00
evanboyle fba783caf9 move pkg/resource -> sdk/go/common/resource, but leave nested resource packages 2020-03-18 13:36:19 -07:00
Pat Gavlin 137fd54f1c
Propagate inputs to outputs during preview. ()
These changes restore a more-correct version of the behavior that was
disabled with . The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes .
2019-11-11 12:09:34 -08:00
Pat Gavlin 834e583c95
Revert "Propagate inputs to outputs during preview. ()" ()
This reverts commit 80504bf0bc.
2019-10-10 10:33:05 -07:00
Pat Gavlin 80504bf0bc
Propagate inputs to outputs during preview. ()
These changes restore a more-correct version of the behavior that was
disabled with . The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes .
2019-09-30 11:03:58 -07:00
Pat Gavlin 8745440c1b
Allow users to explicitly disable delete-before-replace. ()
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
2019-08-20 15:51:02 -07:00
Pat Gavlin c6916051f0
Use a bag for misc. resource options in deploytest ()
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
2019-07-25 11:18:40 -07:00
Paul Stack 02ffff8840
Addition of Custom Timeouts ()
* Plumbing the custom timeouts from the engine to the providers

* Plumbing the CustomTimeouts through to the engine and adding test to show this

* Change the provider proto to include individual timeouts

* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface

* Change how the CustomTimeouts are sent across RPC

These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest

```
req=&pulumirpc.RegisterResourceRequest{
           Type:                    "aws:s3/bucket:Bucket",
           Name:                    "my-bucket",
           Parent:                  "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
           Custom:                  true,
           Object:                  &structpb.Struct{},
           Protect:                 false,
           Dependencies:            nil,
           Provider:                "",
           PropertyDependencies:    {},
           DeleteBeforeReplace:     false,
           Version:                 "",
           IgnoreChanges:           nil,
           AcceptSecrets:           true,
           AdditionalSecretOutputs: nil,
           Aliases:                 nil,
           CustomTimeouts:          &pulumirpc.RegisterResourceRequest_CustomTimeouts{
               Create:               300,
               Update:               400,
               Delete:               500,
               XXX_NoUnkeyedLiteral: struct {}{},
               XXX_unrecognized:     nil,
               XXX_sizecache:        0,
           },
           XXX_NoUnkeyedLiteral: struct {}{},
           XXX_unrecognized:     nil,
           XXX_sizecache:        0,
       }
```

* Changing the design to use strings

* CHANGELOG entry to include the CustomTimeouts work

* Changing custom timeouts to be passed around the engine as converted value

We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-16 00:26:28 +03:00
Pat Gavlin e1a52693dc
Add support for importing existing resources. ()
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.

There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
   resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
   resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
   different from the ID to import. In this case, the new resource is
   imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
   the same as the ID to import. In this case, the import ID is ignored
   and the resource is treated as it would be in all cases except for
   changes that would replace the resource. In that case, the step
   generator issues an error that indicates that the import ID should be
   removed: were we to move forward with the replace, the new state of
   the stack would fall under case (3), which is almost certainly not
   what the user intends.

Fixes .
2019-07-12 11:12:01 -07:00
Luke Hoban 15e924b5cf
Support aliases for renaming, re-typing, or re-parenting resources ()
Adds a new resource option `aliases` which can be used to rename a resource.  When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.

There are two key places this change is implemented. 

The first is the step generator in the engine.  When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource.  That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order).  This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.

The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs.  This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases.  It is similar to how many other resource options are "inherited" implicitly from the parent.

Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
2019-05-31 23:01:01 -07:00
Luke Hoban 0550f71a35
Add an ignoreChanges resource option ()
Fixes .

Adds a new ignoreChanges resource option that allows specifying a list of property names whose values will be ignored during updates. The property values will be used for Create, but will be ignored for purposes of updates, and as a result also cannot trigger replacements.

This is a feature of the Pulumi engine, not of the resource providers, so no new logic is needed in providers to support this feature. Instead, the engine simply replaces the values of input properties in the goal state with old inputs for properties marked as ignoreChanges.

Currently, only top level properties may be specified in ignoreChanges. In the future, this could be extended to support paths to nested properties (including into array elements) with a JSONPath/JMESPath syntax.
2019-04-22 13:54:48 -07:00