Commit Graph

69 Commits

Author SHA1 Message Date
Fraser Waters ca12edfb4e
Send old inputs to Delete ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14115.

This was missed as part of https://github.com/pulumi/pulumi/pull/13139.

Adds a new configure flag (sends_old_inputs_to_delete) which the engine
will now always set to true. If that's set providers can rely on the old
inputs being sent to delete, otherwise they'll get nil.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-13 14:12:26 +00:00
Fraser Waters 183b47bf06
More efficent mapping lookup ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Inspired by a comment from Zaid. This allows providers to return what
providers they have mapping information for without having to marshal
all their mapping data to the engine at the same time, this could save
transmitting a lot of data that the engine might not ever need (for
example if it's not converting code for that specific provider). It also
allows provider to support mulitple mappings.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-21 11:45:07 +00:00
Fraser Waters a254a53206
Pass provider checksums in requests and save to state ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This extends the resource monitor interface with fields for plugin
checksums (on top of the existing plugin version and download url
fields). These fields are threaded through the engine and are persisted
in resource state. The sent or saved data is then used when installing
plugins to ensure that the checksums match what was recorded at the time
the SDK was built.

Similar to https://github.com/pulumi/pulumi/pull/13776 nothing is using
this yet, but this lays the engine side plumbing for them.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-11 15:54:07 +00:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
Fraser Waters 841c99a9dd Send old inputs to diff and update
Fixes https://github.com/pulumi/pulumi/issues/5973

This adds two new properties to the RPC interfaces with providers.
Firstly we pass a new bool flag to Configure to tell the provider if it
will get sent old inputs as part of Diff.
Secondly we now pass the old inputs and the old outputs to Diff and
Update as opposed to just the old outputs.

The flag passed to Configure isn't strictly needed. Providers should be
able to tell that the old inputs aren't being sent to Diff and Updae by
virtue of them being `null/nil/None` rather than an empty map. For
DiffConfig this is the only way to detect this case because it's called
before Configure. But its still useful to send this explictly to
Configure to allow providers to return a configure error that they need
to be ran against a newer engine version.

There is one fairly major failure case that can happen with this feature
and providers who update to make use of this feature should ensure they
call this out in their changelog.
That is if a provider has been saving old inputs in outputs as a way to
get access to them in diff, and then updates to use these new protocol
properties instead and stops saving the inputs in outputs _and then_ the
user downgrades back to an old provider version things will probably get
_very_ confused.
2023-06-21 21:04:50 +01:00
Abhinav Gupta 880e22f66d
proto/ConstructRequest: Add resource options fields
Adds additional resource options fields to ConstructRequest
and regenerates proto files.

These will be plumbed from the SDK down to remote providers.
2023-04-21 10:57:19 -07:00
Fraser Waters fe2c6c0515 Support subdirectories in proto 2023-03-05 16:05:58 +00:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Aaron Friel 9f9989af97 Update Go gRPC, require impl to embed UnimplementedXServer 2023-01-05 18:00:16 -08:00
Fraser Waters 1839e15e4a Phase 3 of the convert mapper 2022-12-01 23:43:43 +00:00
Fraser Waters 14d8b59ea2 First pass of adding the convert mapper 2022-11-29 09:46:44 +00:00
Fraser Waters 920e928886 Fix the go_package option on the .proto files
This doesn't really effect anything with our current usage, but if we
ever put proto files in another package and try to import the current
set it wouldn't actually be able to find them.

I noticed this while working on
https://github.com/pulumi/pulumi/pull/10792 where I added a new
"pulumirpc.engine" package. The proto file could refer to things like
"pulumirpc.PluginDependency" but the go code then tried to import it
like `_go "github.com/pulumi/pulumi/v3/proto/go/pulumirpc"` which isn't
actually the correct import path (missing the sdk folder part, and there
isn't acutally a folder called pulumirpc).
2022-09-20 09:31:03 +01:00
Fraser Waters 615ae91477
Add getOrganization to nodejs and python ()
* Add getOrganization to nodejs and python

* Add to CHANGELOG

* lint

* lint

* format

* Fix python test

* Fix nodejs settings

* Add contextproperty

* Backcompat python kwargs
2022-08-31 10:33:29 +01:00
Fraser Waters 9dfbee3b2d
Remove sequence numbers ()
* Remove sequenceNumber from protobufs

* Regenerate protobufs

* Remove setting and reading of sequence number in Check

* Remove sequence numbers from state

* Replace sequenceNumber with randomSeed in Check

* Fix tests

* Add to CHANGELOG
2022-07-25 12:08:03 +01:00
Fraser Waters a922d51a7f
Add random seed to Check ()
* Add randomSeed to check

* Regenerate protobufs

* Update checksum
2022-07-20 12:25:43 +01:00
Fraser Waters 1beb4f827e
Compile protobufs with "pulumi" namespace ()
* Move proto file to top level

* Regenerate go protobufs

* Regenerate python protobufs

* Fix dotnet protobuf references

* Update README

* Fix up nodejs protobufs

* Add to CHANGELOG

* lint

* Remove google/protobuf/status

* Revert server changes

* Restore pyi files

* Revert "Remove google/protobuf/status" for nodejs

This reverts commit d692f00bd2.
2022-07-12 14:45:03 +01:00
Fraser Waters d56ba6c918
Update protoc-gen-go to 1.5.2 ()
* Update protoc-gen-go to 1.5.2

* lint
2022-06-11 08:52:24 +01:00
Fraser Waters 54497ce32b
Update protoc ()
* Update protoc

* Regenerate protobufs

* go mod tidy
2022-05-31 15:41:40 +01:00
Levi Blackstone be26786e1d
Update docstring for Cancel gRPC method ()
Add a more comprehensive description about the semantics of the provider Cancel operation. This description was copied from the pulumi-kubernetes provider where this feature was originally implemented.
2022-05-25 19:17:21 -06:00
Fraser Waters c9ba17d450
Allow attaching to running providers ()
* Add Attach call

* Regenerate grpc

* Start plumbing in changes

* Main doens't need a port

* Split Attach into grpc interface

* Change envvar format

* Type test for attach

* lint

* Reformat python

* Implement provider debug for nodejs

* Fix plugin close

* lint

* Add to CHANGELOG

* Set Kill

Co-authored-by: Daniel Bradley <daniel@pulumi.com>
2022-04-19 12:41:18 +01:00
Fraser Waters 32dcaa58d6
Monitor.Invoke and Provider.Invoke take different arguments () 2022-04-14 10:59:46 +01:00
Fraser Waters ec3ef44841
Make resource autonames determinstic ()
* Start adding SequenceNumber

* Start adding sequence number to state

* New generate functions

* notes

* Don't increment if unknown

* Deterministic name test

* Check replace

* typo

* lint

* Increment on targetted replace

* Some comments and external fixes

* Add test for resetting sequence number after replace

* Reset sequence numbers after replace

* assert check we never pass -1 to check

* Add to dynamic providers

* lint

* Add to changelog
2022-01-20 11:18:54 +00:00
Ian Wahbe 8e5e4caf6e
Pipe serverURL through register resource ()
* [engine] Pipe serverURL through register resource

* Fix lint

* Thread serverURL through default provider calls

* Change tag from "serverURL" to "pluginDownloadURL"

* Update CHANGELOG_PENDING.md

* Allow provider to be null

* Fix tests

* Include server url passthrough in test

* Fix parseProviderRequest

* Add test for url pass through

* Fix lint

* Correct small nits from @justinp

* Add test for default providers

* Move special helpers to providers

* Partial conversion serverURL -> pluginDownloadURL

* Remove serverURL

* Remove more serverURL instances

* const correctness

* Add url to ProviderRequest.Name()

I also canonicalize the url by removing any trailing '/'

* Fix typo + lint

* Add test for url canonicalization

* Fix ProviderRequest.Name for version=nil
2021-12-17 14:52:01 -08:00
Justin Van Patten ed4b53d3ae
Add monitor feature for output values () 2021-09-15 14:16:00 -07:00
Justin Van Patten 84b574f0df
Initial support for resource methods (authoring from Node.js, calling from Python) ()
Adds initial support for resource methods (via a new `Call` gRPC method similar to `Invoke`), with support for authoring methods from Node.js, and calling methods from Python.
2021-06-30 07:48:56 -07:00
Evan Boyle 634e97cd55
Include config secret info in Construct calls () 2021-06-24 15:38:01 -07:00
Pat Gavlin 3d2e31289a
Add support for serialized resource references. ()
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.

This is the foundation of cross-process resources.

Related to .

Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
2020-10-27 10:12:12 -07:00
Pat Gavlin 249140242e
Add support for provider-side preview. ()
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.

If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.

If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.

Fixes .
2020-10-09 13:13:55 -07:00
Pat Gavlin 2585b86aa4
Initial support for remote component construction. ()
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.

Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.

The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.

These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).

An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.

This is the core of .
2020-09-07 19:33:55 -07:00
James Nugent a1b55192dd Regenerate protocol buffers code 2020-02-28 11:53:47 +00:00
Pat Gavlin ccde31b713
Add support for extracting schemas from providers. ()
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
2020-02-27 16:10:47 -08:00
Justin Van Patten 3bf9067bac
Expose options, parent, deps, and provider config to policies () 2020-02-07 16:11:34 -08:00
Justin Van Patten 6c84b008d8
Send resource URN and name to analyzer ()
More information we want to make available to policy packs.
2019-11-21 21:01:15 +00:00
Pat Gavlin 137fd54f1c
Propagate inputs to outputs during preview. ()
These changes restore a more-correct version of the behavior that was
disabled with . The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes .
2019-11-11 12:09:34 -08:00
Alex Clemmer 25d27d09f9 Add `StreamInvoke` to Provider gRPC interface 2019-11-05 10:47:48 -08:00
Chris Smith d2805fcb3f
Add support for aggregate resource analysis ()
* Add AnalyzeStack method to Analyze service

* Protobuf generated code

* Hook up AnalyzeStack method

* Address PR feedback

* Address PR feedback
2019-10-25 08:29:02 -07:00
Pat Gavlin 67ec74bdc5
Pass ignoreChanges to providers. ()
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.

In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.

Fixes , .
2019-07-31 11:39:07 -05:00
Paul Stack 02ffff8840
Addition of Custom Timeouts ()
* Plumbing the custom timeouts from the engine to the providers

* Plumbing the CustomTimeouts through to the engine and adding test to show this

* Change the provider proto to include individual timeouts

* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface

* Change how the CustomTimeouts are sent across RPC

These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest

```
req=&pulumirpc.RegisterResourceRequest{
           Type:                    "aws:s3/bucket:Bucket",
           Name:                    "my-bucket",
           Parent:                  "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
           Custom:                  true,
           Object:                  &structpb.Struct{},
           Protect:                 false,
           Dependencies:            nil,
           Provider:                "",
           PropertyDependencies:    {},
           DeleteBeforeReplace:     false,
           Version:                 "",
           IgnoreChanges:           nil,
           AcceptSecrets:           true,
           AdditionalSecretOutputs: nil,
           Aliases:                 nil,
           CustomTimeouts:          &pulumirpc.RegisterResourceRequest_CustomTimeouts{
               Create:               300,
               Update:               400,
               Delete:               500,
               XXX_NoUnkeyedLiteral: struct {}{},
               XXX_unrecognized:     nil,
               XXX_sizecache:        0,
           },
           XXX_NoUnkeyedLiteral: struct {}{},
           XXX_unrecognized:     nil,
           XXX_sizecache:        0,
       }
```

* Changing the design to use strings

* CHANGELOG entry to include the CustomTimeouts work

* Changing custom timeouts to be passed around the engine as converted value

We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-16 00:26:28 +03:00
Pat Gavlin 6e5c4a38d8
Defer all diffs to resource providers. ()
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.

Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:

- Not all providers do input-input comparison; some do input-state
  comparison
- Not all providers are able to update the last deployed set of inputs
  when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
  in resources whose inputs have not changed

All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.

These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
  before calling a resource provider's Diff function, the engine calls
  the Diff function in all cases.
- Providers may now return a list of properties that differ between the
  requested and actual state and the way in which they differ. This
  information will then be used by the CLI to render the diff
  appropriately. A provider may also indicate that a particular diff is
  between old and new inputs rather than old state and new inputs.

Fixes .
2019-07-01 12:34:19 -07:00
Alex Clemmer 346861b9d6 Update Analyzer PB for policy 2019-06-13 16:04:13 -07:00
Luke Hoban 15e924b5cf
Support aliases for renaming, re-typing, or re-parenting resources ()
Adds a new resource option `aliases` which can be used to rename a resource.  When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.

There are two key places this change is implemented. 

The first is the step generator in the engine.  When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource.  That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order).  This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.

The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs.  This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases.  It is similar to how many other resource options are "inherited" implicitly from the parent.

Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
2019-05-31 23:01:01 -07:00
Matt Ellis f705dde7fb Remove `acceptsSecrets` from InvokeRequest
In our system, we model secrets as outputs with an additional bit of
metadata that says they are secret. For Read and Register resource
calls, our RPC interface says if the client side of the interface can
handle secrets being returned (i.e. the language SDK knows how to
sniff for the special signiture and resolve the output with the
special bit set).

For Invoke, we have no such model. Instead, we return a `Promise<T>`
where T's shape has just regular property fields.  There's no place
for us to tack the secretness onto, since there are no Outputs.

So, for now, don't even return secret values back across the invoke
channel. We can still take them as arguments (which is good) but we
can't even return secrets as part of invoke calls. This is not ideal,
but given the way we model these sources, there's no way around
this.  Fortunately, the result of these invoke calls are not stored in
the checkpoint and since the type is not Output<T> it will be clear
that the underlying value is just present in plaintext. A user that
wants to pass the result of an invoke into a resource can turn an
existing property into a secret via `pulumi.secret`.
2019-05-10 17:07:52 -07:00
Matt Ellis 2ad9afc572 Regnerate gRPC generated code 2019-05-10 17:07:52 -07:00
Alex Clemmer f21da64f96 Add query mode to the language protocol
`pulumi query` requires that language plugins know about "query mode" so
that they don't do things like try to register the default stack
resource.

To communicate that a language host should boot into query mode, we
augment the language plugin protocol to include this information.
2019-05-02 18:08:08 -07:00
Sean Gillespie 47a2acaa7b
Protobuf changes for provider versioning ()
In pursuit of , this commit adds the necessary changes
to the resource monitor protocol so that language hosts can communicate
exactly what version of a provider should be used when servicing an
Invoke, ReadResource, or RegisterResource. The expectation here is that,
if a language host provides a version, the engine MUST use EXACTLY that
version of a provider plugin in order to service the request.
2019-04-16 10:06:43 -07:00
Pat Gavlin 7ebd70a3e6
Refresh inputs ()
These changes take advantage of the newly-added support for returning
inputs from Read to update a resource's inputs as part of a refresh.
As a consequence, the Pulumi engine will now properly detect drift
between the actual state of a resource and the desired state described
by the program and generate appropriate update or replace steps.

As part of these changes, a resource's old inputs are now passed to the
provider when performing a refresh. The provider can take advantage of
this to maintain the accuracy of any additional data or metadata in the
resource's inputs that may need to be updated during the refresh.

This is required for the complete implementation of
https://github.com/pulumi/pulumi-terraform/pull/349. Without access to
the old inputs for a resource, TF-based providers would lose all
information about default population during a refresh.
2019-03-11 13:50:00 -07:00
Pat Gavlin 905e7353e4
Update the provider RPC interface ()
These changes add two new methods to the provider interface and extend
the results of three others.

The new methods are `CheckConfig` and `DiffConfig`, which fill out the
set of methods required for a complete implementation of the
first-class provider design. Though these methods are optional for
backwards compatibility, they should be implemented by all future
providers for the best possible user experience.

The adjusted result types are `DiffResponse`, `ReadResponse`, and
`ErrorResourceInitFailed`. The first has been updated to include a list
of the properties that changed (if any). The latter two now include
an estimated set of inputs for the resource as well as the resource's
state. Together, these three changes enable the engine to determine the
set of inputs that should be specified by a user in order to match those
that describe the resource's current state.

This contributes to , , , and .
2019-03-05 10:49:24 -08:00
Pat Gavlin a222705143
Implement first-class providers. ()
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.

### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.

These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
  or unknown properties. This is necessary because existing plugins
  only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
  values are known or "must replace" if any configuration value is
  unknown. The justification for this behavior is given
  [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
  configures the provider plugin if all config values are known. If any
  config value is unknown, the underlying plugin is not configured and
  the provider may only perform `Check`, `Read`, and `Invoke`, all of
  which return empty results. We justify this behavior becuase it is
  only possible during a preview and provides the best experience we
  can manage with the existing gRPC interface.

### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.

All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.

Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.

### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
  provider plugins. It must be possible to add providers from a stack's
  checkpoint to this map and to register new/updated providers during
  the execution of a plan in response to CRUD operations on provider
  resources.
- In order to support updating existing stacks using existing Pulumi
  programs that may not explicitly instantiate providers, the engine
  must be able to manage the "default" providers for each package
  referenced by a checkpoint or Pulumi program. The configuration for
  a "default" provider is taken from the stack's configuration data.

The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").

The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.

During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.

While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.

### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
  to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
  a particular provider instance to manage a `CustomResource`'s CRUD
  operations.
- A new type, `InvokeOptions`, can be used to specify options that
  control the behavior of a call to `pulumi.runtime.invoke`. This type
  includes a `provider` field that is analogous to
  `ResourceOptions.provider`.
2018-08-06 17:50:29 -07:00
Alex Clemmer 0e39b3c868 Add `Cancel` to gRPC resource provider interface 2018-07-15 11:05:44 -10:00
Sean Gillespie 8b9e24cd85 Allow dynamic-provider to send structured errors
A critical part of the partial update protocol is to return a structured
error when a resource is successfully created, but fails to initialize.
This structured error contains the properties of the
partially-initialized resource, and instructs the engine to halt.

Most languages implement this by attaching "details" to the error, i.e.,
an arbitrary proto message attached to the error. The JavaScript
implementation is not mature enough to include all the facilities
required to use this, so here we must add a `Status` message, which
protobuf requires as part of its structure for returning details.
2018-07-02 13:32:23 -07:00