pulumi/sdk/go/common/resource/plugin/provider_plugin.go

2049 lines
70 KiB
Go
Raw Permalink Normal View History

2018-05-22 19:43:36 +00:00
// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package plugin
import (
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
"context"
"encoding/json"
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
"errors"
"fmt"
2019-10-22 07:20:26 +00:00
"io"
"os"
2022-11-01 15:15:09 +00:00
"strconv"
"strings"
"github.com/blang/semver"
"github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc"
multierror "github.com/hashicorp/go-multierror"
"github.com/opentracing/opentracing-go"
2022-11-01 15:15:09 +00:00
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
"google.golang.org/protobuf/types/known/emptypb"
"google.golang.org/protobuf/types/known/structpb"
"github.com/pulumi/pulumi/sdk/v3/go/common/apitype"
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/promise"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
Move assets and archives to their own package (#15157) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description This PR is motivated by https://github.com/pulumi/pulumi/pull/15145. `resource.*` should be built on top of `property.Value`,[^1] which means that `resource` needs to be able to import `property.Value`, and so `property` cannot import `resource`. Since Assets and Archives are both types of properties, they must be moved out of `resource`. [^1]: For example: https://github.com/pulumi/pulumi/blob/a1d686227cd7e3c70c51bd772450cb0cd57c1479/sdk/go/common/resource/resource_state.go#L35-L36 ## Open Question This PR moves them to their own sub-folders in `resource`. Should `asset` and `archive` live somewhere more high level, like `sdk/go/property/{asset,archive}`? <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-25 20:39:31 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/archive"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/asset"
"github.com/pulumi/pulumi/sdk/v3/go/common/slice"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/logging"
2022-11-01 15:15:09 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/util/rpcutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/rpcutil/rpcerror"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
pulumirpc "github.com/pulumi/pulumi/sdk/v3/proto/go"
)
// The `Type()` for the NodeJS dynamic provider. Logically, this is the same as calling
// providers.MakeProviderType(tokens.Package("pulumi-nodejs")), but does not depend on the providers package
// (a direct dependency would cause a cyclic import issue.
//
// This is needed because we have to handle some buggy behavior that previous versions of this provider implemented.
const nodejsDynamicProviderType = "pulumi:providers:pulumi-nodejs"
// The `Type()` for the Kubernetes provider. Logically, this is the same as calling
// providers.MakeProviderType(tokens.Package("kubernetes")), but does not depend on the providers package
// (a direct dependency would cause a cyclic import issue.
//
// This is needed because we have to handle some buggy behavior that previous versions of this provider implemented.
const kubernetesProviderType = "pulumi:providers:kubernetes"
// provider reflects a resource plugin, loaded dynamically for a single package.
type provider struct {
NotForwardCompatibleProvider
ctx *Context // a plugin context for caching, etc.
pkg tokens.Package // the Pulumi package containing this provider's resources.
plug *plugin // the actual plugin process wrapper.
clientRaw pulumirpc.ResourceProviderClient // the raw provider client; usually unsafe to use directly.
disableProviderPreview bool // true if previews for Create and Update are disabled.
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
legacyPreview bool // enables legacy behavior for unconfigured provider previews.
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
configSource *promise.CompletionSource[pluginConfig] // the source for the provider's configuration.
}
// pluginConfig holds the configuration of the provider
// as specified by the Configure call.
type pluginConfig struct {
known bool // true if all configuration values are known.
acceptSecrets bool // true if this plugin accepts strongly-typed secrets.
acceptResources bool // true if this plugin accepts strongly-typed resource refs.
acceptOutputs bool // true if this plugin accepts output values.
supportsPreview bool // true if this plugin supports previews for Create and Update.
}
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
// Checks PULUMI_DEBUG_PROVIDERS environment variable for any overrides for the provider identified
// by pkg. If the user has requested to attach to a live provider, returns the port number from the
// env var. For example, `PULUMI_DEBUG_PROVIDERS=aws:12345,gcp:678` will result in 12345 for aws.
func GetProviderAttachPort(pkg tokens.Package) (*int, error) {
var optAttach string
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
if providersEnvVar, has := os.LookupEnv("PULUMI_DEBUG_PROVIDERS"); has {
for _, provider := range strings.Split(providersEnvVar, ",") {
parts := strings.SplitN(provider, ":", 2)
if parts[0] == pkg.String() {
optAttach = parts[1]
break
}
}
}
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
if optAttach == "" {
return nil, nil
}
port, err := strconv.Atoi(optAttach)
if err != nil {
return nil, fmt.Errorf("Expected a numeric port, got %s in PULUMI_DEBUG_PROVIDERS: %w",
optAttach, err)
}
return &port, nil
}
// NewProvider attempts to bind to a given package's resource plugin and then creates a gRPC connection to it. If the
// plugin could not be found, or an error occurs while creating the child process, an error is returned.
func NewProvider(host Host, ctx *Context, pkg tokens.Package, version *semver.Version,
options map[string]interface{}, disableProviderPreview bool, jsonConfig string,
) (Provider, error) {
// See if this is a provider we just want to attach to
var plug *plugin
attachPort, err := GetProviderAttachPort(pkg)
if err != nil {
return nil, err
}
prefix := fmt.Sprintf("%v (resource)", pkg)
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
if attachPort != nil {
port := *attachPort
2022-11-01 15:15:09 +00:00
conn, err := dialPlugin(port, pkg.String(), prefix, providerPluginDialOptions(ctx, pkg, ""))
if err != nil {
return nil, err
}
// Done; store the connection and return the plugin info.
plug = &plugin{
Conn: conn,
// Nothing to kill
Kill: func() error { return nil },
}
} else {
// Load the plugin's path by using the standard workspace logic.
path, err := workspace.GetPluginPath(ctx.Diag,
apitype.ResourcePlugin, strings.ReplaceAll(string(pkg), tokens.QNameDelimiter, "_"),
version, host.GetProjectPlugins())
if err != nil {
return nil, err
}
contract.Assertf(path != "", "unexpected empty path for plugin %s", pkg)
// Runtime options are passed as environment variables to the provider, this is _currently_ used by
// dynamic providers to do things like lookup the virtual environment to use.
env := os.Environ()
for k, v := range options {
env = append(env, fmt.Sprintf("PULUMI_RUNTIME_%s=%v", strings.ToUpper(k), v))
}
if jsonConfig != "" {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
env = append(env, "PULUMI_CONFIG="+jsonConfig)
}
plug, err = newPlugin(ctx, ctx.Pwd, path, prefix,
apitype.ResourcePlugin, []string{host.ServerAddr()}, env, providerPluginDialOptions(ctx, pkg, ""))
if err != nil {
return nil, err
}
}
contract.Assertf(plug != nil, "unexpected nil resource plugin for %s", pkg)
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
legacyPreview := cmdutil.IsTruthy(os.Getenv("PULUMI_LEGACY_PROVIDER_PREVIEW"))
p := &provider{
ctx: ctx,
pkg: pkg,
plug: plug,
clientRaw: pulumirpc.NewResourceProviderClient(plug.Conn),
disableProviderPreview: disableProviderPreview,
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
legacyPreview: legacyPreview,
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
configSource: &promise.CompletionSource[pluginConfig]{},
}
// If we just attached (i.e. plugin bin is nil) we need to call attach
if plug.Bin == "" {
err := p.Attach(host.ServerAddr())
if err != nil {
return nil, err
}
}
return p, nil
}
2022-11-01 15:15:09 +00:00
func providerPluginDialOptions(ctx *Context, pkg tokens.Package, path string) []grpc.DialOption {
dialOpts := append(
rpcutil.OpenTracingInterceptorDialOptions(otgrpc.SpanDecorator(decorateProviderSpans)),
grpc.WithTransportCredentials(insecure.NewCredentials()),
2022-11-01 15:15:09 +00:00
rpcutil.GrpcChannelOptions(),
)
if ctx.DialOptions != nil {
metadata := map[string]interface{}{
"mode": "client",
"kind": "resource",
}
if pkg != "" {
metadata["name"] = pkg.String()
}
if path != "" {
metadata["path"] = path
}
dialOpts = append(dialOpts, ctx.DialOptions(metadata)...)
}
return dialOpts
}
// NewProviderFromPath creates a new provider by loading the plugin binary located at `path`.
2022-09-14 14:23:03 +00:00
func NewProviderFromPath(host Host, ctx *Context, path string) (Provider, error) {
env := os.Environ()
2022-11-01 15:15:09 +00:00
2022-09-14 14:23:03 +00:00
plug, err := newPlugin(ctx, ctx.Pwd, path, "",
apitype.ResourcePlugin, []string{host.ServerAddr()}, env, providerPluginDialOptions(ctx, "", path))
2022-09-14 14:23:03 +00:00
if err != nil {
return nil, err
}
contract.Assertf(plug != nil, "unexpected nil resource plugin at %q", path)
legacyPreview := cmdutil.IsTruthy(os.Getenv("PULUMI_LEGACY_PROVIDER_PREVIEW"))
p := &provider{
ctx: ctx,
plug: plug,
clientRaw: pulumirpc.NewResourceProviderClient(plug.Conn),
legacyPreview: legacyPreview,
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
configSource: &promise.CompletionSource[pluginConfig]{},
2022-09-14 14:23:03 +00:00
}
// If we just attached (i.e. plugin bin is nil) we need to call attach
if plug.Bin == "" {
err := p.Attach(host.ServerAddr())
if err != nil {
return nil, err
}
}
return p, nil
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
func NewProviderWithClient(ctx *Context, pkg tokens.Package, client pulumirpc.ResourceProviderClient,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
disableProviderPreview bool,
) Provider {
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
return &provider{
ctx: ctx,
pkg: pkg,
clientRaw: client,
disableProviderPreview: disableProviderPreview,
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
configSource: &promise.CompletionSource[pluginConfig]{},
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
}
}
func (p *provider) Pkg() tokens.Package { return p.pkg }
// label returns a base label for tracing functions.
func (p *provider) label() string {
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
return fmt.Sprintf("Provider[%s, %p]", p.pkg, p)
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
func (p *provider) requestContext() context.Context {
if p.ctx == nil {
return context.Background()
}
return p.ctx.Request()
}
// isDiffCheckConfigLogicallyUnimplemented returns true when an rpcerror.Error should be treated as if it was an error
// due to a rpc being unimplemented. Due to past mistakes, different providers returned "Unimplemented" in a variaity of
// different ways that don't always result in an Uimplemented error code.
func isDiffCheckConfigLogicallyUnimplemented(err *rpcerror.Error, providerType tokens.Type) bool {
switch string(providerType) {
// The NodeJS dynamic provider implementation incorrectly returned an empty message instead of properly implementing
// Diff/CheckConfig. This gets turned into a error with type: "Internal".
case nodejsDynamicProviderType:
if err.Code() == codes.Internal {
logging.V(8).Infof("treating error %s as unimplemented error", err)
return true
}
// The Kubernetes provider returned an "Unimplmeneted" message, but it did so by returning a status from a different
// package that the provider was expected. That caused the error to be wrapped with an "Unknown" error.
case kubernetesProviderType:
if err.Code() == codes.Unknown && strings.Contains(err.Message(), "Unimplemented") {
logging.V(8).Infof("treating error %s as unimplemented error", err)
return true
}
}
return false
}
func (p *provider) Parameterize(ctx context.Context, request ParameterizeRequest) (ParameterizeResponse, error) {
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
var params pulumirpc.ParameterizeRequest
switch p := request.Parameters.(type) {
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
case ParameterizeArgs:
params.Parameters = &pulumirpc.ParameterizeRequest_Args{
Args: &pulumirpc.ParameterizeRequest_ParametersArgs{
Args: p.Args,
},
}
case ParameterizeValue:
var version string
if p.Version != nil {
version = p.Version.String()
}
value, err := structpb.NewValue(p.Value)
if err != nil {
return ParameterizeResponse{}, err
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
}
params.Parameters = &pulumirpc.ParameterizeRequest_Value{
Value: &pulumirpc.ParameterizeRequest_ParametersValue{
Name: p.Name,
Version: version,
Value: value,
},
}
case nil:
// No args present. That should be Ok.
default:
panic(fmt.Sprintf("Impossible - type is constrained to ParameterizeArgs or ParameterizeValue, found %T", p))
}
resp, err := p.clientRaw.Parameterize(p.requestContext(), &params)
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
if err != nil {
return ParameterizeResponse{}, err
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
}
var version *semver.Version
if resp.Version != "" {
v, err := semver.Parse(resp.Version)
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
if err != nil {
return ParameterizeResponse{}, err
Abstract plugin.Parameterize away from the gRPC interface (#16283) `plugin.Provider` exists to provide a low-level abstraction *on top of* the gRPC interface. It should not expose the gRPC types directly. The key diff in this commit is: ```patch - Parameterize( - ctx context.Context, req *pulumirpc.ParameterizeRequest, - ) (*pulumirpc.ParameterizeResponse, error) + Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error) ``` ```patch +type ParameterizeParameters interface { + isParameterizeParameters() +} + +type ( + ParameterizeArgs struct { + Args []string + } + + ParameterizeValue struct { + Name string + Version *semver.Version + // Value must be one of: + // - nil + // - bool + // - int, int32, int64 + // - uint, uint32, uint64 + // - float32, float64 + // - string + // - []byte + // - map[string]interface{} + // - []interface{} + Value any + } +) + +func (ParameterizeArgs) isParameterizeParameters() {} +func (ParameterizeValue) isParameterizeParameters() {} ``` This is the new interface exposed in `plugin`. The rest of the PR is simply complying with the new interface. While this change is technically breaking (since it was released in v3.116.0), its a heavily experimental feature and the providers team (my team) is the biggest consumer of `plugin`. Before this PR, `Parameterize` was the *only* method that dirrectly exposed the gRPC interface it was "abstracting". <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> ## Checklist - [X] I have run `make tidy` to update any new dependencies - [X] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-30 03:17:49 +00:00
}
version = &v
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
}
return ParameterizeResponse{Name: resp.Name, Version: version}, err
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
}
// GetSchema fetches the schema for this resource provider, if any.
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
func (p *provider) GetSchema(request GetSchemaRequest) ([]byte, error) {
var subpackageVersion string
if request.SubpackageVersion != nil {
subpackageVersion = request.SubpackageVersion.String()
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := p.clientRaw.GetSchema(p.requestContext(), &pulumirpc.GetSchemaRequest{
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
Version: int32(request.Version),
SubpackageName: request.SubpackageName,
SubpackageVersion: subpackageVersion,
})
if err != nil {
return nil, err
}
return []byte(resp.GetSchema()), nil
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// CheckConfig validates the configuration for this resource provider.
func (p *provider) CheckConfig(urn resource.URN, olds,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
news resource.PropertyMap, allowUnknowns bool,
) (resource.PropertyMap, []CheckFailure, error) {
label := fmt.Sprintf("%s.CheckConfig(%s)", p.label(), urn)
logging.V(7).Infof("%s executing (#olds=%d,#news=%d)", label, len(olds), len(news))
molds, err := MarshalProperties(olds, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".olds",
KeepUnknowns: allowUnknowns,
})
if err != nil {
return nil, nil, err
}
mnews, err := MarshalProperties(news, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".news",
KeepUnknowns: allowUnknowns,
})
if err != nil {
return nil, nil, err
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := p.clientRaw.CheckConfig(p.requestContext(), &pulumirpc.CheckRequest{
Urn: string(urn),
Olds: molds,
News: mnews,
})
if err != nil {
rpcError := rpcerror.Convert(err)
code := rpcError.Code()
if code == codes.Unimplemented || isDiffCheckConfigLogicallyUnimplemented(rpcError, urn.Type()) {
// For backwards compatibility, just return the news as if the provider was okay with them.
logging.V(7).Infof("%s unimplemented rpc: returning news as is", label)
return news, nil, nil
}
logging.V(8).Infof("%s provider received rpc error `%s`: `%s`", label, rpcError.Code(),
rpcError.Message())
return nil, nil, err
}
// Unmarshal the provider inputs.
var inputs resource.PropertyMap
if ins := resp.GetInputs(); ins != nil {
inputs, err = UnmarshalProperties(ins, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".inputs",
KeepUnknowns: allowUnknowns,
RejectUnknowns: !allowUnknowns,
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
return nil, nil, err
}
}
// And now any properties that failed verification.
failures := slice.Prealloc[CheckFailure](len(resp.GetFailures()))
for _, failure := range resp.GetFailures() {
failures = append(failures, CheckFailure{resource.PropertyKey(failure.Property), failure.Reason})
}
// Copy over any secret annotations, since we could not pass any to the provider, and return.
annotateSecrets(inputs, news)
logging.V(7).Infof("%s success: inputs=#%d failures=#%d", label, len(inputs), len(failures))
return inputs, failures, nil
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
}
Defer all diffs to resource providers. (#2849) Thse changes make a subtle but critical adjustment to the process the Pulumi engine uses to determine whether or not a difference exists between a resource's actual and desired states, and adjusts the way this difference is calculated and displayed accordingly. Today, the Pulumi engine get the first chance to decide whether or not there is a difference between a resource's actual and desired states. It does this by comparing the current set of inputs for a resource (i.e. the inputs from the running Pulumi program) with the last set of inputs used to update the resource. If there is no difference between the old and new inputs, the engine decides that no change is necessary without consulting the resource's provider. Only if there are changes does the engine consult the resource's provider for more information about the difference. This can be problematic for a number of reasons: - Not all providers do input-input comparison; some do input-state comparison - Not all providers are able to update the last deployed set of inputs when performing a refresh - Some providers--either intentionally or due to bugs--may see changes in resources whose inputs have not changed All of these situations are confusing at the very least, and the first is problematic with respect to correctness. Furthermore, the display code only renders diffs it observes rather than rendering the diffs observed by the provider, which can obscure the actual changes detected at runtime. These changes address both of these issues: - Rather than comparing the current inputs against the last inputs before calling a resource provider's Diff function, the engine calls the Diff function in all cases. - Providers may now return a list of properties that differ between the requested and actual state and the way in which they differ. This information will then be used by the CLI to render the diff appropriately. A provider may also indicate that a particular diff is between old and new inputs rather than old state and new inputs. Fixes #2453.
2019-07-01 19:34:19 +00:00
func decodeDetailedDiff(resp *pulumirpc.DiffResponse) map[string]PropertyDiff {
if !resp.GetHasDetailedDiff() {
return nil
}
detailedDiff := make(map[string]PropertyDiff)
for k, v := range resp.GetDetailedDiff() {
var d DiffKind
switch v.GetKind() {
case pulumirpc.PropertyDiff_ADD:
d = DiffAdd
case pulumirpc.PropertyDiff_ADD_REPLACE:
d = DiffAddReplace
case pulumirpc.PropertyDiff_DELETE:
d = DiffDelete
case pulumirpc.PropertyDiff_DELETE_REPLACE:
d = DiffDeleteReplace
case pulumirpc.PropertyDiff_UPDATE:
d = DiffUpdate
case pulumirpc.PropertyDiff_UPDATE_REPLACE:
d = DiffUpdateReplace
default:
// Consider unknown diff kinds to be simple updates.
d = DiffUpdate
}
detailedDiff[k] = PropertyDiff{
Kind: d,
InputDiff: v.GetInputDiff(),
}
}
return detailedDiff
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// DiffConfig checks what impacts a hypothetical change to this provider's configuration will have on the provider.
func (p *provider) DiffConfig(urn resource.URN, oldInputs, oldOutputs, newInputs resource.PropertyMap,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
allowUnknowns bool, ignoreChanges []string,
) (DiffResult, error) {
label := fmt.Sprintf("%s.DiffConfig(%s)", p.label(), urn)
logging.V(7).Infof("%s: executing (#oldInputs=%d#oldOutputs=%d,#newInputs=%d)",
label, len(oldInputs), len(oldOutputs), len(newInputs))
mOldInputs, err := MarshalProperties(oldInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldInputs",
KeepUnknowns: true,
})
if err != nil {
return DiffResult{}, err
}
mOldOutputs, err := MarshalProperties(oldOutputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldOutputs",
KeepUnknowns: true,
})
if err != nil {
return DiffResult{}, err
}
mNewInputs, err := MarshalProperties(newInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".newInputs",
KeepUnknowns: true,
})
if err != nil {
return DiffResult{}, err
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := p.clientRaw.DiffConfig(p.requestContext(), &pulumirpc.DiffRequest{
Urn: string(urn),
OldInputs: mOldInputs,
Olds: mOldOutputs,
News: mNewInputs,
IgnoreChanges: ignoreChanges,
})
if err != nil {
rpcError := rpcerror.Convert(err)
code := rpcError.Code()
if code == codes.Unimplemented || isDiffCheckConfigLogicallyUnimplemented(rpcError, urn.Type()) {
logging.V(7).Infof("%s unimplemented rpc: returning DiffUnknown with no replaces", label)
// In this case, the provider plugin did not implement this and we have to provide some answer:
//
// There are two interesting scenarios with the present gRPC interface:
// 1. Configuration differences in which all properties are known
// 2. Configuration differences in which some new property is unknown.
//
// In both cases, we return a diff result that indicates that the provider _should not_ be replaced.
// Although this decision is not conservative--indeed, the conservative decision would be to always require
// replacement of a provider if any input has changed--we believe that it results in the best possible user
// experience for providers that do not implement DiffConfig functionality. If we took the conservative
// route here, any change to a provider's configuration (no matter how inconsequential) would cause all of
// its resources to be replaced. This is clearly a bad experience, and differs from how things worked prior
// to first-class providers.
return DiffResult{Changes: DiffUnknown, ReplaceKeys: nil}, nil
}
logging.V(8).Infof("%s provider received rpc error `%s`: `%s`", label, rpcError.Code(),
rpcError.Message())
// https://github.com/pulumi/pulumi/issues/14529: Old versions of kubernetes would error on this
// call if "kubeconfig" was set to a file. This didn't cause issues later when the same config was
// passed to Configure, and for many years silently "worked".
// https://github.com/pulumi/pulumi/pull/14436 fixed this method to start returning errors which
// exposed this issue with the kubernetes provider, new versions will be fixed to not error on
// this (https://github.com/pulumi/pulumi-kubernetes/issues/2663) but so that the CLI continues to
// work for old versions we have an explicit ignore for this one error here.
if p.pkg == "kubernetes" &&
strings.Contains(rpcError.Error(), "cannot unmarshal string into Go value of type struct") {
logging.V(8).Infof("%s ignoring error from kubernetes provider", label)
return DiffResult{Changes: DiffUnknown}, nil
}
Return errors from DiffConfig (#14436) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14342. I suspect this has just been a long standing typo. I can't see any reason to ignore this error. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-30 09:28:17 +00:00
return DiffResult{}, err
}
replaces := slice.Prealloc[resource.PropertyKey](len(resp.GetReplaces()))
for _, replace := range resp.GetReplaces() {
replaces = append(replaces, resource.PropertyKey(replace))
}
stables := slice.Prealloc[resource.PropertyKey](len(resp.GetStables()))
for _, stable := range resp.GetStables() {
stables = append(stables, resource.PropertyKey(stable))
}
diffs := slice.Prealloc[resource.PropertyKey](len(resp.GetDiffs()))
for _, diff := range resp.GetDiffs() {
diffs = append(diffs, resource.PropertyKey(diff))
}
changes := resp.GetChanges()
deleteBeforeReplace := resp.GetDeleteBeforeReplace()
logging.V(7).Infof("%s success: changes=%d #replaces=%v #stables=%v delbefrepl=%v, diffs=#%v",
label, changes, replaces, stables, deleteBeforeReplace, diffs)
return DiffResult{
Changes: DiffChanges(changes),
ReplaceKeys: replaces,
StableKeys: stables,
ChangedKeys: diffs,
Defer all diffs to resource providers. (#2849) Thse changes make a subtle but critical adjustment to the process the Pulumi engine uses to determine whether or not a difference exists between a resource's actual and desired states, and adjusts the way this difference is calculated and displayed accordingly. Today, the Pulumi engine get the first chance to decide whether or not there is a difference between a resource's actual and desired states. It does this by comparing the current set of inputs for a resource (i.e. the inputs from the running Pulumi program) with the last set of inputs used to update the resource. If there is no difference between the old and new inputs, the engine decides that no change is necessary without consulting the resource's provider. Only if there are changes does the engine consult the resource's provider for more information about the difference. This can be problematic for a number of reasons: - Not all providers do input-input comparison; some do input-state comparison - Not all providers are able to update the last deployed set of inputs when performing a refresh - Some providers--either intentionally or due to bugs--may see changes in resources whose inputs have not changed All of these situations are confusing at the very least, and the first is problematic with respect to correctness. Furthermore, the display code only renders diffs it observes rather than rendering the diffs observed by the provider, which can obscure the actual changes detected at runtime. These changes address both of these issues: - Rather than comparing the current inputs against the last inputs before calling a resource provider's Diff function, the engine calls the Diff function in all cases. - Providers may now return a list of properties that differ between the requested and actual state and the way in which they differ. This information will then be used by the CLI to render the diff appropriately. A provider may also indicate that a particular diff is between old and new inputs rather than old state and new inputs. Fixes #2453.
2019-07-01 19:34:19 +00:00
DetailedDiff: decodeDetailedDiff(resp),
DeleteBeforeReplace: deleteBeforeReplace,
}, nil
}
// annotateSecrets copies the "secretness" from the ins to the outs. If there are values with the same keys for the
// outs and the ins, if they are both objects, they are transformed recursively. Otherwise, if the value in the ins
// contains a secret, the entire out value is marked as a secret. This is very close to how we project secrets
// in the programming model, with one small difference, which is how we treat the case where both are objects. In the
// programming model, we would say the entire output object is a secret. Here, we actually recur in. We do this because
// we don't want a single secret value in a rich structure to taint the entire object. Doing so would mean things like
// the entire value in the deployment would be encrypted instead of a small chunk. It also means the entire property
// would be displayed as `[secret]` in the CLI instead of a small part.
//
// NOTE: This means that for an array, if any value in the input version is a secret, the entire output array is
// marked as a secret. This is actually a very nice result, because often arrays are treated like sets by providers
// and the order may not be preserved across an operation. This means we do end up encrypting the entire array
// but that's better than accidentally leaking a value which just moved to a different location.
func annotateSecrets(outs, ins resource.PropertyMap) {
if outs == nil || ins == nil {
return
}
for key, inValue := range ins {
outValue, has := outs[key]
if !has {
continue
}
if outValue.IsObject() && inValue.IsObject() {
annotateSecrets(outValue.ObjectValue(), inValue.ObjectValue())
} else if !outValue.IsSecret() && inValue.ContainsSecrets() {
outs[key] = resource.MakeSecret(outValue)
}
}
}
func removeSecrets(v resource.PropertyValue) interface{} {
switch {
case v.IsNull():
return nil
case v.IsBool():
return v.BoolValue()
case v.IsNumber():
return v.NumberValue()
case v.IsString():
return v.StringValue()
case v.IsArray():
arr := []interface{}{}
for _, v := range v.ArrayValue() {
arr = append(arr, removeSecrets(v))
}
return arr
case v.IsAsset():
return v.AssetValue()
case v.IsArchive():
return v.ArchiveValue()
case v.IsComputed():
return v.Input()
case v.IsOutput():
return v.OutputValue()
case v.IsSecret():
return removeSecrets(v.SecretValue().Element)
default:
contract.Assertf(v.IsObject(), "v is not Object '%v' instead", v.TypeString())
obj := map[string]interface{}{}
for k, v := range v.ObjectValue() {
obj[string(k)] = removeSecrets(v)
}
return obj
}
}
[cli/engine] Restore elided asset contents from returned inputs from Read operations (#14078) # Description When we send inputs to `Read` operations due to `pulumi refresh` we skip sending asset contents. However, `Read` implementation might return the asset-valued input properties unchanged (without their asset contents) and the engine will simply write out the empty assets to the state causing an invalid shape of asset in the inputs of resources. This PR implements a function `restoreElidedAssetContents` and uses it in the `Read` implementation on the new inputs and new state returned from the provider plugins. Testing the CLI locally using this fix, it does correctly write out the asset contents to the state _after_ a `pulumi refresh`. ~However, when running `preview` after `refresh`, the nodejs SDK still fails with the same error in #6691 because `RegisterResource` still returns the inputs that contain empty/invalid assets~ **EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are missing asset contents (thanks @Frassle 🙏) Fixes #6691 ### TO-DO - [x] Figure out why `RegisterResource` is sending empty assets for during `preview` after `refresh` ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-06 16:47:01 +00:00
func traverseProperty(element resource.PropertyValue, f func(resource.PropertyValue)) {
f(element)
if element.IsSecret() {
traverseSecret(element.SecretValue(), f)
} else if element.IsObject() {
traverseMap(element.ObjectValue(), f)
} else if element.IsArray() {
traverseArray(element.ArrayValue(), f)
}
}
func traverseArray(elements []resource.PropertyValue, f func(resource.PropertyValue)) {
for _, element := range elements {
traverseProperty(element, f)
}
}
func traverseSecret(v *resource.Secret, f func(resource.PropertyValue)) {
traverseProperty(v.Element, f)
}
func traverseMap(m resource.PropertyMap, f func(resource.PropertyValue)) {
for _, value := range m {
traverseProperty(value, f)
}
}
// restoreElidedAssetContents is used to restore contents of assets inside resource property maps after
// we have skipped serializing contents of assets in order to avoid sending them over the wire to resource
// providers. Mainly used in `Read` operations after we receive the live inputs from the resource provider plugin.
// Those inputs may echo back the input assets and the engine writes them out to the state. We need to make sure that
// we don't write out empty assets to the state, so we restore the asset contents from the original inputs.
func restoreElidedAssetContents(original resource.PropertyMap, transformed resource.PropertyMap) {
Move assets and archives to their own package (#15157) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description This PR is motivated by https://github.com/pulumi/pulumi/pull/15145. `resource.*` should be built on top of `property.Value`,[^1] which means that `resource` needs to be able to import `property.Value`, and so `property` cannot import `resource`. Since Assets and Archives are both types of properties, they must be moved out of `resource`. [^1]: For example: https://github.com/pulumi/pulumi/blob/a1d686227cd7e3c70c51bd772450cb0cd57c1479/sdk/go/common/resource/resource_state.go#L35-L36 ## Open Question This PR moves them to their own sub-folders in `resource`. Should `asset` and `archive` live somewhere more high level, like `sdk/go/property/{asset,archive}`? <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-25 20:39:31 +00:00
isEmptyAsset := func(v *asset.Asset) bool {
[cli/engine] Restore elided asset contents from returned inputs from Read operations (#14078) # Description When we send inputs to `Read` operations due to `pulumi refresh` we skip sending asset contents. However, `Read` implementation might return the asset-valued input properties unchanged (without their asset contents) and the engine will simply write out the empty assets to the state causing an invalid shape of asset in the inputs of resources. This PR implements a function `restoreElidedAssetContents` and uses it in the `Read` implementation on the new inputs and new state returned from the provider plugins. Testing the CLI locally using this fix, it does correctly write out the asset contents to the state _after_ a `pulumi refresh`. ~However, when running `preview` after `refresh`, the nodejs SDK still fails with the same error in #6691 because `RegisterResource` still returns the inputs that contain empty/invalid assets~ **EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are missing asset contents (thanks @Frassle 🙏) Fixes #6691 ### TO-DO - [x] Figure out why `RegisterResource` is sending empty assets for during `preview` after `refresh` ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-06 16:47:01 +00:00
return v.Text == "" && v.Path == "" && v.URI == ""
}
Move assets and archives to their own package (#15157) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description This PR is motivated by https://github.com/pulumi/pulumi/pull/15145. `resource.*` should be built on top of `property.Value`,[^1] which means that `resource` needs to be able to import `property.Value`, and so `property` cannot import `resource`. Since Assets and Archives are both types of properties, they must be moved out of `resource`. [^1]: For example: https://github.com/pulumi/pulumi/blob/a1d686227cd7e3c70c51bd772450cb0cd57c1479/sdk/go/common/resource/resource_state.go#L35-L36 ## Open Question This PR moves them to their own sub-folders in `resource`. Should `asset` and `archive` live somewhere more high level, like `sdk/go/property/{asset,archive}`? <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-25 20:39:31 +00:00
isEmptyArchive := func(v *archive.Archive) bool {
[cli/engine] Restore elided asset contents from returned inputs from Read operations (#14078) # Description When we send inputs to `Read` operations due to `pulumi refresh` we skip sending asset contents. However, `Read` implementation might return the asset-valued input properties unchanged (without their asset contents) and the engine will simply write out the empty assets to the state causing an invalid shape of asset in the inputs of resources. This PR implements a function `restoreElidedAssetContents` and uses it in the `Read` implementation on the new inputs and new state returned from the provider plugins. Testing the CLI locally using this fix, it does correctly write out the asset contents to the state _after_ a `pulumi refresh`. ~However, when running `preview` after `refresh`, the nodejs SDK still fails with the same error in #6691 because `RegisterResource` still returns the inputs that contain empty/invalid assets~ **EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are missing asset contents (thanks @Frassle 🙏) Fixes #6691 ### TO-DO - [x] Figure out why `RegisterResource` is sending empty assets for during `preview` after `refresh` ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-06 16:47:01 +00:00
return v.Path == "" && v.URI == "" && v.Assets == nil
}
Move assets and archives to their own package (#15157) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description This PR is motivated by https://github.com/pulumi/pulumi/pull/15145. `resource.*` should be built on top of `property.Value`,[^1] which means that `resource` needs to be able to import `property.Value`, and so `property` cannot import `resource`. Since Assets and Archives are both types of properties, they must be moved out of `resource`. [^1]: For example: https://github.com/pulumi/pulumi/blob/a1d686227cd7e3c70c51bd772450cb0cd57c1479/sdk/go/common/resource/resource_state.go#L35-L36 ## Open Question This PR moves them to their own sub-folders in `resource`. Should `asset` and `archive` live somewhere more high level, like `sdk/go/property/{asset,archive}`? <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-25 20:39:31 +00:00
originalAssets := map[string]*asset.Asset{}
originalArchives := map[string]*archive.Archive{}
[cli/engine] Restore elided asset contents from returned inputs from Read operations (#14078) # Description When we send inputs to `Read` operations due to `pulumi refresh` we skip sending asset contents. However, `Read` implementation might return the asset-valued input properties unchanged (without their asset contents) and the engine will simply write out the empty assets to the state causing an invalid shape of asset in the inputs of resources. This PR implements a function `restoreElidedAssetContents` and uses it in the `Read` implementation on the new inputs and new state returned from the provider plugins. Testing the CLI locally using this fix, it does correctly write out the asset contents to the state _after_ a `pulumi refresh`. ~However, when running `preview` after `refresh`, the nodejs SDK still fails with the same error in #6691 because `RegisterResource` still returns the inputs that contain empty/invalid assets~ **EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are missing asset contents (thanks @Frassle 🙏) Fixes #6691 ### TO-DO - [x] Figure out why `RegisterResource` is sending empty assets for during `preview` after `refresh` ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-06 16:47:01 +00:00
traverseMap(original, func(value resource.PropertyValue) {
if value.IsAsset() {
originalAsset := value.AssetValue()
originalAssets[originalAsset.Hash] = originalAsset
}
if value.IsArchive() {
originalArchive := value.ArchiveValue()
originalArchives[originalArchive.Hash] = originalArchive
}
})
traverseMap(transformed, func(value resource.PropertyValue) {
if value.IsAsset() {
transformedAsset := value.AssetValue()
originalAsset, has := originalAssets[transformedAsset.Hash]
if has && isEmptyAsset(transformedAsset) {
transformedAsset.Sig = originalAsset.Sig
transformedAsset.Text = originalAsset.Text
transformedAsset.Path = originalAsset.Path
transformedAsset.URI = originalAsset.URI
}
}
if value.IsArchive() {
transformedArchive := value.ArchiveValue()
originalArchive, has := originalArchives[transformedArchive.Hash]
if has && isEmptyArchive(transformedArchive) {
transformedArchive.Sig = originalArchive.Sig
transformedArchive.URI = originalArchive.URI
transformedArchive.Path = originalArchive.Path
transformedArchive.Assets = originalArchive.Assets
}
}
})
}
// Configure configures the resource provider with "globals" that control its behavior.
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
func (p *provider) Configure(inputs resource.PropertyMap) error {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
label := p.label() + ".Configure()"
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
logging.V(7).Infof("%s executing (#vars=%d)", label, len(inputs))
// Convert the inputs to a config map. If any are unknown, do not configure the underlying plugin: instead, leave
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// the cfgknown bit unset and carry on.
config := make(map[string]string)
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
for k, v := range inputs {
if k == "version" {
continue
}
if v.ContainsUnknowns() {
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
p.configSource.MustFulfill(pluginConfig{
known: false,
acceptSecrets: false,
acceptResources: false,
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
})
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
return nil
}
mapped := removeSecrets(v)
if _, isString := mapped.(string); !isString {
marshalled, err := json.Marshal(mapped)
if err != nil {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
err := fmt.Errorf("marshaling configuration property '%v': %w", k, err)
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
p.configSource.MustReject(err)
return err
}
mapped = string(marshalled)
}
// Pass the older spelling of a configuration key across the RPC interface, for now, to support
// providers which are on the older plan.
config[string(p.Pkg())+":config:"+string(k)] = mapped.(string)
}
minputs, err := MarshalProperties(inputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".inputs",
KeepUnknowns: true,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
err := fmt.Errorf("marshaling provider inputs: %w", err)
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
p.configSource.MustReject(err)
return err
}
// Spawn the configure to happen in parallel. This ensures that we remain responsive elsewhere that might
// want to make forward progress, even as the configure call is happening.
go func() {
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := p.clientRaw.Configure(p.requestContext(), &pulumirpc.ConfigureRequest{
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
AcceptSecrets: true,
AcceptResources: true,
SendsOldInputs: true,
SendsOldInputsToDelete: true,
Variables: config,
Args: minputs,
2019-04-12 20:25:01 +00:00
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", label, rpcError.Message())
err = createConfigureError(rpcError)
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
p.configSource.MustReject(err)
return
}
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
p.configSource.MustFulfill(pluginConfig{
known: true,
acceptSecrets: resp.GetAcceptSecrets(),
acceptResources: resp.GetAcceptResources(),
supportsPreview: resp.GetSupportsPreview(),
acceptOutputs: resp.GetAcceptOutputs(),
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
})
}()
return nil
}
// Check validates that the given property bag is valid for a resource of the given type.
func (p *provider) Check(urn resource.URN,
olds, news resource.PropertyMap,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
allowUnknowns bool, randomSeed []byte,
) (resource.PropertyMap, []CheckFailure, error) {
label := fmt.Sprintf("%s.Check(%s)", p.label(), urn)
logging.V(7).Infof("%s executing (#olds=%d,#news=%d)", label, len(olds), len(news))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return nil, nil, err
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// If the configuration for this provider was not fully known--e.g. if we are doing a preview and some input
// property was sourced from another resource's output properties--don't call into the underlying provider.
if !pcfg.known {
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
return news, nil, nil
}
molds, err := MarshalProperties(olds, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".olds",
KeepUnknowns: allowUnknowns,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, nil, err
}
mnews, err := MarshalProperties(news, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".news",
KeepUnknowns: allowUnknowns,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, nil, err
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Check(p.requestContext(), &pulumirpc.CheckRequest{
Urn: string(urn),
Olds: molds,
News: mnews,
RandomSeed: randomSeed,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", label, rpcError.Message())
return nil, nil, rpcError
}
// Unmarshal the provider inputs.
var inputs resource.PropertyMap
if ins := resp.GetInputs(); ins != nil {
inputs, err = UnmarshalProperties(ins, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".inputs",
KeepUnknowns: allowUnknowns,
RejectUnknowns: !allowUnknowns,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
return nil, nil, err
}
}
// If we could not pass secrets to the provider, retain the secret bit on any property with the same name. This
// allows us to retain metadata about secrets in many cases, even for providers that do not understand secrets
// natively.
if !pcfg.acceptSecrets {
annotateSecrets(inputs, news)
}
// And now any properties that failed verification.
failures := slice.Prealloc[CheckFailure](len(resp.GetFailures()))
for _, failure := range resp.GetFailures() {
failures = append(failures, CheckFailure{resource.PropertyKey(failure.Property), failure.Reason})
}
logging.V(7).Infof("%s success: inputs=#%d failures=#%d", label, len(inputs), len(failures))
return inputs, failures, nil
}
// Diff checks what impacts a hypothetical update will have on the resource's properties.
func (p *provider) Diff(urn resource.URN, id resource.ID,
oldInputs, oldOutputs, newInputs resource.PropertyMap, allowUnknowns bool,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
ignoreChanges []string,
) (DiffResult, error) {
contract.Assertf(urn != "", "Diff requires a URN")
contract.Assertf(id != "", "Diff requires an ID")
contract.Assertf(oldInputs != nil, "Diff requires old input properties")
contract.Assertf(newInputs != nil, "Diff requires new input properties")
contract.Assertf(oldOutputs != nil, "Diff requires old output properties")
label := fmt.Sprintf("%s.Diff(%s,%s)", p.label(), urn, id)
logging.V(7).Infof("%s: executing (#oldInputs=%d#oldOutputs=%d,#newInputs=%d)",
label, len(oldInputs), len(oldOutputs), len(newInputs))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
if err != nil {
return DiffResult{}, err
}
// If the configuration for this provider was not fully known--e.g. if we are doing a preview and some input
// property was sourced from another resource's output properties--don't call into the underlying provider.
// Instead, indicate that the diff is unavailable and write a message
if !pcfg.known {
logging.V(7).Infof("%s: cannot diff due to unknown config", label)
const message = "The provider for this resource has inputs that are not known during preview.\n" +
"This preview may not correctly represent the changes that will be applied during an update."
return DiffResult{}, DiffUnavailable(message)
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
mOldInputs, err := MarshalProperties(oldInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldInputs",
ElideAssetContents: true,
KeepUnknowns: allowUnknowns,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return DiffResult{}, err
}
mOldOutputs, err := MarshalProperties(oldOutputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldOutputs",
ElideAssetContents: true,
KeepUnknowns: allowUnknowns,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return DiffResult{}, err
}
mNewInputs, err := MarshalProperties(newInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".newInputs",
ElideAssetContents: true,
KeepUnknowns: allowUnknowns,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return DiffResult{}, err
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Diff(p.requestContext(), &pulumirpc.DiffRequest{
Id: string(id),
Urn: string(urn),
OldInputs: mOldInputs,
Olds: mOldOutputs,
News: mNewInputs,
IgnoreChanges: ignoreChanges,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: %v", label, rpcError.Message())
return DiffResult{}, rpcError
}
// nil is semantically important to a lot of the pulumi system so we only pre-allocate if we have non-zero length.
replaces := slice.Prealloc[resource.PropertyKey](len(resp.GetReplaces()))
for _, replace := range resp.GetReplaces() {
replaces = append(replaces, resource.PropertyKey(replace))
}
stables := slice.Prealloc[resource.PropertyKey](len(resp.GetStables()))
for _, stable := range resp.GetStables() {
stables = append(stables, resource.PropertyKey(stable))
}
diffs := slice.Prealloc[resource.PropertyKey](len(resp.GetDiffs()))
for _, diff := range resp.GetDiffs() {
diffs = append(diffs, resource.PropertyKey(diff))
}
changes := resp.GetChanges()
deleteBeforeReplace := resp.GetDeleteBeforeReplace()
logging.V(7).Infof("%s success: changes=%d #replaces=%v #stables=%v delbefrepl=%v, diffs=#%v, detaileddiff=%v",
label, changes, replaces, stables, deleteBeforeReplace, diffs, resp.GetDetailedDiff())
return DiffResult{
Changes: DiffChanges(changes),
ReplaceKeys: replaces,
StableKeys: stables,
ChangedKeys: diffs,
Defer all diffs to resource providers. (#2849) Thse changes make a subtle but critical adjustment to the process the Pulumi engine uses to determine whether or not a difference exists between a resource's actual and desired states, and adjusts the way this difference is calculated and displayed accordingly. Today, the Pulumi engine get the first chance to decide whether or not there is a difference between a resource's actual and desired states. It does this by comparing the current set of inputs for a resource (i.e. the inputs from the running Pulumi program) with the last set of inputs used to update the resource. If there is no difference between the old and new inputs, the engine decides that no change is necessary without consulting the resource's provider. Only if there are changes does the engine consult the resource's provider for more information about the difference. This can be problematic for a number of reasons: - Not all providers do input-input comparison; some do input-state comparison - Not all providers are able to update the last deployed set of inputs when performing a refresh - Some providers--either intentionally or due to bugs--may see changes in resources whose inputs have not changed All of these situations are confusing at the very least, and the first is problematic with respect to correctness. Furthermore, the display code only renders diffs it observes rather than rendering the diffs observed by the provider, which can obscure the actual changes detected at runtime. These changes address both of these issues: - Rather than comparing the current inputs against the last inputs before calling a resource provider's Diff function, the engine calls the Diff function in all cases. - Providers may now return a list of properties that differ between the requested and actual state and the way in which they differ. This information will then be used by the CLI to render the diff appropriately. A provider may also indicate that a particular diff is between old and new inputs rather than old state and new inputs. Fixes #2453.
2019-07-01 19:34:19 +00:00
DetailedDiff: decodeDetailedDiff(resp),
DeleteBeforeReplace: deleteBeforeReplace,
}, nil
}
// Create allocates a new instance of the provided resource and assigns its unique resource.ID and outputs afterwards.
func (p *provider) Create(urn resource.URN, props resource.PropertyMap, timeout float64, preview bool) (resource.ID,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
resource.PropertyMap, resource.Status, error,
) {
contract.Assertf(urn != "", "Create requires a URN")
contract.Assertf(props != nil, "Create requires properties")
label := fmt.Sprintf("%s.Create(%s)", p.label(), urn)
logging.V(7).Infof("%s executing (#props=%v)", label, len(props))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return "", nil, resource.StatusOK, err
}
// If this is a preview and the plugin does not support provider previews, or if the configuration for the provider
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
// is not fully known, hand back an empty property map. This will force the language SDK will to treat all properties
// as unknown, which is conservatively correct.
//
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
// If the provider does not support previews, return the inputs as the state. Note that this can cause problems for
// the language SDKs if there are input and state properties that share a name but expect differently-shaped values.
if preview {
// TODO: it would be great to swap the order of these if statements. This would prevent a behavioral change for
// providers that do not support provider previews, which will always return the inputs as state regardless of
// whether or not the config is known. Unfortunately, we can't, since the `supportsPreview` bit depends on the
// result of `Configure`, which we won't call if the `cfgknown` is false. It may be worth fixing this catch-22
// by extending the provider gRPC interface with a `SupportsFeature` API similar to the language monitor.
if !pcfg.known {
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
if p.legacyPreview {
return "", props, resource.StatusOK, nil
}
return "", resource.PropertyMap{}, resource.StatusOK, nil
}
if !pcfg.supportsPreview || p.disableProviderPreview {
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
return "", props, resource.StatusOK, nil
}
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// We should only be calling {Create,Update,Delete} if the provider is fully configured.
contract.Assertf(pcfg.known, "Create cannot be called if the configuration is unknown")
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
mprops, err := MarshalProperties(props, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".inputs",
KeepUnknowns: preview,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return "", nil, resource.StatusOK, err
}
var id resource.ID
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
var liveObject *structpb.Struct
var resourceError error
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
resourceStatus := resource.StatusOK
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Create(p.requestContext(), &pulumirpc.CreateRequest{
Urn: string(urn),
Properties: mprops,
2019-07-15 21:26:28 +00:00
Timeout: timeout,
Preview: preview,
})
if err != nil {
resourceStatus, id, liveObject, _, resourceError = parseError(err)
2018-07-06 22:17:32 +00:00
logging.V(7).Infof("%s failed: %v", label, resourceError)
if resourceStatus != resource.StatusPartialFailure {
return "", nil, resourceStatus, resourceError
}
2018-07-06 22:17:32 +00:00
// Else it's a `StatusPartialFailure`.
} else {
id = resource.ID(resp.GetId())
2018-07-06 22:17:32 +00:00
liveObject = resp.GetProperties()
}
if id == "" && !preview {
return "", nil, resource.StatusUnknown,
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
fmt.Errorf("plugin for package '%v' returned empty resource.ID from create '%v'", p.pkg, urn)
}
2018-07-06 22:17:32 +00:00
outs, err := UnmarshalProperties(liveObject, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".outputs",
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
RejectUnknowns: !preview,
KeepUnknowns: preview,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
2018-07-06 22:17:32 +00:00
return "", nil, resourceStatus, err
}
// If we could not pass secrets to the provider, retain the secret bit on any property with the same name. This
// allows us to retain metadata about secrets in many cases, even for providers that do not understand secrets
// natively.
if !pcfg.acceptSecrets {
annotateSecrets(outs, props)
}
logging.V(7).Infof("%s success: id=%s; #outs=%d", label, id, len(outs))
if resourceError == nil {
return id, outs, resourceStatus, nil
}
return id, outs, resourceStatus, resourceError
}
// read the current live state associated with a resource. enough state must be include in the inputs to uniquely
// identify the resource; this is typically just the resource id, but may also include some properties.
func (p *provider) Read(urn resource.URN, id resource.ID,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
inputs, state resource.PropertyMap,
) (ReadResult, resource.Status, error) {
contract.Assertf(urn != "", "Read URN was empty")
contract.Assertf(id != "", "Read ID was empty")
label := fmt.Sprintf("%s.Read(%s,%s)", p.label(), id, urn)
logging.V(7).Infof("%s executing (#inputs=%v, #state=%v)", label, len(inputs), len(state))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return ReadResult{}, resource.StatusUnknown, err
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// If the provider is not fully configured, return an empty bag.
if !pcfg.known {
return ReadResult{
Outputs: resource.PropertyMap{},
Inputs: resource.PropertyMap{},
}, resource.StatusUnknown, nil
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
}
// Marshal the resource inputs and state so we can perform the RPC.
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
var minputs *structpb.Struct
if inputs != nil {
m, err := MarshalProperties(inputs, MarshalOptions{
Label: label,
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return ReadResult{}, resource.StatusUnknown, err
}
minputs = m
}
mstate, err := MarshalProperties(state, MarshalOptions{
Label: label,
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return ReadResult{}, resource.StatusUnknown, err
}
// Now issue the read request over RPC, blocking until it finished.
var readID resource.ID
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
var liveObject *structpb.Struct
var liveInputs *structpb.Struct
var resourceError error
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
resourceStatus := resource.StatusOK
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Read(p.requestContext(), &pulumirpc.ReadRequest{
Id: string(id),
Urn: string(urn),
Properties: mstate,
Inputs: minputs,
})
if err != nil {
resourceStatus, readID, liveObject, liveInputs, resourceError = parseError(err)
logging.V(7).Infof("%s failed: %v", label, err)
if resourceStatus != resource.StatusPartialFailure {
return ReadResult{}, resourceStatus, resourceError
}
// Else it's a `StatusPartialFailure`.
} else {
readID = resource.ID(resp.GetId())
liveObject = resp.GetProperties()
liveInputs = resp.GetInputs()
}
// If the resource was missing, simply return a nil property map.
if string(readID) == "" {
return ReadResult{}, resourceStatus, nil
}
// Finally, unmarshal the resulting state properties and return them.
newState, err := UnmarshalProperties(liveObject, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".outputs",
RejectUnknowns: true,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
return ReadResult{}, resourceStatus, err
}
var newInputs resource.PropertyMap
if liveInputs != nil {
newInputs, err = UnmarshalProperties(liveInputs, MarshalOptions{
Label: label + ".inputs",
RejectUnknowns: true,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
return ReadResult{}, resourceStatus, err
}
}
// If we could not pass secrets to the provider, retain the secret bit on any property with the same name. This
// allows us to retain metadata about secrets in many cases, even for providers that do not understand secrets
// natively.
if !pcfg.acceptSecrets {
annotateSecrets(newInputs, inputs)
annotateSecrets(newState, state)
}
[cli/engine] Restore elided asset contents from returned inputs from Read operations (#14078) # Description When we send inputs to `Read` operations due to `pulumi refresh` we skip sending asset contents. However, `Read` implementation might return the asset-valued input properties unchanged (without their asset contents) and the engine will simply write out the empty assets to the state causing an invalid shape of asset in the inputs of resources. This PR implements a function `restoreElidedAssetContents` and uses it in the `Read` implementation on the new inputs and new state returned from the provider plugins. Testing the CLI locally using this fix, it does correctly write out the asset contents to the state _after_ a `pulumi refresh`. ~However, when running `preview` after `refresh`, the nodejs SDK still fails with the same error in #6691 because `RegisterResource` still returns the inputs that contain empty/invalid assets~ **EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are missing asset contents (thanks @Frassle 🙏) Fixes #6691 ### TO-DO - [x] Figure out why `RegisterResource` is sending empty assets for during `preview` after `refresh` ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-06 16:47:01 +00:00
// make sure any echoed properties restore their original asset contents if they have not changed
restoreElidedAssetContents(inputs, newInputs)
restoreElidedAssetContents(inputs, newState)
logging.V(7).Infof("%s success; #outs=%d, #inputs=%d", label, len(newState), len(newInputs))
return ReadResult{
ID: readID,
Outputs: newState,
Inputs: newInputs,
}, resourceStatus, resourceError
}
// Update updates an existing resource with new values.
func (p *provider) Update(urn resource.URN, id resource.ID,
oldInputs, oldOutputs, newInputs resource.PropertyMap, timeout float64,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
ignoreChanges []string, preview bool,
) (resource.PropertyMap, resource.Status, error) {
contract.Assertf(urn != "", "Update requires a URN")
contract.Assertf(id != "", "Update requires an ID")
contract.Assertf(oldInputs != nil, "Update requires old inputs")
contract.Assertf(oldOutputs != nil, "Update requires old outputs")
contract.Assertf(newInputs != nil, "Update requires new properties")
label := fmt.Sprintf("%s.Update(%s,%s)", p.label(), id, urn)
logging.V(7).Infof("%s executing (#oldInputs=%v,#oldOutputs=%v,#newInputs=%v)",
label, len(oldInputs), len(oldOutputs), len(newInputs))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return newInputs, resource.StatusOK, err
}
// If this is a preview and the plugin does not support provider previews, or if the configuration for the provider
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
// is not fully known, hand back an empty property map. This will force the language SDK to treat all properties
// as unknown, which is conservatively correct.
//
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
// If the provider does not support previews, return the inputs as the state. Note that this can cause problems for
// the language SDKs if there are input and state properties that share a name but expect differently-shaped values.
if preview {
// TODO: it would be great to swap the order of these if statements. This would prevent a behavioral change for
// providers that do not support provider previews, which will always return the inputs as state regardless of
// whether or not the config is known. Unfortunately, we can't, since the `supportsPreview` bit depends on the
// result of `Configure`, which we won't call if the `cfgknown` is false. It may be worth fixing this catch-22
// by extending the provider gRPC interface with a `SupportsFeature` API similar to the language monitor.
if !pcfg.known {
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
if p.legacyPreview {
return newInputs, resource.StatusOK, nil
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
}
return resource.PropertyMap{}, resource.StatusOK, nil
}
if !pcfg.supportsPreview || p.disableProviderPreview {
return newInputs, resource.StatusOK, nil
[sdk/providers] Fix update previews (#7560) Do not return the inputs as the state for update previews that use an unconfigured provider. Returning the inputs as the state allows the language SDKs to incorrectly treat unknown properties as known (because we can't call `Update` on an unconfigured provider, we can't know which properties are unknown). Users can re-enable the existing behavior by setting the `PULUMI_LEGACY_PROVIDER_PREVIEW` environment variable to a truthy value (e.g. `1`, `true`, etc.). Most users will be unaffected by these changes. The most common programs that may be affected are those that combine the creation of a managed Kubernetes cluster with the deployment of applications to that cluster. These programs generally need to configure a k8s provider instance by constructing a kubeconfig from the output of the managed k8s cluster. Any changes to the cluster that cause the kubeconfig to be unknown then cause the provider to go unconfigured at runtime. Prior to these changes, resources managed by the k8s provider would have some known outputs in this scenario, as the engine would treat the resource's input values as its output values. After these changes, the resource's outputs will be treated as unknown. The most frequent affect that this has is that applies/stack outputs that depend on the outputs of a k8s resource managed by a provider with an unknown kubeconfig will not run/be displayed as `output`s during previews, respectively. We might be able to improve on this by taking advantage of schema information and filling in unknown values for properties that do not exist in the inputs. Fixes #7521. Co-authored-by: Justin Van Patten <jvp@justinvp.com> Co-authored-by: Luke Hoban <luke@pulumi.com>
2021-08-11 00:44:15 +00:00
}
}
// We should only be calling {Create,Update,Delete} if the provider is fully configured.
contract.Assertf(pcfg.known, "Update cannot be called if the configuration is unknown")
mOldInputs, err := MarshalProperties(oldInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldInputs",
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, resource.StatusOK, err
}
mOldOutputs, err := MarshalProperties(oldOutputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".oldOutputs",
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, resource.StatusOK, err
}
mNewInputs, err := MarshalProperties(newInputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".newInputs",
KeepUnknowns: preview,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, resource.StatusOK, err
}
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
var liveObject *structpb.Struct
var resourceError error
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
resourceStatus := resource.StatusOK
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Update(p.requestContext(), &pulumirpc.UpdateRequest{
Id: string(id),
Urn: string(urn),
Olds: mOldOutputs,
News: mNewInputs,
Timeout: timeout,
IgnoreChanges: ignoreChanges,
Preview: preview,
OldInputs: mOldInputs,
})
if err != nil {
resourceStatus, _, liveObject, _, resourceError = parseError(err)
logging.V(7).Infof("%s failed: %v", label, resourceError)
if resourceStatus != resource.StatusPartialFailure {
return nil, resourceStatus, resourceError
}
2018-07-06 22:17:32 +00:00
// Else it's a `StatusPartialFailure`.
} else {
2018-07-06 22:17:32 +00:00
liveObject = resp.GetProperties()
}
2018-07-06 22:17:32 +00:00
outs, err := UnmarshalProperties(liveObject, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".outputs",
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
RejectUnknowns: !preview,
KeepUnknowns: preview,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
2018-07-06 22:17:32 +00:00
return nil, resourceStatus, err
}
// If we could not pass secrets to the provider, retain the secret bit on any property with the same name. This
// allows us to retain metadata about secrets in many cases, even for providers that do not understand secrets
// natively.
if !pcfg.acceptSecrets {
annotateSecrets(outs, newInputs)
}
logging.V(7).Infof("%s success; #outs=%d", label, len(outs))
if resourceError == nil {
return outs, resourceStatus, nil
}
return outs, resourceStatus, resourceError
}
// Delete tears down an existing resource.
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
func (p *provider) Delete(urn resource.URN, id resource.ID, oldInputs, oldOutputs resource.PropertyMap,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
timeout float64,
) (resource.Status, error) {
contract.Assertf(urn != "", "Delete requires a URN")
contract.Assertf(id != "", "Delete requires an ID")
label := fmt.Sprintf("%s.Delete(%s,%s)", p.label(), urn, id)
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
logging.V(7).Infof("%s executing (#inputs=%d, #outputs=%d)", label, len(oldInputs), len(oldOutputs))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return resource.StatusOK, err
}
// We should never call delete at preview time, so we should never see unknowns here
contract.Assertf(pcfg.known, "Delete cannot be called if the configuration is unknown")
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
minputs, err := MarshalProperties(oldInputs, MarshalOptions{
Label: label,
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return resource.StatusOK, err
}
moutputs, err := MarshalProperties(oldOutputs, MarshalOptions{
Label: label,
ElideAssetContents: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return resource.StatusOK, err
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// We should only be calling {Create,Update,Delete} if the provider is fully configured.
contract.Assertf(pcfg.known, "Delete cannot be called if the configuration is unknown")
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
if _, err := client.Delete(p.requestContext(), &pulumirpc.DeleteRequest{
2017-07-19 14:57:22 +00:00
Id: string(id),
Urn: string(urn),
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
Properties: moutputs,
2019-07-15 21:26:28 +00:00
Timeout: timeout,
Send old inputs to Delete (#14051) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14115. This was missed as part of https://github.com/pulumi/pulumi/pull/13139. Adds a new configure flag (sends_old_inputs_to_delete) which the engine will now always set to true. If that's set providers can rely on the old inputs being sent to delete, otherwise they'll get nil. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-13 14:12:26 +00:00
OldInputs: minputs,
}); err != nil {
resourceStatus, rpcErr := resourceStateAndError(err)
logging.V(7).Infof("%s failed: %v", label, rpcErr)
return resourceStatus, rpcErr
}
logging.V(7).Infof("%s success", label)
return resource.StatusOK, nil
}
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
// Construct creates a new component resource from the given type, name, parent, options, and inputs, and returns
// its URN and outputs.
Allow anything in resource names (#14107) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/13968. Fixes https://github.com/pulumi/pulumi/issues/8949. This requires changing the parsing of URN's slightly, it is _very_ likely that providers will need to update to handle URNs like this correctly. This changes resource names to be `string` not `QName`. We never validated this before and it turns out that users have put all manner of text for resource names so we just updating the system to correctly reflect that. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-20 08:59:00 +00:00
func (p *provider) Construct(info ConstructInfo, typ tokens.Type, name string, parent resource.URN,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
inputs resource.PropertyMap, options ConstructOptions,
) (ConstructResult, error) {
contract.Assertf(typ != "", "Construct requires a type")
contract.Assertf(name != "", "Construct requires a name")
contract.Assertf(inputs != nil, "Construct requires input properties")
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
label := fmt.Sprintf("%s.Construct(%s, %s, %s)", p.label(), typ, name, parent)
logging.V(7).Infof("%s executing (#inputs=%v)", label, len(inputs))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
if err != nil {
return ConstructResult{}, err
}
// If the provider is not fully configured, we need to error. We can't support unknown URNs but if the
// provider isn't configured we can't call into it to get the URN.
if !pcfg.known {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
return ConstructResult{}, errors.New("cannot construct components if the provider is configured with unknown values")
}
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
if !pcfg.acceptSecrets {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
return ConstructResult{}, errors.New("plugins that can construct components must support secrets")
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
}
// Marshal the input properties.
minputs, err := MarshalProperties(inputs, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".inputs",
KeepUnknowns: true,
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
// To initially scope the use of this new feature, we only keep output values for
// Construct and Call (when the client accepts them).
KeepOutputValues: pcfg.acceptOutputs,
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
})
if err != nil {
return ConstructResult{}, err
}
// Marshal the aliases.
aliasURNs := make([]string, len(options.Aliases))
for i, alias := range options.Aliases {
aliasURNs[i] = string(alias.URN)
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
}
// Marshal the dependencies.
dependencies := make([]string, len(options.Dependencies))
for i, dep := range options.Dependencies {
dependencies[i] = string(dep)
}
// Marshal the property dependencies.
inputDependencies := map[string]*pulumirpc.ConstructRequest_PropertyDependencies{}
for name, dependencies := range options.PropertyDependencies {
urns := make([]string, len(dependencies))
for i, urn := range dependencies {
urns[i] = string(urn)
}
inputDependencies[string(name)] = &pulumirpc.ConstructRequest_PropertyDependencies{Urns: urns}
}
// Marshal the config.
config := map[string]string{}
for k, v := range info.Config {
config[k.String()] = v
}
configSecretKeys := []string{}
for _, k := range info.ConfigSecretKeys {
configSecretKeys = append(configSecretKeys, k.String())
}
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
req := &pulumirpc.ConstructRequest{
Project: info.Project,
Stack: info.Stack,
Config: config,
ConfigSecretKeys: configSecretKeys,
DryRun: info.DryRun,
Parallel: int32(info.Parallel),
MonitorEndpoint: info.MonitorAddress,
Type: string(typ),
Allow anything in resource names (#14107) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/13968. Fixes https://github.com/pulumi/pulumi/issues/8949. This requires changing the parsing of URN's slightly, it is _very_ likely that providers will need to update to handle URNs like this correctly. This changes resource names to be `string` not `QName`. We never validated this before and it turns out that users have put all manner of text for resource names so we just updating the system to correctly reflect that. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-20 08:59:00 +00:00
Name: name,
Parent: string(parent),
Inputs: minputs,
Protect: options.Protect,
Providers: options.Providers,
InputDependencies: inputDependencies,
Aliases: aliasURNs,
Dependencies: dependencies,
AdditionalSecretOutputs: options.AdditionalSecretOutputs,
DeletedWith: string(options.DeletedWith),
DeleteBeforeReplace: options.DeleteBeforeReplace,
IgnoreChanges: options.IgnoreChanges,
ReplaceOnChanges: options.ReplaceOnChanges,
RetainOnDelete: options.RetainOnDelete,
Allow component providers to return outputs to the engine (#15408) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to provider Construct and Call requests to indicate that the engine will accept output values in the response. If a component provider sees that flag set they can return Output PropertyValues and optionally skip filling in the dependency map in the response. The engine will use the combined information of the dependency map and any dependencies found on any output values in the property to build the final dependency map that is returned to programs. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-14 08:15:24 +00:00
AcceptsOutputValues: true,
}
if ct := options.CustomTimeouts; ct != nil {
req.CustomTimeouts = &pulumirpc.ConstructRequest_CustomTimeouts{
Create: ct.Create,
Update: ct.Update,
Delete: ct.Delete,
}
}
resp, err := client.Construct(p.requestContext(), req)
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
if err != nil {
return ConstructResult{}, err
}
outputs, err := UnmarshalProperties(resp.GetState(), MarshalOptions{
Allow component providers to return outputs to the engine (#15408) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to provider Construct and Call requests to indicate that the engine will accept output values in the response. If a component provider sees that flag set they can return Output PropertyValues and optionally skip filling in the dependency map in the response. The engine will use the combined information of the dependency map and any dependencies found on any output values in the property to build the final dependency map that is returned to programs. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-14 08:15:24 +00:00
Label: label + ".outputs",
KeepUnknowns: info.DryRun,
KeepSecrets: true,
KeepResources: true,
KeepOutputValues: true,
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 02:33:55 +00:00
})
if err != nil {
return ConstructResult{}, err
}
outputDependencies := map[resource.PropertyKey][]resource.URN{}
for k, rpcDeps := range resp.GetStateDependencies() {
urns := make([]resource.URN, len(rpcDeps.Urns))
for i, d := range rpcDeps.Urns {
urns[i] = resource.URN(d)
}
outputDependencies[resource.PropertyKey(k)] = urns
}
logging.V(7).Infof("%s success: #outputs=%d", label, len(outputs))
return ConstructResult{
URN: resource.URN(resp.GetUrn()),
Outputs: outputs,
OutputDependencies: outputDependencies,
}, nil
}
// Invoke dynamically executes a built-in function in the provider.
func (p *provider) Invoke(tok tokens.ModuleMember, args resource.PropertyMap) (resource.PropertyMap,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
[]CheckFailure, error,
) {
contract.Assertf(tok != "", "Invoke requires a token")
label := fmt.Sprintf("%s.Invoke(%s)", p.label(), tok)
logging.V(7).Infof("%s executing (#args=%d)", label, len(args))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return nil, nil, err
}
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
// If the provider is not fully configured, return an empty property map.
if !pcfg.known {
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
return resource.PropertyMap{}, nil, nil
}
margs, err := MarshalProperties(args, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".args",
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
})
if err != nil {
return nil, nil, err
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
resp, err := client.Invoke(p.requestContext(), &pulumirpc.InvokeRequest{
Tok: string(tok),
Args: margs,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: %v", label, rpcError.Message())
return nil, nil, rpcError
}
// Unmarshal any return values.
ret, err := UnmarshalProperties(resp.GetReturn(), MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".returns",
RejectUnknowns: true,
KeepSecrets: true,
KeepResources: true,
})
if err != nil {
return nil, nil, err
}
// And now any properties that failed verification.
failures := slice.Prealloc[CheckFailure](len(resp.GetFailures()))
for _, failure := range resp.GetFailures() {
failures = append(failures, CheckFailure{resource.PropertyKey(failure.Property), failure.Reason})
}
logging.V(7).Infof("%s success (#ret=%d,#failures=%d) success", label, len(ret), len(failures))
return ret, failures, nil
}
2019-10-22 07:20:26 +00:00
// StreamInvoke dynamically executes a built-in function in the provider, which returns a stream of
// responses.
func (p *provider) StreamInvoke(
tok tokens.ModuleMember,
args resource.PropertyMap,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
onNext func(resource.PropertyMap) error,
) ([]CheckFailure, error) {
contract.Assertf(tok != "", "StreamInvoke requires a token")
2019-10-22 07:20:26 +00:00
label := fmt.Sprintf("%s.StreamInvoke(%s)", p.label(), tok)
logging.V(7).Infof("%s executing (#args=%d)", label, len(args))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
2019-10-22 07:20:26 +00:00
if err != nil {
return nil, err
}
// If the provider is not fully configured, return an empty property map.
if !pcfg.known {
2019-10-22 07:20:26 +00:00
return nil, onNext(resource.PropertyMap{})
}
margs, err := MarshalProperties(args, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".args",
KeepSecrets: pcfg.acceptSecrets,
KeepResources: pcfg.acceptResources,
2019-10-22 07:20:26 +00:00
})
if err != nil {
return nil, err
}
streamClient, err := client.StreamInvoke(
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
p.requestContext(), &pulumirpc.InvokeRequest{
Tok: string(tok),
Args: margs,
})
2019-10-22 07:20:26 +00:00
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: %v", label, rpcError.Message())
return nil, rpcError
}
for {
in, err := streamClient.Recv()
if err == io.EOF {
return nil, nil
}
if err != nil {
return nil, err
}
// Unmarshal response.
ret, err := UnmarshalProperties(in.GetReturn(), MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".returns",
2019-10-22 07:20:26 +00:00
RejectUnknowns: true,
KeepSecrets: true,
KeepResources: true,
2019-10-22 07:20:26 +00:00
})
if err != nil {
return nil, err
}
// Check properties that failed verification.
var failures []CheckFailure
for _, failure := range in.GetFailures() {
failures = append(failures, CheckFailure{resource.PropertyKey(failure.Property), failure.Reason})
}
if len(failures) > 0 {
return failures, nil
}
// Send stream message back to whoever is consuming the stream.
if err := onNext(ret); err != nil {
return nil, err
}
}
}
// Call dynamically executes a method in the provider associated with a component resource.
func (p *provider) Call(tok tokens.ModuleMember, args resource.PropertyMap, info CallInfo,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
options CallOptions,
) (CallResult, error) {
contract.Assertf(tok != "", "Call requires a token")
label := fmt.Sprintf("%s.Call(%s)", p.label(), tok)
logging.V(7).Infof("%s executing (#args=%d)", label, len(args))
// Ensure that the plugin is configured.
client := p.clientRaw
Add a strict opinionated promise library (#14552) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by having another case in https://github.com/pulumi/pulumi/pull/14548 that looked like "this is just a promise". We already had a mini-promise library for provider config in sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of places where we did a poor mans Go promise by having a WaitGroup plus a variable to set (see the two changes in /pkg). This adds a little promise library to safely cover all three of these cases plus the case I'll be adding in #14548. It is _minimal_ adding just the features of promises needed for these initial cases. We can add to it as needed. I suspect we've got a number of places in test code that could probably use this as well, but I haven't gone through that. I also suspect that having this type available will result in more places in the future being simpler because "its just a promise" is a fairly common scenario in async systems. In fact Output's internally _are just a promise_ so we could probably rewrite their internals using this, add a `Then()` like method for apply and all the async complexity gets handled in the promise layer while the output layer just cares about unknowns/secrets/etc. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - This is in sdk/go/common and I'm considering it an non-public api for now. <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 14:53:12 +00:00
pcfg, err := p.configSource.Promise().Result(context.Background())
if err != nil {
return CallResult{}, err
}
// If the provider is not fully configured, return an empty property map.
if !pcfg.known {
return CallResult{}, nil
}
margs, err := MarshalProperties(args, MarshalOptions{
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
Label: label + ".args",
KeepUnknowns: true,
KeepSecrets: true,
KeepResources: true,
// To initially scope the use of this new feature, we only keep output values for
// Construct and Call (when the client accepts them).
KeepOutputValues: pcfg.acceptOutputs,
})
if err != nil {
return CallResult{}, err
}
// Marshal the arg dependencies.
argDependencies := map[string]*pulumirpc.CallRequest_ArgumentDependencies{}
for name, dependencies := range options.ArgDependencies {
urns := make([]string, len(dependencies))
for i, urn := range dependencies {
urns[i] = string(urn)
}
argDependencies[string(name)] = &pulumirpc.CallRequest_ArgumentDependencies{Urns: urns}
}
// Marshal the config.
config := map[string]string{}
for k, v := range info.Config {
config[k.String()] = v
}
resp, err := client.Call(p.requestContext(), &pulumirpc.CallRequest{
Allow component providers to return outputs to the engine (#15408) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to provider Construct and Call requests to indicate that the engine will accept output values in the response. If a component provider sees that flag set they can return Output PropertyValues and optionally skip filling in the dependency map in the response. The engine will use the combined information of the dependency map and any dependencies found on any output values in the property to build the final dependency map that is returned to programs. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-14 08:15:24 +00:00
Tok: string(tok),
Args: margs,
ArgDependencies: argDependencies,
Project: info.Project,
Stack: info.Stack,
Config: config,
DryRun: info.DryRun,
Parallel: int32(info.Parallel),
MonitorEndpoint: info.MonitorAddress,
AcceptsOutputValues: true,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: %v", label, rpcError.Message())
return CallResult{}, rpcError
}
// Unmarshal any return values.
ret, err := UnmarshalProperties(resp.GetReturn(), MarshalOptions{
Allow component providers to return outputs to the engine (#15408) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to provider Construct and Call requests to indicate that the engine will accept output values in the response. If a component provider sees that flag set they can return Output PropertyValues and optionally skip filling in the dependency map in the response. The engine will use the combined information of the dependency map and any dependencies found on any output values in the property to build the final dependency map that is returned to programs. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-14 08:15:24 +00:00
Label: label + ".returns",
KeepUnknowns: info.DryRun,
KeepSecrets: true,
KeepResources: true,
KeepOutputValues: true,
})
if err != nil {
return CallResult{}, err
}
returnDependencies := map[resource.PropertyKey][]resource.URN{}
for k, rpcDeps := range resp.GetReturnDependencies() {
urns := make([]resource.URN, len(rpcDeps.Urns))
for i, d := range rpcDeps.Urns {
urns[i] = resource.URN(d)
}
returnDependencies[resource.PropertyKey(k)] = urns
}
// And now any properties that failed verification.
failures := slice.Prealloc[CheckFailure](len(resp.GetFailures()))
for _, failure := range resp.GetFailures() {
failures = append(failures, CheckFailure{resource.PropertyKey(failure.Property), failure.Reason})
}
logging.V(7).Infof("%s success (#ret=%d,#failures=%d) success", label, len(ret), len(failures))
return CallResult{Return: ret, ReturnDependencies: returnDependencies, Failures: failures}, nil
}
// GetPluginInfo returns this plugin's information.
func (p *provider) GetPluginInfo() (workspace.PluginInfo, error) {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
label := p.label() + ".GetPluginInfo()"
logging.V(7).Infof("%s executing", label)
// Calling GetPluginInfo happens immediately after loading, and does not require configuration to proceed.
// Thus, we access the clientRaw property, rather than calling getClient.
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
resp, err := p.clientRaw.GetPluginInfo(p.requestContext(), &emptypb.Empty{})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", label, rpcError.Message())
return workspace.PluginInfo{}, rpcError
}
var version *semver.Version
if v := resp.Version; v != "" {
sv, err := semver.ParseTolerant(v)
if err != nil {
return workspace.PluginInfo{}, err
}
version = &sv
}
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
path := ""
if p.plug != nil {
path = p.plug.Bin
}
2023-06-02 21:06:57 +00:00
logging.V(7).Infof("%s success (#version=%v) success", label, version)
return workspace.PluginInfo{
Name: string(p.pkg),
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
Path: path,
Kind: apitype.ResourcePlugin,
Version: version,
}, nil
}
// Attach attaches this plugin to the engine
func (p *provider) Attach(address string) error {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
label := p.label() + ".Attach()"
logging.V(7).Infof("%s executing", label)
// Calling Attach happens immediately after loading, and does not require configuration to proceed.
// Thus, we access the clientRaw property, rather than calling getClient.
_, err := p.clientRaw.Attach(p.requestContext(), &pulumirpc.PluginAttach{Address: address})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", label, rpcError.Message())
return rpcError
}
return nil
}
func (p *provider) SignalCancellation() error {
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
_, err := p.clientRaw.Cancel(p.requestContext(), &emptypb.Empty{})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(8).Infof("provider received rpc error `%s`: `%s`", rpcError.Code(),
rpcError.Message())
turn on the golangci-lint exhaustive linter (#15028) Turn on the golangci-lint exhaustive linter. This is the first step towards catching more missing cases during development rather than in tests, or in production. This might be best reviewed commit-by-commit, as the first commit turns on the linter with the `default-signifies-exhaustive: true` option set, which requires a lot less changes in the current codebase. I think it's probably worth doing the second commit as well, as that will get us the real benefits, even though we end up with a little bit more churn. However it means all the `switch` statements are covered, which isn't the case after the first commit, since we do have a lot of `default` statements that just call `assert.Fail`. Fixes #14601 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 16:50:41 +00:00
if rpcError.Code() == codes.Unimplemented {
// For backwards compatibility, do nothing if it's not implemented.
return nil
}
}
return err
}
// Close tears down the underlying plugin RPC connection and process.
func (p *provider) Close() error {
gRPC bridge: fix unknowns in `Update` previews (#6006) These changes are a combination of three commits, each of which contributes to the testing and/or fixing of a problem with marshaling unknowns in `plugin.provider.Update` when `preview` is true. ## deploytest: add support for gRPC adapters. These changes add support for communicating with providers using the gRPC adapters to the deploytest pacakage. This makes it easier to test the gRPC adapters across typical lifecycle patterns. Supporting these changes are two additions to the `resource/plugin` package: 1. A type that bridges between the `plugin.Provider` interface and the `pulumirpc.ResourceProviderServer` 2. A function to create a `plugin.Provider` given a `pulumirpc.ResourceProviderClient` The deploytest package uses these to wrap an in-process `plugin.Provider` in a gRPC interface and connect to it without using the default plugin host, respectively. ## pulumi_test: test provider preview over gRPC. Add a test that runs the provider preview lifecycle, but using a provider that communicates over gRPC. ## gRPC bridge: fix unknowns in `Update` previews Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions` used when unmarshaling update results to preserve unknowns during a preview and reject them otherwise. These changes also set the `RejectUnknowns` bit in the `MarshalOptions` used by `Create` if `preview` is false, and fix a bug in the array unmarshaler that could cause out-of-bounds accesses. Fixes https://github.com/pulumi/pulumi/issues/6004.
2020-12-23 21:25:48 +00:00
if p.plug == nil {
return nil
}
return p.plug.Close()
}
// createConfigureError creates a nice error message from an RPC error that
// originated from `Configure`.
//
// If we requested that a resource configure itself but omitted required configuration
// variables, resource providers will respond with a list of missing variables and their descriptions.
// If that is what occurred, we'll use that information here to construct a nice error message.
func createConfigureError(rpcerr *rpcerror.Error) error {
var err error
for _, detail := range rpcerr.Details() {
if missingKeys, ok := detail.(*pulumirpc.ConfigureErrorMissingKeys); ok {
for _, missingKey := range missingKeys.MissingKeys {
singleError := fmt.Errorf("missing required configuration key \"%s\": %s\n"+
"Set a value using the command `pulumi config set %s <value>`.",
missingKey.Name, missingKey.Description, missingKey.Name)
err = multierror.Append(err, singleError)
}
}
}
if err != nil {
return err
}
return rpcerr
}
// resourceStateAndError interprets an error obtained from a gRPC endpoint.
//
// gRPC gives us a `status.Status` structure as an `error` whenever our
// gRPC servers serve up an error. Each `status.Status` contains a code
// and a message. Based on the error code given to us, we can understand
// the state of our system and if our resource status is truly unknown.
//
// In general, our resource state is only really unknown if the server
// had an internal error, in which case it will serve one of `codes.Internal`,
// `codes.DataLoss`, or `codes.Unknown` to us.
func resourceStateAndError(err error) (resource.Status, *rpcerror.Error) {
rpcError := rpcerror.Convert(err)
logging.V(8).Infof("provider received rpc error `%s`: `%s`", rpcError.Code(), rpcError.Message())
turn on the golangci-lint exhaustive linter (#15028) Turn on the golangci-lint exhaustive linter. This is the first step towards catching more missing cases during development rather than in tests, or in production. This might be best reviewed commit-by-commit, as the first commit turns on the linter with the `default-signifies-exhaustive: true` option set, which requires a lot less changes in the current codebase. I think it's probably worth doing the second commit as well, as that will get us the real benefits, even though we end up with a little bit more churn. However it means all the `switch` statements are covered, which isn't the case after the first commit, since we do have a lot of `default` statements that just call `assert.Fail`. Fixes #14601 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 16:50:41 +00:00
//nolint:exhaustive // We want to handle only some error codes specially
switch rpcError.Code() {
case codes.Internal, codes.DataLoss, codes.Unknown:
logging.V(8).Infof("rpc error kind `%s` may not be recoverable", rpcError.Code())
return resource.StatusUnknown, rpcError
}
logging.V(8).Infof("rpc error kind `%s` is well-understood and recoverable", rpcError.Code())
return resource.StatusOK, rpcError
}
2018-07-06 22:17:32 +00:00
// parseError parses a gRPC error into a set of values that represent the state of a resource. They
// are: (1) the `resourceStatus`, indicating the last known state (e.g., `StatusOK`, representing
// success, `StatusUnknown`, representing internal failure); (2) the `*rpcerror.Error`, our internal
// representation for RPC errors; and optionally (3) `liveObject`, containing the last known live
// version of the object that has successfully created but failed to initialize (e.g., because the
// object was created, but app code is continually crashing and the resource never achieves
// liveness).
func parseError(err error) (
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
resourceStatus resource.Status, id resource.ID, liveInputs, liveObject *structpb.Struct, resourceErr error,
2018-07-06 22:17:32 +00:00
) {
var responseErr *rpcerror.Error
resourceStatus, responseErr = resourceStateAndError(err)
contract.Assertf(responseErr != nil, "resourceStateAndError must never return a nil error")
2018-07-06 22:17:32 +00:00
// If resource was successfully created but failed to initialize, the error will be packed
// with the live properties of the object.
resourceErr = responseErr
for _, detail := range responseErr.Details() {
2018-07-06 22:17:32 +00:00
if initErr, ok := detail.(*pulumirpc.ErrorResourceInitFailed); ok {
id = resource.ID(initErr.GetId())
2018-07-06 22:17:32 +00:00
liveObject = initErr.GetProperties()
liveInputs = initErr.GetInputs()
2018-07-06 22:17:32 +00:00
resourceStatus = resource.StatusPartialFailure
resourceErr = &InitError{Reasons: initErr.Reasons}
2018-07-06 22:17:32 +00:00
break
}
}
return resourceStatus, id, liveObject, liveInputs, resourceErr
}
// InitError represents a failure to initialize a resource, i.e., the resource has been successfully
// created, but it has failed to initialize.
type InitError struct {
Reasons []string
}
var _ error = (*InitError)(nil)
func (ie *InitError) Error() string {
var err error
for _, reason := range ie.Reasons {
err = multierror.Append(err, errors.New(reason))
}
return err.Error()
2018-07-06 22:17:32 +00:00
}
func decorateSpanWithType(span opentracing.Span, urn string) {
if urn := resource.URN(urn); urn.IsValid() {
span.SetTag("pulumi-decorator", urn.Type())
}
}
func decorateProviderSpans(span opentracing.Span, method string, req, resp interface{}, grpcError error) {
if req == nil {
return
}
switch method {
case "/pulumirpc.ResourceProvider/Check", "/pulumirpc.ResourceProvider/CheckConfig":
decorateSpanWithType(span, req.(*pulumirpc.CheckRequest).Urn)
case "/pulumirpc.ResourceProvider/Diff", "/pulumirpc.ResourceProvider/DiffConfig":
decorateSpanWithType(span, req.(*pulumirpc.DiffRequest).Urn)
case "/pulumirpc.ResourceProvider/Create":
decorateSpanWithType(span, req.(*pulumirpc.CreateRequest).Urn)
case "/pulumirpc.ResourceProvider/Update":
decorateSpanWithType(span, req.(*pulumirpc.UpdateRequest).Urn)
case "/pulumirpc.ResourceProvider/Delete":
decorateSpanWithType(span, req.(*pulumirpc.DeleteRequest).Urn)
case "/pulumirpc.ResourceProvider/Invoke":
span.SetTag("pulumi-decorator", req.(*pulumirpc.InvokeRequest).Tok)
}
}
// GetMapping fetches the conversion mapping (if any) for this resource provider.
More efficent mapping lookup (#13975) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Inspired by a comment from Zaid. This allows providers to return what providers they have mapping information for without having to marshal all their mapping data to the engine at the same time, this could save transmitting a lot of data that the engine might not ever need (for example if it's not converting code for that specific provider). It also allows provider to support mulitple mappings. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-09-21 11:45:07 +00:00
func (p *provider) GetMapping(key, provider string) ([]byte, string, error) {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
label := p.label() + ".GetMapping"
More efficent mapping lookup (#13975) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Inspired by a comment from Zaid. This allows providers to return what providers they have mapping information for without having to marshal all their mapping data to the engine at the same time, this could save transmitting a lot of data that the engine might not ever need (for example if it's not converting code for that specific provider). It also allows provider to support mulitple mappings. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-09-21 11:45:07 +00:00
logging.V(7).Infof("%s executing: key=%s, provider=%s", label, key, provider)
2022-12-01 23:03:25 +00:00
resp, err := p.clientRaw.GetMapping(p.requestContext(), &pulumirpc.GetMappingRequest{
More efficent mapping lookup (#13975) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Inspired by a comment from Zaid. This allows providers to return what providers they have mapping information for without having to marshal all their mapping data to the engine at the same time, this could save transmitting a lot of data that the engine might not ever need (for example if it's not converting code for that specific provider). It also allows provider to support mulitple mappings. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-09-21 11:45:07 +00:00
Key: key,
Provider: provider,
2022-12-01 23:03:25 +00:00
})
if err != nil {
rpcError := rpcerror.Convert(err)
code := rpcError.Code()
if code == codes.Unimplemented {
// For backwards compatibility, just return nothing as if the provider didn't have a mapping for
// the given key
logging.V(7).Infof("%s unimplemented", label)
2022-12-01 23:03:25 +00:00
return nil, "", nil
}
logging.V(7).Infof("%s failed: %v", label, rpcError)
2022-12-01 23:03:25 +00:00
return nil, "", err
}
logging.V(7).Infof("%s success: data=#%d provider=%s", label, len(resp.Data), resp.Provider)
2022-12-01 23:03:25 +00:00
return resp.Data, resp.Provider, nil
}
More efficent mapping lookup (#13975) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Inspired by a comment from Zaid. This allows providers to return what providers they have mapping information for without having to marshal all their mapping data to the engine at the same time, this could save transmitting a lot of data that the engine might not ever need (for example if it's not converting code for that specific provider). It also allows provider to support mulitple mappings. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-09-21 11:45:07 +00:00
func (p *provider) GetMappings(key string) ([]string, error) {
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
label := p.label() + ".GetMappings"
More efficent mapping lookup (#13975) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Inspired by a comment from Zaid. This allows providers to return what providers they have mapping information for without having to marshal all their mapping data to the engine at the same time, this could save transmitting a lot of data that the engine might not ever need (for example if it's not converting code for that specific provider). It also allows provider to support mulitple mappings. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-09-21 11:45:07 +00:00
logging.V(7).Infof("%s executing: key=%s", label, key)
resp, err := p.clientRaw.GetMappings(p.requestContext(), &pulumirpc.GetMappingsRequest{
Key: key,
})
if err != nil {
rpcError := rpcerror.Convert(err)
code := rpcError.Code()
if code == codes.Unimplemented {
// For backwards compatibility just return nil to indicate unimplemented.
logging.V(7).Infof("%s unimplemented", label)
return nil, nil
}
logging.V(7).Infof("%s failed: %v", label, rpcError)
return nil, err
}
logging.V(7).Infof("%s success: providers=%v", label, resp.Providers)
// Ensure we don't return nil here because we use it as an "unimplemented" flag elsewhere in the system
if resp.Providers == nil {
resp.Providers = []string{}
}
return resp.Providers, nil
}