pulumi/pkg/resource/deploy
Pat Gavlin 9d0b0fed91 [engine] Support remapping envvars for providers
These changes add support for remapping environment variables when
launching providers. This allows users to work around problems with
dynamic provider configuration that is stored in statefiles causing
problems during refresh and destroy operations.

For a bit of background: `pulumi up` is distinctly different from
`pulumi destroy` and `pulumi refresh` in that it involves running the
Pulumi program associated with the stack's project. As it runs, the
Pulumi program defines the desired state for resources--including
provider resources--using values computed by the program in coordination
with the Pulumi engine. When the program creates a provider resource,
the inputs for the provider are either sourced from the program itself
(i.e. from values provided by the program) or are read out-of-band by
the provider plugin. The exact set of configuration that may be sourced
from the environment is particular to each provider--for example, the
Kubernetes provider uses the ambient `kubeconfig` by default, the AWS
provider reads various AWS-specific environment variables, etc. Any
_explicitly-provided inputs_ are written into the stack's statefile.

For example, consider the following program:

```typescript
import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", { region: "us-east-1" });
const defaultRegion = new aws.Provider("default-region");
```

The `usEast1` provider's `region` is explicitly specified by the
program, but the `defaultRegion` provider's `region` will be read from
the environment (e.g. from the `AWS_REGION` environment variable). In
the resulting statefile, the `usEast1` provider's state will include the
`region` input, but the `defaultRegion` provider's state will not.

Because `pulumi refresh` and `pulumi destroy` do not run the Pulumi
program associated with the stack's project, they are unable to
recompute configuration values that were explicitly provided by the
program, and must use the values stored in the statefile. Unfortunately,
this may include credential information, which is what causes the errors
described here. The current workaround--which is certainly not
sufficient for explicitly-instantiated providers--is to use environment
variables to provide credentials out-of-band.

The clearest/most complete solution here is to run the Pulumi program
associated with a stack's project as part of `pulumi refresh` and
`pulumi destroy`. Unfortunately, this is a _major_ behavioral change,
and the exact semantics of the run are not clear.

These changes allow explicitly-instantiated providers to make use of the
same workaround that is available to default providers: pass dynamic,
environmentally-sourced provider configuration in environment variables
rather than as provider inputs. The environment variable remapping allows
users to replace the value for a provider environment variable with the
value of a different environment variable before the provider is loaded.
This allows users to place configuration in environment variables that
the provider would not normally read and remap them to
provider-supported envvars, which allows multiple distinct sets of
environment variables for providers.

For the example above, this might look like so:

```typescript
import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", {
    pluginEnvVars: { "AWS_REGION": { from: "US_EAST_1_REGION" } },
});
const defaultRegion = new aws.Provider("default-region");
```

Or, if the providers needed different credentials (much more common):

```typescript
import * as aws from "@pulumi/aws";

const usEast1 = new aws.Provider("us-east-1", {
    pluginEnvVars: {
	"AWS_ACCESS_KEY_ID": { from: "US_EAST_1_AWS_ACCESS_KEY_ID" },
	"AWS_SECRET_ACCESS_KEY": { from: "US_EAST_1_AWS_SECRET_ACCESS_KEY" },
	"AWS_SESSION_TOKEN": { from: "US_EAST_1_AWS_SESSION_TOKEN" },
    },
});
const defaultRegion = new aws.Provider("default-region");
```
2024-06-13 13:44:11 -07:00
..
deploytest [engine] Support remapping envvars for providers 2024-06-13 13:44:11 -07:00
providers [engine] Support remapping envvars for providers 2024-06-13 13:44:11 -07:00
builtins.go Plugin interface cleanup (#16293) 2024-05-31 00:28:48 +00:00
builtins_test.go Add Paramaterize to provider interface (#16174) 2024-05-15 16:22:39 +00:00
deployment.go Use a generic wrapper around `sync.Map` (#15878) 2024-04-09 10:56:25 +00:00
deployment_executor.go fix panic with --continue-on-error on delete after failed create (#16261) 2024-05-30 17:18:34 +00:00
deployment_executor_test.go Better handle property dependencies and `deletedWith` (#16088) 2024-05-03 17:08:06 +00:00
deployment_test.go Allow anything in resource names (#14107) 2023-11-20 08:59:00 +00:00
doc.go Document Go packages (#6009) 2021-01-11 11:07:59 -07:00
import.go Revert "Revert "Run integration tests and dev builds with race detection" (#15998)" (#16148) 2024-05-09 16:15:41 +00:00
import_test.go upgrade to latest version of golangci-lint (#15977) 2024-04-19 06:20:33 +00:00
manifest.go Split Manifest type to it's own file (#8712) 2022-01-10 12:25:24 +00:00
manifest_test.go Refactor: move plugin kind to apitype (#15946) 2024-04-25 17:30:30 +00:00
plan.go Enable perfsprint linter (#14813) 2023-12-12 12:19:42 +00:00
plan_test.go [ci] `pkg/resource/deploy` coverage (#14831) 2023-12-19 16:14:40 +00:00
snapshot.go Revert "Revert "Run integration tests and dev builds with race detection" (#15998)" (#16148) 2024-05-09 16:15:41 +00:00
snapshot_test.go Revert "Revert "Run integration tests and dev builds with race detection" (#15998)" (#16148) 2024-05-09 16:15:41 +00:00
source.go Implement up --continue-on-error (#15740) 2024-04-22 11:12:45 +00:00
source_error.go Change Source.Iterate to return an error (#13996) 2023-09-20 14:34:24 +00:00
source_error_test.go [ci] `pkg/resource/deploy/source_error.go` coverage (#14982) 2023-12-22 20:57:52 +00:00
source_eval.go RegisterProvider engine work (#16241) 2024-05-23 06:16:59 +00:00
source_eval_test.go RegisterProvider engine work (#16241) 2024-05-23 06:16:59 +00:00
source_null.go Change Source.Iterate to return an error (#13996) 2023-09-20 14:34:24 +00:00
source_query.go RegisterProvider engine work (#16241) 2024-05-23 06:16:59 +00:00
source_query_test.go Refactor: move plugin kind to apitype (#15946) 2024-04-25 17:30:30 +00:00
state_builder.go Revert "Revert "Run integration tests and dev builds with race detection" (#15998)" (#16148) 2024-05-09 16:15:41 +00:00
state_builder_test.go When changing parents also fix URNs (#13935) 2023-09-14 19:52:27 +00:00
step.go Propagate deleted dependencies of untargeted resources (#16247) 2024-05-23 12:31:03 +00:00
step_executor.go Revert "Revert "Run integration tests and dev builds with race detection" (#15998)" (#16148) 2024-05-09 16:15:41 +00:00
step_executor_test.go Use a generic wrapper around `sync.Map` (#15878) 2024-04-09 10:56:25 +00:00
step_generator.go Propagate deleted dependencies of untargeted resources (#16247) 2024-05-23 12:31:03 +00:00
step_generator_test.go Check property dependencies and `DeletedWith` for target dependents (#16220) 2024-05-17 15:07:13 +00:00
step_test.go Use a generic wrapper around `sync.Map` (#15878) 2024-04-09 10:56:25 +00:00
target.go Add tokens.StackName (#14487) 2023-11-15 07:44:54 +00:00
target_test.go upgrade to latest version of golangci-lint (#15977) 2024-04-19 06:20:33 +00:00