pulumi/tests/integration
Justin Van Patten 472e6bdd50
Allow Python dynamic providers to capture secrets (#15864)
As of https://github.com/pulumi/pulumi/pull/13315 (which shipped in
[v3.75.0](https://github.com/pulumi/pulumi/releases/tag/v3.75.0)),
`__provider` (the serialized provider string) is _always_ set to a
secret in the state. This can lead to poor performance when there are a
lot of dynamic resources and the serialized provider does not actually
have any secrets.

This change does two things:
1. Provides a way to opt-out of always serializing the provider as a
secret.
2. Allows Outputs to be captured during serialization of the provider,
which wasn't previously possible.

## 1. Opt-out of always serializing as secret

A new attribute, `serialize_as_secret_always`, can be set on a subclass
of `ResourceProvider` to opt-out of always serializing the provider as a
secret. If you know you don't have any secrets being serialized into the
provider and want to avoid the encryption overhead, you can set this
attribute to `False`.

```python
class MyProvider(ResourceProvider):
    serialize_as_secret_always = False

    def create(self, props):
        # Doesn't have any secrets that need encrypting
        ...
```

## 2. Allow Outputs to be captured

If you currently try to capture an `Output`, it fails when serializing
the provider with:

```
TypeError: cannot pickle '_asyncio.Future' object
```

This change allows Outputs to be captured/serialized for dynamic
providers, including secret Outputs.

This aligns Python dynamic providers with
[Node.js](https://github.com/pulumi/pulumi/pull/13329).

```python
import pulumi
from pulumi.dynamic import CreateResult, Resource, ResourceProvider

config = pulumi.Config()
password = config.require_secret("password")

class SimpleProvider(ResourceProvider):
    def create(self, props):
        # Need to use `password.get()` to get the underlying value of the secret from within the serialized code.
        # This simulates using this as a credential to talk to an external system.
        return CreateResult("0", { "authenticated": "200" if password.get() == "s3cret" else "401" })

class SimpleResource(Resource):
    authenticated: pulumi.Output[str]

    def __init__(self, name):
        super().__init__(SimpleProvider(), name, { "authenticated": None })


r = SimpleResource("foo")
pulumi.export("out", r.authenticated)
```

Note: In the above example, we didn't have to specify
`serialize_as_secret_always` since the default behavior is to always
serialize the provider as a secret. If we wanted to, we could have
specified `serialize_as_secret_always = False` and it still would have
serialized the provider as a secret, since it captured `password` which
is a secret Output. If `serialize_as_secret_always = False` was
specified and no secrets were captured, then the provider would not be
serialized as a secret.

We plan to recommend this approach to capturing secrets for both Node.js
and Python dynamic providers after this change.

Fixes #15539
2024-08-02 23:51:52 +00:00
..
about Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
aliases Tiny cleanup to an integration test (#16856) 2024-08-01 23:15:22 +00:00
backend/diy Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
cloud_secrets_provider Enable full strict mode. (#3218) 2019-09-11 16:21:35 -07:00
component_provider_schema Add RunPlugin support for python 2023-03-06 21:35:39 +00:00
config_basic Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
config_capture_e2e/nodejs Revert package.json lookup fix. 2023-05-15 14:14:57 -04:00
config_missing Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
config_secrets_warn Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_configure_provider Changelog and go.mod updates for v3.127.0 (#16805) 2024-07-25 23:47:22 +00:00
construct_component_error_apply Update @types/node version in tests 2023-02-07 14:20:27 +00:00
construct_component_id_output [python/sdk] Allow remote components to use output property called id (#15115) 2024-01-18 13:54:09 +00:00
construct_component_methods Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_methods_errors Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_methods_provider Bump the go_modules group across 5 directories with 1 update (#16054) 2024-04-25 21:11:41 +00:00
construct_component_methods_resources Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_methods_unknown Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_output_values Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_plain Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_provider Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_provider_explicit Bump go modules (#16051) 2024-04-25 14:30:00 +00:00
construct_component_provider_propagation Query language runtime for options during “pulumi new” (#16346) 2024-06-17 17:10:55 +00:00
construct_component_resource_options Query language runtime for options during “pulumi new” (#16346) 2024-06-17 17:10:55 +00:00
construct_component_slow Update @types/node version in tests 2023-02-07 14:20:27 +00:00
construct_component_unknown Bump the go_modules group across 5 directories with 1 update (#16054) 2024-04-25 21:11:41 +00:00
construct_nested_component/go Query language runtime for options during “pulumi new” (#16346) 2024-06-17 17:10:55 +00:00
custom_timeouts Rename "Smoke" test to "Acceptance" tests 2023-01-30 15:38:37 -05:00
delete_before_create Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
deleted_with Bump the go_modules group across 5 directories with 1 update (#16054) 2024-04-25 21:11:41 +00:00
dependency_steps Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
double_pending_delete Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
duplicate_urns Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
dynamic Allow Python dynamic providers to capture secrets (#15864) 2024-08-02 23:51:52 +00:00
ee_perf Use prefered `new pulumi.Config()` form 2019-01-31 16:11:57 -08:00
empty Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
enums [tests/integration] Remove `pulumi` from `requirements.txt` (#15226) 2024-01-24 14:16:40 +00:00
environments_basic [environments] Add integration tests (#14144) 2023-10-12 16:39:26 +00:00
environments_merge [environments] Add integration tests (#14144) 2023-10-12 16:39:26 +00:00
exclude_protected Implement the --exclude-protected feature (#8359) 2021-11-15 11:45:14 -08:00
explicit_provider Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
gather_plugin Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
get_created Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
get_resource Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
go Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
invalid_package_json Revert package.json lookup fix. 2023-05-15 14:14:57 -04:00
large_resource Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
nodejs Detect possible OOM errors when nodejs crashes (#16700) 2024-07-23 16:10:43 +00:00
partial_state Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
partial_values Propagate inputs to outputs during preview. (#3327) 2019-11-11 12:09:34 -08:00
policy Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
printf Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
project_main Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
project_main_abs Enable absolute and relative parent paths for pulumi main (#6734) 2021-04-08 21:39:52 -07:00
project_main_parent/foo Enable absolute and relative parent paths for pulumi main (#6734) 2021-04-08 21:39:52 -07:00
protect_resources Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
provider_secret_config Enable full strict mode. (#3218) 2019-09-11 16:21:35 -07:00
python Add support for parameterized invokes in Python (#16832) 2024-07-29 14:51:07 +00:00
python_await Cleanup output futures list as they complete (#16057) 2024-04-26 16:35:28 +00:00
query Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
read Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
recreate_resource_check Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
refresh/go Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
resource_refs_get_resource Bump the go_modules group across 24 directories with 1 update (#15685) 2024-03-14 07:52:34 +00:00
rotate_passphrase Bump the go_modules group across 5 directories with 1 update (#16054) 2024-04-25 21:11:41 +00:00
secret_outputs [sdk/python] Fix secret regression (#5496) 2020-10-01 14:57:51 -07:00
single_resource Suppress JSON outputs in preview correctly (#2771) 2019-05-25 12:10:38 +02:00
stack_bad_parenting Consistent dependencies (#2517) 2019-03-05 20:34:51 -08:00
stack_dependencies Remove existing lock files 2018-11-12 15:33:58 -08:00
stack_outputs Remove dotnet 2022-12-13 16:13:53 +00:00
stack_outputs_program_error Fix: Don't delete stack outputs on failed deployments (#15754) 2024-03-25 22:37:46 +00:00
stack_outputs_resource_error Fix: Don't delete stack outputs on failed deployments (#15754) 2024-03-25 22:37:46 +00:00
stack_parenting Consistent dependencies (#2517) 2019-03-05 20:34:51 -08:00
stack_project_name Actually fix the naming test (#10554) 2022-08-31 20:52:23 +01:00
state_rename_parent Bump the go_modules group across 5 directories with 1 update (#16054) 2024-04-25 21:11:41 +00:00
steps Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
targets Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
transformations Remove unused `package.json` files from tests (#13162) 2024-07-08 14:09:16 +00:00
transforms sdk/python: add support for stack invoke transforms (#16701) 2024-07-19 08:01:23 +00:00
tsconfig Add tsconfig option to specify tsconfig path (#8452) 2021-11-22 11:42:39 -08:00
types Fix remaining lint issues (#14688) 2023-12-02 17:16:09 +00:00
unsafe_snapshot_tests/bad_resource Update tests/integration/unsafe_snapshot_tests/bad_resource/resource.ts 2022-09-15 08:00:36 -07:00
valid-property-names [testing] Add integration tests for regressions in property name parsing (#14681) 2023-12-01 17:27:33 +00:00
.gitignore ci: Enable async component builds 2022-09-14 10:06:05 -07:00
appdash_test.go Reimport appdash from our mirror (#14701) 2023-11-30 14:21:35 +00:00
component_setup.sh Engine and Golang support for shimless providers 2022-11-14 11:25:41 +00:00
integration_acceptance_test.go Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
integration_go_acceptance_test.go test: Update deps (#16010) 2024-04-24 05:07:52 +00:00
integration_go_test.go Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
integration_nodejs_acceptance_test.go Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
integration_nodejs_test.go Detect possible OOM errors when nodejs crashes (#16700) 2024-07-23 16:10:43 +00:00
integration_python_acceptance_test.go Run Environment.DeleteIfNotFailed after tests complete (#16730) 2024-07-23 10:37:01 +00:00
integration_python_test.go Allow Python dynamic providers to capture secrets (#15864) 2024-08-02 23:51:52 +00:00
integration_test.go Ensure internal provider state doesn't clash with user config (#16837) 2024-07-30 12:22:32 +00:00
integration_util_test.go Replace some more uses of assert.Contains(err.Error()) with assert.ErrorContains (#14863) 2023-12-15 17:45:32 +00:00