Commit Graph

39 Commits

Author SHA1 Message Date
kvthr 086a75d18d
More descriptive exception in serialize_property ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-10 11:34:18 +00:00
Fraser Waters f0085de0c7
Handful of mypy fixes in the python test code () 2024-05-02 17:46:52 +00:00
Fraser Waters 3d3729006c
Start mypy linting lib/test ()
This fixes up a load of '[un]expected return statement' errors in the
tests. Mostly property getters which I've filled in with `pulumi.get`
but some of the tests were explictly checking things worked when the
getter was an empty method body.
2024-04-30 12:54:13 +00:00
Fraser Waters 48dbd6c596
Use black to format lib/test ()
Test code should be formatted and linted the same as library code. This
is the first step of that, simply including ./lib/test to the folder
that the black formatter runs on.
2024-04-23 08:29:58 +00:00
Thomas Gummerer c91d40ac5f
allow unknowns in apply during update ()
Currently during updates we try to make the values known during apply,
even if they are marked unknown. This results in knows that should
really still be unknown.

Note that this behaviour is currently encoded in tests, which also
needed to change.
2024-04-10 16:48:38 +00:00
Fraser Waters 8db3087d2b Make pythons RPCManager a context variable 2023-07-13 09:59:12 +01:00
Fraser Waters b150120506 Allow tuples as sequences in Python serialization
Fixes https://github.com/pulumi/pulumi/issues/7029

For some reason the python RPC serializer explictly disallowed tuple,
despite python codegen outputing types using `Sequence` rather than
`List` suggesting that `tuple` would be acceptable.
2023-06-20 14:26:00 +01:00
Kyle Pitzen 122d7c25b9
fix(sdk/python): Makes SETTINGS attributes context-aware () 2022-08-23 13:45:37 -04:00
Ian Wahbe d3b2dedd1d
[sdk/python] Unmarshal output values in component providers () 2021-11-15 10:12:12 -08:00
Justin Van Patten 3027d01f25
Enable output values by default ()
* Enable output values by default

Enable output values by default in the resource monitor and change the polarity of the envvar from `PULUMI_ENABLE_OUTPUT_VALUES` to `PULUMI_DISABLE_OUTPUT_VALUES`.

* Marshal unknown as unknown string when `!KeepOutputValues`

Marshal all unknown output values as `resource.MakeComputed(resource.NewStringProperty(""))` when not keeping output values, which is consistent with what the SDKs do.

Otherwise, when `v.OutputValue().Element` is nil, `resource.MakeComputed(v.OutputValue().Element)` will be marshaled as a null value rather than as an unknown sentinel.

* Add MarshalOptions.DontSkipOutputs and use where needed

Before we expanded the meaning of `resource.Output`, `MarshalProperties` always skipped output values:

```go
if v.IsOutput() {
    logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```

As part of expanding the meaning of `resource.Output`, I'd adjusted `MarshalProperties` to only skip output values when the value was unknown and when not keeping output values:

```go
if v.IsOutput() && !v.OutputValue().Known && !opts.KeepOutputValues {
    logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```

However, this doesn't work the way we want when marshaling properties that include unknown output values to a provider that does not accept outputs. In that case, `opts.KeepOutputValues` will be `false` because we want the marshaler to fall back to returning non-output-values (e.g. unknown sentinel value for unknown output values), but instead of getting the intended fallback values, the unknown output values are skipped (not what we want).

I suspect we may be able to delete the output value skipping in `MarshalProperties` altogether (it's odd that it is skipping `resource.Output` but not `resource.Computed`), but to avoid any unintended side effects of doing that, instead, this commit introduces a new `MarshalOptions.DontSkipOutputs` option that can be set to `true` to opt-in to not skipping output values when marshaling. The check in `MarshalProperties` now looks like this:

```go
if !opts.DontSkipOutputs && v.IsOutput() && !v.OutputValue().Known {
    logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```

`opts.DontSkipOutputs` is set to `true` when marshaling properties for calls to a provider's `Construct` and `Call`.

* [sdk/nodejs] Deserialize output values

This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.

* [sdk/python] Deserialize output values

This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.
2021-09-24 08:57:04 -07:00
Justin Van Patten df50c4492e
[sdk/python] Fix serializing output values within dicts ()
Pass along the `keep_output_values` param when serializing values within dicts.
2021-09-17 20:53:29 -07:00
Justin Van Patten 4f3f366695
[sdk/python] Marshal output values ()
This change adds support for marshaling outputs as output values in the Python SDK.
2021-09-15 21:49:23 -07:00
Paul Stack e955a6b06a Refactor Mock newResource and call to accept property bag rather than individual args () 2021-04-14 19:32:18 +01:00
Justin Van Patten 1112c513c0 [sdk/python] Improved dict key translation support ()
This change addresses Python dictionary key translation issues. When the
type of `props` passed to the resource is decorated with `@input_type`,
the type's and resource's property name metadata will be used for dict
key translations instead of the resource's `translate_input_property`
and `translate_output_property` methods.

The generated provider SDKs will be updated to opt-in to this new
behavior:

- FIX: Keys in user-defined dicts will no longer be unintentionally
  translated/modified.

- BREAKING: Dictionary keys in nested output classes are now
  consistently snake_case. If accessing camelCase keys from such output
  classes, move to accessing the values via the snake_case property
  getters (or snake_case keys). Generated SDKs will log a warning
  when accessing camelCase keys.

When serializing inputs:

  - If a value is a dict and the associated type is an input type, the
    dict's keys will be translated based on the input type's property
    name metadata.

  - If a value is a dict and the associated type is a dict (or Mapping),
    the dict's keys will _not_ be translated.

When resolving outputs:

  - If a value is a dict and the associated type is an output type, the
    dict's keys will be translated based on the output type's property
    name metadata.

  - If a value is a dict and the associated type is a dict (or Mapping),
    the dict's keys will _not_ be translated.
2021-04-14 19:32:18 +01:00
Komal 7f226c1cb6
Fix serializing bug if output contains 'items' property () 2021-04-05 20:36:56 -07:00
Komal ca5ab18f6d
[sdk/python] - Allow Output.all with no inputs () 2021-02-18 10:11:13 -08:00
Komal 1a9050eaa6
[sdk/python] - Allow kwargs in Output.all() to create a dict () 2021-02-11 08:52:46 -07:00
Komal f374b8a953
[python/sdk] - Correctly handle outputs with properties named "values" ()
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2021-02-05 17:16:13 -07:00
Justin Van Patten 2779de38ea
[sdk/python] from_input: Unwrap nested outputs in input types ()
`Output.from_input` deeply unwraps nested output values in dicts and lists, but doesn't currently do that for the more recently added "input types" (i.e. args classes). This leads to errors when using args classes with output values with `Provider` resources, which uses `Output.from_input` on each input property and then serializes the value to JSON in an `apply`. This changes fixes `Output.from_input` to recurse into values within args classes to properly unwrap any nested outputs.
2021-01-29 15:44:00 -08:00
Paul Stack 1cff1a4bcf
[sdk/python] Adding pulumi.unsecret and pulumi.is_secret () 2021-01-14 20:37:05 +00:00
Pat Gavlin 268e50d00e
Add Python resource ref unit tests. ()
- Improve the existing coverage to use real resources and mocks
- Add tests for deserialization as well as serialization
- Add tests that serialize custom resources during preview

Contributes to .
2020-12-18 23:02:48 -08:00
Pat Gavlin 8e278ca9d3
Improve Python serialization test harness. ()
- Reset the runtime prior to each test
- Use the SDK's `test` decorator instead of `async_test`
- Rename a couple classes to avoid warnings from pytest
2020-12-17 14:45:18 -08:00
Komal 48f43906f4
[sdk/python] - Support enums ()
Co-authored-by: Pat Gavlin <pat@pulumi.com>
2020-11-24 19:15:11 -06:00
Pat Gavlin 3d2e31289a
Add support for serialized resource references. ()
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.

This is the foundation of cross-process resources.

Related to .

Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
2020-10-27 10:12:12 -07:00
Justin Van Patten d0ba9fbdcd
[sdk/python] Add support for Sequence ()
We currently emit array types as `List[T]` for Python, but `List[T]` is invariant, which causes type checkers like mypy to produce errors when values like `["foo", "bar"]` are passed as args typed as `List[pulumi.Input[str]]` (since `Input[str]` is an alias for `Union[T, Awaitable[T], Output[T]]`. To address this, we should move to using [`Sequence[T]`](https://docs.python.org/3/library/typing.html#typing.Sequence) which is covariant, and does not have this problem.

We actually already do this for `Dict` vs. `Mapping`, emitting map types as `Mapping[str, T]` rather than `Dict[str, T]` because `Mapping[str, T]` is covariant for the value. This change makes us consistent for array types.

These are the SDK changes necessary to support `Sequence[T]`.
2020-09-08 22:22:35 -07:00
Justin Van Patten 69fbd70330
Python: Ignore internal properties when unmarshaling () 2020-08-28 18:26:16 -07:00
Justin Van Patten cd9fae599d
Python SDK changes to support input/output classes ()
Python SDK changes to support strongly-typed input/output "dataclasses".
2020-08-19 01:15:56 -07:00
Justin Van Patten dd104a00a7
Propagate secretness correctly in Python `apply` ()
* Propagate secretness correctly in Python `apply`

* Improve `apply` test coverage

* Update CHANGELOG.md
2020-04-02 13:01:29 -07:00
CyrusNajmabadi f4fc00ad0e
Output.apply should lift resources from inner Outputs to the top level output. () 2019-12-17 14:11:45 -08:00
Pat Gavlin 137fd54f1c
Propagate inputs to outputs during preview. ()
These changes restore a more-correct version of the behavior that was
disabled with . The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes .
2019-11-11 12:09:34 -08:00
Pat Gavlin 834e583c95
Revert "Propagate inputs to outputs during preview. ()" ()
This reverts commit 80504bf0bc.
2019-10-10 10:33:05 -07:00
Pat Gavlin 80504bf0bc
Propagate inputs to outputs during preview. ()
These changes restore a more-correct version of the behavior that was
disabled with . The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes .
2019-09-30 11:03:58 -07:00
Pat Gavlin 7fef102bc3
Check for valid PB types in serialize_property ()
Just what it says on the tin. This allows us to return an incrementally
better error.

Fixes .
2019-08-09 16:48:28 -07:00
Chris Smith 208dfc29b4
Update references to pulumi.io ()
* Remove pulumi.io reference in tests

* Remove pulumi.io references in Dockerfiles

* Remove pulumi.io references in CONTRIBUTING.md

* Update README.md's

* Use correct logo
2019-07-25 09:58:12 -07:00
Matt Ellis 858517a7ca Correctly push secretness up during deserialization in runtimes
There current RPC model for Pulumi allows secret values to be deeply
embedded in lists or maps, however at the language level, since we
track secrets via `Output<T>` we need to ensure that during
deserialization, if a list or a map contains a secret, we need to
instead treat it as if the entire list or map was a secret.

We have logic in the language runtimes to do this as part of
serialization. There were a few issues this commit addresses:

- We were not promoting secretness across arrays in either Node or
  Python
- For Python, our promotion logic was buggy and caused it to behave in
  a manner where if any value was secret, the output values of the
  object would be corrupted, because we'd incorrectly treat the
  outputs as a secret who's value was a map, instead of a map of
  values (some of which may be secret).

This caused very confusing behavior, because it would appear that a
resource creation call just did not set various output properties when
one or more of them ended up containing a secret.
2019-07-09 10:40:27 -07:00
Matt Ellis ae4d126d8c Support Secrets in Python SDK
A new static method, `secret` now exists on `pulumi.Output` which can
be used to create secrets.
2019-05-10 17:07:52 -07:00
Pat Gavlin 35c60d61eb
Follow up on ()
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
  layers and language SDKs
2019-01-28 17:38:16 -08:00
Sean Gillespie 81c0de1e4e
Add 'Output.all' combinator for Python ()
* Add 'Output.all' combinator for Python

Output.all is a useful combinator that we already have in Node that
allows the composition of a list of outputs into an output of a list.
This is very useful when authoring components and its lack of presence
in Python was an oversight.

This commit adds 'Output.all' and 'Output.from_input', adding tests and
documentation for each.

* start unwrap

* Add functionality and test for nested inputs
2018-12-18 13:22:04 -08:00
Sean Gillespie 56be1a6677
Implement RPC for Python 3 ()
* Implement RPC for Python 3

* Try not setting PYTHONPATH

* Remove PYTHONPATH line

* Implement Invoke for Python 3

* Implement register resource

* progress

* Rewrite the whole thing

* Fix a few bugs

* All tests pass

* Fix an abnormal shutdown bug

* CR feedback

* Provide a hook for resources to rename properties

As dictionaries and other classes come from the engine, the
translate_property hook can be used to intercept them and rename
properties if desired.

* Fix variable names and comments

* Disable Python integration tests for now
2018-10-31 13:35:31 -07:00