Fix an issue with UpgradeToOutputValues that tripped an assert in
ResourceReferences.
Also shouldn't matter much but made UpgradeToOutputValues require
KeepOutputValues set as well.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds a new option `UpgradeToOutputValues` to `MarshalOptions`. When
set all `Computed` and `Unknown` values are _upgraded_ to `Output`
values. This reduces the number of cases that need to be handled later
in the codebase.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
This PR is motivated by https://github.com/pulumi/pulumi/pull/15145.
`resource.*` should be built on top of `property.Value`,[^1] which means
that `resource`
needs to be able to import `property.Value`, and so `property` cannot
import
`resource`. Since Assets and Archives are both types of properties, they
must be moved out
of `resource`.
[^1]: For example:
a1d686227c/sdk/go/common/resource/resource_state.go (L35-L36)
## Open Question
This PR moves them to their own sub-folders in `resource`. Should
`asset` and `archive`
live somewhere more high level, like `sdk/go/property/{asset,archive}`?
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
github.com/golang/protobuf is marked deprecated and I was getting
increasingly triggered by the inconsistency of importing the `Empty`
type from "github.com/golang/protobuf/ptypes/empty" or
"google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty"
or "emptypb". Similar for the struct type.
So this replaces all the Protobufs imports with ones from
"google.golang.org/protobuf", normalises the import name to always just
be the module name (emptypb), and adds the depguard linter to ensure we
don't use the deprecated package anymore.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This is a pass over all of /sdk to replace asserts that just checked we
had an error with asserts for what the error value is.
Just checking for an error is a weak test that can result in error paths
being broken and tests not detecting it.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Likewise `require.NoError` instead of `require.Nil`, and `assert.Error`
rather than `assert.NotNil`.
The error variants of these functions print the errors nicer for test
failures using `Error()` rather than `GoString()`.
For bail errors this is _much_ better than the `result.Result` days
where we now get errors like:
```
Error: Received unexpected error:
BAIL: inner error
```
instead of:
```
Error: Expected nil, but got: &simpleResult{}
```
Also print the bail error in `TestPlan.Run` so we can see the
description of it.
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
Migrates all uses of contract.{Assert, AssertNoError, Require} in sdk/
to the `*f` variants that are required to provide more error context.
Step towards deprecating non-f variants entirely.
For context, `contract.Require` is similar to `contract.Assert`,
except it has a required parameter name as an argument:
func Require(cond bool, param string)
func Requiref(cond bool, param string, msg string, args ...any)
It includes the parameter name in the error message by default,
so the `msg` and `args` should only describe the constraint
without naming the parameter.
Refs #12132
TestAssetSerialize prints to stdout from the test.
This seems like a leftover from debugging.
Drop that in favor of a t.Logf so that if the test fails,
the print is still available in the output.
* about error string: error string should not be capitalized or end with punctuation mark
* apply suggestions from code review
* lint: fix format error and simplify the code
Co-authored-by: Fraser Waters <frassle@gmail.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
* Enable output values by default
Enable output values by default in the resource monitor and change the polarity of the envvar from `PULUMI_ENABLE_OUTPUT_VALUES` to `PULUMI_DISABLE_OUTPUT_VALUES`.
* Marshal unknown as unknown string when `!KeepOutputValues`
Marshal all unknown output values as `resource.MakeComputed(resource.NewStringProperty(""))` when not keeping output values, which is consistent with what the SDKs do.
Otherwise, when `v.OutputValue().Element` is nil, `resource.MakeComputed(v.OutputValue().Element)` will be marshaled as a null value rather than as an unknown sentinel.
* Add MarshalOptions.DontSkipOutputs and use where needed
Before we expanded the meaning of `resource.Output`, `MarshalProperties` always skipped output values:
```go
if v.IsOutput() {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
As part of expanding the meaning of `resource.Output`, I'd adjusted `MarshalProperties` to only skip output values when the value was unknown and when not keeping output values:
```go
if v.IsOutput() && !v.OutputValue().Known && !opts.KeepOutputValues {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
However, this doesn't work the way we want when marshaling properties that include unknown output values to a provider that does not accept outputs. In that case, `opts.KeepOutputValues` will be `false` because we want the marshaler to fall back to returning non-output-values (e.g. unknown sentinel value for unknown output values), but instead of getting the intended fallback values, the unknown output values are skipped (not what we want).
I suspect we may be able to delete the output value skipping in `MarshalProperties` altogether (it's odd that it is skipping `resource.Output` but not `resource.Computed`), but to avoid any unintended side effects of doing that, instead, this commit introduces a new `MarshalOptions.DontSkipOutputs` option that can be set to `true` to opt-in to not skipping output values when marshaling. The check in `MarshalProperties` now looks like this:
```go
if !opts.DontSkipOutputs && v.IsOutput() && !v.OutputValue().Known {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
`opts.DontSkipOutputs` is set to `true` when marshaling properties for calls to a provider's `Construct` and `Call`.
* [sdk/nodejs] Deserialize output values
This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.
* [sdk/python] Deserialize output values
This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.
This change expands the definition of `resource.Output` in the Go SDK with additional information about the output, i.e. dependencies and secretness, and adds support in the core Go RPC code for (un)marshaling output values.
Output values are marshaled as special objects ala archives, assets, and resource refs and are unmarshaled as `resource.Output` values.
Subsequent PRs will add:
- A monitor feature for output values, which will initially be disabled by default but available to turn on via an envvar
- Support for (un)marshaling output values in each language SDKs
- A way for providers to indicate support for receiving output values
- E2E tests
- Turn the monitor feature on by default (w/ env var to disable) (Note: the current plan is to initially scope this to only be used when marshaling inputs to a multi-language component)
Currently whenever an issue occurs in `UnmarshalProperties` and
`MarshalProperties` the offending property is hidden and very difficult
to track down.
This commit changes the behavior. For `Assets` and `Archives` the error
message now includes the URI and for other properties it includes the
key of the `PropertyMap`.
* Fix resource-ref-as-ID marshaling. (#6125)
This reapplies 2f0dba23ab.
* Fix malformed resource value bug
PR #6125 introduced a bug by marshaling resource
ids as PropertyValues, but not handling that case on
the unmarshaling side. The previous code assumed
that the id was a simple string value. This bug prevents
any stack update operations (preview, update, destroy,
refresh). Since this change was already
released, we must now handle both cases in the
unmarshaling code.
* Add resource ref unit tests for the Go SDK. (#6142)
This reapplies 3d505912b8.
Co-authored-by: Pat Gavlin <pat@pulumi.com>
When marshaling a resource reference as its ID (i.e. when
opts.KeepResources is false, as it will be in the case of downlevel SDKs
and resource providers), we must take care to marshal/unmarshal an empty
ID as the unknown property value.
This includes the following changes to the resource ref APIs:
- Bifurcate resource reference creation into two methods: one for
creating references to custom resources and one for creating
references to component resources.
- Store the ID in a resource reference as a PropertyValue s.t. it can be
computed.
- Add a helper method for retrieving the ID as a string + an indicator of
whether or not the reference has an ID.
Fixes#5939.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)