<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/14631.
This adds a couple of new tests to codegen. Firstly to check that if you
use "urn" or "id" as a resource output property we fail schema binding.
This is because these clash with the urn and id fields that every
resource already has to have.
Secondly a schema gen test that checks that urn and id _do_ work in
other places, i.e. resource inputs, nested types, etc.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
Support returning plain values from methods.
Implements Node, Python and Go support.
Remaining:
- [x] test receiving unknowns
- [x] acceptance tests written and passing locally for Node, Python, Go
clients against a Go server
- [x] acceptance tests passing in CI
- [x] tickets filed for remaining languages
- [x] https://github.com/pulumi/pulumi-yaml/issues/499
- [x] https://github.com/pulumi/pulumi-java/issues/1193
- [x] https://github.com/pulumi/pulumi-dotnet/issues/170
Known limitations:
- this is technically a breaking change in case there is code out there
that already uses methods that return Plain: true
- struct-wrapping limitation: the provider for the component resource
needs to still wrap the plain-returning Method response with a 1-arg
struct; by convention the field is named "res", and this is how it
travels through the plumbing
- resources cannot return plain values yet
- the provider for the component resource cannot have unknown
configuration, if it does, the methods will not be called
- Per Luke https://github.com/pulumi/pulumi/issues/11520 this might not
be supported/realizable yet
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/12709
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Fixes https://github.com/pulumi/pulumi/issues/12738https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.
Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
Fixes https://github.com/pulumi/pulumi-terraform-bridge/issues/1201
hcl.Diagnostics implements the Error interface, but this means a list of
zero diagnostics still looks like a non-nil error, which throws of
normal "if err == nil" checks.
This changes a few use sites of hcl.Diagnostics to ensure we return nil
when there aren't any diagnostics rather than an empty slice. This
ensures if they get cast to `Error` they don't trip up standard `err ==
nil` checks.
Migrates all remaining usages of
`contract.Assert*` and `contract.Require*` to the f variants,
which require adding meaningful error messages.
There were a couple cases where a `testing.T` or `testing.B`
was already available.
For those, this uses t.FailNow or require.NoError.
Refs #12132
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
11819: Cleanup diag error for protected resources r=dixler a=iwahbe
Change the error message for deleting protected resources so it sounds less like we tried and failed to delete the resource, and more like we chose not to.
Fixes#11816
11834: Preallocate slices with a known capacity. r=dixler a=RobbieMcKinstry
Enable the prealloc linter, which identifies slices with a known capacity, but are not preallocated, which results in unnecessary allocations and memcpys.
<!---
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->
## Checklist
<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] N/A: enabling a linter. I have added tests that prove my fix is effective or that my feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] **N/A: not a user-facing change.** I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
<!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->
11842: Changelog and go.mod updates for v3.51.1 r=dixler a=justinvp
11845: Move SecretsProvider to pkg/secrets r=dixler a=Frassle
<!---
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->
Trying to push DefaultSecretsProvider up past the backends resulted in a module loop between pkg/resource/stack and pkg/resource/deploy. This places SecretProvider in the secrets module to avoid that.
## Checklist
<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works - N/A Just moving interface declaration
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - N/A Internal code changes
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version - No
<!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->
Co-authored-by: Ian Wahbe <ian@wahbe.com>
Co-authored-by: Robbie McKinstry <robbie@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Robbie McKinstry <thesnowmancometh@gmail.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
Go treats comments that match the following regex as directives.
//[a-z0-9]+:[a-z0-9]
Comments that are directives don't show in an entity's documentation.
5a550b6951 (diff-f56160fd9fcea272966a8a1d692ad9f49206fdd8dbcbfe384865a98cd9bc2749R165)
Our code has `//nolint` directives that now show in the API Reference.
This is because these directives are in one of the following forms,
which don't get this special treatment.
// nolint:foo
//nolint: foo
This change fixes all such directives found by the regex:
`// nolint|//nolint: `.
See bottom of commit for command used for the fix.
Verification:
Here's the output of `go doc` on some entities
before and after this change.
Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
nolint: lll, interfacer
nolint: lll, interfacer
const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
```
After
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
func BoolRef(v bool) *bool
func Float64Ref(v float64) *float64
func IntRef(v int) *int
func IsSecret(o Output) bool
```
Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
func URN_(o string) ResourceOption
URN_ is an optional URN of a previously-registered resource of this type to
read from the engine. nolint: revive
```
After:
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
func URN_(o string) ResourceOption
URN_ is an optional URN of a previously-registered resource of this type to
read from the engine.
```
Note that golangci-lint offers a 'nolintlint' linter
that finds such miuses of nolint,
but it also finds other issues so I've deferred that to a follow up PR.
Resolves#11785
Related: https://github.com/golangci/golangci-lint/issues/892
[git-generate]
FILES=$(mktemp)
rg -l '// nolint|//nolint: ' |
tee "$FILES" |
xargs perl -p -i -e '
s|// nolint|//nolint|g;
s|//nolint: |//nolint:|g;
'
rg '.go$' < "$FILES" | xargs gofmt -w -s
These changes extend the public API of `pkg/codegen/schema` to support
on-demand binding of package members. On-demand binding is appropriate for
scenarios that do not require the entire package, especially those such as
program code generation or the YAML LSP server that require only specific
types/functions/etc.
The extensions to the public API consist of two new types and several new
methods. The most notable of these are `PackageReference` and
`Loader.LoadPackageReference`. The former provides the on-demand binding
interface, while the latter creates instances of the former (n.b. it was
my intent to make a breaking change to the signature of `Loader.LoadPackage`
s.t. it returns a `PackageReference`, but the circular dependency between
this Go module and those for YAML and Java prevented that change).
These changes _dramatically_ reduce the memory required to interace with
Pulumi Packages, and only require memory proportional to the number of
accessed package members. We may be able to improve on this in the future
by removing type/resource/function interning, which would allow those
values to be garbage collected at a granaular level rather than at a
package level. That is a more radical change, though, as it requires new
equality semantics for each of the affected types (some of which are
currently used as map keys).