Commit Graph

27 Commits

Author SHA1 Message Date
Fraser Waters 8b26393163
Validate that resources don't use urn or id as output properties ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14631.

This adds a couple of new tests to codegen. Firstly to check that if you
use "urn" or "id" as a resource output property we fail schema binding.
This is because these clash with the urn and id fields that every
resource already has to have.

Secondly a schema gen test that checks that urn and id _do_ work in
other places, i.e. resource inputs, nested types, etc.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-23 02:43:38 +00:00
Anton Tayanovskyy ba039c20f8
Support returning plain values from methods ()
Support returning plain values from methods.

Implements Node, Python and Go support.

Remaining:

- [x] test receiving unknowns
- [x] acceptance tests written and passing locally for Node, Python, Go
clients against a Go server
- [x] acceptance tests passing in CI
- [x] tickets filed for remaining languages
   - [x] https://github.com/pulumi/pulumi-yaml/issues/499
   - [x] https://github.com/pulumi/pulumi-java/issues/1193
   - [x] https://github.com/pulumi/pulumi-dotnet/issues/170 

Known limitations:

- this is technically a breaking change in case there is code out there
that already uses methods that return Plain: true

- struct-wrapping limitation: the provider for the component resource
needs to still wrap the plain-returning Method response with a 1-arg
struct; by convention the field is named "res", and this is how it
travels through the plumbing

- resources cannot return plain values yet

- the provider for the component resource cannot have unknown
configuration, if it does, the methods will not be called
- Per Luke https://github.com/pulumi/pulumi/issues/11520 this might not
be supported/realizable yet

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/12709

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-18 06:02:06 +00:00
Fraser Waters c27d768bb1
Add matrix testing ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Adds the first pass of matrix testing.

Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.

Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).

Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.

This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property

These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).

There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.

1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
Kyle Dixler f2a92e19e4 Fix bug in provider schema where default int properties could not be int
Fixes 
2023-07-26 14:14:26 -07:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
Fraser Waters ef7b123bba Don't return empty hcl.Diagnostics
Fixes https://github.com/pulumi/pulumi-terraform-bridge/issues/1201

hcl.Diagnostics implements the Error interface, but this means a list of
zero diagnostics still looks like a non-nil error, which throws of
normal "if err == nil" checks.

This changes a few use sites of hcl.Diagnostics to ensure we return nil
when there aren't any diagnostics rather than an empty slice. This
ensures if they get cast to `Error` they don't trip up standard `err ==
nil` checks.
2023-06-08 19:14:41 +01:00
Abhinav Gupta acaf79bc10
all: Drop //nolint:goconst
Drops the nolint:goconst directive
now that the goconst linter is no longer enabled.
2023-03-09 11:15:21 -08:00
Abhinav Gupta e395deef6b
all: Assert => Assertf
Migrates all remaining usages of
`contract.Assert*` and `contract.Require*` to the f variants,
which require adding meaningful error messages.

There were a couple cases where a `testing.T` or `testing.B`
was already available.
For those, this uses t.FailNow or require.NoError.

Refs 
2023-03-03 14:37:43 -08:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Fraser Waters 6d7f4ee992
Update pkg/codegen/schema/bind.go
Co-authored-by: Ian Wahbe <ian@wahbe.com>
2023-01-31 17:23:58 +00:00
Fraser Waters 55aad4436a Add tests for resources called "provider"
Fixes https://github.com/pulumi/pulumi/issues/11961.

Fix in so much as it shouldn't panic anymore, but it will now error out
on reading the schema.
2023-01-29 20:04:07 +00:00
bors[bot] 8915911825
Merge
11819: Cleanup diag error for protected resources r=dixler a=iwahbe

Change the error message for deleting protected resources so it sounds less like we tried and failed to delete the resource, and more like we chose not to. 

Fixes 


11834: Preallocate slices with a known capacity. r=dixler a=RobbieMcKinstry

Enable the prealloc linter, which identifies slices with a known capacity, but are not preallocated, which results in unnecessary allocations and memcpys.

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] N/A: enabling a linter. I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] **N/A: not a user-facing change.** I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11842: Changelog and go.mod updates for v3.51.1 r=dixler a=justinvp



11845: Move SecretsProvider to pkg/secrets r=dixler a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Trying to push DefaultSecretsProvider up past the backends resulted in a module loop between pkg/resource/stack and pkg/resource/deploy. This places SecretProvider in the secrets module to avoid that.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works - N/A Just moving interface declaration 
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - N/A Internal code changes
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version - No
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Ian Wahbe <ian@wahbe.com>
Co-authored-by: Robbie McKinstry <robbie@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Robbie McKinstry <thesnowmancometh@gmail.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-01-12 20:11:42 +00:00
Robbie McKinstry 4959522a53
Repair tests expecting nil slices 2023-01-11 21:53:04 -08:00
Zaid Ajaj 330676a0d0 Initial implementation of simplified invokes for dotnet and nodejs 2023-01-11 14:17:14 -08:00
Robbie McKinstry 1f78baae71
Preallocate slices with a known capacity.
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
2023-01-11 12:52:51 -08:00
Abhinav Gupta 0bff0b8716 sdk/go: Remove 'nolint' directives from package docs
Go treats comments that match the following regex as directives.

    //[a-z0-9]+:[a-z0-9]

Comments that are directives don't show in an entity's documentation.
5a550b6951 (diff-f56160fd9fcea272966a8a1d692ad9f49206fdd8dbcbfe384865a98cd9bc2749R165)

Our code has `//nolint` directives that now show in the API Reference.
This is because these directives are in one of the following forms,
which don't get this special treatment.

    // nolint:foo
    //nolint: foo

This change fixes all such directives found by the regex:
`// nolint|//nolint: `.
See bottom of commit for command used for the fix.

Verification:
Here's the output of `go doc` on some entities
before and after this change.

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

nolint: lll, interfacer

nolint: lll, interfacer

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
```

After
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
func BoolRef(v bool) *bool
func Float64Ref(v float64) *float64
func IntRef(v int) *int
func IsSecret(o Output) bool
```

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine. nolint: revive
```

After:
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine.
```

Note that golangci-lint offers a 'nolintlint'  linter
that finds such miuses of nolint,
but it also finds other issues so I've deferred that to a follow up PR.

Resolves 

Related: https://github.com/golangci/golangci-lint/issues/892

[git-generate]
FILES=$(mktemp)
rg -l '// nolint|//nolint: ' |
  tee "$FILES" |
  xargs perl -p -i -e '
    s|// nolint|//nolint|g;
    s|//nolint: |//nolint:|g;
  '
rg '.go$' < "$FILES" | xargs gofmt -w -s
2023-01-06 09:06:47 -08:00
Aaron Friel a682a1ab06
Revert "Simplified invokes: SDK-gen and program-gen implementation for dotnet and nodejs" 2022-12-20 17:47:29 -08:00
Zaid Ajaj a6f26d2355 Initial implementation of simplified invokes for dotnet and nodejs 2022-12-16 13:39:03 +01:00
Ian Wahbe 878cf9055a Delete `*Package` fields from schema types
Fix NeedsGoOutputVersion.
2022-12-14 10:29:30 +01:00
Ian Wahbe 65b6b01abd Don't use *schema.Package in python codegen 2022-12-07 17:54:38 +01:00
Ian Wahbe 453222555e Higher fidelity schema marshalling
I noticed this when trying to round-trip a schema out of a provider SDK
2022-10-24 16:16:13 -07:00
Aaron Friel a4b1d6b2a7 ci: gofmt 1.18+ clean 2022-09-21 09:48:39 -07:00
Ian Wahbe df2dbbdaf4 Get extract schema working 2022-09-14 16:23:03 +02:00
Ian Wahbe e4cbb79e23
Include pkg reference in bound schema types ()
* Include type references in schema types

* Fix lint

* Fix enum tests
2022-06-23 16:27:44 -07:00
Pat Gavlin 36cbf572f4
[schema] Add support for on-demand binding. ()
These changes extend the public API of `pkg/codegen/schema` to support
on-demand binding of package members. On-demand binding is appropriate for
scenarios that do not require the entire package, especially those such as
program code generation or the YAML LSP server that require only specific
types/functions/etc.

The extensions to the public API consist of two new types and several new
methods. The most notable of these are `PackageReference` and
`Loader.LoadPackageReference`. The former provides the on-demand binding
interface, while the latter creates instances of the former (n.b. it was
my intent to make a breaking change to the signature of `Loader.LoadPackage`
s.t. it returns a `PackageReference`, but the circular dependency between
this Go module and those for YAML and Java prevented that change).

These changes _dramatically_ reduce the memory required to interace with
Pulumi Packages, and only require memory proportional to the number of
accessed package members. We may be able to improve on this in the future
by removing type/resource/function interning, which would allow those
values to be garbage collected at a granaular level rather than at a
package level. That is a more radical change, though, as it requires new
equality semantics for each of the affected types (some of which are
currently used as map keys).
2022-05-23 15:44:35 -07:00
Ian Wahbe b281b482dc
Pass through resource replacement in the schema ()
* Pass through resource replacement in the schema

* Add property

* Ignore files in .pytest_cache

* Remove .pytest_cache files

* Add a test + Fix
2022-05-17 13:59:47 -07:00
Pat Gavlin e6095194a9
[schema] Move the binder into its own file. ()
Just what it says on the tin.
2022-05-09 19:08:43 -07:00