Commit Graph

86 Commits

Author SHA1 Message Date
Zaid Ajaj ac6adb98bf
[go/program-gen] Fix using inline invoke expressions inside resources, objects and arrays ()
# Description

This PR fixes an issue in Go program where if users are writing invoke
expressions inline inside other expressions, then that invoke is
extracted into a temporary variable and then later referenced the same
way it was used. This is because non-output-versioned invokes cannot be
used directly since they return a tuple (InvokeResult, err) so we need
to check for the error before proceeding.

Fixes 

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-03 14:22:43 +00:00
Thomas Gummerer 0d6fff3995
codegen/go: output correct array type ()
Currently for an array of objects, codegen for Go outputs a
`pulumi.AnyArray` type. However that does not exist. Make sure it always
outputs the correct `pulumi.Array` type.

While there give Maps the same treatment, as `pulumi.AnyMap` also
doesn't exist.

Fixes #https://github.com/pulumi/pulumi/issues/8072

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-20 16:12:25 +00:00
Zaid Ajaj ed02926277 Allow generating code for unknown invokes in non-strict mode 2023-07-10 15:05:18 +02:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
Abhinav Gupta da184df24a
fix(codegen/go): Use raw string literals for multiline-strings
We already have logic in place to generate mutliline strings with
backticks if they meet certain conditions in genTemplateExpression.
However, the logic came into effect only if the string was complex
enough to warrant use of `Sprintf`
because the function short circuits to just printing a string literal
if the string doesn't need a `Sprintf`.

This changes genStringLiteral to be responsible for the decision
of whether the string warrants backticks or not,
retaining the prior conditions on whether the string would benefit
from being split across multiple lines.

Resolves 
2023-06-22 10:55:45 -07:00
Justin Van Patten 31d106980e [programgen/dotnet,go] Implement singleOrNone intrinsic
This change adds support for the `singleOrNone` intrinsic to Go and C# programgen.
2023-06-11 16:23:33 -07:00
Zaid Ajaj 2ce1fdb091 Fix panic in go program-gen when encountering splat expressions 2023-06-07 13:22:41 +02:00
Zaid Ajaj 703c53cfeb Turn error diagnostics into warnings 2023-06-02 17:17:59 +02:00
Zaid Ajaj af09d174cd Do not error out when generaing not yet implemented ForExpressions 2023-06-02 16:59:46 +02:00
Zaid Ajaj 6528a9fe7a Go module support as component resources in program-gen 2023-05-08 16:20:04 +02:00
Zaid Ajaj b90b5a5bf9 Add "NotImplemented" PCL function intrinsic 2023-03-10 12:14:28 +01:00
Abhinav Gupta acaf79bc10
all: Drop //nolint:goconst
Drops the nolint:goconst directive
now that the goconst linter is no longer enabled.
2023-03-09 11:15:21 -08:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta cc32691bc1
chore(all): strings.Replace(..., -1) => strings.Replace(...)
Replaces all instances of `strings.Replace(s, old, new, -1)`
with the equivalent `strings.ReplaceAll(s, old, new)`.
This function has been available since Go 1.12.
2023-03-01 13:22:33 -08:00
Abhinav Gupta aefcd86af2
pkg/codegen/go: Prefer contract.Assertf over contract.Assert
Migrates uses of contract.{Assert, AssertNoError} in pkg/codegen/go
to use Assertf and AssertNoErrorf so that
we write more meaningful messages when these contracts are violated.

Incremental step towards deprecating the non-f variants.

Refs 
2023-02-13 16:27:39 -08:00
Fraser Waters 3936ed9b4f Add unsecret intrinsic function 2023-01-31 14:18:01 +00:00
Abhinav Gupta 42e7dc1fa3
pcl/codegen: Fix typeNameID data race
generator.argumentTypeName increments a typeNameID global variable.
This causes a data race when tests run in parallel:

```
WARNING: DATA RACE
Read at 0x000005d9dd28 by goroutine 20:
  github.com/pulumi/pulumi/pkg/v3/codegen/go.(*generator).argumentTypeName()
      /home/runner/work/pulumi/pulumi/pkg/codegen/go/gen_program_expressions.go:775 +0x84
  github.com/pulumi/pulumi/pkg/v3/codegen/go.(*generator).genObjectConsExpression()
  ...

Previous write at 0x000005d9dd28 by goroutine 19:
  github.com/pulumi/pulumi/pkg/v3/codegen/go.(*generator).argumentTypeName()
      /home/runner/work/pulumi/pulumi/pkg/codegen/go/gen_program_expressions.go:775 +0xa4
  github.com/pulumi/pulumi/pkg/v3/codegen/go.(*generator).genTemplateExpression()
      /home/runner/work/pulumi/pulumi/pkg/codegen/go/gen_program_expressions.go:672 +0xbe
```

This variable is not used anywhere else in the system.
Fix the data race by deleting the variable.
2023-01-19 11:07:26 -08:00
bors[bot] 8915911825
Merge
11819: Cleanup diag error for protected resources r=dixler a=iwahbe

Change the error message for deleting protected resources so it sounds less like we tried and failed to delete the resource, and more like we chose not to. 

Fixes 


11834: Preallocate slices with a known capacity. r=dixler a=RobbieMcKinstry

Enable the prealloc linter, which identifies slices with a known capacity, but are not preallocated, which results in unnecessary allocations and memcpys.

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] N/A: enabling a linter. I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] **N/A: not a user-facing change.** I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11842: Changelog and go.mod updates for v3.51.1 r=dixler a=justinvp



11845: Move SecretsProvider to pkg/secrets r=dixler a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Trying to push DefaultSecretsProvider up past the backends resulted in a module loop between pkg/resource/stack and pkg/resource/deploy. This places SecretProvider in the secrets module to avoid that.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works - N/A Just moving interface declaration 
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - N/A Internal code changes
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version - No
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Ian Wahbe <ian@wahbe.com>
Co-authored-by: Robbie McKinstry <robbie@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Robbie McKinstry <thesnowmancometh@gmail.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-01-12 20:11:42 +00:00
Abhinav Gupta 3d64521cf2
unused: Remove unused functions and types
Several functions and types were reported as unused.

Most notably, jsonTemp and the method in jsonSpiller that uses it
were all unused.
2023-01-12 09:55:34 -08:00
Robbie McKinstry 4959522a53
Repair tests expecting nil slices 2023-01-11 21:53:04 -08:00
Zaid Ajaj 330676a0d0 Initial implementation of simplified invokes for dotnet and nodejs 2023-01-11 14:17:14 -08:00
Robbie McKinstry 1f78baae71
Preallocate slices with a known capacity.
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
2023-01-11 12:52:51 -08:00
Abhinav Gupta 1158d4acee
all: Drop ioutil
Stop using io/ioutil across the entire repository.
The io/ioutil package was deprecated in Go 1.16 (2021-02)
with replacements provided in other packages.
Specifically:

    ioutil.Discard   => io.Discard
    ioutil.NopCloser => io.NopCloser
    ioutil.ReadAll   => io.ReadAll
    ioutil.ReadFile  => os.ReadFile
    ioutil.TempDir   => os.MkdirTemp
    ioutil.TempFile  => os.CreateTemp
    ioutil.WriteFile => os.WriteFile

This change switches all of these entities
across the repository.

Following this change,
the only references to ioutil are in schema files:

    % rg -l ioutil
    pkg/codegen/testing/test/testdata/aws-4.26.0.json
    pkg/codegen/testing/test/testdata/aws-4.36.0.json
    pkg/codegen/testing/test/testdata/aws-4.37.1.json
    pkg/codegen/testing/test/testdata/aws-5.4.0.json
    pkg/codegen/testing/test/testdata/aws-5.16.2.json

The bulk of this change was generated automatically
with manual touch ups afterwards.
2023-01-06 16:35:14 -08:00
Abhinav Gupta 0bff0b8716 sdk/go: Remove 'nolint' directives from package docs
Go treats comments that match the following regex as directives.

    //[a-z0-9]+:[a-z0-9]

Comments that are directives don't show in an entity's documentation.
5a550b6951 (diff-f56160fd9fcea272966a8a1d692ad9f49206fdd8dbcbfe384865a98cd9bc2749R165)

Our code has `//nolint` directives that now show in the API Reference.
This is because these directives are in one of the following forms,
which don't get this special treatment.

    // nolint:foo
    //nolint: foo

This change fixes all such directives found by the regex:
`// nolint|//nolint: `.
See bottom of commit for command used for the fix.

Verification:
Here's the output of `go doc` on some entities
before and after this change.

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

nolint: lll, interfacer

nolint: lll, interfacer

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
```

After
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
func BoolRef(v bool) *bool
func Float64Ref(v float64) *float64
func IntRef(v int) *int
func IsSecret(o Output) bool
```

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine. nolint: revive
```

After:
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine.
```

Note that golangci-lint offers a 'nolintlint'  linter
that finds such miuses of nolint,
but it also finds other issues so I've deferred that to a follow up PR.

Resolves 

Related: https://github.com/golangci/golangci-lint/issues/892

[git-generate]
FILES=$(mktemp)
rg -l '// nolint|//nolint: ' |
  tee "$FILES" |
  xargs perl -p -i -e '
    s|// nolint|//nolint|g;
    s|//nolint: |//nolint:|g;
  '
rg '.go$' < "$FILES" | xargs gofmt -w -s
2023-01-06 09:06:47 -08:00
Aaron Friel a682a1ab06
Revert "Simplified invokes: SDK-gen and program-gen implementation for dotnet and nodejs" 2022-12-20 17:47:29 -08:00
bors[bot] 43f7e4d6b9
Merge
11670: Fix incorrect pointers for array of objects [go/codegen] r=aq17 a=aq17

Fixes 
Fix incorrect pointer argument codegen when objects are nested inside an array whose type is not a pointer.

Example:
`&s3.BucketLoggingArgs` is valid here
```
bucket, err := s3.NewBucket(ctx, "bucket", &s3.BucketArgs{
			Loggings: s3.BucketLoggingArray{
				&s3.BucketLoggingArgs{
					TargetBucket: logs.Bucket,
				},
			},
		})

```
but `&ec2.SubnetSpecArgs{}` is invalid here:
```
SubnetSpecs: []ec2.SubnetSpecArgs{
				&ec2.SubnetSpecArgs{
					Type:     ec2.SubnetTypePublic,
					CidrMask: 22,
				},
				&ec2.SubnetSpecArgs{
					Type:     ec2.SubnetTypePrivate,
					CidrMask: 20,
				},
			},
```
Instead, it should be: 
```
SubnetSpecs: []ec2.SubnetSpecArgs{
				{
					Type:     ec2.SubnetTypePublic,
					CidrMask: 22,
				},
				{
					Type:     ec2.SubnetTypePrivate,
					CidrMask: 20,
				},
			},
```



Co-authored-by: aq17 <aqiu@pulumi.com>
2022-12-19 20:47:21 +00:00
aq17 825de881b8 Fix incorrect pointers for array of objects [go/codegen] 2022-12-19 12:45:27 -08:00
Zaid Ajaj a6f26d2355 Initial implementation of simplified invokes for dotnet and nodejs 2022-12-16 13:39:03 +01:00
aq17 d780e3a074 Check for optional/ Ptr types within Unions [go/programgen] 2022-12-12 15:49:20 -08:00
Ian Wahbe 7ae9c181d0 Don't use *schema.Package in go codegen 2022-12-08 17:51:50 +01:00
aq17 927e9d4a69 Improve error msg for invalid enum values on convert 2022-12-01 15:29:22 -08:00
Ian Wahbe 095f5a9fbf Convert invoke results to ouputs when needed 2022-11-28 16:14:07 -08:00
Ian Wahbe 724fae5c48 Mark external modules as forign
Previously, the names of forign types in the top level were omitted,
since the top level module is always "".

Apply the same fix for enum types.
2022-11-22 13:46:18 -08:00
aq17 575c2549aa Fix codegen for __apply functions 2022-09-28 10:32:33 -07:00
Eng Zer Jun 57b25b45d1
refactor: use `os.ReadDir` for lightweight directory reading
`os.ReadDir` was added in Go 1.16 as part of the deprecation of `ioutil`
package. It is a more efficient implementation than `ioutil.ReadDir`.

Reference: https://pkg.go.dev/io/ioutil#ReadDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-09-24 15:20:43 +08:00
Alex Qiu 2a81117dd6
Fix invalid resource type on `pulumi convert` to Go () 2022-09-09 16:11:38 -07:00
Alex Qiu 15d86e53d8
Fix import path for non-pulumi owner providers () 2022-08-29 17:16:22 -07:00
Ian Wahbe 94422f97f8
Expose external package cache + global default (Take 2) ()
* Expose external package cache

* CL

* Add a default global cache

* More cache passing

* Compare `Cache` entries by pointer equality

* Add a check for providers when doing imports

* Pass the cache

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-08-12 20:04:21 +02:00
Aaron Friel cca0a39e70
Revert "Expose external package cache ()" ()
This reverts commit 16c173963a.
2022-08-11 15:04:14 -07:00
Ian Wahbe 16c173963a
Expose external package cache ()
* Expose external package cache

* CL

* Add a default global cache

* More cache passing

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-08-11 13:28:27 -07:00
Ian Wahbe a4803f43ea
Remove programgen conversion assertions () 2022-07-13 14:29:34 -07:00
Zaid Ajaj 515cf6191c
Fixes panic when generating go from pulumi yaml ()
* Fixes  panic when generating go from pulumi yaml

Co-authored-by: Aaron Friel <friel@pulumi.com>
Co-authored-by: Zaid Ajaj <zaid@pulumi.com>

* rewrite rewriteConversions to accumulate diagnostics, improve errors

* rewrite diags

* update java dependency

* chore: fix go.mod/go.sum

* fix: traversal of multi-part template string expressions

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
Co-authored-by: Aaron Friel <friel@pulumi.com>
Co-authored-by: Zaid Ajaj <zaid@pulumi.com>
2022-07-07 13:15:47 -07:00
Ian Wahbe 6db525c91d
Add external enums all languages ()
* Enable tests for all languages

* Enable go to work around ungenerated types

This relies on the resource `.pkg` field bieng accurate. Since this is
not the case, we need to make it accurate. I will do this in a seperate
PR since it is invasive.

* Include type references in schema types

* Fix lint

* Generate the correct pkg

* Check if we generated the external type

* Fix enum tests

* Perform the same change for outputs

* Cache global analysis
2022-07-06 11:35:31 -07:00
Aaron Friel 5e7e39a941
[codegen] simplify opaque types to string newtype () 2022-06-13 11:13:03 -07:00
Ian Wahbe 633d586aeb
Handle long traversals in a nicer way ()
* Handle long traversals in a nicer way

* CL

* assert no error

* Update tests
2022-06-02 09:36:33 -07:00
Aaron Friel 1da317dcc9
[codegen] Improve handling of resources with outputs that are resources. () 2022-05-20 10:29:05 -07:00
Aaron Friel be2501e245
codegen for all assets and archives types ()
* codegen for all assets and archives types

* chore: add changelog entry

* Update CHANGELOG_PENDING.md

Co-authored-by: Fraser Waters <fraser@pulumi.com>

* fix C# codegen with explicit AssetOrArchive type

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2022-04-25 12:59:30 -07:00
Ian Wahbe 183fa89163
Add an EnumType to the PCL model ()
* Add an EnumType to the PCL model

Languages implementation:
- [X] C#
- [X] Python
- [X] TypeScript
- [X] Go

* Fix assert

* Take namespace settings into account

* Implement enums for Go

* Update tests and add documentation.

* Add a test + handle output&unsafe

* Get outputs compiling for Go

* Line up types for Go

* Add nodejs

* Add Python

* Change expression form

* Fix nil check (`==` -> `!=`)

* Standardize nodejs & python checks

This should fix the Node OOM error.

* Rename enum-py to typed-enum-pp

* Don't reference schema types in `hcl2/model`

* Fix nits
2022-04-18 11:03:42 +02:00
Ian Wahbe 66f8b357e2
Expand pcl intrinsic functions ()
* Add TS support

* Add python support

* Add go support

* Add C# support
2022-03-15 17:05:36 -07:00
Anton Tayanovskyy 8ba737574b
Fix 8940 invalid go code generated () 2022-02-08 08:45:24 -05:00