Fixes https://github.com/pulumi/pulumi/issues/16859.
Standard issue map iteration being undefined order and thus giving
inconsistent codegen results. This fixes the map iteration to use
`SortedKeys` to get a consistent result.
By fixing the emitted function for `getOutput` in Go program-gen which
is specific to stack references.
Locally the generated program compiles but I get a runtime error:
```
Error: Not equal:
expected: resource.PropertyValue{V:"plain"}
actual : resource.PropertyValue{V:(*resource.Secret)(0x14000aa6050)}
Diff:
--- Expected
+++ Actual
@@ -1,3 +1,7 @@
(resource.PropertyValue) {
- V: (string) (len=5) "plain"
+ V: (*resource.Secret)({
+ Element: (resource.PropertyValue) {
+ V: (string) (len=5) "plain"
+ }
+ })
}
Messages: expected property "plain" to be {plain}
```
However I _think_ this is a local problem with some cached Go SDK
artifacts because when go the implementation of `GetOutput` (from the
generated program) it brings me to a different implementation than the
one I find on latest SDK so testing in CI to make sure
EDIT: fixed by removing hardcoded `/` in the used path when replacing
Pulumi SDK reference
Fixes#16840
Enables the l1-main conformance test for Go. This required fixing up Go
programgen to respect the "main" attribute in the projects Pulumi.yaml
if present.
This commit extends PCL so that it knows about the `DeletedWith`
resource option. With this, we can give `pulumi convert` the ability to
preserve `DeletedWith` resource options when converting programs.
Part of https://github.com/pulumi/pulumi-yaml/pull/437
Enable the l1-empty test for Go.
This required some changes in reporting versions from the Go language
host (replaced dependencies don't really have a version anymore, because
they've been replaced by a local artefact and that doesn't contain any
version info itself). This then cascaded that the conformance test had
to be less strict about checking versions from GetDependencies because
Go now returns empty versions for these local deps.
Rather than writing a go.mod with string lerping, this uses the
"golang.org/x/mod/modfile" to construct and render the go.mod files we
generate for programs.
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
For resources that don't have a schema, we emit a best guess for how
they are declared and how imports should be emitted. For these resources
we emit `index.{ResourceName}` if that resource is from the index
module, however this is not how resources are usually generated, instead
we should generate `{packageName}.{ResourceName}` for resources from the
index module which is what this PR resolves.
Fixes#16461
# Description
Partially addressing
https://github.com/pulumi/pulumi-azure-native/issues/3308
Implementing a new go language option called `importPathPattern` which
can be used to override how the base import path and modules are
concatenated to create imports in generated Go programs. By convention
this used to be `{baseImportPath}/{module}` which has worked for all of
our providers. However, azure-native v2 has introduced a new import
scheme where the convention above causes incorrect import paths to be
generated.
This is where `importPathPattern` comes into play and allows for a
different convention. In the case the of azure-native v2, the pattern
_must_ be `github.com/pulumi/pulumi-azure-native-sdk/{module}/v2`.
This PR implements `importPathPattern` and tests it using a squashed
down azure-native v2 schema containing only contents from the
`eventgrid` module. This schema sets the option like this:
```json
"importPathPattern": "github.com/pulumi/pulumi-azure-native-sdk/{module}/v2"
```
This schema also modifies `packageImportAliases` from the current
azure-native v2 schema to exclude /v2 before the module path (see the
file below). This change is needed in the actual azure-native v2
provider cc @danielrbradley and it is the second part of fully fixing
https://github.com/pulumi/pulumi-azure-native/issues/3308
> We cannot just use `{baseImportPath}/{module}` because the base import
path for azure-native v2 has a suffix v2 (it's required)
Also implemented a small feature in ProgramTest to allow overriding the
used plugin host for the specific program test, this is because I wanted
to test the program against the simplified azure-native v2 schema
without changing how other test load the previous azure-native v1.x
schemas (I tested that without this, binding programs fails)
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
For an MLC package such as `aws-static-website`, it has some types which
are referenced from the `aws` package. Program-gen assumes packages are
inferred from resources and invokes, not types which caused panics in Go
(#15597), dotnet and python
(https://github.com/pulumi/pulumi-converter-constructor-syntax/issues/2)
This PR fixes those panics. In Go the panic was due to using package
name instead of the package reference from the imported type. In dotnet
and python was due to assuming no external type references. Now we
generate nice code for all these languages.
That said, there is still an issue of resolving imports for the packages
of these external types. It works in Go, TypeScript doesn't need it but
dotnet and python do. That is why the latter are added in `SkipCompile`
in the test program.
Fixes#15597
Fixes
https://github.com/pulumi/pulumi-converter-constructor-syntax/issues/3
Fixes
https://github.com/pulumi/pulumi-converter-constructor-syntax/issues/2
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
For C#, Python and TypeScript: emit missing comments for local variable
declaration
For Go: emit missing comments for local variables and resources
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
While covering more parts of go codegen, I've seen that config variables
are broken 😓 specifically when requiring config variables using
`RequireFloat` it should be `RequireFloat64` and `RequireBoolean` should
be `RequireBool`.
Moreover, it seems that `RequireObject` doesn't work at all since the
function signature doesn't match the way it was generated (see #14957)
C# has a similar issue with optional untyped objects as config
variables. For now have skipped compilation for those.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945
This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.
Most of these edits were made by running `perfsprint --fix`.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
This PR fixes an issue in Go program where if users are writing invoke
expressions inline inside other expressions, then that invoke is
extracted into a temporary variable and then later referenced the same
way it was used. This is because non-output-versioned invokes cannot be
used directly since they return a tuple (InvokeResult, err) so we need
to check for the error before proceeding.
Fixes#14064
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
We added program generation that assumes availability of generics as
part of https://github.com/pulumi/pulumi/pull/13149, but didn't bump the
version of Go required in the generated go.mod. Currently, Pulumi is
supported on the supported versions of Go, which includes Go 1.20 and
1.21, so here we bump the Go version to the minimum currently supported
version, which also supports all of the code we generate today.
I was curious how this was not already failing tests, and it appears
it's because we delete the `go.mod` file in our tests and recreate it
with `go mod init`. I am unclear why we do that, but it feels like we
should not, exactly to avoid this sort of problem in the future and to
test the actual code we generate.
8696695cdb/pkg/codegen/go/test.go (L25-L28)Fixes#13728.
This isn't currently actually used anywhere. I've just threaded it
through to all the program gen functions where it will be needed.
Matrix testing will be using and testing this.
This fixes how programgen generates import statements
to handle conflicting imports when two imported packages
have the same name, e.g.
github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ecs
github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecs
To do this, we add a fileImporter type that tracks these imports.
It prefers unnamed imports for packages unless one of the following is
true:
- the name of the package has already been used by another import
- the name of the package does not match the last component
of the import path (e.g., `example.com/foo-go` with `package foo`).
If the name has already been used by another import,
it attempts the following in-order:
- Combine the last two path components of the import path
into an identifier and use that if available.
e.g., `awsxs3` from `sdk/go/awsx/s3`.
- Append a number to the package name and increment it
until an unused name is found.
e.g. `ecs2`, `ecs3`, and so on.
There's a change in how this information is tracked as well.
Previously, this was a pull approach: various calls returned
programImports objects which all got merged together.
This change switches to a push approach:
as code is generated and imports are requested,
they're submitted to the fileImporter which keeps track of them
until the next `Reset()` call.
The above also has a nice side effect of dropping a parameter.
Another change worth explicitly calling out:
Previously, getModOrAlias partially duplicated some of the logic
implemented in getPulumiImport, and used `mod`, `originalMod`
in a non-obvious way.
This generated incorrect imports like the following
(note the two `/aws` at the end):
github.com/pulumi/pulumi-aws/sdk/v5/go/aws/aws
This change replicates more of the logic of getPulumiImport
(now called addPulumiImport) into this function,
and addresses the discrepancy in codegen caused by `mod`/`originalMod`.
The result leaves most existing code unchanged,
except in a couple existing cases where the resulting changes make sense
given the logic for named imports outlined above.
Resolves#11176
Uses the newly added `options.retainOnDelete` in PCL
to generate the RetainOnDelete resource option
for Go, NodeJS, .NET, and Python.
Java support cannot be added without a release of pulumi/pulumi
because the new field in pcl.ResourceOptions is not yet visible
to Java.
Resolves#12304
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
Replace `buffer.WriteString(fmt.Sprintf(..))` calls,
where buffer is one of `bytes.Buffer`, `strings.Builder`, or `bufio.Writer`,
with equivalent `fmt.Fprintf` calls -- all those types are io.Writers.
Migrates uses of contract.{Assert, AssertNoError} in pkg/codegen/go
to use Assertf and AssertNoErrorf so that
we write more meaningful messages when these contracts are violated.
Incremental step towards deprecating the non-f variants.
Refs #12132
Stop using io/ioutil across the entire repository.
The io/ioutil package was deprecated in Go 1.16 (2021-02)
with replacements provided in other packages.
Specifically:
ioutil.Discard => io.Discard
ioutil.NopCloser => io.NopCloser
ioutil.ReadAll => io.ReadAll
ioutil.ReadFile => os.ReadFile
ioutil.TempDir => os.MkdirTemp
ioutil.TempFile => os.CreateTemp
ioutil.WriteFile => os.WriteFile
This change switches all of these entities
across the repository.
Following this change,
the only references to ioutil are in schema files:
% rg -l ioutil
pkg/codegen/testing/test/testdata/aws-4.26.0.json
pkg/codegen/testing/test/testdata/aws-4.36.0.json
pkg/codegen/testing/test/testdata/aws-4.37.1.json
pkg/codegen/testing/test/testdata/aws-5.4.0.json
pkg/codegen/testing/test/testdata/aws-5.16.2.json
The bulk of this change was generated automatically
with manual touch ups afterwards.
Go treats comments that match the following regex as directives.
//[a-z0-9]+:[a-z0-9]
Comments that are directives don't show in an entity's documentation.
5a550b6951 (diff-f56160fd9fcea272966a8a1d692ad9f49206fdd8dbcbfe384865a98cd9bc2749R165)
Our code has `//nolint` directives that now show in the API Reference.
This is because these directives are in one of the following forms,
which don't get this special treatment.
// nolint:foo
//nolint: foo
This change fixes all such directives found by the regex:
`// nolint|//nolint: `.
See bottom of commit for command used for the fix.
Verification:
Here's the output of `go doc` on some entities
before and after this change.
Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
nolint: lll, interfacer
nolint: lll, interfacer
const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
```
After
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
func BoolRef(v bool) *bool
func Float64Ref(v float64) *float64
func IntRef(v int) *int
func IsSecret(o Output) bool
```
Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
func URN_(o string) ResourceOption
URN_ is an optional URN of a previously-registered resource of this type to
read from the engine. nolint: revive
```
After:
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
func URN_(o string) ResourceOption
URN_ is an optional URN of a previously-registered resource of this type to
read from the engine.
```
Note that golangci-lint offers a 'nolintlint' linter
that finds such miuses of nolint,
but it also finds other issues so I've deferred that to a follow up PR.
Resolves#11785
Related: https://github.com/golangci/golangci-lint/issues/892
[git-generate]
FILES=$(mktemp)
rg -l '// nolint|//nolint: ' |
tee "$FILES" |
xargs perl -p -i -e '
s|// nolint|//nolint|g;
s|//nolint: |//nolint:|g;
'
rg '.go$' < "$FILES" | xargs gofmt -w -s