Issue #10659 lists a number of extra linting checks that we could enable
in order to make our Go code more robust. This commit implements as many
as seem sensible:
* `durationcheck`, which checks for multiplication of `time.Duration`s,
which can lead to unexpected behaviour (e.g. `time.Second * time.Second`
is *not* one second)
* `goprintffuncname`, which checks that `Printf`-like functions are
appropriately suffixed with `f` to indicate as such
* `tenv`, which checks for `os.Setenv` in tests where `t.Setenv` is
generally a better solution
* `wastedassign`, which checks for assignments whose values are never
used (such as initial values before an `if` where both branches then
overwrite the value)
* `whitespace`, which checks for blank lines at the beginning and end of
blocks such as functions, `if`s, `for`s and so on.
This commit does *not* enable the following checks listed in #10659:
* `wrapcheck`, which insists that third-party library errors are always
`%w`rapped -- we have a lot of cases where we don't do this and it's
probably a bit more involved than "just wrap them" in terms of making
sure we don't break anything (maybe)
* `predeclared`, which checks for shadowing of existing Go identifiers
-- we use `old` and `new` a lot, especially in step generation, so this
is probably a slightly bigger clean-up/one we might want to opt out of
* `mnd` (magic number detection) -- we have a lot of failures on this
* `nilnil` -- we only have a couple of failures on this; these could
probably be handled with `//nolint` but for now I've opted not to take
this route.
We have added `getOrganization` fucntions to all SDKs. So we can now
lift this into PCL. This is needed to support `pulumi.organization` in
YAML, so that convert from YAML can target PCL and other downstream
languages.
Part of https://github.com/pulumi/pulumi-yaml/issues/461.
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
When running .NET conformance tests, we use local nuget packages for
testing, generating from the `Pack` method of language plugins. This PR
extracts the version and package name correctly from the local
dependencies map and adds to the generated program and generated sdks
# Description
Fixes#16191
The original issue is that the intermediate PCL we generate used enum
names instead of enum values for numeric enum inputs. This PR changes it
so that the PCL program now uses the first numeric value for the first
enum case then subsequently fixing downstream program-gen bugs that
didn't know how to handle numeric values as inputs for enums.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
Fixes#15769 by specifically handling object expressions that are
annotated as `Any` and emitting the correct code.
- In case of Go, expression `X` is emitted as `pulumi.Any(X)` where the
type name here is `map[string]interface{}`.
- In case of C#, we generate `Dictionary<string, object?>` expressions
(similar to what we do inside `toJSON` calls)
- Updates the test aws-native schema from 0.13.0 to 0.99.0
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
This PR improves enum type resolution from strings. When we try to
resolve `Union[string, Enum]` for a string expression, we choose
`string` because it is the more general type since not every string is
assignable to `Enum`. However, here we spacial case strings that are
actually part of that `Enum`.
The result is that `pcl.LowerConversion` will choose `Enum` from
`Union[string, Enum]` when the value of the input string is compatible
with the enum. This greatly improves program-gen for all of typescript,
python, csharp and go which now will emit the fully qualified enum cases
instead of emitting strings.
Closes https://github.com/pulumi/pulumi-dotnet/issues/41 which is
supposed to be a duplicate of
https://github.com/pulumi/pulumi-azure-native/issues/2616 but that is
not the case (the former is about unions of objects, the latter is
unions of enums and strings)
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
### Description
A while ago we started implementing [specialized JSON serialization
methods](https://github.com/pulumi/pulumi/issues/12519) for Pulumi
programs which can accept nested outputs without having to rewrite and
combine applies.
- `Output.SerializeJson` in .NET
- `pulumi.jsonStringify` in nodejs
- `pulumi.Output.json_dumps` in Python
This PR extends program-gen for TypeScript, C# and Python to start
emitting these JSON serialization functions (when necessary). The PR
special-cases the `toJSON` PCL function when rewriting applies so that
nested outputs aren't rewritted.
Example PCL program and generated results:
> Also check out the downstream codegen tests to see improved generated
examples
```
resource vpc "aws:ec2:Vpc" {
cidrBlock = "10.100.0.0/16"
instanceTenancy = "default"
}
resource policy "aws:iam/policy:Policy" {
description = "test"
policy = toJSON({
"Version" = "2012-10-17"
"Interpolated" = "arn:${vpc.arn}:value"
"Value" = vpc.id
})
}
```
### Generated TypeScript Before
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
const vpc = new aws.ec2.Vpc("vpc", {
cidrBlock: "10.100.0.0/16",
instanceTenancy: "default",
});
const policy = new aws.iam.Policy("policy", {
description: "test",
policy: pulumi.all([vpc.arn, vpc.id]).apply(([arn, id]) => JSON.stringify({
Version: "2012-10-17",
Interpolated: `arn:${arn}:value`,
Value: id,
})),
});
```
### Generated TypeScript After
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
const vpc = new aws.ec2.Vpc("vpc", {
cidrBlock: "10.100.0.0/16",
instanceTenancy: "default",
});
const policy = new aws.iam.Policy("policy", {
description: "test",
policy: pulumi.jsonStringify({
Version: "2012-10-17",
Interpolated: pulumi.interpolate`arn:${vpc.arn}:value`,
Value: vpc.id,
}),
});
```
### Generated Python Before
```python
import pulumi
import json
import pulumi_aws as aws
vpc = aws.ec2.Vpc("vpc",
cidr_block="10.100.0.0/16",
instance_tenancy="default")
policy = aws.iam.Policy("policy",
description="test",
policy=pulumi.Output.all(vpc.arn, vpc.id).apply(lambda arn, id: json.dumps({
"Version": "2012-10-17",
"Interpolated": f"arn:{arn}:value",
"Value": id,
})))
```
### Generated Python After
```python
import pulumi
import json
import pulumi_aws as aws
vpc = aws.ec2.Vpc("vpc",
cidr_block="10.100.0.0/16",
instance_tenancy="default")
policy = aws.iam.Policy("policy",
description="test",
policy=pulumi.Output.json_dumps({
"Version": "2012-10-17",
"Interpolated": vpc.arn.apply(lambda arn: f"arn:{arn}:value"),
"Value": vpc.id,
}))
```
### Generated C# Before
```csharp
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using Pulumi;
using Aws = Pulumi.Aws;
return await Deployment.RunAsync(() =>
{
var vpc = new Aws.Ec2.Vpc("vpc", new()
{
CidrBlock = "10.100.0.0/16",
InstanceTenancy = "default",
});
var policy = new Aws.Iam.Policy("policy", new()
{
Description = "test",
PolicyDocument = Output.Tuple(vpc.Arn, vpc.Id).Apply(values =>
{
var arn = values.Item1;
var id = values.Item2;
return JsonSerializer.Serialize(new Dictionary<string, object?>
{
["Version"] = "2012-10-17",
["Interpolated"] = $"arn:{arn}:value",
["Value"] = id,
});
}),
});
});
```
### Generated C# After
```csharp
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using Pulumi;
using Aws = Pulumi.Aws;
return await Deployment.RunAsync(() =>
{
var vpc = new Aws.Ec2.Vpc("vpc", new()
{
CidrBlock = "10.100.0.0/16",
InstanceTenancy = "default",
});
var policy = new Aws.Iam.Policy("policy", new()
{
Description = "test",
PolicyDocument = Output.JsonSerialize(Output.Create(new Dictionary<string, object?>
{
["Version"] = "2012-10-17",
["Interpolated"] = vpc.Arn.Apply(arn => $"arn:{arn}:value"),
["Value"] = vpc.Id,
})),
});
});
```
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
I've noticed while working with PCL generated from Kubernetes manifests
that when an object property key is quoted for example `{ "key" = value
}` instead of `{ key = value }` then program-gen for csharp and python
_panic_ because they assume the object keys to be strictly a literal
value expression and this type assertion fails:
```go
lit := item.Key.(*model.LiteralValueExpression)
```
This PR fixes this panic and allows program-gen to handle cases where
the object keys are `TemplateExpression` assuming that it has one part
which is a literal value expression (handling cases like `{ "key" =
value }`)
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945
This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.
Most of these edits were made by running `perfsprint --fix`.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
Fixes#13998
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
Migrates all remaining usages of
`contract.Assert*` and `contract.Require*` to the f variants,
which require adding meaningful error messages.
There were a couple cases where a `testing.T` or `testing.B`
was already available.
For those, this uses t.FailNow or require.NoError.
Refs #12132
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
* dotnet program-gen targets .NET 6
* changelog entry
* Update GH workflows to use .NET 6
* Enable DOTNET_ROLL_FORWARD=Major for GH workflows
* Expose GeneratePrograms with ability to provide generation options. Use specialized options for import
* Bump dotnet test projects TFM and relax warnings as errors
* refactor code according to Ian's comments
* Include error in generator diagnostics when unable to find function schema
* remove specialized version of GenerateProgram for import and refactor lowerExpression
* remove AcceptCodegenChanges from program driver
* dotnet integration tests updated to net6.0
* Generate return await on top-level Deployment.RunAsync
* Add an EnumType to the PCL model
Languages implementation:
- [X] C#
- [X] Python
- [X] TypeScript
- [X] Go
* Fix assert
* Take namespace settings into account
* Implement enums for Go
* Update tests and add documentation.
* Add a test + handle output&unsafe
* Get outputs compiling for Go
* Line up types for Go
* Add nodejs
* Add Python
* Change expression form
* Fix nil check (`==` -> `!=`)
* Standardize nodejs & python checks
This should fix the Node OOM error.
* Rename enum-py to typed-enum-pp
* Don't reference schema types in `hcl2/model`
* Fix nits
* Added filebase64 support for Golang
* Fixed function signature
* Added filebase64 support for Typescript
* Added filebase64 support for Python
* Added filebase64 support for Dotnet, fixed Sha1
* Fixed helper method list
Co-authored-by: Vova Ivanov <jetvova@gmail.com>
These changes support arbitrary combinations of input + plain types
within a schema. Handling plain types at the property level was not
sufficient to support such combinations. Reifying these types
required updating quite a bit of code. This is likely to have caused
some temporary complications, but should eventually lead to
substantial simplification in the SDK and program code generators.
With the new design, input and optional types are explicit in the schema
type system. Optionals will only appear at the outermost level of a type
(i.e. Input<Optional<>>, Array<Optional<>>, etc. will not occur). In
addition to explicit input types, each object type now has a "plain"
shape and an "input" shape. The former uses only plain types; the latter
uses input shapes wherever a plain type is not specified. Plain types
are indicated in the schema by setting the "plain" property of a type spec
to true.
The PCL binder has supported resource options for some time, but these
options haven't been used or processed by the various code generators.
These options--particularly the parent and provider options0--are
critical for import codegen. These changes implement the basic set of
options, and add a note about fleshing out the rest as necessary.
One component of these changes is a new rewriter that rewrites property
references into property paths that are understood by the Pulumi engine.
This rewriter is used to preprocess the contents of the `ignoreChanges`
resource option.
These changes also hack around a weakness in the HCL2 type system:
In Go, references to resources should be typed as `hcl2.ResourceType`.
Unfortunately, this breaks the existing collection semantics associated
with resources. Because of this, the Go code generator does not have
enough information to know that it should generate a `[]pulumi.Resource`
for lists of resources. These changes hack around that limitation using
a Go-specific opaque type and some hardcoded comparisons in
`argumentTypeName`.
Fixes#4923.