Commit Graph

44 Commits

Author SHA1 Message Date
Will Jones 6f8cfbab9f
Don't publish test code in `pkg/codegen` ()
Presently, we implement code generation (e.g. for SDKs and
programs/`pulumi convert`) on a per-language basis in `pkg/codegen`.
Alongside these implementations, we have a set of tests built using a
common framework that use snapshots to verify that code generation
doesn't break when changes are made. Unfortunately, due to the way
things are currently laid out in our repository, these tests and their
dependencies are shipped as part of the `pkg/codegen` package. This
commit brings @blampe's work in
https://github.com/pulumi/pulumi/pull/16011 up to date and fixes this by
taking the following actions:

* Test harnesses that were previously located in
`pkg/codegen/<language>/test.go` are moved to
`pkg/codegen/testing/test/<language>.go`.

By default, Go excludes files ending in `_test.go`, but *not* files
named e.g. `test.go`. It might seem logical therefore to just rename
these files (e.g. to `codegen_test.go`), so that they can continue to
live alongside their language implementations. Unfortunately, while this
fixes one problem (dependencies pulling in test code), it introduces
another -- the existing `test.go` files actually exist to implement an
interface which is used by the common codegen-test framework we have.
Moving to `*_test.go` files would make them invisible to the actual
modules which run the tests through that framework.

* Test code is consequently refactored to clean up the separation of
test details (e.g. relevant fixtures) and execution implementation
(working directory, means of program generation).

* The `gen_program_test/generate.go` program, which we use to generate
the "batch tests", has been updated so that it respects the new code
layout and organisation.

As laid out by @blampe in  (and shamelessly copied here), this
brings a number of benefits. Test dependencies are no longer included in
non-test packages that are consumed downstream. As an example, this
takes the `pulumi-language-go` binary from ~61MB down to ~36MB, and
speeds up build times:

```
go clean -cache && time go build .
(master) go build .  81.06s user 17.36s system 470% cpu 20.933 total
(PR)     go build .  47.77s user 9.20s system 530% cpu 10.737 total
```

This doesn't completely remove these dependencies from downstream
because most also include `pkg/v3/testing/integration` in tests. It does
only compile these dependencies into the test binary, though.

Before:

```
❯ go mod why github.com/aws/aws-sdk-go-v2/service/sts
github.com/pulumi/pulumi-random/provider/v4/cmd/pulumi-tfgen-random
github.com/pulumi/pulumi-terraform-bridge/pf/tfgen
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen
github.com/pulumi/pulumi/pkg/v3/codegen/dotnet
github.com/pulumi/pulumi/pkg/v3/testing/integration
github.com/pulumi/pulumi/pkg/v3/resource/stack
github.com/pulumi/pulumi/pkg/v3/secrets/cloud
github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test
github.com/aws/aws-sdk-go-v2/service/sts
```

After:

```
github.com/pulumi/pulumi-random/provider/v4
github.com/pulumi/pulumi-random/provider/v4.test     <- expected
github.com/pulumi/providertest
github.com/pulumi/pulumi/pkg/v3/testing/integration
github.com/pulumi/pulumi/pkg/v3/resource/stack
github.com/pulumi/pulumi/pkg/v3/secrets/cloud
github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test
github.com/aws/aws-sdk-go-v2/service/sts
```

Note: this PR has been split into a number of commits, roughly one per
language, to make reviewing a bit easier. The commits *do not build
individually* due to the nature of the changes, but hopefully it makes
understanding the work a bit more feasible.

Closes 

---------

Co-authored-by: Bryce Lampe <bryce@pulumi.com>
2024-10-09 11:09:54 +00:00
Julien 68524af701
Enable goheader rule and add missing license headers ()
This commit adds the `goheader` rule to `golangci-lint` to enforce that
all our Go source code includes appropriate licence headers, fixing up
files that currently fail that check.

---------

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-09-09 12:05:45 +00:00
Fraser Waters 977ea57247
Enable the l1-empty test for Go ()
Enable the l1-empty test for Go.

This required some changes in reporting versions from the Go language
host (replaced dependencies don't really have a version anymore, because
they've been replaced by a local artefact and that doesn't contain any
version info itself). This then cascaded that the conformance test had
to be less strict about checking versions from GetDependencies because
Go now returns empty versions for these local deps.
2024-07-17 13:15:07 +00:00
Anton Tayanovskyy d62c398bfb
Move codegen testdata ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

It appears that Go copies testdata into every GOMODCACHE of a project
that depends on pulumi/pkg; the schemas in codegen testdata add 300MB of
weight to the GOMODCACHE needed for download. What if we moved the
testdata out from under the tree.

The move looks like this:

```
from=pkg/codegen/testing/test/testdata
to=tests/testdata/codegen/
git mv "$from" "$to"
(cd pkg/codegen/testing/test && ln -s ../../../../tests/testdata/codegen ./testdata)
git add "$from"
```

The previous location is symlinked to the new location.

Evidence of `GOMODCACHE` pressure reduction:
https://gist.github.com/t0yv0/05dd8be5880171045aed01e123ae2b09

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-06 20:36:50 +00:00
Fraser Waters 7302311c2c
Renable tests disabled due to appdash () 2024-01-17 09:09:53 +00:00
Justin Van Patten b0c276ece0
Update esc to v0.6.1 ()
First step in addressing https://github.com/pulumi/pulumi/issues/14873

Note: I suspect I'll need to temporarily disable some codegen tests to
get this merged, and then once we release v3.98.0 and the next version
of esc, we can re-enable.
2023-12-19 08:05:16 +00:00
Zaid Ajaj ac6adb98bf
[go/program-gen] Fix using inline invoke expressions inside resources, objects and arrays ()
# Description

This PR fixes an issue in Go program where if users are writing invoke
expressions inline inside other expressions, then that invoke is
extracted into a temporary variable and then later referenced the same
way it was used. This is because non-output-versioned invokes cannot be
used directly since they return a tuple (InvokeResult, err) so we need
to check for the error before proceeding.

Fixes 

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-03 14:22:43 +00:00
Abhinav Gupta a24ad1839b
programgen(go): Handle conflicting names in imported packages
This fixes how programgen generates import statements
to handle conflicting imports when two imported packages
have the same name, e.g.

    github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ecs
    github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecs

To do this, we add a fileImporter type that tracks these imports.
It prefers unnamed imports for packages unless one of the following is
true:

- the name of the package has already been used by another import
- the name of the package does not match the last component
  of the import path (e.g., `example.com/foo-go` with `package foo`).

If the name has already been used by another import,
it attempts the following in-order:

- Combine the last two path components of the import path
  into an identifier and use that if available.
  e.g., `awsxs3` from `sdk/go/awsx/s3`.
- Append a number to the package name and increment it
  until an unused name is found.
  e.g. `ecs2`, `ecs3`, and so on.

There's a change in how this information is tracked as well.
Previously, this was a pull approach: various calls returned
programImports objects which all got merged together.

This change switches to a push approach:
as code is generated and imports are requested,
they're submitted to the fileImporter which keeps track of them
until the next `Reset()` call.
The above also has a nice side effect of dropping a parameter.

Another change worth explicitly calling out:
Previously, getModOrAlias partially duplicated some of the logic
implemented in getPulumiImport, and used `mod`, `originalMod`
in a non-obvious way.
This generated incorrect imports like the following
(note the two `/aws` at the end):

    github.com/pulumi/pulumi-aws/sdk/v5/go/aws/aws

This change replicates more of the logic of getPulumiImport
(now called addPulumiImport) into this function,
and addresses the discrepancy in codegen caused by `mod`/`originalMod`.
The result leaves most existing code unchanged,
except in a couple existing cases where the resulting changes make sense
given the logic for named imports outlined above.

Resolves 
2023-07-25 12:49:37 -07:00
Zaid Ajaj a5975ca585 lint 2023-07-14 01:47:25 +02:00
Zaid Ajaj cd65005389 Fix panic in GenerateProject when version is not set in schema 2023-07-14 01:23:27 +02:00
Zaid Ajaj 6528a9fe7a Go module support as component resources in program-gen 2023-05-08 16:20:04 +02:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta 1158d4acee
all: Drop ioutil
Stop using io/ioutil across the entire repository.
The io/ioutil package was deprecated in Go 1.16 (2021-02)
with replacements provided in other packages.
Specifically:

    ioutil.Discard   => io.Discard
    ioutil.NopCloser => io.NopCloser
    ioutil.ReadAll   => io.ReadAll
    ioutil.ReadFile  => os.ReadFile
    ioutil.TempDir   => os.MkdirTemp
    ioutil.TempFile  => os.CreateTemp
    ioutil.WriteFile => os.WriteFile

This change switches all of these entities
across the repository.

Following this change,
the only references to ioutil are in schema files:

    % rg -l ioutil
    pkg/codegen/testing/test/testdata/aws-4.26.0.json
    pkg/codegen/testing/test/testdata/aws-4.36.0.json
    pkg/codegen/testing/test/testdata/aws-4.37.1.json
    pkg/codegen/testing/test/testdata/aws-5.4.0.json
    pkg/codegen/testing/test/testdata/aws-5.16.2.json

The bulk of this change was generated automatically
with manual touch ups afterwards.
2023-01-06 16:35:14 -08:00
Aaron Friel abb3763229 fix(ci): Split programgen tests over more jobs 2022-10-16 23:20:45 -07:00
Aaron Friel 4e7a887f3f chore: update test to use aws v5 2022-10-13 15:59:06 -07:00
Aaron Friel 2d90969b58 feat(ci): Enable fully offline codegen tests with versioned plugins
Commit 1 of 2: this makes the changes to every file except the schemas,
for ease of review.
2022-10-11 05:16:23 -07:00
Ian Wahbe b2ce84c195 Add `modulePath` to go 2022-10-10 16:01:53 -07:00
aq17 525849bf17 Support options.version on pulumi convert 2022-09-23 09:41:17 -07:00
Ian Wahbe 94422f97f8
Expose external package cache + global default (Take 2) ()
* Expose external package cache

* CL

* Add a default global cache

* More cache passing

* Compare `Cache` entries by pointer equality

* Add a check for providers when doing imports

* Pass the cache

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-08-12 20:04:21 +02:00
Aaron Friel cca0a39e70
Revert "Expose external package cache ()" ()
This reverts commit 16c173963a.
2022-08-11 15:04:14 -07:00
Ian Wahbe 16c173963a
Expose external package cache ()
* Expose external package cache

* CL

* Add a default global cache

* More cache passing

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-08-11 13:28:27 -07:00
Ian Wahbe e39998784a
Expose the check function () 2022-03-16 11:42:30 -07:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Ian Wahbe 8666d9c0b7
Normalize providers merge ()
* Normalize providers merge

* Update CHANGELOG_PENDING.md

* Fix CHANGELOG_PENDING.md

* Add unit tests

* Ensure go lints

* Remove `Go Lint` as redundant

* Fix go.mod for sdk

* Fix some tests

* Correctly fetch the package name from a token

* Restore unnecessary changes

* Use the correct pkg for provider
2022-02-08 15:35:36 +01:00
Ian Wahbe a1e18dae4d
export codegen tests for internal use ()
* Export Codegen test modules

* Document pkg stability guarantee

* Expose programgen

* Recommendation: Improve wording

* Move `internal` to `testing`

* Re-rout references to codegen/internal

* Fix some other "internal" references
2022-02-07 12:10:04 +01:00
Aaron Friel ee709d6c52 [codegen/go] Improve optional params in invoke
As described in , docs generated for helper functions can be incorrect because optional arguments to parameter objects are not correctly handled.

Previously, code would be generated like so:

```go
    policyDocument, err := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
      Statements: []iam.GetPolicyDocumentStatement{
        iam.GetPolicyDocumentStatement{
          Sid: "1",
          //...
```

However "Sid" is of type `*string`.

This four helper conversion functions, to handle the primitive types we lower from, and modifies codegen to recursively apply these functions inside of an invoke call.

In the new code generation, the above is instead rendered as:

```go
    policyDocument, err := iam.GetPolicyDocument(ctx, &iam.GetPolicyDocumentArgs{
      Statements: []iam.GetPolicyDocumentStatement{
        iam.GetPolicyDocumentStatement{
          Sid: pulumi.StringRef("1"),
          //...
```
2022-02-01 11:44:40 -08:00
Ian Wahbe f12b8fa00c
Cleanup program-gen test dependency gathering () 2021-10-19 12:17:19 -07:00
Pat Gavlin f21eda521f
[codegen] Rename the PCL package. ()
It's just confusing that PCL lives in a package named `hcl2`.
2021-09-29 20:11:56 -07:00
Ian Wahbe 9f6589ed8c
iwahbe/7802/compile program generator test output ()
* Move program tests into folders

* update package schema

* Enabled tests pass

* Fix lints and begin to update test cases

* Re-enable tests

* Update aws version to v4

* Refactor language specific parts

* Hook up dotnet and nodejs

* Update tests from master

* SSOT for schema/version

* Name blocking errors. Leave tests in valid state

* Give each language its own folder

* Remove SkipCompile for azure-sa (bug was fixed)

* Fix nits + changes asserts to require

* Remove unused import

* One last assert => require
2021-09-29 11:33:57 -07:00
T-Vova 90c0fa30af
Implemented file reading support for Go code generator ()
Co-authored-by: Vova Ivanov <jetvova@gmail.com>
2021-09-09 22:10:25 +03:00
Komal fddeea8a88
[codegen] Unify program codegen testing ()
* ProgramGen test driver

* Refactor to avoid exporting hcl2 entities
2021-07-27 18:00:39 -07:00
Pat Gavlin 7b1d6ec1ac
Reify `Input` and `Optional` types in the schema type system. ()
These changes support arbitrary combinations of input + plain types
within a schema. Handling plain types at the property level was not
sufficient to support such combinations. Reifying these types
required updating quite a bit of code. This is likely to have caused
some temporary complications, but should eventually lead to
substantial simplification in the SDK and program code generators.

With the new design, input and optional types are explicit in the schema
type system. Optionals will only appear at the outermost level of a type
(i.e. Input<Optional<>>, Array<Optional<>>, etc. will not occur). In
addition to explicit input types, each object type now has a "plain"
shape and an "input" shape. The former uses only plain types; the latter
uses input shapes wherever a plain type is not specified. Plain types
are indicated in the schema by setting the "plain" property of a type spec
to true.
2021-06-24 09:17:55 -07:00
Mikhail Shilkov 1378c45849 Discriminated unions in schema and programgen 2021-05-05 10:01:22 +02:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
evanboyle 2f71fbbf2d traverse object and tuple cons expressions for imports 2020-08-05 15:23:36 -07:00
Evan Boyle 31770c3300
go program gen: prompt array conversion, unused range vars, id handling () 2020-06-24 11:07:26 -07:00
Evan Boyle b302dbcbca
go program gen: lift expressions where prompt optionals are required () 2020-06-23 10:59:26 -07:00
Evan Boyle 8b8170252b
[Go Program Gen] multiline strings, get/lookup disambiguation, webserver example () 2020-06-18 13:34:22 -07:00
Evan Boyle 9cdddd2566
[Go Program Gen] Support for EKS example () 2020-06-17 17:19:39 -07:00
Evan Boyle 2d61852e4f
Go program gen: resource range, readDir, template strings, etc () 2020-06-15 23:00:02 -07:00
Evan Boyle 94cd67b564
[Go Program Gen] Improved type handling and invokes () 2020-06-05 18:52:00 -07:00
Evan Boyle ea114b04ee
output generation for s3 logging example () 2020-05-26 07:07:54 -07:00
Evan Boyle 8dd329f57f
Go basic program gen () 2020-05-21 10:23:33 -07:00
Evan Boyle 0d257bb32d
go gen_program shell + test + sample () 2020-05-14 08:43:12 -07:00