mirror of https://github.com/pulumi/pulumi.git
6f8cfbab9f
Presently, we implement code generation (e.g. for SDKs and programs/`pulumi convert`) on a per-language basis in `pkg/codegen`. Alongside these implementations, we have a set of tests built using a common framework that use snapshots to verify that code generation doesn't break when changes are made. Unfortunately, due to the way things are currently laid out in our repository, these tests and their dependencies are shipped as part of the `pkg/codegen` package. This commit brings @blampe's work in https://github.com/pulumi/pulumi/pull/16011 up to date and fixes this by taking the following actions: * Test harnesses that were previously located in `pkg/codegen/<language>/test.go` are moved to `pkg/codegen/testing/test/<language>.go`. By default, Go excludes files ending in `_test.go`, but *not* files named e.g. `test.go`. It might seem logical therefore to just rename these files (e.g. to `codegen_test.go`), so that they can continue to live alongside their language implementations. Unfortunately, while this fixes one problem (dependencies pulling in test code), it introduces another -- the existing `test.go` files actually exist to implement an interface which is used by the common codegen-test framework we have. Moving to `*_test.go` files would make them invisible to the actual modules which run the tests through that framework. * Test code is consequently refactored to clean up the separation of test details (e.g. relevant fixtures) and execution implementation (working directory, means of program generation). * The `gen_program_test/generate.go` program, which we use to generate the "batch tests", has been updated so that it respects the new code layout and organisation. As laid out by @blampe in #16011 (and shamelessly copied here), this brings a number of benefits. Test dependencies are no longer included in non-test packages that are consumed downstream. As an example, this takes the `pulumi-language-go` binary from ~61MB down to ~36MB, and speeds up build times: ``` go clean -cache && time go build . (master) go build . 81.06s user 17.36s system 470% cpu 20.933 total (PR) go build . 47.77s user 9.20s system 530% cpu 10.737 total ``` This doesn't completely remove these dependencies from downstream because most also include `pkg/v3/testing/integration` in tests. It does only compile these dependencies into the test binary, though. Before: ``` ❯ go mod why github.com/aws/aws-sdk-go-v2/service/sts github.com/pulumi/pulumi-random/provider/v4/cmd/pulumi-tfgen-random github.com/pulumi/pulumi-terraform-bridge/pf/tfgen github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfgen github.com/pulumi/pulumi/pkg/v3/codegen/dotnet github.com/pulumi/pulumi/pkg/v3/testing/integration github.com/pulumi/pulumi/pkg/v3/resource/stack github.com/pulumi/pulumi/pkg/v3/secrets/cloud github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test github.com/aws/aws-sdk-go-v2/service/sts ``` After: ``` github.com/pulumi/pulumi-random/provider/v4 github.com/pulumi/pulumi-random/provider/v4.test <- expected github.com/pulumi/providertest github.com/pulumi/pulumi/pkg/v3/testing/integration github.com/pulumi/pulumi/pkg/v3/resource/stack github.com/pulumi/pulumi/pkg/v3/secrets/cloud github.com/pulumi/pulumi/pkg/v3/secrets/cloud.test github.com/aws/aws-sdk-go-v2/service/sts ``` Note: this PR has been split into a number of commits, roughly one per language, to make reviewing a bit easier. The commits *do not build individually* due to the nature of the changes, but hopefully it makes understanding the work a bit more feasible. Closes #16011 --------- Co-authored-by: Bryce Lampe <bryce@pulumi.com> |
||
|---|---|---|
| .. | ||
| cgstrings | ||
| convert | ||
| docs | ||
| dotnet | ||
| gen_program_test | ||
| go | ||
| hcl2 | ||
| nodejs | ||
| pcl | ||
| python | ||
| report | ||
| schema | ||
| testing | ||
| README.md | ||
| docs.go | ||
| docs_test.go | ||
| programs.md | ||
| sdks.md | ||
| utilities.go | ||
| utilities_test.go | ||
| utilities_types.go | ||
README.md
(codegen)= (crosscode)=
Code generation
Code generation is essential to Pulumi's ability to support both a variety of programming languages and a variety of cloud providers. This package defines the core components of Pulumi's code generation functionality (known as Pulumi CrossCode). At a high level, code generation is used to manage three categories of output: SDKs, programs, and documentation. At a lower level, these all make use of a number of shared concepts such as schema and Pulumi Configuration Language (PCL).
:::{toctree} :maxdepth: 1 :titlesonly:
/pkg/codegen/sdks.md /pkg/codegen/programs.md /pkg/codegen/docs/README /pkg/codegen/schema/README /pkg/codegen/pcl/README :::