pulumi/cmd/pulumi-test-language
Fraser Waters 3043adeeb3
Add SupportPack to schemas to write out in the new style (#15713)
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->


This adds a new flag to the schema metadata to tell codegen to use the
new proposed style of SDKs where we fill in versions and write go.mods
etc.

I've reworked pack to operate on packages assuming they're in this new
style. That is pack no longer has the responsibility to fill in any
version information.

This updates python and node codegen to write out SDKs in this new
style, and fixes their core libraries to still be buildable via pack.
There are two approaches to fixing those, I've chosen option 1 below but
could pretty easily rework for option 2.

1) Write the version information directly to the SDKs at the same time
as we edit the .version file. To simplify this I've added a new
'set-version.py' script that takes a version string an writes it to all
the relevant places (.version, package.json, etc).

2) Write "pack" in the language host to search up the directory tree for
the ".version" file and then fill in the version information as we we're
doing before with envvar tricks and copying and editing package.json.

I think 1 is simpler long term, but does force some amount of cleanup in
unrelated bits of the system right now (release makefiles need a small
edit). 2 is much more localised but keeps this complexity that
sdk/nodejs sdk/python aren't actually valid source modules.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-22 09:25:46 +00:00
..
testdata Add snapshot edit facilities to conformance testing (#15747) 2024-03-21 15:26:58 +00:00
README.md Add sequence diagram about pulumi-test-language (#15289) 2024-01-29 22:08:14 +00:00
bad_provider.go Test GetDependencies and library versions in conformance testing (#15324) 2024-02-06 12:38:44 +00:00
go.mod Bump google.golang.org/protobuf, golang.org/x/crypto, and github.com/moby/moby (#15717) 2024-03-17 22:20:32 +00:00
go.sum Bump google.golang.org/protobuf, golang.org/x/crypto, and github.com/moby/moby (#15717) 2024-03-17 22:20:32 +00:00
interface.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
interface_test.go Test GetDependencies and library versions in conformance testing (#15324) 2024-02-06 12:38:44 +00:00
internal_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
l1empty_test.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
l1main_test.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
l2destroy_test.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
l2resourcesimple_test.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
main.go Add matrix testing (#13705) 2023-09-13 15:17:46 +00:00
runtime_options_test.go Add SupportPack to schemas to write out in the new style (#15713) 2024-03-22 09:25:46 +00:00
simple_provider.go Allow multiple updates in a single conformance test (#15504) 2024-03-01 12:20:12 +00:00
snapshot_test.go Add snapshot edit facilities to conformance testing (#15747) 2024-03-21 15:26:58 +00:00
snapshots.go Add snapshot edit facilities to conformance testing (#15747) 2024-03-21 15:26:58 +00:00
testing.go Use mapset in pulumi-test-language (#14738) 2023-12-04 20:49:34 +00:00
tests.go l2-destroy test: sort the correct part of the slice (#15660) 2024-03-13 10:05:38 +00:00

README.md

pulumi-language-test runs a gRPC interface that language plugins can use to run a suite of standard tests.

Architecture

pulumi-language-test is used to run a standard suite of tests against any compliant language plugin.

The diagram below shows the main interactions and data flows for how this system works.

There are three main actors involved. Firstly test which is a test function coordinating the language plugin and pulumi-language-test. Secondly ptl which is the pulumi-language-test process. Finally uut which is the language plugin actually being tested. This will generally be a grpc server running in the same process as the test method.


sequenceDiagram
    test->>ptl: Start ptl process
    ptl-->>test: Read stdout for ptl address to connect to

    Note right of test: All future calls to ptl are via grpc

    test->>+ptl: GetLanguageTests()
    ptl-->>-test: Returns list of test names

    test->>+uut: Serve
    uut-->>-test: Returns uut server address

    test->>+ptl: PrepareLanguageTests(uut)
    ptl->>+uut: Pack(core)
    uut-->>-ptl: Returns name of core artifact
    ptl-->>-test: Returns `token`

    loop for each test
        test->>+ptl: RunLanguageTest(token, test)
        loop for each sdk used in test
            ptl->>+uut: GeneratePackage(sdk)
            uut-->>-ptl: Write package code to temporary directory
            ptl->>ptl: Verify sdk snapshot
            ptl->>+uut: Pack(sdk)
            uut-->>-ptl: Returns name of sdk artifact
        end

        ptl->>+uut: GenerateProject(test)
        uut-->>-ptl: Write project code to temporary directory
        ptl->>ptl: Verify project snapshot

        ptl->>+uut: InstallDependencies(project)
        uut-->>-ptl: 

        note right of ptl: Execute test with engine
        activate ptl
        ptl->>+uut: Run(project)
        uut-->>-ptl: Return run result
        ptl->>ptl: Run test asserts against run result and snapshot
        deactivate ptl

        ptl-->>-test: Returns test result from asserts
    end

    test->>ptl: sigkill

Meta tests

This module contains a number of _test.go files. These are tests of the conformance test system itself. The actual conformance tests are all defined in tests.go.