pulumi/cmd/pulumi-test-language
Will Jones 1be4888f7d
Replace `result.Result` with native errors (#17044)
`result.Result` is a type that was introduced to enable us to
distinguish between *expected* errors (or *bails* in Pulumi parlance)
and *unexpected* errors or exceptions. Prior to Go 1.13, this made a lot
of sense, since there was no standard story on "wrapping" errors, and
thus no way to answer the question "did I end up here (in an error path)
because I meant to, or did I end up here because of a program bug?".

With the introduction of `%w`, `interface { Unwrap() []error }` and
company in Go 1.13, a simpler interface is available: one can use
`errors.Is` and `errors.As` to ask if an error at any point wraps an
error of a certain type. With this, we can simply ask "is there a bail
at any point in this error tree?" rather than having to track this
explicitly using a type such as `result.Result`. The `IsBail` function
was introduced a while ago to this end, but its rollout was not
completed and several uses of `result.Result` remained.

This commit completes the rollout of this simplified interface,
replacing all uses of `result.Result` with native Go errors that may or
may not wrap bails, and all uses of e.g. `res.IsBail` with
`IsBail(err)`. Doing so allows us to remove `result.Result` entirely.
2024-08-22 14:39:59 +00:00
..
providers Parse conformance programs to decide which packages are needed for SDKs (#16872) 2024-08-05 12:07:06 +00:00
testdata Add an `organization` intrinsic to PCL (#16948) 2024-08-19 03:58:19 +00:00
README.md Add sequence diagram about pulumi-test-language (#15289) 2024-01-29 22:08:14 +00:00
go.mod Changelog and go.mod updates for v3.129.0 (#16947) 2024-08-12 19:28:33 +00:00
go.sum upgrade pulumi-yaml to 1.9.2 (#16925) 2024-08-09 15:09:23 +00:00
interface.go Install missing python versions using pyenv during installation (#16855) 2024-08-19 15:55:54 +00:00
interface_test.go Add a test that all our conformance test programs are bindable (#16770) 2024-07-24 06:55:18 +00:00
internal_test.go Parse conformance programs to decide which packages are needed for SDKs (#16872) 2024-08-05 12:07:06 +00:00
l1empty_test.go Run pulumi-test-language tests in parallel (#16060) 2024-04-25 21:32:39 +00:00
l1main_test.go Add asset/archive to conformance tests and fix engine working dir issues (#16100) 2024-05-02 11:32:54 +00:00
l2continue_on_error_test.go Run pulumi-test-language tests in parallel (#16060) 2024-04-25 21:32:39 +00:00
l2destroy_test.go Run pulumi-test-language tests in parallel (#16060) 2024-04-25 21:32:39 +00:00
l2large_test.go Refactor: move plugin kind to apitype (#15946) 2024-04-25 17:30:30 +00:00
l2resourceasset_test.go Add conformance tests for remote assets (#16467) 2024-07-03 09:03:24 +00:00
l2resourcesimple_test.go Replace `result.Result` with native errors (#17044) 2024-08-22 14:39:59 +00:00
main.go Add matrix testing (#13705) 2023-09-13 15:17:46 +00:00
runtime_options_test.go Run pulumi-test-language tests in parallel (#16060) 2024-04-25 21:32:39 +00:00
snapshot_test.go Add asset/archive to conformance tests and fix engine working dir issues (#16100) 2024-05-02 11:32:54 +00:00
snapshots.go Add asset/archive to conformance tests and fix engine working dir issues (#16100) 2024-05-02 11:32:54 +00:00
test_host.go Use grpc providers in conformance tests (#16952) 2024-08-13 18:48:44 +00:00
testing.go Replace `result.Result` with native errors (#17044) 2024-08-22 14:39:59 +00:00
tests.go Replace `result.Result` with native errors (#17044) 2024-08-22 14:39:59 +00:00

README.md

pulumi-language-test runs a gRPC interface that language plugins can use to run a suite of standard tests.

Architecture

pulumi-language-test is used to run a standard suite of tests against any compliant language plugin.

The diagram below shows the main interactions and data flows for how this system works.

There are three main actors involved. Firstly test which is a test function coordinating the language plugin and pulumi-language-test. Secondly ptl which is the pulumi-language-test process. Finally uut which is the language plugin actually being tested. This will generally be a grpc server running in the same process as the test method.


sequenceDiagram
    test->>ptl: Start ptl process
    ptl-->>test: Read stdout for ptl address to connect to

    Note right of test: All future calls to ptl are via grpc

    test->>+ptl: GetLanguageTests()
    ptl-->>-test: Returns list of test names

    test->>+uut: Serve
    uut-->>-test: Returns uut server address

    test->>+ptl: PrepareLanguageTests(uut)
    ptl->>+uut: Pack(core)
    uut-->>-ptl: Returns name of core artifact
    ptl-->>-test: Returns `token`

    loop for each test
        test->>+ptl: RunLanguageTest(token, test)
        loop for each sdk used in test
            ptl->>+uut: GeneratePackage(sdk)
            uut-->>-ptl: Write package code to temporary directory
            ptl->>ptl: Verify sdk snapshot
            ptl->>+uut: Pack(sdk)
            uut-->>-ptl: Returns name of sdk artifact
        end

        ptl->>+uut: GenerateProject(test)
        uut-->>-ptl: Write project code to temporary directory
        ptl->>ptl: Verify project snapshot

        ptl->>+uut: InstallDependencies(project)
        uut-->>-ptl: 

        note right of ptl: Execute test with engine
        activate ptl
        ptl->>+uut: Run(project)
        uut-->>-ptl: Return run result
        ptl->>ptl: Run test asserts against run result and snapshot
        deactivate ptl

        ptl-->>-test: Returns test result from asserts
    end

    test->>ptl: sigkill

Meta tests

This module contains a number of _test.go files. These are tests of the conformance test system itself. The actual conformance tests are all defined in tests.go.