pulumi/cmd/pulumi-test-language
Thomas Gummerer 898a682ef6
Make sure non-targeted resources are not updated (#15476)
When the `--target` option is used, resources that already exist in the
snapshot, but aren't directly targeted should not be updated at all.
Internally in the engine, this is done by turning them into a
`SameStep`, meaning no updates will actually be preformed, and we will
make it look like the resource stayed the same.

However, we currently still write the "new" state of the resource (e.g.
updated dependencies, inputs, etc.) into the snapshot. This is mostly
fine as long as the new dependencies already exist. If a dependency on a
resource is that doesn't already exist is added however this breaks.
Since the resource that's being depended on doesn't exist in the
snapshot and isn't targeted, we won't create it. At the same time we're
adding a dependency on that virtually non-existing resource, which makes
the snapshot invalid.

Since we're in `--target` mode, we should do what we promised the user,
and only update the targeted resources, nothing else. Introduce a new
`NonTargetedSameStep` here, which does exactly that. It's essentially
the same as a `SameStep`, but we always use the *old* state instead of
the new one when writing it out. Since the resource is not targeted,
this leaves it in the same state as before.

Fixes #12096
Fixes #15382
2024-03-05 07:49:11 +00:00
..
testdata Make sure non-targeted resources are not updated (#15476) 2024-03-05 07:49:11 +00:00
README.md Add sequence diagram about pulumi-test-language (#15289) 2024-01-29 22:08:14 +00:00
bad_provider.go Test GetDependencies and library versions in conformance testing (#15324) 2024-02-06 12:38:44 +00:00
go.mod Changelog and go.mod updates for v3.108.1 (#15571) 2024-03-01 22:33:54 +00:00
go.sum Auto-fix encrypted keys in the wrong format due to gocloud.dev upgrade regression (#15334) 2024-02-01 09:39:41 +00:00
interface.go Allow multiple updates in a single conformance test (#15504) 2024-03-01 12:20:12 +00:00
interface_test.go Test GetDependencies and library versions in conformance testing (#15324) 2024-02-06 12:38:44 +00:00
internal_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
l1empty_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
l1main_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
l2destroy_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
l2resourcesimple_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
main.go Add matrix testing (#13705) 2023-09-13 15:17:46 +00:00
runtime_options_test.go don't run conformance test tests in parallel (#15584) 2024-03-04 14:26:51 +00:00
simple_provider.go Allow multiple updates in a single conformance test (#15504) 2024-03-01 12:20:12 +00:00
testing.go Use mapset in pulumi-test-language (#14738) 2023-12-04 20:49:34 +00:00
tests.go Make sure non-targeted resources are not updated (#15476) 2024-03-05 07:49:11 +00:00

README.md

pulumi-language-test runs a gRPC interface that language plugins can use to run a suite of standard tests.

Architecture

pulumi-language-test is used to run a standard suite of tests against any compliant language plugin.

The diagram below shows the main interactions and data flows for how this system works.

There are three main actors involved. Firstly test which is a test function coordinating the language plugin and pulumi-language-test. Secondly ptl which is the pulumi-language-test process. Finally uut which is the language plugin actually being tested. This will generally be a grpc server running in the same process as the test method.


sequenceDiagram
    test->>ptl: Start ptl process
    ptl-->>test: Read stdout for ptl address to connect to

    Note right of test: All future calls to ptl are via grpc

    test->>+ptl: GetLanguageTests()
    ptl-->>-test: Returns list of test names

    test->>+uut: Serve
    uut-->>-test: Returns uut server address

    test->>+ptl: PrepareLanguageTests(uut)
    ptl->>+uut: Pack(core)
    uut-->>-ptl: Returns name of core artifact
    ptl-->>-test: Returns `token`

    loop for each test
        test->>+ptl: RunLanguageTest(token, test)
        loop for each sdk used in test
            ptl->>+uut: GeneratePackage(sdk)
            uut-->>-ptl: Write package code to temporary directory
            ptl->>ptl: Verify sdk snapshot
            ptl->>+uut: Pack(sdk)
            uut-->>-ptl: Returns name of sdk artifact
        end

        ptl->>+uut: GenerateProject(test)
        uut-->>-ptl: Write project code to temporary directory
        ptl->>ptl: Verify project snapshot

        ptl->>+uut: InstallDependencies(project)
        uut-->>-ptl: 

        note right of ptl: Execute test with engine
        activate ptl
        ptl->>+uut: Run(project)
        uut-->>-ptl: Return run result
        ptl->>ptl: Run test asserts against run result and snapshot
        deactivate ptl

        ptl-->>-test: Returns test result from asserts
    end

    test->>ptl: sigkill

Meta tests

This module contains a number of _test.go files. These are tests of the conformance test system itself. The actual conformance tests are all defined in tests.go.