mirror of https://github.com/pulumi/pulumi.git
173 lines
7.6 KiB
Markdown
173 lines
7.6 KiB
Markdown
(language-conformance-tests)=
|
|
# Language conformance tests
|
|
|
|
*Language conformance tests* (often just *conformance tests*) are a class of
|
|
integration test that assert various properties that should hold true across all
|
|
[language runtimes](language-runtimes), in essence providing a specification
|
|
that language runtimes must conform to. They are structured as follows:
|
|
|
|
* The "program under test" is expressed using [PCL](pcl). The program can
|
|
specify resources and functions supplied by one or more test resource
|
|
providers with fixed, known implementations, as well as provider-agnostic
|
|
entities such as stack outputs.
|
|
* A set of assertions are made about the [state](state-snapshots) of the
|
|
resources, functions, outputs, etc. in the program before and after it has
|
|
been executed. These assertions should hold true regardless of the language
|
|
being used to define and execute the program.
|
|
|
|
For each test run and language, then:
|
|
|
|
* If the test requires one or more providers, SDKs are generated from the
|
|
relevant test providers, exercising SDK generation for the given language.
|
|
* The PCL program is converted into a program in the target language, exercising
|
|
program generation for the given language.
|
|
* The generated program is executed using a compiled language host running as a
|
|
separate process and the assertions are checked, exercising program execution
|
|
for the given language.
|
|
* Generated code is snapshot tested to ensure that it doesn't change
|
|
unexpectedly.
|
|
|
|
## Running Language Conformance Tests
|
|
|
|
To run the language conformance tests, for example for Python, run:
|
|
|
|
```bash
|
|
go test github.com/pulumi/pulumi/sdk/python/cmd/pulumi-language-python/v3 -count 1
|
|
```
|
|
|
|
Note: to run this from the root of the repository, make sure you have an uptodate `go.work` file by running `make work`.
|
|
|
|
The conformance tests are named `TestLanguage/${TEST_NAME}`, for example `TestLanguage/l1-output-string`.
|
|
|
|
Python can generate SDKs in multiple ways, either using `setup.py` or `project.toml` to define the python package metadata, as well as using different input types (`classes-and-dicts` or `classes`). Python also supports either `MyPy` or `PyRight` as typechecker. To ensure we cover all of these options, each test is run in 3 variants (the options are orthogonal to each other, so we don't run all combinations):
|
|
|
|
* `default` (uses `setup.py`, `MyPy` and the default input types)
|
|
* `toml` (uses `pyproject.toml`, `PyRight` and `classes-and-dicts`)
|
|
* `classes` (uses `setup`, `PyRight` and `classes`)
|
|
|
|
Test names follow the pattern `TestLanguage/${VARIANT}/${TEST_NAME}`, for example `TestLanguage/classes/l1-output-string`.
|
|
|
|
For Nodejs we have two variants, using TypeScript (`forceTsc=false`) or plain Javascript (`forceTsc=true`, so named because the test setup runs `tsc` on the project so it's runnable as plain Javascript). Tests are named for example `TestLanguage/forceTsc=true/l1-output-string` or `TestLanguage/forceTsc=false/l1-output-string`.
|
|
|
|
To run a single test case:
|
|
|
|
```bash
|
|
go test github.com/pulumi/pulumi/sdk/python/cmd/pulumi-language-python/v3 -count 1 -run TestLanguage/classes/l1-output-string
|
|
```
|
|
|
|
To update the snapshots for generated code, run the tests with the `PULUMI_ACCEPT` environment variable set to a truthy value:
|
|
|
|
```bash
|
|
PULUMI_ACCEPT=1 go test github.com/pulumi/pulumi/sdk/python/cmd/pulumi-language-python/v3 -count 1
|
|
```
|
|
|
|
## Architecture
|
|
|
|
Test providers are defined in
|
|
<gh-file:pulumi#cmd/pulumi-test-language/providers>. PCL programs for language
|
|
conformance tests are defined in
|
|
<gh-file:pulumi#cmd/pulumi-test-language/testdata>.
|
|
<gh-file:pulumi#cmd/pulumi-test-language/tests.go> then references these
|
|
programs and defines the assertions to be made about each. Tests are categorised
|
|
as follows:
|
|
|
|
* *L1 tests* are those which *do not* exercise provider code paths and use only
|
|
the most basic of features (e.g. stack outputs).
|
|
* *L2 tests* are those which *do* exercise provider code paths and use things
|
|
such as custom resources, function invocations, and so on.
|
|
* *L3 tests* exercise features that require more advanced language support such
|
|
as first-class functions -- `apply` is a good example of this.
|
|
|
|
Each language defines a test function (the *language test host*) responsible for
|
|
running its conformance test suite, if it implements one. For core languages
|
|
whose runtimes are written in Go, this typically lives in a `language_test.go`
|
|
file next to the relevant language host executable code -- see for example
|
|
<gh-file:pulumi#sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go> for
|
|
NodeJS/TypeScript and
|
|
<gh-file:pulumi#sdk/python/cmd/pulumi-language-python/language_test.go> for
|
|
Python. This function works as follows:
|
|
|
|
* The relevant [language runtime](language-runtimes) (e.g.
|
|
`pulumi-language-nodejs` for NodeJS) is booted up as a separate process.
|
|
* The `pulumi-test-language` executable is booted up, with the language
|
|
runtime's gRPC server address as a parameter. The test language executable
|
|
itself exposes a gRPC server which allows clients to e.g. retrieve a list of
|
|
tests (`GetLanguageTests`) and execute a test (`RunLanguageTest`).
|
|
* In preparation for test execution, the language test host retrieves the list
|
|
of tests from the `pulumi-test-language` server. It
|
|
[](pulumirpc.LanguageRuntime.Pack)s the core SDK (e.g. `@pulumi/pulumi` in
|
|
TypeScript/NodeJS).
|
|
* For each test:
|
|
* SDKs required by the test are generated by calling the language host's
|
|
[](pulumirpc.LanguageRuntime.GeneratePackage) method. The generated code is
|
|
written to a temporary directory where it is
|
|
[](pulumirpc.LanguageRuntime.Pack)ed for use in the test.
|
|
* The [](pulumirpc.LanguageRuntime.GenerateProject) method is invoked to
|
|
convert the test's PCL code into a program in the target language.
|
|
Dependencies are installed with
|
|
[](pulumirpc.LanguageRuntime.InstallDependencies) and the test program is
|
|
[](pulumirpc.LanguageRuntime.Run).
|
|
* Assertions are verified and the next test is processed until there are no
|
|
more remaining.
|
|
|
|
```mermaid
|
|
:caption: The lifecycle of a language conformance test suite
|
|
:zoom:
|
|
|
|
sequenceDiagram
|
|
participant LTH as Language test host
|
|
participant PTL as pulumi-test-language
|
|
participant LH as Language host
|
|
|
|
LTH->>+PTL: Start pulumi-test-language process
|
|
PTL-->>-LTH: Return gRPC server address
|
|
|
|
note right of LTH: All future calls to<br>pulumi-test-language are via gRPC
|
|
|
|
LTH->>+PTL: GetLanguageTests()
|
|
PTL-->>-LTH: List of test names
|
|
|
|
LTH->>+LH: Start language host
|
|
LH-->>-LTH: Return gRPC server address
|
|
|
|
note right of LTH: All future calls to<br>language host are via gRPC
|
|
|
|
LTH->>+PTL: PrepareLanguageTests(Language host)
|
|
PTL->>+LH: Pack(Core SDK)
|
|
LH-->>-PTL: Name of core SDK artifact
|
|
PTL->>-LTH: Token
|
|
|
|
loop For each test
|
|
LTH->>+PTL: RunLanguageTest(Token, Language test host)
|
|
|
|
loop For each SDK
|
|
PTL->>+LH: GeneratePackage(SDK)
|
|
LH-->>-PTL: Write package code to temporary directory
|
|
PTL->>PTL: Verify SDK snapshot
|
|
PTL->>+LH: Pack(SDK)
|
|
LH-->>-PTL: Name of SDK artifact
|
|
end
|
|
|
|
PTL->>+LH: GenerateProject(Test)
|
|
LH-->>-PTL: Write project code to temporary directory
|
|
PTL->>PTL: Verify project snapshot
|
|
|
|
PTL->>+LH: InstallDependencies(Project)
|
|
LH-->>-PTL: Dependencies installed
|
|
|
|
note right of PTL: Execute test with engine
|
|
|
|
PTL->>+LH: Run(Project)
|
|
LH-->>-PTL: Run result
|
|
PTL->>PTL: Check assertions against run result and snapshot
|
|
|
|
PTL-->>-LTH: Test result
|
|
end
|
|
```
|
|
|
|
## Meta tests
|
|
|
|
This module contains a number of `_test.go` files. These are tests of the
|
|
conformance test system itself. The actual conformance tests are all defined in
|
|
`tests.go`.
|