mirror of https://github.com/pulumi/pulumi.git
143 lines
6.0 KiB
Markdown
143 lines
6.0 KiB
Markdown
(language-conformance-tests)=
|
|
# Language conformance tests
|
|
|
|
*Language conformance tests* (often just *conformance tests*) are a class of
|
|
integration test that assert various properties that should hold true across all
|
|
[language runtimes](language-runtimes), in essence providing a specification
|
|
that language runtimes must conform to. They are structured as follows:
|
|
|
|
* The "program under test" is expressed using [PCL](pcl). The program can
|
|
specify resources and functions supplied by one or more test resource
|
|
providers with fixed, known implementations, as well as provider-agnostic
|
|
entities such as stack outputs.
|
|
* A set of assertions are made about the [state](state-snapshots) of the
|
|
resources, functions, outputs, etc. in the program before and after it has
|
|
been executed. These assertions should hold true regardless of the language
|
|
being used to define and execute the program.
|
|
|
|
For each test run and language, then:
|
|
|
|
* If the test requires one or more providers, SDKs are generated from the
|
|
relevant test providers, exercising SDK generation for the given language.
|
|
* The PCL program is converted into a program in the target language, exercising
|
|
program generation for the given language.
|
|
* The generated program is executed using a compiled language host running as a
|
|
separate process and the assertions are checked, exercising program execution
|
|
for the given language.
|
|
* Generated code is snapshot tested to ensure that it doesn't change
|
|
unexpectedly.[^pulumi-accept]
|
|
|
|
[^pulumi-accept]:
|
|
Snapshots can typically be updated by running tests with the `PULUMI_ACCEPT`
|
|
environment set to a truthy value (e.g. `PULUMI_ACCEPT=1 go test ...`).
|
|
|
|
## Architecture
|
|
|
|
Test providers are defined in
|
|
<gh-file:pulumi#cmd/pulumi-test-language/providers>. PCL programs for language
|
|
conformance tests are defined in
|
|
<gh-file:pulumi#cmd/pulumi-test-language/testdata>.
|
|
<gh-file:pulumi#cmd/pulumi-test-language/tests.go> then references these
|
|
programs and defines the assertions to be made about each. Tests are categorised
|
|
as follows:
|
|
|
|
* *L1 tests* are those which *do not* exercise provider code paths and use only
|
|
the most basic of features (e.g. stack outputs).
|
|
* *L2 tests* are those which *do* exercise provider code paths and use things
|
|
such as custom resources, function invocations, and so on.
|
|
* *L3 tests* exercise features that require more advanced language support such
|
|
as first-class functions -- `apply` is a good example of this.
|
|
|
|
Each language defines a test function (the *language test host*) responsible for
|
|
running its conformance test suite, if it implements one. For core languages
|
|
whose runtimes are written in Go, this typically lives in a `language_test.go`
|
|
file next to the relevant language host executable code -- see for example
|
|
<gh-file:pulumi#sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go> for
|
|
NodeJS/TypeScript and
|
|
<gh-file:pulumi#sdk/python/cmd/pulumi-language-python/language_test.go> for
|
|
Python. This function works as follows:
|
|
|
|
* The relevant [language runtime](language-runtimes) (e.g.
|
|
`pulumi-language-nodejs` for NodeJS) is booted up as a separate process.
|
|
* The `pulumi-test-language` executable is booted up, with the language
|
|
runtime's gRPC server address as a parameter. The test language executable
|
|
itself exposes a gRPC server which allows clients to e.g. retrieve a list of
|
|
tests (`GetLanguageTests`) and execute a test (`RunLanguageTest`).
|
|
* In preparation for test execution, the language test host retrieves the list
|
|
of tests from the `pulumi-test-language` server. It
|
|
[](pulumirpc.LanguageRuntime.Pack)s the core SDK (e.g. `@pulumi/pulumi` in
|
|
TypeScript/NodeJS).
|
|
* For each test:
|
|
* SDKs required by the test are generated by calling the language host's
|
|
[](pulumirpc.LanguageRuntime.GeneratePackage) method. The generated code is
|
|
written to a temporary directory where it is
|
|
[](pulumirpc.LanguageRuntime.Pack)ed for use in the test.
|
|
* The [](pulumirpc.LanguageRuntime.GenerateProject) method is invoked to
|
|
convert the test's PCL code into a program in the target language.
|
|
Dependencies are installed with
|
|
[](pulumirpc.LanguageRuntime.InstallDependencies) and the test program is
|
|
[](pulumirpc.LanguageRuntime.Run).
|
|
* Assertions are verified and the next test is processed until there are no
|
|
more remaining.
|
|
|
|
```{mermaid}
|
|
:caption: The lifecycle of a language conformance test suite
|
|
:zoom:
|
|
|
|
sequenceDiagram
|
|
participant LTH as Language test host
|
|
participant PTL as pulumi-test-language
|
|
participant LH as Language host
|
|
|
|
LTH->>+PTL: Start pulumi-test-language process
|
|
PTL-->>-LTH: Return gRPC server address
|
|
|
|
note right of LTH: All future calls to<br>pulumi-test-language are via gRPC
|
|
|
|
LTH->>+PTL: GetLanguageTests()
|
|
PTL-->>-LTH: List of test names
|
|
|
|
LTH->>+LH: Start language host
|
|
LH-->>-LTH: Return gRPC server address
|
|
|
|
note right of LTH: All future calls to<br>language host are via gRPC
|
|
|
|
LTH->>+PTL: PrepareLanguageTests(Language host)
|
|
PTL->>+LH: Pack(Core SDK)
|
|
LH-->>-PTL: Name of core SDK artifact
|
|
PTL->>-LTH: Token
|
|
|
|
loop For each test
|
|
LTH->>+PTL: RunLanguageTest(Token, Language test host)
|
|
|
|
loop For each SDK
|
|
PTL->>+LH: GeneratePackage(SDK)
|
|
LH-->>-PTL: Write package code to temporary directory
|
|
PTL->>PTL: Verify SDK snapshot
|
|
PTL->>+LH: Pack(SDK)
|
|
LH-->>-PTL: Name of SDK artifact
|
|
end
|
|
|
|
PTL->>+LH: GenerateProject(Test)
|
|
LH-->>-PTL: Write project code to temporary directory
|
|
PTL->>PTL: Verify project snapshot
|
|
|
|
PTL->>+LH: InstallDependencies(Project)
|
|
LH-->>-PTL: Dependencies installed
|
|
|
|
note right of PTL: Execute test with engine
|
|
|
|
PTL->>+LH: Run(Project)
|
|
LH-->>-PTL: Run result
|
|
PTL->>PTL: Check assertions against run result and snapshot
|
|
|
|
PTL-->>-LTH: Test result
|
|
end
|
|
```
|
|
|
|
## Meta tests
|
|
|
|
This module contains a number of `_test.go` files. These are tests of the
|
|
conformance test system itself. The actual conformance tests are all defined in
|
|
`tests.go`.
|