Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
module github.com/pulumi/pulumi/cmd/pulumi-test-language
|
|
|
|
|
|
|
|
go 1.20
|
|
|
|
|
|
|
|
replace github.com/pulumi/pulumi/sdk/v3 => ../../sdk
|
|
|
|
|
|
|
|
replace github.com/pulumi/pulumi/pkg/v3 => ../../pkg
|
|
|
|
|
|
|
|
require (
|
|
|
|
github.com/blang/semver v3.5.1+incompatible
|
|
|
|
github.com/pulumi/pulumi/pkg/v3 v3.81.0
|
2023-11-19 12:43:09 +00:00
|
|
|
github.com/pulumi/pulumi/sdk/v3 v3.94.2
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/segmentio/encoding v0.3.6
|
|
|
|
github.com/stretchr/testify v1.8.4
|
2023-10-28 02:10:00 +00:00
|
|
|
google.golang.org/grpc v1.57.1
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
google.golang.org/protobuf v1.31.0
|
|
|
|
gopkg.in/yaml.v2 v2.4.0
|
|
|
|
)
|
|
|
|
|
|
|
|
require (
|
|
|
|
cloud.google.com/go v0.110.4 // indirect
|
|
|
|
cloud.google.com/go/compute v1.20.1 // indirect
|
|
|
|
cloud.google.com/go/compute/metadata v0.2.3 // indirect
|
|
|
|
cloud.google.com/go/iam v1.1.1 // indirect
|
|
|
|
cloud.google.com/go/kms v1.12.1 // indirect
|
|
|
|
cloud.google.com/go/logging v1.7.0 // indirect
|
|
|
|
cloud.google.com/go/longrunning v0.5.1 // indirect
|
|
|
|
cloud.google.com/go/storage v1.30.1 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
dario.cat/mergo v1.0.0 // indirect
|
2023-09-20 14:48:41 +00:00
|
|
|
github.com/AlecAivazis/survey/v2 v2.3.7 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/Azure/azure-sdk-for-go v66.0.0+incompatible // indirect
|
|
|
|
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.1 // indirect
|
|
|
|
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.0.0 // indirect
|
|
|
|
github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.0 // indirect
|
|
|
|
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1 // indirect
|
|
|
|
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest v0.11.28 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/adal v0.9.21 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/azure/auth v0.5.11 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/azure/cli v0.4.6 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/to v0.4.0 // indirect
|
|
|
|
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
|
|
|
|
github.com/Azure/go-autorest/logger v0.2.1 // indirect
|
|
|
|
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
|
|
|
|
github.com/AzureAD/microsoft-authentication-library-for-go v0.4.0 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
github.com/Microsoft/go-winio v0.6.1 // indirect
|
|
|
|
github.com/ProtonMail/go-crypto v0.0.0-20230828082145-3c4c8a2d2371 // indirect
|
|
|
|
github.com/acomagu/bufpipe v1.0.4 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da // indirect
|
|
|
|
github.com/agext/levenshtein v1.2.3 // indirect
|
|
|
|
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
|
|
|
|
github.com/armon/go-metrics v0.4.0 // indirect
|
|
|
|
github.com/armon/go-radix v1.0.0 // indirect
|
|
|
|
github.com/atotto/clipboard v0.1.4 // indirect
|
|
|
|
github.com/aws/aws-sdk-go v1.44.298 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2 v1.17.3 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.3 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/config v1.15.15 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/credentials v1.12.10 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.9 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.21 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.27 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.21 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.16 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.6 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.3 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.10 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.9 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.9 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/kms v1.18.1 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.27.2 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/sso v1.11.13 // indirect
|
|
|
|
github.com/aws/aws-sdk-go-v2/service/sts v1.16.10 // indirect
|
|
|
|
github.com/aws/smithy-go v1.13.5 // indirect
|
|
|
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
|
|
|
github.com/cenkalti/backoff/v3 v3.2.2 // indirect
|
|
|
|
github.com/charmbracelet/bubbles v0.16.1 // indirect
|
|
|
|
github.com/charmbracelet/bubbletea v0.24.2 // indirect
|
|
|
|
github.com/charmbracelet/lipgloss v0.7.1 // indirect
|
|
|
|
github.com/cheggaaa/pb v1.0.29 // indirect
|
|
|
|
github.com/cloudflare/circl v1.3.3 // indirect
|
|
|
|
github.com/containerd/console v1.0.4-0.20230313162750-1ae8d489ac81 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/davecgh/go-spew v1.1.1 // indirect
|
|
|
|
github.com/dimchansky/utfbom v1.1.1 // indirect
|
|
|
|
github.com/djherbis/times v1.5.0 // indirect
|
|
|
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
|
|
github.com/edsrzf/mmap-go v1.1.0 // indirect
|
|
|
|
github.com/emirpasic/gods v1.18.1 // indirect
|
|
|
|
github.com/fatih/color v1.13.0 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
|
|
|
|
github.com/go-git/go-billy/v5 v5.5.0 // indirect
|
|
|
|
github.com/go-git/go-git/v5 v5.9.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/gofrs/uuid v4.2.0+incompatible // indirect
|
|
|
|
github.com/gogo/protobuf v1.3.2 // indirect
|
|
|
|
github.com/golang-jwt/jwt v3.2.1+incompatible // indirect
|
|
|
|
github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
|
|
|
|
github.com/golang/glog v1.1.0 // indirect
|
|
|
|
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
|
|
|
|
github.com/golang/protobuf v1.5.3 // indirect
|
|
|
|
github.com/golang/snappy v0.0.4 // indirect
|
|
|
|
github.com/google/go-cmp v0.5.9 // indirect
|
|
|
|
github.com/google/go-querystring v1.1.0 // indirect
|
|
|
|
github.com/google/s2a-go v0.1.4 // indirect
|
|
|
|
github.com/google/uuid v1.3.0 // indirect
|
|
|
|
github.com/google/wire v0.5.0 // indirect
|
|
|
|
github.com/googleapis/enterprise-certificate-proxy v0.2.3 // indirect
|
|
|
|
github.com/googleapis/gax-go/v2 v2.11.0 // indirect
|
|
|
|
github.com/gorilla/mux v1.8.0 // indirect
|
|
|
|
github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 // indirect
|
|
|
|
github.com/hashicorp/errwrap v1.1.0 // indirect
|
|
|
|
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
|
|
|
|
github.com/hashicorp/go-hclog v1.2.2 // indirect
|
|
|
|
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
|
|
|
|
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
|
|
|
github.com/hashicorp/go-plugin v1.4.6 // indirect
|
|
|
|
github.com/hashicorp/go-retryablehttp v0.7.1 // indirect
|
|
|
|
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
|
|
|
|
github.com/hashicorp/go-secure-stdlib/mlock v0.1.2 // indirect
|
|
|
|
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.6 // indirect
|
|
|
|
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
|
|
|
|
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
|
|
|
|
github.com/hashicorp/go-uuid v1.0.3 // indirect
|
|
|
|
github.com/hashicorp/go-version v1.6.0 // indirect
|
|
|
|
github.com/hashicorp/golang-lru v0.5.4 // indirect
|
|
|
|
github.com/hashicorp/hcl v1.0.0 // indirect
|
2023-10-10 01:35:39 +00:00
|
|
|
github.com/hashicorp/hcl/v2 v2.17.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/hashicorp/vault/api v1.8.2 // indirect
|
|
|
|
github.com/hashicorp/vault/sdk v0.6.1 // indirect
|
|
|
|
github.com/hashicorp/yamux v0.1.1 // indirect
|
|
|
|
github.com/ijc/Gotty v0.0.0-20170406111628-a8b993ba6abd // indirect
|
|
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
|
|
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
|
|
|
|
github.com/jmespath/go-jmespath v0.4.0 // indirect
|
|
|
|
github.com/json-iterator/go v1.1.12 // indirect
|
|
|
|
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
|
|
|
|
github.com/kevinburke/ssh_config v1.2.0 // indirect
|
|
|
|
github.com/kylelemons/godebug v1.1.0 // indirect
|
|
|
|
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
|
|
|
|
github.com/mattn/go-colorable v0.1.12 // indirect
|
|
|
|
github.com/mattn/go-isatty v0.0.18 // indirect
|
|
|
|
github.com/mattn/go-localereader v0.0.1 // indirect
|
|
|
|
github.com/mattn/go-runewidth v0.0.14 // indirect
|
|
|
|
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect
|
|
|
|
github.com/mitchellh/copystructure v1.2.0 // indirect
|
|
|
|
github.com/mitchellh/go-homedir v1.1.0 // indirect
|
|
|
|
github.com/mitchellh/go-ps v1.0.0 // indirect
|
|
|
|
github.com/mitchellh/go-testing-interface v1.14.1 // indirect
|
|
|
|
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
|
|
|
|
github.com/mitchellh/mapstructure v1.5.0 // indirect
|
|
|
|
github.com/mitchellh/reflectwalk v1.0.2 // indirect
|
|
|
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
|
|
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
|
|
|
github.com/muesli/ansi v0.0.0-20211018074035-2e021307bc4b // indirect
|
|
|
|
github.com/muesli/cancelreader v0.2.2 // indirect
|
|
|
|
github.com/muesli/reflow v0.3.0 // indirect
|
|
|
|
github.com/muesli/termenv v0.15.1 // indirect
|
|
|
|
github.com/natefinch/atomic v1.0.1 // indirect
|
|
|
|
github.com/oklog/run v1.1.0 // indirect
|
|
|
|
github.com/opentracing/basictracer-go v1.1.0 // indirect
|
|
|
|
github.com/opentracing/opentracing-go v1.2.0 // indirect
|
|
|
|
github.com/pgavlin/goldmark v1.1.33-0.20200616210433-b5eb04559386 // indirect
|
|
|
|
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
|
|
|
|
github.com/pjbgf/sha1cd v0.3.0 // indirect
|
|
|
|
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect
|
|
|
|
github.com/pkg/errors v0.9.1 // indirect
|
|
|
|
github.com/pkg/term v1.1.0 // indirect
|
|
|
|
github.com/pmezard/go-difflib v1.0.0 // indirect
|
2023-11-17 07:39:36 +00:00
|
|
|
github.com/pulumi/esc v0.6.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/rivo/uniseg v0.4.4 // indirect
|
2023-10-10 01:35:39 +00:00
|
|
|
github.com/rogpeppe/go-internal v1.11.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/ryanuber/go-glob v1.0.0 // indirect
|
|
|
|
github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect
|
|
|
|
github.com/santhosh-tekuri/jsonschema/v5 v5.0.0 // indirect
|
|
|
|
github.com/segmentio/asm v1.1.3 // indirect
|
|
|
|
github.com/sergi/go-diff v1.3.1 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
github.com/skeema/knownhosts v1.2.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
github.com/spf13/cobra v1.7.0 // indirect
|
|
|
|
github.com/spf13/pflag v1.0.5 // indirect
|
|
|
|
github.com/texttheater/golang-levenshtein v1.0.1 // indirect
|
|
|
|
github.com/tweekmonster/luser v0.0.0-20161003172636-3fa38070dbd7 // indirect
|
|
|
|
github.com/uber/jaeger-client-go v2.30.0+incompatible // indirect
|
|
|
|
github.com/uber/jaeger-lib v2.4.1+incompatible // indirect
|
|
|
|
github.com/xanzy/ssh-agent v0.3.3 // indirect
|
|
|
|
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
|
|
|
|
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
|
|
|
|
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
|
|
|
|
github.com/zclconf/go-cty v1.13.2 // indirect
|
|
|
|
go.opencensus.io v0.24.0 // indirect
|
|
|
|
go.uber.org/atomic v1.9.0 // indirect
|
|
|
|
gocloud.dev v0.27.0 // indirect
|
|
|
|
gocloud.dev/secrets/hashivault v0.27.0 // indirect
|
2023-10-19 19:42:54 +00:00
|
|
|
golang.org/x/crypto v0.14.0 // indirect
|
2023-10-11 16:57:30 +00:00
|
|
|
golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
golang.org/x/mod v0.13.0 // indirect
|
2023-10-19 19:42:54 +00:00
|
|
|
golang.org/x/net v0.17.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
golang.org/x/oauth2 v0.8.0 // indirect
|
2023-10-28 07:25:27 +00:00
|
|
|
golang.org/x/sync v0.4.0 // indirect
|
2023-10-11 16:57:30 +00:00
|
|
|
golang.org/x/sys v0.13.0 // indirect
|
2023-10-19 19:42:54 +00:00
|
|
|
golang.org/x/term v0.13.0 // indirect
|
|
|
|
golang.org/x/text v0.13.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
golang.org/x/time v0.0.0-20220722155302-e5dcc9cfc0b9 // indirect
|
2023-11-01 17:21:52 +00:00
|
|
|
golang.org/x/tools v0.14.0 // indirect
|
Add matrix testing (#13705)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Adds the first pass of matrix testing.
Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.
Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).
Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.
This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property
These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).
There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.
1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
|
|
|
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
|
|
|
|
google.golang.org/api v0.126.0 // indirect
|
|
|
|
google.golang.org/appengine v1.6.7 // indirect
|
|
|
|
google.golang.org/genproto v0.0.0-20230726155614-23370e0ffb3e // indirect
|
|
|
|
google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130 // indirect
|
|
|
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20230706204954-ccb25ca9f130 // indirect
|
|
|
|
gopkg.in/square/go-jose.v2 v2.6.0 // indirect
|
|
|
|
gopkg.in/warnings.v0 v0.1.2 // indirect
|
|
|
|
gopkg.in/yaml.v3 v3.0.1 // indirect
|
|
|
|
lukechampine.com/frand v1.4.2 // indirect
|
|
|
|
sourcegraph.com/sourcegraph/appdash v0.0.0-20211028080628-e2786a622600 // indirect
|
|
|
|
)
|