<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
This PR fixes a small problem with ProgramTest, that the `With` function
doesn't propagate the `PostPrepareProject` option as it should. e.g.
```go
options := baseOptions.With(integration.ProgramTestOptions{
PostPrepareProject: func(p *engine.Projinfo) error {
// ...
},
})
```
A follow-up to: https://github.com/pulumi/pulumi/pull/12302
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
With the introduction of component programs there will be mulitple
"pulumi:pulumi:stack" resources in a program. We should therefore check
against the qualified type of resources to see if they are the root
stack, not just their direct type.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This PR changes the behavior of `ProgramTester` in
`pkg/testing/integration` package, to detect test failures more eagerly
and then to skip remaining steps. Specifically it checks whether the
test been marked as failed by a validation function (as defined by
[`t.Failed()`](https://pkg.go.dev/testing#T.Failed)).
The rationale is that, for most integration tests, each step assumes
that the previous step was successful. It is simply noisy to continue
with the subsequent steps. A workaround is to call `t.FailNow()` in the
validation function (or use `require`).
An option is provided `ExpectTestFailure` to continue with steps after
test failure (as is the current behavior).
There exists an option `ExpectFailures` to continue with steps after a
pulumi deployment failure. I took such failures to be orthogonal to test
failure.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
## Example
Here's a snippet of output in the event that a validation function marks
the test as failed (e.g. using an assertion).
```
...
/Users/eronwright/Pulumi/pulumi/tests/examples/program.go:1742: Performing extra runtime validation.
/Users/eronwright/Pulumi/pulumi/tests/examples/examples_test.go:44:
Error Trace: ...
Error: Expected nil, but got ...
Test: TestAccMinimal
/Users/eronwright/Pulumi/pulumi/tests/examples/program.go:1744: Extra runtime validation complete.
/Users/eronwright/Pulumi/pulumi/tests/examples/program.go:1164: Canceling further steps due to test failure
/Users/eronwright/Pulumi/pulumi/tests/examples/program.go:1349: Destroying stack
...
/Users/eronwright/Pulumi/pulumi/tests/examples/program.go:1365: Test failed, retaining stack '...'
--- FAIL: TestAccMinimal (5.61s)
```
**Overview**
This re-enables tracking of code coverage.
For Go, there are two kinds of coverage at play:
unit test and integration test coverage.
Unit tests follow the usual pattern of running
`go test -cover -coverprofile=whatever.cov`.
For integration tests, we use the new integration test profiling support
[added in Go 1.20](https://go.dev/testing/coverage/).
In short, the way it works is:
# Build a coverage instrumented binary:
go build -cover
# Set GOCOVERDIR to a directory and run the integration tests
# that will invoke this coverage-instrumented binary.
GOCOVERDIR=$(pwd)/coverage
go test ./tests
# $GOCOVERDIR will now be filled with coverage data
# from every invocation of the coverage-instrumented binary.
# Combine it into a single coverage file:
go tool covdata textfmt -i=$(GOCOVERDIR) -o=out.cov
# The resulting file can be uploaded to codecov as-is.
The above replaces the prior, partially working hacks we had in place
to get coverage-instrumented binaries with `go test -c`
and hijacking the TestMain.
**Notable changes**
- TestMain hijacking is deleted from the Pulumi CLI.
We no longer need this to build coverage-instrumented binaries.
- ProgramTest no longer tracks or passes PULUMI_TEST_COVERAGE_PATH
because the Pulumi binary no longer accepts a test.coverprofile flag.
This information is now in the GOCOVERDIR environment variable.
- We add an `enable-coverage` parameter to the `ci-build-binaries`
workflow to mirror some of the other workflows.
It will produce coverage-instrumented binaries if this is true.
These binaries are then used by `ci-run-test` which will set
`GOCOVERDIR` and merge the coverage results from it.
- Coverage configuration no longer counts tests, testdata,
and Protobuf-generated code against coverage.
- go-wrapper.sh:
Because we're no longer relying on the `go test -c` hack,
this no longer excludes Windows and language providers
from coverage tracking.
- go-test.py and go-wrapper.sh will include pulumi-language-go and
pulumi-language-nodejs in covered packages.
*Other changes*
- go-test.py:
Fixed a bug where `args` parameters added for coverage were ignored.
Note that this change DOES NOT track coverage for calls made to Pulumi
packages by plugins downloaded from external sources,
e.g. provider plugins. Arguably, that's out of scope of coverage
trackcing for the Pulumi repository.
Resolves#8615, #11419
If a resource is renamed with `pulumi state rename`,
in addition to updating references in resources that refer to it
as a dependency,
also update references to the resource as a parent of another resource.
Testing:
Includes an integration test that reproduces the panic,
and a unit test that verifies the new behavior at a more isolated level.
To implement the integration test,
a new RunPulumiCommand method was added to ProgramTester
because we don't have other means of running `pulumi state rename`
with ProgramTest.
Resolves#13179
This commit fixes a bug where integration tests specifying
RelativeWorkDir would not load because the path to Pulumi.yaml was not
applying the RelativeWorkDir parameter when searching for the file.
12699: test(go): Integration test for MLC options r=abhinav a=abhinav
Adds a minimal integration test that verifies
options passed to an MLC in Go are propagated properly.
Currently, two options are checked: DependsOn and Protect.
Other options will be added to this test as part of #12154.
To aid in the `Protect` test,
this also adds an option to ProgramTest to set the `--exclude-protected` flag
when running `pulumi destroy` at the end of the run.
This will allow integration tests that create protected resources
without going back and unprotecting them for cleanup.
12713: refactor(go/host): Extract global state into a struct r=abhinav a=abhinav
Refactors the Go language host plugin
to extract global state (e.g. stdout, command line args)
into a mainCmd struct.
The command line parsing logic is placed into a separate,
independently tested function.
This change does not modify any behavior.
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
The function `writePackageJSON` used by ProgramTest
always returns a non-nil error--even when encode succeded
because it unconditionally wraps the result with an error.
Adds an option to ProgramTest to set the `--exclude-protected` flag
when running `pulumi destroy` at the end of the run.
This will allow integration tests that create protected resources
without going back and unprotecting them.
Migrates all remaining usages of
`contract.Assert*` and `contract.Require*` to the f variants,
which require adding meaningful error messages.
There were a couple cases where a `testing.T` or `testing.B`
was already available.
For those, this uses t.FailNow or require.NoError.
Refs #12132
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
Replaces all instances of `strings.Replace(s, old, new, -1)`
with the equivalent `strings.ReplaceAll(s, old, new)`.
This function has been available since Go 1.12.
This change to ProgramTest allows Go module replacements to omit the major
version number, simplifying the process to release a provider with a new major
version.
When the major version is incremented in a PR, the examples fail to compile if
they reference the incorrect SDK version.
This enables the examples to be written to target a versionless `/sdk`, and have
a Dependencies line such as:
```go
integration.ProgramTestOptions{
Dependencies: []string{
"github.com/pulumi/pulumi-keycloak/sdk=../sdk",
},
}
```
This versionless import mapping means that examples run regardless of the major
version of the provider, and updates to major versions in PRs will require only
one PR.
A provider PR proving out the implementation was created against this commit
here:
- https://github.com/pulumi/pulumi-keycloak/pull/165
Removes unnecessary appends in pkg.
There were two instances:
- append into a slice that's never used again
- append with nothing to append (`append([]{foo}) == []{foo}`)
Issue found by staticcheck:
```
testing/integration/program.go:2190:15: SA4021: x = append(y) is equivalent to x = y (staticcheck)
testing/integration/program.go:2192:15: SA4021: x = append(y) is equivalent to x = y (staticcheck)
codegen/pcl/utilities.go:158:11: SA4010: this result of append is never used, except maybe in other appends (staticcheck)
```
Refs #11808
This replaces for loops and slice appends reported by gosimple
with simpler variants.
Specifically,
for _, x := range src {
dst = append(dst, x)
}
// can be replaced with
dst = append(dst, src...)
And,
for i, x := range src {
dst[i] = x
}
// can be replaced with
copy(dst, src)
And,
for true { ... }
// can be replaced with
for { ... }
And, given a string `s`,
for _, r := range []rune(s) { .. }
// can be replaced with
for _, r := range s { .. }
Lastly, this fixes in ineffective break statement
also reported by the linter.
Inside a switch block,
`break` affects the current `case` only.
The outer loop needs a label.
Stop using io/ioutil across the entire repository.
The io/ioutil package was deprecated in Go 1.16 (2021-02)
with replacements provided in other packages.
Specifically:
ioutil.Discard => io.Discard
ioutil.NopCloser => io.NopCloser
ioutil.ReadAll => io.ReadAll
ioutil.ReadFile => os.ReadFile
ioutil.TempDir => os.MkdirTemp
ioutil.TempFile => os.CreateTemp
ioutil.WriteFile => os.WriteFile
This change switches all of these entities
across the repository.
Following this change,
the only references to ioutil are in schema files:
% rg -l ioutil
pkg/codegen/testing/test/testdata/aws-4.26.0.json
pkg/codegen/testing/test/testdata/aws-4.36.0.json
pkg/codegen/testing/test/testdata/aws-4.37.1.json
pkg/codegen/testing/test/testdata/aws-5.4.0.json
pkg/codegen/testing/test/testdata/aws-5.16.2.json
The bulk of this change was generated automatically
with manual touch ups afterwards.
11524: test: use T.TempDir to create temporary test directory r=Frassle a=Juneezee
# Description
A testing cleanup.
This pull request replaces `ioutil.TempDir` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
```go
func TestFoo(t *testing.T) {
// before
tmpDir, err := ioutil.TempDir("", "")
assert.NoError(t, err)
defer os.RemoveAll(tmpDir)
// now
tmpDir := t.TempDir()
}
```
## Checklist
- [ ] ~~I have added tests that prove my fix is effective or that my feature works~~: Tests refactoring only, no new features added.
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] ~~I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change~~: This PR is a non user-facing change.
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] ~~Yes, there are changes in this PR that warrants bumping the Pulumi Service API version~~: N/A
<!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->
11586: ci: Fix extraneous alpha draft releases r=AaronFriel a=AaronFriel
The [PR workflow to update Pulumi YAML to 1.0.4](https://github.com/pulumi/pulumi/actions/runs/3642973677) shows that it created a draft release, and this has been the case for all contributors' PRs.
This fixes the extra alpha releases created as a result of pull requests. The job "prepare-release" shouldn't run unless the ci/test label is set, indicating an intent to evaluate the full CI pipeline.
Co-authored-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
The cache key is based on hashing the contents of "requirements.txt" for each
python test. This should result in unchanged test results among tests
sharing venvs, and maintains environment isolation across requirement sets
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
10725: Use set-all r=t0yv0 a=t0yv0
<!---
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->
# Description
Similar to what Automation API does. It does not look like Automation API does any escaping so perhaps this is OK. The change saves lots of precious seconds on CI when a lot of examples and config values are involved, like in pulumi/templates.
<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->
Fixes # (issue)
## Checklist
<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
<!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
10720: ci: Enable programtests against local backend, improve isolation r=AaronFriel a=AaronFriel
These changes make integration tests run much more quickly on a local dev loop, for easily parallelized program tests like in the `integration_go_test.go` file the improvement was on the order of minutes across all tests.
The changes rely on setting `PULUMI_BACKEND_URL` to override the backend during particular tests. The backend is set to a temporary directory which is cleaned up on exit.
A helper function `NewBackendUrl(t *testing.T)` is added to enable
When `PULUMI_TEST_USE_SERVICE=true`, the `RequireService` option is set to true.
When `RequireService == true`, the test is skipped if an access token is not present, improving local dev experience by skipping tests which would error very loudly.
When `RequireService == false and CloudURL == ""`, then we use the helper function to create a temporary directory and point the filestate backend to it.
The CloudURL check allows tests which, even in the presence of `PULUMI_TEST_USE_SERVICE=true`, to still run against a local backend. E.g.:
```go
localTestOptions := testOptions.With(integration.ProgramTestOptions{
CloudURL: integration.NewBackendURL(t),
})
```
10734: prepare for next release (v3.41.0) r=AaronFriel a=pulumi-bot
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
Co-authored-by: github-actions <github-actions@github.com>
* Consistently disable HTML escaping in JSON output
* Add to CHANGELOG
* Add test
* fix warning
* Add RawJSON
* Change to RawJSON
* Change MarshalIndent to JSON
* Don't try to replace everywhere
* Targeted fix for stack state