<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
In the filestate code we would always run loaded snapshots through
VerifyIntegrity before using them. The httpstate didn't have any similar
checks. This brings the two backends into alignment, reusing the option
from before that was just for filestate (--disable-integrity-checking).
At some point we should further align these so that httpstate also
validates the snapshots it has written out, like filestate does today.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
Fixes https://github.com/pulumi/pulumi/issues/13242.
The checks for if we should auto-opt-in to project mode was stricter
than intended. This now checks for if there's any legacy stacks rather
than if there's any files/folders at all.
The filestate backend implementation drops context in many places
instead opting to use context.TODO.
context.TODO is a TODO--something intended to be addressed.
context.Backgruond is appropriate for process-scoped operations.
In this case, we can resolve the issues and drop all uses of
context.TODO by plumbing the received contexts appropriately.
For localSnapshotPersister, we add a new `ctx` field
because the interface doesn't yet accept a context argument.
This is tracked in #12594.
Result:
```
% rg context.TODO pkg/backend/filestate
[empty]
```
Resolves#12594
This is an alternative take on what #12473 was for.
Specifically, the current filestate.New constructor
makes it tedious to inject optional hooks
to control external state like the environment, time, etc.
This introduces a private constructor:
func newLocalBackend(..., *localBackendOptions) (*localBackendReference, error)
The filestate.New constructor just calls newLocalBackend
with the default options.
The only available option is Getenv: an override for os.Getenv.
We replace all direct uses of os.Getenv with this function reference,
so this allows us to control environment variables in tests
without *actually* changing them with t.Setenv.
That, in turn, allows these tests to run in parallel again.
To further demonstrate the value of doing this,
this change also includes tests for previously untested functionality:
the PULUMI_RETAIN_CHECKPOINTS and PULUMI_SELF_MANAGED_STATE_GZIP
environment variables.
The only remaining uses of t.Setenv are in tests
that cross boundaries to stack.DefaultSecretsProvider.
That dependency is also easy to break with localBackendOptions
in a future change.
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Adds a referenceStore abstraction to control the layout of the storage,
and a legacyReferenceStore implementation based on the current layout
(that does not support projects).
This allows us to move code to determine file paths of stacks, their
histories, and their backups, all into a single component that we can
swap out for project support.
localBackendReferences keep track of the referenceStore that built them.
The primary reason for this is that when we add support for migrating a
stack state from legacy to project mode, `backend.store` will become
mutable.
For references created before the store for a backend was changed, we
still need to be able to access their original file paths, so we need to
hold onto the original referenceStore.
However, as a side-effect of this,
it's more convenient to acess paths from `ref.Foo()` rather than
`backend.foo(ref)` or `backend.store.Foo(ref)`.
In the future, we may also move stackPath to the store,
since right now the .json/.json.gz logic is duplicated in a couple
places.
Extracted from #12134
filestate backend currently operates exclusively with stack names.
All its internal pass around just the stack name, and nothing else.
This makes it a bit difficult to add project support to the backend.
This is a refactor in advance of adding project support,
changing the internals of filestate to pass a stack reference around.
It inspects the reference directly for all its operations.
Note: This contains no behavioral changes.
Name and FullyQualifiedName currently both return just the stack name.
In a future change, once project name is incorporated into the object,
FullyQualifiedName will be able to return `organization/$project/$name`.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
Incremental step towards #12132
Migrates uses of contract.{Assert, AssertNoError, Require} in pkg/engine
to `*f` variants so that we're required to provide more error context.
Refs #12132
This is a first pass at supporting projects for the file state backend.
Projects must be given explictly, that is unlike the service backend where the stack reference "teststack" would implictly parse as the stack "teststack" in the current project in the current organisation.
"teststack" with the filestate backend will continue to parse as the stack "teststack" not associated with a project, but you can now give the stack reference now as "testproj/teststack" to get a project scoped stack.
This makes no effort to try automatically moving stack files to be associated with projects. Users can use pulumi stack rename to do that if they so wish.
Fixes https://github.com/pulumi/pulumi/issues/2522
* ci: include required deps in test artifact builds
* ci: reduce retention on other build artifacts to match pulumi CLI
* ci: move lang providers acquisition to script/goreleaser
* ci: combine goreleaser configs, move OS-dependency to script
* ci: re-enable Windows and MacOS tests
* [cli] Fix Windows path handling for gzip filestate backend
* Revert "[ci] Name the archive to push to buckets"
This reverts commit 73f23fbc39.
* Revert "[ci] Fix extra extension in goreleaser templates"
This reverts commit 03888a33fa.
* Revert #10149
This reverts commit f6ee2fd20c.
This reverts commit 33a84df3cf.
This reverts commit 01c1719c1c.
This reverts commit e7b96115be.
This reverts commit 60fb289408.
This reverts commit 7a81b1435f.
This reverts commit b1f2e41b56.
* chore: changelog for 3.37.1
* add gzip compression in filestate backend
New versions of pulumi can now read json files wrapped in .gz
Compression is disabled by default, can be enabled via:
PULUMI_SELF_MANAGED_STATE_GZIP=true
* functionnal test for gzip filestate
* update CHANGELOG_PENDING.md
* Move compression to encoding
* Update changelog
* Clean up tests
* Adjust IsCompressed check
* Don't leave out of data state files
Co-authored-by: Arthur Woimbée <arthur@extrality.ai>
Co-authored-by: Arthur Woimbée <arthur.woimbee@gmail.com>
* Readd "Make StackReference.Name a tokens.Name (#9088)"
This reverts commit f0aa4df149.
This also removes the AsName asserting casts for stack names. We do want
to add them in at some point to be sure that bad names don't slip in
somehow but they don't need adding with this.
* Update sdk/go/common/util/fsutil/qname.go
Co-authored-by: Ian Wahbe <ian@wahbe.com>
Co-authored-by: Ian Wahbe <ian@wahbe.com>
Adds a `--limit` flag to `pulumi stack history. This allows limiting to the last few entries rather than fetching the entirety of a stack's update history (which can be quite slow for stacks with lots of updates). Example: `pulumi stack history --limit 1` fetches the last history entry only.
`stack.up` and related operations in the Automation API have been updated to consume this change, drastically reducing overhead.
When writing the snapshot to the filestate bucket, we can retry in the
event of an error, which helps users who are experiencing issues around
write rates to GCS
This probably seems like a trivial change, but while debugging #4258 it
was apparent that an error could have come from 3 different places. This
rewords an error message to make it slightly clearer what the error is.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)