Fixes https://github.com/pulumi/pulumi/issues/13242.
The checks for if we should auto-opt-in to project mode was stricter
than intended. This now checks for if there's any legacy stacks rather
than if there's any files/folders at all.
This is an alternative take on what #12473 was for.
Specifically, the current filestate.New constructor
makes it tedious to inject optional hooks
to control external state like the environment, time, etc.
This introduces a private constructor:
func newLocalBackend(..., *localBackendOptions) (*localBackendReference, error)
The filestate.New constructor just calls newLocalBackend
with the default options.
The only available option is Getenv: an override for os.Getenv.
We replace all direct uses of os.Getenv with this function reference,
so this allows us to control environment variables in tests
without *actually* changing them with t.Setenv.
That, in turn, allows these tests to run in parallel again.
To further demonstrate the value of doing this,
this change also includes tests for previously untested functionality:
the PULUMI_RETAIN_CHECKPOINTS and PULUMI_SELF_MANAGED_STATE_GZIP
environment variables.
The only remaining uses of t.Setenv are in tests
that cross boundaries to stack.DefaultSecretsProvider.
That dependency is also easy to break with localBackendOptions
in a future change.
With project support added,
filestate will default to project-scoped stacks
for all newly initialized buckets.
This is desirable long-term, but for the initial release,
we'd like users to have an escape hatch to go back to the old layout
until they've had a change to migrate.
This adds the ability for users to opt-out of this feature
by setting an environment variable.
Note that this only applies to new buckets.
You cannot opt out of this feature for a bucket
that is already using project-scoped stacks.
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.
We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.
To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:
# .pulumi/meta.yaml
version: 1
Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.
Notes:
- Stack references produced by filestate will shorten to
just the stack name if the project name for the stack
matches the currently selected project.
This required turning currentProject on localBackend
into an atomic pointer because otherwise
SetCurrentProject and localBackendReference.String may race.
Extracted from #12134
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
In #12472, we added a .pulumi/Pulumi.yaml file to filestate backends
to hold metadata about the storage layout format.
It has been brought up multiple times that this is a confusing name,
so we're renaming it to meta.yaml.
Backwards compatibility:
Although we've already shipped #12472,
and have started writing Pulumi.yaml files,
we don't currently use it for anything.
Its first use will be in #12437 when we add project-scoped stacks.
Therefore, this change is completely backwards compatible.
We'll end up leaving the extraneous .pulumi/Pulumi.yaml file behind
if the bucket was initialized with Pulumi v3.59.0 or newer,
before this change was released. This is not a high cost.
I don't think we should go back and delete these files;
I'm reluctant for the backend to perform destructive operations
for files that are technically not managed by it.
In #12472, we added a new file to filestate backends:
.pulumi/Pulumi.yaml.
This file is intended to store metadata about the layout
in anticipation of support for project-scoped stacks.
This had the unintended side-effect of breaking users
who use tight access control on their S3 buckets:
users don't get write access outside specific stack directories
so they get an error like the following:
error: write ".pulumi/Pulumi.yaml": blob (key ".pulumi/Pulumi.yaml") (code=Unknown): AccessDenied: Access Denied
This changes filestate.New to never write the file
if the version is zero
so users that don't currently have write access to that directory
will continue to be able to use the backend.
Resolves#12534
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437