Commit Graph

8 Commits

Author SHA1 Message Date
Fraser Waters 2c74dddc91
Switch to use env.Env in filestate ()
Internal refactor to use `env.Env` directly in filestate rather than
mocking `os.Getenv`.
2023-10-18 10:52:54 +00:00
Fraser Waters 51d36e1eb4 Fix auto-opt-in for local project stacks
Fixes https://github.com/pulumi/pulumi/issues/13242.

The checks for if we should auto-opt-in to project mode was stricter
than intended. This now checks for if there's any legacy stacks rather
than if there's any files/folders at all.
2023-06-22 13:36:52 +01:00
Abhinav Gupta 82c82c6848
refactor(filestate): Don't use t.Setenv in tests
This is an alternative take on what  was for.
Specifically, the current filestate.New constructor
makes it tedious to inject optional hooks
to control external state like the environment, time, etc.

This introduces a private constructor:

    func newLocalBackend(..., *localBackendOptions) (*localBackendReference, error)

The filestate.New constructor just calls newLocalBackend
with the default options.

The only available option is Getenv: an override for os.Getenv.
We replace all direct uses of os.Getenv with this function reference,
so this allows us to control environment variables in tests
without *actually* changing them with t.Setenv.
That, in turn, allows these tests to run in parallel again.

To further demonstrate the value of doing this,
this change also includes tests for previously untested functionality:
the PULUMI_RETAIN_CHECKPOINTS and PULUMI_SELF_MANAGED_STATE_GZIP
environment variables.

The only remaining uses of t.Setenv are in tests
that cross boundaries to stack.DefaultSecretsProvider.
That dependency is also easy to break with localBackendOptions
in a future change.
2023-03-31 16:29:06 -07:00
Fraser Waters 20e89176d9
filestate: Support opting into legacy layout
With project support added,
filestate will default to project-scoped stacks
for all newly initialized buckets.

This is desirable long-term, but for the initial release,
we'd like users to have an escape hatch to go back to the old layout
until they've had a change to migrate.

This adds the ability for users to opt-out of this feature
by setting an environment variable.

Note that this only applies to new buckets.
You cannot opt out of this feature for a bucket
that is already using project-scoped stacks.
2023-03-31 13:21:37 -07:00
Fraser Waters 9d0fba3a7e
filestate: Re-add project support
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.

We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.

To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:

    # .pulumi/meta.yaml
    version: 1

Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.

Notes:

- Stack references produced by filestate will shorten to
  just the stack name if the project name for the stack
  matches the currently selected project.
  This required turning currentProject on localBackend
  into an atomic pointer because otherwise
  SetCurrentProject and localBackendReference.String may race.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-31 13:21:36 -07:00
Abhinav Gupta 32bd1f9635
filestate: Rename Pulumi.yaml to meta.yaml
In , we added a .pulumi/Pulumi.yaml file to filestate backends
to hold metadata about the storage layout format.
It has been brought up multiple times that this is a confusing name,
so we're renaming it to meta.yaml.

Backwards compatibility:
Although we've already shipped ,
and have started writing Pulumi.yaml files,
we don't currently use it for anything.
Its first use will be in  when we add project-scoped stacks.
Therefore, this change is completely backwards compatible.

We'll end up leaving the extraneous .pulumi/Pulumi.yaml file behind
if the bucket was initialized with Pulumi v3.59.0 or newer,
before this change was released. This is not a high cost.
I don't think we should go back and delete these files;
I'm reluctant for the backend to perform destructive operations
for files that are technically not managed by it.
2023-03-28 12:13:55 -07:00
Abhinav Gupta 450f35586c
filestate: Don't write metadata for legacy layouts
In , we added a new file to filestate backends:
.pulumi/Pulumi.yaml.
This file is intended to store metadata about the layout
in anticipation of support for project-scoped stacks.

This had the unintended side-effect of breaking users
who use tight access control on their S3 buckets:
users don't get write access outside specific stack directories
so they get an error like the following:

    error: write ".pulumi/Pulumi.yaml": blob (key ".pulumi/Pulumi.yaml") (code=Unknown): AccessDenied: Access Denied

This changes filestate.New to never write the file
if the version is zero
so users that don't currently have write access to that directory
will continue to be able to use the backend.

Resolves 
2023-03-28 12:13:54 -07:00
Abhinav Gupta e47c5823c0
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.

To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.

Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".

This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.

In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.

If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.

Note that this differs slightly
from the initial implementation of this functionality in .
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.

This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.

Extracted from 
2023-03-22 12:25:13 -07:00