filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
// Copyright 2016-2023, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
|
|
|
package filestate
|
|
|
|
|
|
|
|
import (
|
|
|
|
"context"
|
|
|
|
"fmt"
|
|
|
|
"path/filepath"
|
|
|
|
|
2023-10-18 10:52:54 +00:00
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/env"
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
|
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
|
|
|
|
"gocloud.dev/gcerrors"
|
|
|
|
"gopkg.in/yaml.v3"
|
|
|
|
)
|
|
|
|
|
|
|
|
// Path inside the bucket where we store the metadata file.
|
2023-03-28 17:52:53 +00:00
|
|
|
var pulumiMetaPath = filepath.Join(workspace.BookkeepingDir, "meta.yaml")
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
|
2023-03-28 17:52:53 +00:00
|
|
|
// pulumiMeta holds the contents of the .pulumi/meta.yaml file
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
// in a filestate backend.
|
|
|
|
//
|
|
|
|
// This file specifies metadata for the backend,
|
|
|
|
// including a version number that the backend can use
|
|
|
|
// to maintain compatibility with older versions of the CLI.
|
2023-03-28 17:31:35 +00:00
|
|
|
//
|
|
|
|
// The metadata file is not written for legacy layouts.
|
|
|
|
// However, there was a short period of time where it was written,
|
|
|
|
// so we should still allow for Version 0 when reading these files.
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
type pulumiMeta struct {
|
|
|
|
// Version is the current version of the state store.
|
|
|
|
//
|
|
|
|
// Version 0 is the starting version.
|
|
|
|
// It does not support project-scoped stacks.
|
2023-02-10 12:24:28 +00:00
|
|
|
// Version 1 adds support for project-scoped stacks.
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
//
|
|
|
|
// Does not use "omitempty" to differentiate
|
|
|
|
// between a missing field and a zero value.
|
|
|
|
Version int `json:"version" yaml:"version"`
|
|
|
|
}
|
|
|
|
|
2023-03-28 17:31:35 +00:00
|
|
|
// ensurePulumiMeta loads the Pulumi state metadata file from the bucket.
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
//
|
|
|
|
// Unlike [readPulumiMeta],
|
|
|
|
// the result of this function will always be non-nil if the error is nil.
|
|
|
|
//
|
|
|
|
// If the bucket is empty, this will create a new metadata file
|
|
|
|
// with the latest version number.
|
2023-02-10 12:24:28 +00:00
|
|
|
// This can be overridden by setting the environment variable
|
|
|
|
// "PULUMI_SELF_MANAGED_STATE_LEGACY_LAYOUT" to "1".
|
|
|
|
// ensurePulumiMeta uses the provided 'getenv' function
|
|
|
|
// to read the environment variable.
|
2023-10-18 10:52:54 +00:00
|
|
|
func ensurePulumiMeta(ctx context.Context, b Bucket, e env.Env) (*pulumiMeta, error) {
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
meta, err := readPulumiMeta(ctx, b)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
if meta != nil {
|
|
|
|
return meta, nil
|
|
|
|
}
|
|
|
|
|
2023-02-10 12:24:28 +00:00
|
|
|
// If there's no metadata file, we need to create one.
|
|
|
|
// The version we pick for the new file decides how we lay out the state.
|
2023-03-28 17:31:35 +00:00
|
|
|
//
|
2023-02-10 12:24:28 +00:00
|
|
|
// - Version 0 is legacy mode, which is the old layout.
|
|
|
|
// To avoid breaking old stacks, we want to use version 0
|
|
|
|
// if the bucket is not empty.
|
|
|
|
//
|
|
|
|
// - Version 1 added support for project-scoped stacks.
|
|
|
|
// For entirely new buckets, we'll use version 1
|
|
|
|
// to give new users access to the latest features.
|
2023-06-22 12:33:03 +00:00
|
|
|
refs, err := newLegacyReferenceStore(b).ListReferences(ctx)
|
2023-02-10 12:24:28 +00:00
|
|
|
if err != nil {
|
2023-06-22 12:33:03 +00:00
|
|
|
// If there's an error listing don't fail, just don't print the warnings
|
2023-02-10 12:24:28 +00:00
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
2023-06-22 12:33:03 +00:00
|
|
|
useLegacy := len(refs) > 0
|
|
|
|
if !useLegacy {
|
2023-02-10 12:24:28 +00:00
|
|
|
// Allow opting into legacy mode for new states
|
|
|
|
// by setting the environment variable.
|
2023-10-18 10:52:54 +00:00
|
|
|
useLegacy = e.GetBool(env.SelfManagedStateLegacyLayout)
|
2023-02-10 12:24:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if useLegacy {
|
2023-02-10 12:24:28 +00:00
|
|
|
meta = &pulumiMeta{Version: 0}
|
2023-02-10 12:24:28 +00:00
|
|
|
} else {
|
|
|
|
meta = &pulumiMeta{Version: 1}
|
2023-02-10 12:24:28 +00:00
|
|
|
}
|
|
|
|
|
2023-03-28 17:31:35 +00:00
|
|
|
// Implementation detail:
|
2023-02-10 12:24:28 +00:00
|
|
|
// For version 0, WriteTo won't write the metadata file.
|
|
|
|
// See [pulumiMeta.WriteTo] for details on why.
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
if err := meta.WriteTo(ctx, b); err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
return meta, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// readPulumiMeta loads the Pulumi state metadata from the bucket.
|
|
|
|
// If the file does not exist, it returns nil and no error.
|
|
|
|
func readPulumiMeta(ctx context.Context, b Bucket) (*pulumiMeta, error) {
|
|
|
|
metaBody, err := b.ReadAll(ctx, pulumiMetaPath)
|
|
|
|
if err != nil {
|
|
|
|
if gcerrors.Code(err) == gcerrors.NotFound {
|
|
|
|
return nil, nil
|
|
|
|
}
|
|
|
|
return nil, fmt.Errorf("read %q: %w", pulumiMetaPath, err)
|
|
|
|
}
|
|
|
|
|
|
|
|
// State is a copy of the pulumiMeta shape,
|
|
|
|
// but with pointers to fields where we need to differentiate
|
|
|
|
// between a missing field and a zero value.
|
|
|
|
// Don't use pointers for fields where the zero value is invalid.
|
|
|
|
//
|
|
|
|
// This is necessary because the YAML unmarshaler
|
|
|
|
// will read a zero value for a missing field or an empty file.
|
|
|
|
var state struct {
|
|
|
|
// Version 0 is valid, so we need to use a pointer.
|
|
|
|
Version *int `yaml:"version"`
|
|
|
|
}
|
|
|
|
|
|
|
|
if err := yaml.Unmarshal(metaBody, &state); err != nil {
|
|
|
|
return nil, fmt.Errorf("corrupt store: unmarshal %q: %w", pulumiMetaPath, err)
|
|
|
|
}
|
|
|
|
|
|
|
|
if state.Version == nil {
|
|
|
|
return nil, fmt.Errorf("corrupt store: missing version in %q", pulumiMetaPath)
|
|
|
|
}
|
|
|
|
|
|
|
|
return &pulumiMeta{
|
|
|
|
Version: *state.Version,
|
|
|
|
}, nil
|
|
|
|
}
|
|
|
|
|
|
|
|
// WriteTo writes the metadata to the bucket, overwriting any existing metadata.
|
|
|
|
func (m *pulumiMeta) WriteTo(ctx context.Context, b Bucket) error {
|
2023-03-28 17:31:35 +00:00
|
|
|
if m.Version == 0 {
|
|
|
|
// We don't want to write a metadata file
|
|
|
|
// for legacy layouts.
|
|
|
|
//
|
|
|
|
// This allows for cases where a user has
|
|
|
|
// strict permission controls on their bucket,
|
|
|
|
// and doesn't expect a file outside .pulumi/stacks/.
|
|
|
|
return nil
|
|
|
|
}
|
|
|
|
|
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.
To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.
Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".
This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.
In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.
If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.
Note that this differs slightly
from the initial implementation of this functionality in #12134.
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.
This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.
Extracted from #12437
2023-03-22 19:14:05 +00:00
|
|
|
bs, err := yaml.Marshal(m)
|
|
|
|
contract.AssertNoErrorf(err, "Could not marshal filestate.pulumiMeta to YAML")
|
|
|
|
|
|
|
|
if err := b.WriteAll(ctx, pulumiMetaPath, bs, nil); err != nil {
|
|
|
|
return fmt.Errorf("write %q: %w", pulumiMetaPath, err)
|
|
|
|
}
|
|
|
|
return nil
|
|
|
|
}
|