Commit Graph

195 Commits

Author SHA1 Message Date
Fraser Waters afb287d2fb
Rename filestate to DIY ()
This goes through the codebase to try and be consistent about names for
the diy/filestate/local/selfmanaged backend. Every reference to this
backend should now use the terms "DIY". There are a couple of places
that still say "local DIY backend" this is referring to a DIY backend
using the local filesystem (i.e. `pulumi login --local`).
2024-01-30 15:53:10 +00:00
Fraser Waters 7ba0541e1c
Rename self-managed to diy-backend ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

We're moving away from referring to filestate as "self managed"
backends, preferring to refer to this as "DIY" backends going forward.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-30 09:00:15 +00:00
Komal 87d1090d9f
Replace a type used by the service ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

A type that is used by the Pulumi Cloud code was removed in
[](https://github.com/pulumi/pulumi/pull/15028/files#diff-52b25675ac0d98614d062fc1d1c9e60759f905cf7c60a9339402a62c23cc2c55L32-L33)
- this PR replaces it.

Fixes # (issue)

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-23 19:15:29 +00:00
Thomas Gummerer baecc85eaf
turn on the golangci-lint exhaustive linter ()
Turn on the golangci-lint exhaustive linter.  This is the first step
towards catching more missing cases during development rather than
in tests, or in production.

This might be best reviewed commit-by-commit, as the first commit turns
on the linter with the `default-signifies-exhaustive: true` option set,
which requires a lot less changes in the current codebase.

I think it's probably worth doing the second commit as well, as that
will get us the real benefits, even though we end up with a little bit
more churn. However it means all the `switch` statements are covered,
which isn't the case after the first commit, since we do have a lot of
`default` statements that just call `assert.Fail`.
 
Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-17 16:50:41 +00:00
Fraser Waters 0c11bd2fd6
Rewrite filestate.RenameStack logic in terms of checkpoints ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

The RenameStack function in filestate was one of the few places outside
of "cmd" still using `stack.DefaultSecretsProvider`, which it was only
using because the stack rename function was written in terms of
deserialised snapshots instead of serialised deployments.

This change rewrites the RenameStack function to work on the raw
deployment data rather than deserialised snapshot. This allows us to
remove the use of `stack.DefaultSecretsProvider` from filestate.

This would also allow the service to update their use of RenameStack to
work on the raw deployment as well, saving them the use of their
NoopSecretsProvider.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-22 12:28:41 +00:00
Fraser Waters 16d9f4c167
Enable perfsprint linter ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945

This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.

Most of these edits were made by running `perfsprint --fix`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-12 12:19:42 +00:00
Fraser Waters f36ce248b2
Add --import-file to pulumi preview ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/12768.

This will generate an import file for every resource the preview wants
to Create.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-05 08:32:40 +00:00
Fraser Waters d771acf707
Add tokens.StackName ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new type `tokens.StackName` which is a relatively strongly
typed container for a stack name. The only weakly typed aspect of it is
Go will always allow the "zero" value to be created for a struct, which
for a stack name is the empty string which is invalid. To prevent
introducing unexpected empty strings when working with stack names the
`String()` method will panic for zero initialized stack names.
 
Apart from the zero value, all other instances of `StackName` are via
`ParseStackName` which returns a descriptive error if the string is not
valid.

This PR only updates "pkg/" to use this type. There are a number of
places in "sdk/" which could do with this type as well, but there's no
harm in doing a staggered roll out, and some parts of "sdk/" are user
facing and will probably have to stay on the current `tokens.Name` and
`tokens.QName` types.

There are two places in the system where we panic on invalid stack
names, both in the http backend. This _should_ be fine as we've had long
standing validation that stacks created in the service are valid stack
names.

Just in case people have managed to introduce invalid stack names, there
is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn
off the validation _and_ panicing for stack names. Users can use that to
temporarily disable the validation and continue working, but it should
only be seen as a temporary measure. If they have invalid names they
should rename them, or if they think they should be valid raise an issue
with us to change the validation code.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-15 07:44:54 +00:00
Justin Van Patten c88667bb4d
Revert "Move filestate locking to apply time ()" ()
This reverts commit 7f1b75d236.
2023-11-03 01:25:20 +00:00
Fraser Waters 7f1b75d236
Move filestate locking to apply time ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This aligns the filestate behavior with httpstate in that we won't take
a lock during the preview phase of an update. Instead we take the lock
at the point we start doing a non-preview apply

We've tried this once before
(https://github.com/pulumi/pulumi/pull/8806) but got a couple of user
reports about locking not working soon after it released, and so
reverted it just in case. Further investigation into those user issues
suggested they were due to user fault, not this change.

There's one comment from myself on Slack from around that time saying
I'm not sure this change is correct, but even with that nudge and
looking at it with fresh eyes I think it is. But worth having a hard
think about this before hitting approve.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-01 09:12:38 +00:00
Thomas Gummerer f278fa9696
stack ls: remove misleading updateInProgress output for file backend ()
File backends currently do not support indicating whether an operation
is currently in progress or not. In the `stack ls --json` output we
always end up with a "updateInProgress = false" field, even when an
update is in progress. This is misleading.

Disable showing this erroneous and misleading output.

Fixes 

Based on Luke's comments in that issue, I believe we should merge this
as bug fix for the issue, and then open a separate feature request
tracking the implementation of tracking whether an update is in progress
in the file backend.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-27 07:33:07 +00:00
Fraser Waters 2c74dddc91
Switch to use env.Env in filestate ()
Internal refactor to use `env.Env` directly in filestate rather than
mocking `os.Getenv`.
2023-10-18 10:52:54 +00:00
Fraser Waters 10c1e1baaa
Cleanup result.Result in pkg/engine ()
A big change for result.Result cleanup. This removes all references to
the Result type from pkg/engine. This was mostly just search replace.
Points to note, we still map to `result.Result` in pkg/backend (that
will be the next big result change to deal with), and a little bit of
fiddlyness with multiple error values in test_plan.go `runWithContext`.
2023-10-11 14:44:09 +00:00
Fraser Waters 156a89e611
Add SelfManaged env vars ()
Tiny fix up to env vars. This wasn't showing in `pulumi env`.
2023-09-25 14:07:46 +00:00
Fraser Waters 61dcd6f6ca
Add token info to whoami ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi-cloud-requests/issues/236

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-23 12:46:11 +00:00
Fraser Waters cea3d70f2b
Write out snapshot errors to the user error stream ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Currently if the engine encounters an error writing out snapshots it
only logs that to the V11 log.

We'd like to get to the point where the engine doesn't write out invalid
snapshots in the first place, but doing this in small steps to reduce
blast radius of changes to a core engine loop.

This is the first step, we'll log the error to the user error stream so
it's more noticeable than just being in the developer logs at -v11.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-22 20:20:35 +00:00
Fraser Waters 79271cf543
Replace result.Result in Query with error ()
Continued cleanup of result.Bail with error. This cleans up the codepath
using Query.
2023-09-20 15:43:46 +00:00
Fraser Waters 65f3493126
Move SecretsManager from SnapshotPersister to SnapshotManager ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

It should never have really been the role of the persister to decide
what SecretsManager to use, and they were in practice always being setup
with a SnapshotManager anyway so not a lot of code to move the secrets
manager around. This also gets the httpstate and filestate backends in
alignment over which SecretsManager variable they're going to use.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-19 18:31:56 +00:00
Fraser Waters 6e6735ed29
Remove Logout logic from Backend interface ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Backends themselves don't actually need any logout logic. Logging out is
mearly a function of deleting the account from the credentials file and
is the same for filestate, httpstate and probably any other state we
introduce.

This also simplifies the code "logout --all" which used to try and
lookup the project, make up a dummy httpstate object, and then call
LogoutAll to call workspace.DeleteAllAccounts. Now it goes straight to
that last call.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-18 17:21:32 +00:00
Fraser Waters 4ebf61f896
Move sdk/go/common/display to /pkg/display ()
Similar to how https://github.com/pulumi/pulumi/pull/13953 moves some
code from sdk/go/common to /pkg. This display code is only used in /pkg,
another simple reduction of what's in sdk/go/common.
2023-09-18 11:01:28 +00:00
Kyle Dixler 71ec5b7dfa
`pulumi new -s 'org/project/stack'` checks the proper organization for project existence rather than backend.currentUser()'s org. 2023-07-03 09:28:52 -07:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
Abhinav Gupta 03ff557de4
filestate/upgrade: Support backfilling projects for detached stack
Adds an option to the filestate upgrade operation that, when supplied,
allows the caller to fill project names for stacks
for which we could not guess a project name.

The caller supplies a function with the following signature,
taking a list of stack names and returning a list of project names
in the same order.

    func(stacks []Name) (projects []Name, err error)

A caller like the CLI can use this hook to prompt the user for input,
allowing users to fill in project names for stacks.
2023-06-13 10:08:32 -07:00
Abhinav Gupta adf96f057e
refactor(filestate/upgrade): Separate project extraction and upgrade
The "upgrade a stack" function is comprised of two steps:

- guess project name by inspecting stack state
- actually upgrade the stack

This splits that apart into two separate steps
in preparation of filling in the missing projects
between these two steps.
2023-06-13 10:08:32 -07:00
Abhinav Gupta 5f12e947db
filestate/Upgrade: Accept an UpgradeOptions struct
Adds an optional UpgradeOptions struct to the filestate backend's
Upgrade operation.

This changes no behavior and adds no options.
2023-06-13 10:08:32 -07:00
Abhinav Gupta d971a6d10f
filestate: Use worker pool for upgrades
Simplify localBackend.Upgrade logic by making use
of the previously added re-usable worker pool type.
2023-06-12 09:23:24 -07:00
bors[bot] e57d82fd5b
Merge
12856: Support 'pulumi:tags' config to set stack tags r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Support a "pulumi:tags" config property to set the stacks tags on operations (e.g. up, refresh, etc).

Fixes https://github.com/pulumi/pulumi/issues/5004

Note that I've gone with a slightly different approach to . The suggestion in that ticket was to add a new section "tags" to the yaml, but given that we probably want to allow setting tags per-project and per-stack with some way to merge them I figured it better to reuse normal config and benefit from https://github.com/pulumi/pulumi/issues/11547 when that's done.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-06-01 17:10:29 +00:00
Fraser Waters 92d4411327 Fix project renames for filestate 2023-05-30 13:54:28 +01:00
Fraser Waters 89a7922b08 Support 'pulumi:tags' config to set stack tags
Fixes https://github.com/pulumi/pulumi/issues/5004
2023-05-24 09:02:22 +01:00
Fraser Waters 11cd6965d2 Make filestate.Snapshot lazy like httpstate 2023-05-24 08:24:04 +01:00
Fraser Waters d9d7666322 Fix filestates project filter for ListStacks
Fixes https://github.com/pulumi/pulumi/issues/12974
2023-05-22 14:19:14 +01:00
Fraser Waters dbf8151a5c Don't pass a second SecretsProvider to backend.Watch
`backend.Watch` was taking a `SecretsProvider` parameter, but it already
took an `UpdateOperation` parameter which already has a `SecretProvider`
on it. So now we just use that.
2023-05-08 20:38:40 +01:00
bors[bot] f69c473d68
Merge
12798: Update DoesProjectExist to not list all projects r=abhinav a=neverbeenthisweeb

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Add `projectReferenceStore.ProjectExists`.

`projectReferenceStore.ProjectExists` enables us to search for a specific project in `localBackend.DoesProjectExist`.

**Tasks**

- [X] Add `projectReferenceStore.ProjectExists`
- [X] Replace `projectReferenceStore.ListProjects` with `projectReferenceStore.ProjectExists` in `localBackend.DoesProjectExist`

Fixes https://github.com/pulumi/pulumi/issues/12547

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [X] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: neverbeenthisweeb <rf.ramadhana@gmail.com>
2023-05-03 00:19:27 +00:00
Fraser Waters 1376a13d13 Fix destroy without project file
Fixes https://github.com/pulumi/pulumi/issues/12714.

The empty project created for destroy operations when no Pulumi.yaml is
found still needs to have it's name filled in so that project name
consistency checks work.

Our tests didn't pick this up because for filestate we were still
searching from the working directory for consistency checks rather than
checking the backends current project. I have _not_ changed that in this
changeset as previosly that triggered total breakage of the filestate
backend when not in project mode (see
https://github.com/pulumi/pulumi/issues/12760).
2023-05-02 10:36:16 +01:00
neverbeenthisweeb 6d62466115 Update DoesProjectExist
Replace ListProject with Project Exists.
2023-04-30 13:00:58 +07:00
Justin Van Patten 6219b6d833 Revert "Fix destroy without project file"
This reverts commit 5eb04f18ad.
2023-04-26 18:17:08 -07:00
Fraser Waters 5eb04f18ad Fix destroy without project file
Fixes https://github.com/pulumi/pulumi/issues/12714

The empty project created for destroy operations when no Pulumi.yaml is
found still needs to have it's name filled in so that project name
consistency checks work.

Our tests didn't pick this up because for filestate we were still
searching from the working directory for consistency checks rather than
checking the backends current project. I've also fixed that up as part
of this change.
2023-04-24 16:14:56 +01:00
Abhinav Gupta 34961a6a29
pkg/filestate: Plumb context
The filestate backend implementation drops context in many places
instead opting to use context.TODO.
context.TODO is a TODO--something intended to be addressed.
context.Backgruond is appropriate for process-scoped operations.

In this case, we can resolve the issues and drop all uses of
context.TODO by plumbing the received contexts appropriately.

For localSnapshotPersister, we add a new `ctx` field
because the interface doesn't yet accept a context argument.
This is tracked in .

Result:

```
% rg context.TODO pkg/backend/filestate
[empty]
```

Resolves 
2023-04-10 15:53:32 -07:00
Abhinav Gupta 82c82c6848
refactor(filestate): Don't use t.Setenv in tests
This is an alternative take on what  was for.
Specifically, the current filestate.New constructor
makes it tedious to inject optional hooks
to control external state like the environment, time, etc.

This introduces a private constructor:

    func newLocalBackend(..., *localBackendOptions) (*localBackendReference, error)

The filestate.New constructor just calls newLocalBackend
with the default options.

The only available option is Getenv: an override for os.Getenv.
We replace all direct uses of os.Getenv with this function reference,
so this allows us to control environment variables in tests
without *actually* changing them with t.Setenv.
That, in turn, allows these tests to run in parallel again.

To further demonstrate the value of doing this,
this change also includes tests for previously untested functionality:
the PULUMI_RETAIN_CHECKPOINTS and PULUMI_SELF_MANAGED_STATE_GZIP
environment variables.

The only remaining uses of t.Setenv are in tests
that cross boundaries to stack.DefaultSecretsProvider.
That dependency is also easy to break with localBackendOptions
in a future change.
2023-03-31 16:29:06 -07:00
Abhinav Gupta 98ba51d5a4
filestate: Upgrade stacks concurrently
This changes filestate's Upgrade mechanism
to upgrade available stacks concurrently.

Note that we use a fixed number of workers
rather than risk unbounded goroutine growth
because there were too many stacks to upgrade.
2023-03-31 16:29:06 -07:00
Abhinav Gupta ced4b6e374
filestate/Upgrade: Upgrade meta.yaml before the rest
We currently upgrade the meta.yaml *after* upgrading all stack files.
This has a risk of leaving a state store inaccessible.
For example, if someone only has write access to `.pulumi/*/dev`,
and not `.pulumi/*` (), the system would move the stack files,
and fail to write a meta.yaml.
This would leave the stored information completely inaccessible
because lack of meta.yaml would indicate legacy layout,
but the files have all been migrated to the project layout.

This changes filestate.Upgrade to upgrade the meta.yaml first.
If the operation fails, we don't try anything else.
2023-03-31 16:29:06 -07:00
Fraser Waters d0dde0e843
{cli, filestate}: Add 'state upgrade' command
Adds a new `pulumi state upgrade` command
that will upgrade the state for the current state storage
to the latest version supported by that format.
This is a no-op for httpstate,
but for filestate, this will migrate from unversioned storage
to the project-based versioned format.

The upgrade command is backed by a new Upgrade operation
on the filestate Backend.
The upgrade operation operates by effectively renaming stacks
from legacy store to the new project store.

UX notes:

- We'll warn about stacks we failed to upgrade
  but keep going otherwise.
- We'll print a loud warning before upgrading,
  informing users that the state will not be usable
  from older CLIs.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-31 16:29:05 -07:00
Fraser Waters 20e89176d9
filestate: Support opting into legacy layout
With project support added,
filestate will default to project-scoped stacks
for all newly initialized buckets.

This is desirable long-term, but for the initial release,
we'd like users to have an escape hatch to go back to the old layout
until they've had a change to migrate.

This adds the ability for users to opt-out of this feature
by setting an environment variable.

Note that this only applies to new buckets.
You cannot opt out of this feature for a bucket
that is already using project-scoped stacks.
2023-03-31 13:21:37 -07:00
Abhinav Gupta bcc5dfc4aa
filestate: Warn about mixed states
It's possible to end up in a mixed state
where some stacks are project-scoped and others aren't.

For instance, imagine the scenario:

- *User A* sets up a new state with the latest version of the CLI
  and brings up a new stack
- *User B* brings up a stack in that state using an older version;
  the older version doesn't know about the .pulumi/meta.yaml
  so it doesn't explode and writes a legacy stack file

At this point, the state is in mixed state.

To help users detect (and eventually resolve) this issue,
we'll print a warning when the backend is initialized.
It'll find and warn users about any legacy files.
They can opt out of this check by setting an environment variable.

This will help transition users to project-scoped stacks
until we're ready to delete this environment variable
and the check.
2023-03-31 13:21:36 -07:00
Fraser Waters 9d0fba3a7e
filestate: Re-add project support
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.

We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.

To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:

    # .pulumi/meta.yaml
    version: 1

Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.

Notes:

- Stack references produced by filestate will shorten to
  just the stack name if the project name for the stack
  matches the currently selected project.
  This required turning currentProject on localBackend
  into an atomic pointer because otherwise
  SetCurrentProject and localBackendReference.String may race.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-31 13:21:36 -07:00
Abhinav Gupta 29ce91aa8a
pkg/backend: Delete SupportsTeams methods
This deletes the SupportsTeams method added in .
It came up during review that we want to avoid too many new
"SupportsFoo" methods.

Instead, we'll let each backend report whether it supports teams
by returning ErrTeamsNotSupported.

As a result of this change, validateCreateStackOpts cannot error,
so it's been renamed to newCreateStackOptions.

Testing:
There's already a test (added in the  refactor)
that verifies that we report the appropriate error
when the backend doesn't support --teams.
This updates the mock in that test.
2023-03-29 11:07:58 -07:00
Abhinav Gupta 32bd1f9635
filestate: Rename Pulumi.yaml to meta.yaml
In , we added a .pulumi/Pulumi.yaml file to filestate backends
to hold metadata about the storage layout format.
It has been brought up multiple times that this is a confusing name,
so we're renaming it to meta.yaml.

Backwards compatibility:
Although we've already shipped ,
and have started writing Pulumi.yaml files,
we don't currently use it for anything.
Its first use will be in  when we add project-scoped stacks.
Therefore, this change is completely backwards compatible.

We'll end up leaving the extraneous .pulumi/Pulumi.yaml file behind
if the bucket was initialized with Pulumi v3.59.0 or newer,
before this change was released. This is not a high cost.
I don't think we should go back and delete these files;
I'm reluctant for the backend to perform destructive operations
for files that are technically not managed by it.
2023-03-28 12:13:55 -07:00
Abhinav Gupta e4529562cf
filestate: Make currentProject an atomic.Pointer
The currentProject is stored as a plain field on the local backend.
This is fine right now because SetCurrentProject
is not expected to be called concurrently with anything else.

However, in  (project-scoped stacks),
we need to be able to access the current project in a context
where there are no guarantees about SetCurrentProject being called
concurrently with other operations.

This switches the currentProject into an atomic.Pointer
that we can Load and Swap in a thread-safe manner.

Note that this does not protect us from data races caused by
concurrent mutations to the workspace.Project struct's fields,
but that doesn't appear to be a concern;
the code treats that object as a blob of constant values.

Extracted from 
2023-03-22 16:33:57 -07:00
Abhinav Gupta e47c5823c0
filestate: Track a state metadata file (.pulumi/Pulumi.yaml)
We want the filestate backend to support project-scoped stacks,
but we can't make the change as-is because it would break old states
with new CLIs.

To differentiate between old and new states,
we've decided to introduce the concept of state metadata.
This is a file under the path .pulumi/Pulumi.yaml
that tracks metadata necessary for the filestate backend to operate.

Initially, this contains just one field: `version`,
with the initial value of 0 representing non-project or "legacy mode".

This changes the filestate layout to track such a file,
creating it if it doesn't exist with the default value of 0.

In a future change, we'll introduce "version 1",
which adds support for project-scoped stacks.

If we ever need to make breaking changes to the layout,
the version in this file will help the CLI decide
whether it's allowed to handle that state bucket
without corrupting it.

Note that this differs slightly
from the initial implementation of this functionality in .
Particularly, this idempotently ensures that a Pulumi.yaml exists,
allowing `version: 0` to indicate legacy mode,
versus the original implementation that treated absence of the file
in a non-empty bucket as legacy mode.

This drops the bucket.IsAccessible check from filestate.New
because accessibility is now verified
when we try to read the metadata file.

Extracted from 
2023-03-22 12:25:13 -07:00
bors[bot] f7c940e7c2
Merge
11974: Add --teams flag to assign team name to stack. r=RobbieMcKinstry a=RobbieMcKinstry

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

This PR adds a --teams flag to `pulumi stack init` which accepts a string. This flag can be provided multiple times. Each team that is provided is assigned read/write permissions on the stack after it has been initialized.

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Robbie McKinstry <robbie@pulumi.com>
Co-authored-by: Robbie McKinstry <thesnowmancometh@gmail.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-21 19:36:53 +00:00