Commit Graph

245 Commits

Author SHA1 Message Date
Fraser Waters afb287d2fb
Rename filestate to DIY ()
This goes through the codebase to try and be consistent about names for
the diy/filestate/local/selfmanaged backend. Every reference to this
backend should now use the terms "DIY". There are a couple of places
that still say "local DIY backend" this is referring to a DIY backend
using the local filesystem (i.e. `pulumi login --local`).
2024-01-30 15:53:10 +00:00
Fraser Waters 7ba0541e1c
Rename self-managed to diy-backend ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

We're moving away from referring to filestate as "self managed"
backends, preferring to refer to this as "DIY" backends going forward.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-30 09:00:15 +00:00
Komal 87d1090d9f
Replace a type used by the service ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

A type that is used by the Pulumi Cloud code was removed in
[](https://github.com/pulumi/pulumi/pull/15028/files#diff-52b25675ac0d98614d062fc1d1c9e60759f905cf7c60a9339402a62c23cc2c55L32-L33)
- this PR replaces it.

Fixes # (issue)

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-23 19:15:29 +00:00
Thomas Gummerer baecc85eaf
turn on the golangci-lint exhaustive linter ()
Turn on the golangci-lint exhaustive linter.  This is the first step
towards catching more missing cases during development rather than
in tests, or in production.

This might be best reviewed commit-by-commit, as the first commit turns
on the linter with the `default-signifies-exhaustive: true` option set,
which requires a lot less changes in the current codebase.

I think it's probably worth doing the second commit as well, as that
will get us the real benefits, even though we end up with a little bit
more churn. However it means all the `switch` statements are covered,
which isn't the case after the first commit, since we do have a lot of
`default` statements that just call `assert.Fail`.
 
Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-17 16:50:41 +00:00
Fraser Waters 01dd48ed4a
Add Base64SecretsProvider for testing ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
To support plugin based secret providers like (e.g. AGE
https://github.com/pulumi/pulumi/issues/11493) the default secrets
provider will need to become more complicated, requiring access to a
plugin host at least.

As such for tests it will be simpler to have a basic secrets provider
that only supports base64 and can be created without any setup.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-05 08:32:56 +00:00
Fraser Waters 0c11bd2fd6
Rewrite filestate.RenameStack logic in terms of checkpoints ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

The RenameStack function in filestate was one of the few places outside
of "cmd" still using `stack.DefaultSecretsProvider`, which it was only
using because the stack rename function was written in terms of
deserialised snapshots instead of serialised deployments.

This change rewrites the RenameStack function to work on the raw
deployment data rather than deserialised snapshot. This allows us to
remove the use of `stack.DefaultSecretsProvider` from filestate.

This would also allow the service to update their use of RenameStack to
work on the raw deployment as well, saving them the use of their
NoopSecretsProvider.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-22 12:28:41 +00:00
Fraser Waters 16d9f4c167
Enable perfsprint linter ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945

This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.

Most of these edits were made by running `perfsprint --fix`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-12 12:19:42 +00:00
Fraser Waters f36ce248b2
Add --import-file to pulumi preview ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/12768.

This will generate an import file for every resource the preview wants
to Create.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-05 08:32:40 +00:00
Fraser Waters 293f9294a7
Validate snapshots from service on load ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

In the filestate code we would always run loaded snapshots through
VerifyIntegrity before using them. The httpstate didn't have any similar
checks. This brings the two backends into alignment, reusing the option
from before that was just for filestate (--disable-integrity-checking).

At some point we should further align these so that httpstate also
validates the snapshots it has written out, like filestate does today.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-04 15:12:56 +00:00
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Fraser Waters d771acf707
Add tokens.StackName ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new type `tokens.StackName` which is a relatively strongly
typed container for a stack name. The only weakly typed aspect of it is
Go will always allow the "zero" value to be created for a struct, which
for a stack name is the empty string which is invalid. To prevent
introducing unexpected empty strings when working with stack names the
`String()` method will panic for zero initialized stack names.
 
Apart from the zero value, all other instances of `StackName` are via
`ParseStackName` which returns a descriptive error if the string is not
valid.

This PR only updates "pkg/" to use this type. There are a number of
places in "sdk/" which could do with this type as well, but there's no
harm in doing a staggered roll out, and some parts of "sdk/" are user
facing and will probably have to stay on the current `tokens.Name` and
`tokens.QName` types.

There are two places in the system where we panic on invalid stack
names, both in the http backend. This _should_ be fine as we've had long
standing validation that stacks created in the service are valid stack
names.

Just in case people have managed to introduce invalid stack names, there
is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn
off the validation _and_ panicing for stack names. Users can use that to
temporarily disable the validation and continue working, but it should
only be seen as a temporary measure. If they have invalid names they
should rename them, or if they think they should be valid raise an issue
with us to change the validation code.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-15 07:44:54 +00:00
Justin Van Patten c88667bb4d
Revert "Move filestate locking to apply time ()" ()
This reverts commit 7f1b75d236.
2023-11-03 01:25:20 +00:00
Fraser Waters 7f1b75d236
Move filestate locking to apply time ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This aligns the filestate behavior with httpstate in that we won't take
a lock during the preview phase of an update. Instead we take the lock
at the point we start doing a non-preview apply

We've tried this once before
(https://github.com/pulumi/pulumi/pull/8806) but got a couple of user
reports about locking not working soon after it released, and so
reverted it just in case. Further investigation into those user issues
suggested they were due to user fault, not this change.

There's one comment from myself on Slack from around that time saying
I'm not sure this change is correct, but even with that nudge and
looking at it with fresh eyes I think it is. But worth having a hard
think about this before hitting approve.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-01 09:12:38 +00:00
Thomas Gummerer f278fa9696
stack ls: remove misleading updateInProgress output for file backend ()
File backends currently do not support indicating whether an operation
is currently in progress or not. In the `stack ls --json` output we
always end up with a "updateInProgress = false" field, even when an
update is in progress. This is misleading.

Disable showing this erroneous and misleading output.

Fixes 

Based on Luke's comments in that issue, I believe we should merge this
as bug fix for the issue, and then open a separate feature request
tracking the implementation of tracking whether an update is in progress
in the file backend.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-27 07:33:07 +00:00
Fraser Waters 2c74dddc91
Switch to use env.Env in filestate ()
Internal refactor to use `env.Env` directly in filestate rather than
mocking `os.Getenv`.
2023-10-18 10:52:54 +00:00
Fraser Waters cf5b4a2790
Use `assert.NoError` rather than `assert.Nil` ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Likewise `require.NoError` instead of `require.Nil`, and `assert.Error`
rather than `assert.NotNil`.

The error variants of these functions print the errors nicer for test
failures using `Error()` rather than `GoString()`.

For bail errors this is _much_ better than the `result.Result` days
where we now get errors like:
```
Error:      	Received unexpected error:
            	BAIL: inner error
```
instead of:
```
Error:      	Expected nil, but got: &simpleResult{}
```

Also print the bail error in `TestPlan.Run` so we can see the
description of it.
2023-10-13 09:46:07 +00:00
Fraser Waters 10c1e1baaa
Cleanup result.Result in pkg/engine ()
A big change for result.Result cleanup. This removes all references to
the Result type from pkg/engine. This was mostly just search replace.
Points to note, we still map to `result.Result` in pkg/backend (that
will be the next big result change to deal with), and a little bit of
fiddlyness with multiple error values in test_plan.go `runWithContext`.
2023-10-11 14:44:09 +00:00
Fraser Waters 156a89e611
Add SelfManaged env vars ()
Tiny fix up to env vars. This wasn't showing in `pulumi env`.
2023-09-25 14:07:46 +00:00
Fraser Waters 61dcd6f6ca
Add token info to whoami ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi-cloud-requests/issues/236

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-23 12:46:11 +00:00
Fraser Waters cea3d70f2b
Write out snapshot errors to the user error stream ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Currently if the engine encounters an error writing out snapshots it
only logs that to the V11 log.

We'd like to get to the point where the engine doesn't write out invalid
snapshots in the first place, but doing this in small steps to reduce
blast radius of changes to a core engine loop.

This is the first step, we'll log the error to the user error stream so
it's more noticeable than just being in the developer logs at -v11.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-22 20:20:35 +00:00
Fraser Waters 79271cf543
Replace result.Result in Query with error ()
Continued cleanup of result.Bail with error. This cleans up the codepath
using Query.
2023-09-20 15:43:46 +00:00
Fraser Waters 65f3493126
Move SecretsManager from SnapshotPersister to SnapshotManager ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

It should never have really been the role of the persister to decide
what SecretsManager to use, and they were in practice always being setup
with a SnapshotManager anyway so not a lot of code to move the secrets
manager around. This also gets the httpstate and filestate backends in
alignment over which SecretsManager variable they're going to use.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-19 18:31:56 +00:00
Fraser Waters 6e6735ed29
Remove Logout logic from Backend interface ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Backends themselves don't actually need any logout logic. Logging out is
mearly a function of deleting the account from the credentials file and
is the same for filestate, httpstate and probably any other state we
introduce.

This also simplifies the code "logout --all" which used to try and
lookup the project, make up a dummy httpstate object, and then call
LogoutAll to call workspace.DeleteAllAccounts. Now it goes straight to
that last call.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-18 17:21:32 +00:00
Fraser Waters 4ebf61f896
Move sdk/go/common/display to /pkg/display ()
Similar to how https://github.com/pulumi/pulumi/pull/13953 moves some
code from sdk/go/common to /pkg. This display code is only used in /pkg,
another simple reduction of what's in sdk/go/common.
2023-09-18 11:01:28 +00:00
Fraser Waters 501b80911e
Add tests for stackChangeSecretsProviderCmd ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This command is currently untested. This adds a couple of basic tests to
ensure that state and config secrets are changed.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-08-31 08:56:38 +00:00
Kyle Dixler 71ec5b7dfa
`pulumi new -s 'org/project/stack'` checks the proper organization for project existence rather than backend.currentUser()'s org. 2023-07-03 09:28:52 -07:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
Fraser Waters 51d36e1eb4 Fix auto-opt-in for local project stacks
Fixes https://github.com/pulumi/pulumi/issues/13242.

The checks for if we should auto-opt-in to project mode was stricter
than intended. This now checks for if there's any legacy stacks rather
than if there's any files/folders at all.
2023-06-22 13:36:52 +01:00
Abhinav Gupta 03ff557de4
filestate/upgrade: Support backfilling projects for detached stack
Adds an option to the filestate upgrade operation that, when supplied,
allows the caller to fill project names for stacks
for which we could not guess a project name.

The caller supplies a function with the following signature,
taking a list of stack names and returning a list of project names
in the same order.

    func(stacks []Name) (projects []Name, err error)

A caller like the CLI can use this hook to prompt the user for input,
allowing users to fill in project names for stacks.
2023-06-13 10:08:32 -07:00
Abhinav Gupta adf96f057e
refactor(filestate/upgrade): Separate project extraction and upgrade
The "upgrade a stack" function is comprised of two steps:

- guess project name by inspecting stack state
- actually upgrade the stack

This splits that apart into two separate steps
in preparation of filling in the missing projects
between these two steps.
2023-06-13 10:08:32 -07:00
Abhinav Gupta 5f12e947db
filestate/Upgrade: Accept an UpgradeOptions struct
Adds an optional UpgradeOptions struct to the filestate backend's
Upgrade operation.

This changes no behavior and adds no options.
2023-06-13 10:08:32 -07:00
Abhinav Gupta e7aa7c484f
sdk/go: Add tokens.ValidateProjectName
As pointed out in
https://github.com/pulumi/pulumi/pull/13078#discussion_r1227854722,
we repreatedly reimplement the project name validation logic.

This moves it to a central place: `tokens.ValidateProjectName`,
and re-uses it in existing places where copies were found.
2023-06-13 10:08:31 -07:00
Abhinav Gupta d971a6d10f
filestate: Use worker pool for upgrades
Simplify localBackend.Upgrade logic by making use
of the previously added re-usable worker pool type.
2023-06-12 09:23:24 -07:00
Abhinav Gupta d05dc76f69
filestate: Add re-usable worker pool
The filestate backend's Upgrade method currently
does some manual goroutine management
to ensure that it uses a fixed number of goroutines
as it attempts to upgrade all stacks in parallel.

This was fine while the upgrade step was just one phase:
Roughly:

    for _, stack := range stacks {
        go upgrade(stack)
    }
    // Using a pool instead of a new goroutine for each upgrade.

However, this will not suffice with upcoming changes to address 
because the upgrade process will now have multiple phases:
gather information, fill missing information with a prompt, upgrade.
Only the first and last phases of this are parallelizable.

Attempting to do that using the existing pattern in Upgrade
will lead to code that is quite difficult to read,
so this change introduces a simple shared worker pool abstraction.
It operates like a mix of `sync.WaitGroup` and [errgroup][1].
Namely:

- supports multiple `Wait` and `Enqueue` phases (like WaitGroup)
- supports functions that return errors (like errgroup)

[1]: https://pkg.go.dev/golang.org/x/sync@v0.1.0/errgroup

This makes it very easy to adapt code that looks like the following:

    wg := &errgroup.Group{}
    for _, x := range xs {
        x := x
        wg.Go(func() error { return f(x) })
    }
    if err := wg.Wait(); err != nil {
        return err
    }

    wg = &errgroup.Group{}
    for _, y := range ys {
        y := y
        wg.Go(func() error { return f(y) })
    }
    if err := wg.Wait(); err != nil {
        return err
    }

Into the following:

    pool := newWorkerPool(..)
    defer pool.Close()

    for _, x := range xs {
        x := x
        pool.Enqueue(func() error { return f(x) })
    }
    if err := pool.Wait(); err != nil {
        return err
    }

    for _, y := range ys {
        y := y
        pool.Enqueue(func() error { return f(y) })
    }
    if err := pool.Wait(); err != nil {
        return err
    }

The workerPool-based version looks similar,
but it'll spawn a fixed number of goroutines once at the start
and re-use them for all tasks.
2023-06-12 09:20:26 -07:00
bors[bot] e57d82fd5b
Merge
12856: Support 'pulumi:tags' config to set stack tags r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Support a "pulumi:tags" config property to set the stacks tags on operations (e.g. up, refresh, etc).

Fixes https://github.com/pulumi/pulumi/issues/5004

Note that I've gone with a slightly different approach to . The suggestion in that ticket was to add a new section "tags" to the yaml, but given that we probably want to allow setting tags per-project and per-stack with some way to merge them I figured it better to reuse normal config and benefit from https://github.com/pulumi/pulumi/issues/11547 when that's done.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-06-01 17:10:29 +00:00
Fraser Waters 92d4411327 Fix project renames for filestate 2023-05-30 13:54:28 +01:00
Fraser Waters 89a7922b08 Support 'pulumi:tags' config to set stack tags
Fixes https://github.com/pulumi/pulumi/issues/5004
2023-05-24 09:02:22 +01:00
Fraser Waters 11cd6965d2 Make filestate.Snapshot lazy like httpstate 2023-05-24 08:24:04 +01:00
Fraser Waters d9d7666322 Fix filestates project filter for ListStacks
Fixes https://github.com/pulumi/pulumi/issues/12974
2023-05-22 14:19:14 +01:00
Fraser Waters dbf8151a5c Don't pass a second SecretsProvider to backend.Watch
`backend.Watch` was taking a `SecretsProvider` parameter, but it already
took an `UpdateOperation` parameter which already has a `SecretProvider`
on it. So now we just use that.
2023-05-08 20:38:40 +01:00
bors[bot] f69c473d68
Merge
12798: Update DoesProjectExist to not list all projects r=abhinav a=neverbeenthisweeb

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Add `projectReferenceStore.ProjectExists`.

`projectReferenceStore.ProjectExists` enables us to search for a specific project in `localBackend.DoesProjectExist`.

**Tasks**

- [X] Add `projectReferenceStore.ProjectExists`
- [X] Replace `projectReferenceStore.ListProjects` with `projectReferenceStore.ProjectExists` in `localBackend.DoesProjectExist`

Fixes https://github.com/pulumi/pulumi/issues/12547

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [X] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: neverbeenthisweeb <rf.ramadhana@gmail.com>
2023-05-03 00:19:27 +00:00
Fraser Waters 1376a13d13 Fix destroy without project file
Fixes https://github.com/pulumi/pulumi/issues/12714.

The empty project created for destroy operations when no Pulumi.yaml is
found still needs to have it's name filled in so that project name
consistency checks work.

Our tests didn't pick this up because for filestate we were still
searching from the working directory for consistency checks rather than
checking the backends current project. I have _not_ changed that in this
changeset as previosly that triggered total breakage of the filestate
backend when not in project mode (see
https://github.com/pulumi/pulumi/issues/12760).
2023-05-02 10:36:16 +01:00
neverbeenthisweeb b1672afee6 Remove path pgk named import 2023-05-02 07:33:49 +07:00
neverbeenthisweeb 374bb3b57a Rename path pkg import 2023-05-02 06:32:20 +07:00
neverbeenthisweeb 48756f68b1 Update ProjectExists
- Add contract
- Replace filepath.Join with path.Join
- Update returned error
- Update unit test
2023-05-02 06:29:55 +07:00
neverbeenthisweeb 6d62466115 Update DoesProjectExist
Replace ListProject with Project Exists.
2023-04-30 13:00:58 +07:00
neverbeenthisweeb 2a9dd000dc Add ProjectExists 2023-04-29 22:52:04 +07:00
Justin Van Patten 6219b6d833 Revert "Fix destroy without project file"
This reverts commit 5eb04f18ad.
2023-04-26 18:17:08 -07:00
Fraser Waters 5eb04f18ad Fix destroy without project file
Fixes https://github.com/pulumi/pulumi/issues/12714

The empty project created for destroy operations when no Pulumi.yaml is
found still needs to have it's name filled in so that project name
consistency checks work.

Our tests didn't pick this up because for filestate we were still
searching from the working directory for consistency checks rather than
checking the backends current project. I've also fixed that up as part
of this change.
2023-04-24 16:14:56 +01:00
Abhinav Gupta 34961a6a29
pkg/filestate: Plumb context
The filestate backend implementation drops context in many places
instead opting to use context.TODO.
context.TODO is a TODO--something intended to be addressed.
context.Backgruond is appropriate for process-scoped operations.

In this case, we can resolve the issues and drop all uses of
context.TODO by plumbing the received contexts appropriately.

For localSnapshotPersister, we add a new `ctx` field
because the interface doesn't yet accept a context argument.
This is tracked in .

Result:

```
% rg context.TODO pkg/backend/filestate
[empty]
```

Resolves 
2023-04-10 15:53:32 -07:00