Commit Graph

62 Commits

Author SHA1 Message Date
Fraser Waters 293f9294a7
Validate snapshots from service on load ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

In the filestate code we would always run loaded snapshots through
VerifyIntegrity before using them. The httpstate didn't have any similar
checks. This brings the two backends into alignment, reusing the option
from before that was just for filestate (--disable-integrity-checking).

At some point we should further align these so that httpstate also
validates the snapshots it has written out, like filestate does today.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-04 15:12:56 +00:00
Fraser Waters 2c74dddc91
Switch to use env.Env in filestate ()
Internal refactor to use `env.Env` directly in filestate rather than
mocking `os.Getenv`.
2023-10-18 10:52:54 +00:00
Fraser Waters 156a89e611
Add SelfManaged env vars ()
Tiny fix up to env vars. This wasn't showing in `pulumi env`.
2023-09-25 14:07:46 +00:00
Fraser Waters 51d36e1eb4 Fix auto-opt-in for local project stacks
Fixes https://github.com/pulumi/pulumi/issues/13242.

The checks for if we should auto-opt-in to project mode was stricter
than intended. This now checks for if there's any legacy stacks rather
than if there's any files/folders at all.
2023-06-22 13:36:52 +01:00
Fraser Waters 11cd6965d2 Make filestate.Snapshot lazy like httpstate 2023-05-24 08:24:04 +01:00
Abhinav Gupta 34961a6a29
pkg/filestate: Plumb context
The filestate backend implementation drops context in many places
instead opting to use context.TODO.
context.TODO is a TODO--something intended to be addressed.
context.Backgruond is appropriate for process-scoped operations.

In this case, we can resolve the issues and drop all uses of
context.TODO by plumbing the received contexts appropriately.

For localSnapshotPersister, we add a new `ctx` field
because the interface doesn't yet accept a context argument.
This is tracked in .

Result:

```
% rg context.TODO pkg/backend/filestate
[empty]
```

Resolves 
2023-04-10 15:53:32 -07:00
Abhinav Gupta 82c82c6848
refactor(filestate): Don't use t.Setenv in tests
This is an alternative take on what  was for.
Specifically, the current filestate.New constructor
makes it tedious to inject optional hooks
to control external state like the environment, time, etc.

This introduces a private constructor:

    func newLocalBackend(..., *localBackendOptions) (*localBackendReference, error)

The filestate.New constructor just calls newLocalBackend
with the default options.

The only available option is Getenv: an override for os.Getenv.
We replace all direct uses of os.Getenv with this function reference,
so this allows us to control environment variables in tests
without *actually* changing them with t.Setenv.
That, in turn, allows these tests to run in parallel again.

To further demonstrate the value of doing this,
this change also includes tests for previously untested functionality:
the PULUMI_RETAIN_CHECKPOINTS and PULUMI_SELF_MANAGED_STATE_GZIP
environment variables.

The only remaining uses of t.Setenv are in tests
that cross boundaries to stack.DefaultSecretsProvider.
That dependency is also easy to break with localBackendOptions
in a future change.
2023-03-31 16:29:06 -07:00
Fraser Waters 9d0fba3a7e
filestate: Re-add project support
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.

We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.

To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:

    # .pulumi/meta.yaml
    version: 1

Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.

Notes:

- Stack references produced by filestate will shorten to
  just the stack name if the project name for the stack
  matches the currently selected project.
  This required turning currentProject on localBackend
  into an atomic pointer because otherwise
  SetCurrentProject and localBackendReference.String may race.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-31 13:21:36 -07:00
Abhinav Gupta 041721995b
filestate: Introduce referenceStore to control layout
Adds a referenceStore abstraction to control the layout of the storage,
and a legacyReferenceStore implementation based on the current layout
(that does not support projects).

This allows us to move code to determine file paths of stacks, their
histories, and their backups, all into a single component that we can
swap out for project support.

localBackendReferences keep track of the referenceStore that built them.
The primary reason for this is that when we add support for migrating a
stack state from legacy to project mode, `backend.store` will become
mutable.
For references created before the store for a backend was changed, we
still need to be able to access their original file paths, so we need to
hold onto the original referenceStore.
However, as a side-effect of this,
it's more convenient to acess paths from `ref.Foo()` rather than
`backend.foo(ref)` or `backend.store.Foo(ref)`.

In the future, we may also move stackPath to the store,
since right now the .json/.json.gz logic is duplicated in a couple
places.

Extracted from 
2023-03-17 12:05:43 -07:00
Fraser Waters 5ba5f26d70
filestate/internal: Use stack reference, not name
filestate backend currently operates exclusively with stack names.
All its internal pass around just the stack name, and nothing else.
This makes it a bit difficult to add project support to the backend.

This is a refactor in advance of adding project support,
changing the internals of filestate to pass a stack reference around.
It inspects the reference directly for all its operations.

Note: This contains no behavioral changes.
Name and FullyQualifiedName currently both return just the stack name.
In a future change, once project name is incorporated into the object,
FullyQualifiedName will be able to return `organization/$project/$name`.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-16 14:44:33 -07:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta fff7e0bc4b
pkg/backend: Prefer contract.Assertf over Assert
Incremental step towards 

Migrates uses of contract.{Assert, AssertNoError, Require} in pkg/engine
to `*f` variants so that we're required to provide more error context.

Refs 
2023-02-21 15:13:16 -08:00
Justin Van Patten c3ba7082ca Revert "Support projects in the file state backend"
This reverts commit c902b9499f.
2023-02-02 15:26:26 -08:00
Fraser Waters c902b9499f Support projects in the file state backend
This is a first pass at supporting projects for the file state backend.

Projects must be given explictly, that is unlike the service backend where the stack reference "teststack" would implictly parse as the stack "teststack" in the current project in the current organisation.
"teststack" with the filestate backend will continue to parse as the stack "teststack" not associated with a project, but you can now give the stack reference now as "testproj/teststack" to get a project scoped stack.

This makes no effort to try automatically moving stack files to be associated with projects. Users can use pulumi stack rename to do that if they so wish.

Fixes https://github.com/pulumi/pulumi/issues/2522
2023-01-29 07:39:18 +00:00
Fraser Waters fec5e7e205 Don't marshal/unmarshal deployments for import/export 2023-01-27 05:06:52 +00:00
Fraser Waters 4eef34d6c3 Lift DefaultSecretsProvider out of DeserializeCheckpoint
SerializeCheckpoint takes the `secrets.Manager` to use as a parameter.
DeserializeCheckpoint was hardcoded to always use the same `SecretsProvider`.
2023-01-10 14:07:56 +00:00
Fraser Waters 615ae91477
Add getOrganization to nodejs and python ()
* Add getOrganization to nodejs and python

* Add to CHANGELOG

* lint

* lint

* format

* Fix python test

* Fix nodejs settings

* Add contextproperty

* Backcompat python kwargs
2022-08-31 10:33:29 +01:00
Aaron Friel ef898d5691
Reduce macOS jobs on PRs, replace with partial integration tests ()
* ci: include required deps in test artifact builds

* ci: reduce retention on other build artifacts to match pulumi CLI

* ci: move lang providers acquisition to script/goreleaser

* ci: combine goreleaser configs, move OS-dependency to script

* ci: re-enable Windows and MacOS tests

* [cli] Fix Windows path handling for gzip filestate backend
2022-08-16 18:00:14 -07:00
Aaron Friel dd7733c8fa
Address regression in ()
* Revert "[ci] Name the archive to push to buckets"

This reverts commit 73f23fbc39.

* Revert "[ci] Fix extra extension in goreleaser templates"

This reverts commit 03888a33fa.

* Revert 

This reverts commit f6ee2fd20c.
This reverts commit 33a84df3cf.
This reverts commit 01c1719c1c.
This reverts commit e7b96115be.
This reverts commit 60fb289408.
This reverts commit 7a81b1435f.
This reverts commit b1f2e41b56.

* chore: changelog for 3.37.1
2022-07-27 13:05:49 -07:00
Aaron Friel bcf6435149 [cli] Fix Windows path handling for gzip filestate backend 2022-07-19 15:46:15 -07:00
Anton Tayanovskyy 96a3783fb2
Fix orphan bulk decrypt traces ()
* Propagate context through crypter to correctly parent service crypter tracing spans

* Remove one of the context.TODO that is needed

* Fix new_test.go

* Fix pulumi_test.go

* Fix resources_test.go

* Fix pkg/resource/stack tests

* Add CHANGELOG_PENDING

* Fix value_test.go

* Fix stack_test.go

* Feedback
2022-07-18 09:36:31 -04:00
Ian Wahbe 6dc3574d79
Fix master: test local state gzip ()
* Select correct variable

* Simplify conditional

* CL
2022-06-07 22:17:39 +02:00
Kyle Dixler 1c3974d794
improved filestateBackend search prefix ()
* improved prefix

* updated CHANGELOG_PENDING.md

* Fix prefix search in stackPath

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2022-06-07 08:34:44 +01:00
Fraser Waters 9d0b09c229
Fix a panic in renameHistory when encountering unexpected files ()
* Fix a panic in renameHistory when encountering unexpected files

Fixes https://github.com/pulumi/pulumi/issues/9670

* Add to CHANGELOG

* lint
2022-05-25 09:10:36 +01:00
Ian Wahbe 9bb8efe597
Add gzip flag to filestate backend ()
* add gzip compression in filestate backend

New versions of pulumi can now read json files wrapped in .gz
Compression is disabled by default, can be enabled via:
  PULUMI_SELF_MANAGED_STATE_GZIP=true

* functionnal test for gzip filestate

* update CHANGELOG_PENDING.md

* Move compression to encoding

* Update changelog

* Clean up tests

* Adjust IsCompressed check

* Don't leave out of data state files

Co-authored-by: Arthur Woimbée <arthur@extrality.ai>
Co-authored-by: Arthur Woimbée <arthur.woimbee@gmail.com>
2022-05-23 12:13:21 -07:00
Fraser Waters c8a36883d4
Fix backupTarget ()
* Fix backupTarget

src and dst arguments to bucket.Copy were the wrong way round.

* Add test

* Add to CHANGELOG
2022-04-26 19:44:00 +01:00
Fraser Waters 86f3f712aa
Readd "Make StackReference.Name a tokens.Name ()" ()
* Readd "Make StackReference.Name a tokens.Name ()"

This reverts commit f0aa4df149.

This also removes the AsName asserting casts for stack names. We do want
to add them in at some point to be sure that bad names don't slip in
somehow but they don't need adding with this.

* Update sdk/go/common/util/fsutil/qname.go

Co-authored-by: Ian Wahbe <ian@wahbe.com>

Co-authored-by: Ian Wahbe <ian@wahbe.com>
2022-03-17 21:37:11 +00:00
Ian Wahbe f0aa4df149
Revert "Make StackReference.Name a tokens.Name ()" ()
This reverts commit a72df9aefa.
2022-03-09 13:48:52 -08:00
Fraser Waters ca3b9a5727
Ensure filestate stack.json file are atomically replaced ()
* Inline renameObject

* Add keepOriginal to backupTarget

* Add to CHANGELOG

* s/%w/%s/
2022-03-08 11:21:39 +00:00
Fraser Waters a72df9aefa
Make StackReference.Name a tokens.Name ()
* Make StackReference.Name a tokens.Name

* Don't panic in ParseStackReference
2022-03-07 20:59:44 +00:00
Praneet Loke 2c25c3fbd0
[backend/filestate] Don't unwrap go-cloud errors ()
* Don't unwrap go-cloud errors

* Remove unused func

* Add changelog entry

Co-authored-by: Ian Wahbe <ian@wahbe.com>
2021-11-19 12:21:37 -08:00
Ian Wahbe 272c4643b2
Update error handling ()
This is the result of a change applied via `go-rewrap-errors`.
2021-11-12 18:37:17 -08:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Evan Boyle 8e58f5d682
Add pagination to `pulumi stack history` ()
replaces the unreleased `--limit` flag with `--page` and `--page-size` to support full pagination
2021-02-09 16:20:01 -08:00
Evan Boyle eefc104c2b
Add `--limit` flag to `pulumi stack history` and consume from automation api ()
Adds a `--limit` flag to `pulumi stack history. This allows limiting to the last few entries rather than fetching the entirety of a stack's update history (which can be quite slow for stacks with lots of updates). Example: `pulumi stack history --limit 1` fetches the last history entry only. 

`stack.up` and related operations in the Automation API have been updated to consume this change, drastically reducing overhead.
2021-02-08 10:49:57 -08:00
Paul Stack 48f906e026
Allow pulumi stack export to decrypt secrets () 2020-05-11 19:16:30 +01:00
Lee Briggs c07e95e134
Build features/ 2020-04-27 17:43:37 -07:00
Lee Briggs 6a7adc8a16
Set delay and timeout for retries 2020-04-27 17:14:07 -07:00
Lee Briggs 0086edb2d9
Retry bucket writes on error
When writing the snapshot to the filestate bucket, we can retry in the
event of an error, which helps users who are experiencing issues around
write rates to GCS
2020-04-24 16:41:56 -07:00
Lee Briggs e74f797dbd
Change the wording of some IO errors
This probably seems like a trivial change, but while debugging  it
was apparent that an error could have come from 3 different places. This
rewords an error message to make it slightly clearer what the error is.
2020-04-22 16:43:44 -07:00
CyrusNajmabadi 66bd3f4aa8
Breaking changes due to Feature 2.0 work
* Make `async:true` the default for `invoke` calls ()

* Switch away from native grpc impl. ()

* Remove usage of the 'deasync' library from @pulumi/pulumi. ()

* Only retry as long as we get unavailable back.  Anything else continues. ()

* Handle all errors for now. ()


* Do not assume --yes was present when using pulumi in non-interactive mode ()

* Upgrade all paths for sdk and pkg to v2

* Backport C# invoke classes and other recent gen changes ()

Adjust C# generation

* Replace IDeployment with a sealed class ()

Replace IDeployment with a sealed class

* .NET: default to args subtype rather than Args.Empty ()

* Adding system namespace for Dotnet code gen

This is required for using Obsolute attributes for deprecations

```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```

* Fix the nullability of config type properties in C# codegen ()
2020-04-14 09:30:25 +01:00
evanboyle d3f5bbce48 go fmt 2020-03-18 17:27:02 -07:00
evanboyle 30e1a5917e move pkg/util/fsutil -> sdk/go/common/util/fsutil 2020-03-18 15:48:38 -07:00
evanboyle c1d3a8524b move pkg/util/cmdutil -> sdk/go/common/util/cmdutil 2020-03-18 15:39:00 -07:00
evanboyle c3f6ae2451 move pkg/util/logging -> sdk/go/common/util/logging 2020-03-18 15:34:58 -07:00
evanboyle cdfb8608c9 move pkg/encoding -> sdk/go/common/encoding 2020-03-18 15:28:19 -07:00
evanboyle f754b486b8 move pkg/resource/config -> sdk/go/common/resource/config 2020-03-18 15:03:37 -07:00
evanboyle 67cb405c93 move pkg/apitype -> sdk/common/apitype 2020-03-18 15:00:30 -07:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00
evanboyle fccf301d14 move pkg/util/contract -> sdk/go/common/util/contract 2020-03-18 14:40:07 -07:00