pulumi/pkg/backend/httpstate/state.go

405 lines
13 KiB
Go
Raw Normal View History

Add tokens.StackName (#14487) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new type `tokens.StackName` which is a relatively strongly typed container for a stack name. The only weakly typed aspect of it is Go will always allow the "zero" value to be created for a struct, which for a stack name is the empty string which is invalid. To prevent introducing unexpected empty strings when working with stack names the `String()` method will panic for zero initialized stack names. Apart from the zero value, all other instances of `StackName` are via `ParseStackName` which returns a descriptive error if the string is not valid. This PR only updates "pkg/" to use this type. There are a number of places in "sdk/" which could do with this type as well, but there's no harm in doing a staggered roll out, and some parts of "sdk/" are user facing and will probably have to stay on the current `tokens.Name` and `tokens.QName` types. There are two places in the system where we panic on invalid stack names, both in the http backend. This _should_ be fine as we've had long standing validation that stacks created in the service are valid stack names. Just in case people have managed to introduce invalid stack names, there is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn off the validation _and_ panicing for stack names. Users can use that to temporarily disable the validation and continue working, but it should only be seen as a temporary measure. If they have invalid names they should rename them, or if they think they should be valid raise an issue with us to change the validation code. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 07:44:54 +00:00
// Copyright 2016-2023, Pulumi Corporation.
2018-05-22 19:43:36 +00:00
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package httpstate
import (
"context"
"fmt"
"sync"
"time"
Exclude internal events from those sent to the service (#14972) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Following up from https://github.com/pulumi/pulumi/commit/54c956af6d64638983eca50875d1dae704a18240 to also exclude the internal events from being sent to Pulumi Cloud. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-21 00:13:04 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/channel"
"github.com/pulumi/pulumi/sdk/v3/go/common/diag"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/logging"
"github.com/pulumi/pulumi/pkg/v3/backend"
"github.com/pulumi/pulumi/pkg/v3/backend/display"
"github.com/pulumi/pulumi/pkg/v3/backend/httpstate/client"
"github.com/pulumi/pulumi/pkg/v3/engine"
"github.com/pulumi/pulumi/pkg/v3/resource/deploy"
"github.com/pulumi/pulumi/pkg/v3/resource/stack"
"github.com/pulumi/pulumi/pkg/v3/secrets"
"github.com/pulumi/pulumi/sdk/v3/go/common/apitype"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/config"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
)
type cloudQuery struct {
root string
proj *workspace.Project
}
func (q *cloudQuery) GetRoot() string {
return q.root
}
func (q *cloudQuery) GetProject() *workspace.Project {
return q.proj
}
// cloudUpdate is an implementation of engine.Update backed by remote state and a local program.
type cloudUpdate struct {
context context.Context
backend *cloudBackend
update client.UpdateIdentifier
tokenSource *tokenSource
root string
proj *workspace.Project
target *deploy.Target
}
func (u *cloudUpdate) GetRoot() string {
return u.root
}
func (u *cloudUpdate) GetProject() *workspace.Project {
return u.proj
}
func (u *cloudUpdate) GetTarget() *deploy.Target {
return u.target
}
func (u *cloudUpdate) Complete(status apitype.UpdateStatus) error {
defer u.tokenSource.Close()
return u.backend.client.CompleteUpdate(u.context, u.update, status, u.tokenSource)
}
// recordEngineEvents will record the events with the Pulumi Service, enabling things like viewing
// the update logs or drilling into the timeline of an update.
func (u *cloudUpdate) recordEngineEvents(startingSeqNumber int, events []engine.Event) error {
contract.Assertf(u.tokenSource != nil, "cloud update requires a token source")
var apiEvents apitype.EngineEventBatch
for idx, event := range events {
apiEvent, convErr := display.ConvertEngineEvent(event, false /* showSecrets */)
if convErr != nil {
return fmt.Errorf("converting engine event: %w", convErr)
}
// Each event within an update must have a unique sequence number. Any request to
// emit an update with the same sequence number will fail. (Read: the caller needs
// to be accurate about this.)
apiEvent.Sequence = idx + startingSeqNumber
apiEvent.Timestamp = int(time.Now().Unix())
apiEvents.Events = append(apiEvents.Events, apiEvent)
}
return u.backend.client.RecordEngineEvents(u.context, u.update, apiEvents, u.tokenSource)
}
// RecordAndDisplayEvents inspects engine events from the given channel, and prints them to the CLI as well as
// posting them to the Pulumi service.
func (u *cloudUpdate) RecordAndDisplayEvents(
label string, action apitype.UpdateKind, stackRef backend.StackReference, op backend.UpdateOperation,
permalink string, events <-chan engine.Event, done chan<- bool, opts display.Options, isPreview bool,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
) {
// We take the channel of engine events and pass them to separate components that will display
// them to the console or persist them on the Pulumi Service. Both should terminate as soon as
// they see a CancelEvent, and when finished, close the "done" channel.
displayEvents := make(chan engine.Event) // Note: unbuffered, but we assume it won't matter in practice.
displayEventsDone := make(chan bool)
persistEvents := make(chan engine.Event, 100)
persistEventsDone := make(chan bool)
// We close our own done channel when both of the dependent components have finished.
defer func() {
<-displayEventsDone
<-persistEventsDone
close(done)
}()
// Start the Go-routines for displaying and persisting events.
go display.ShowEvents(
label, action, stackRef.Name(), op.Proj.Name, permalink,
displayEvents, displayEventsDone, opts, isPreview)
go persistEngineEvents(
u, opts.Debug, /* persist debug events */
persistEvents, persistEventsDone)
for e := range events {
displayEvents <- e
persistEvents <- e
// We stop reading from the event stream as soon as we see the CancelEvent,
// which will also signal the display/persist components to shutdown too.
if e.Type == engine.CancelEvent {
break
}
}
// Note that we don't return immediately, the defer'd function will block until
// the display and persistence go-routines are finished processing events.
}
func (b *cloudBackend) newQuery(ctx context.Context,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
op backend.QueryOperation,
) (engine.QueryInfo, error) {
return &cloudQuery{root: op.Root, proj: op.Proj}, nil
}
func (b *cloudBackend) newUpdate(ctx context.Context, stackRef backend.StackReference, op backend.UpdateOperation,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
update client.UpdateIdentifier, token string,
) (*cloudUpdate, error) {
// Create a token source for this update if necessary.
var tokenSource *tokenSource
if token != "" {
// TODO[pulumi/pulumi#10482] instead of assuming
// expiration, consider expiration times returned by
// the backend, if any.
duration := 5 * time.Minute
assumedExpires := func() time.Time {
return time.Now().Add(duration)
}
renewLease := func(
ctx context.Context,
duration time.Duration,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
currentToken string,
) (string, time.Time, error) {
tok, err := b.Client().RenewUpdateLease(
ctx, update, currentToken, duration)
if err != nil {
return "", time.Time{}, err
}
return tok, assumedExpires(), err
}
ts, err := newTokenSource(ctx, token, assumedExpires(), duration, renewLease)
if err != nil {
return nil, err
}
tokenSource = ts
}
// Construct the deployment target.
target, err := b.getTarget(ctx, op.SecretsProvider, stackRef,
op.StackConfiguration.Config, op.StackConfiguration.Decrypter)
if err != nil {
return nil, err
}
// Construct and return a new update.
return &cloudUpdate{
context: ctx,
backend: b,
update: update,
tokenSource: tokenSource,
root: op.Root,
proj: op.Proj,
target: target,
}, nil
}
func (b *cloudBackend) getSnapshot(ctx context.Context,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
secretsProvider secrets.Provider, stackRef backend.StackReference,
) (*deploy.Snapshot, error) {
untypedDeployment, err := b.exportDeployment(ctx, stackRef, nil /* get latest */)
if err != nil {
return nil, err
}
snapshot, err := stack.DeserializeUntypedDeployment(ctx, untypedDeployment, secretsProvider)
if err != nil {
return nil, err
}
Validate snapshots from service on load (#14046) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> In the filestate code we would always run loaded snapshots through VerifyIntegrity before using them. The httpstate didn't have any similar checks. This brings the two backends into alignment, reusing the option from before that was just for filestate (--disable-integrity-checking). At some point we should further align these so that httpstate also validates the snapshots it has written out, like filestate does today. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-04 15:12:56 +00:00
// Ensure the snapshot passes verification before returning it, to catch bugs early.
if !backend.DisableIntegrityChecking {
if err := snapshot.VerifyIntegrity(); err != nil {
return nil, fmt.Errorf("snapshot integrity failure; refusing to use it: %w", err)
}
}
Show manifest information for stacks This change supports displaying manifest information for a stack and changes the way we handle Snapshots in our backend. Previously, every call to GetStack would synthesize a Snapshot by taking the set of resources returned from the `/api/stacks/<owner>/<name>` endpoint, combined with an empty manfiest (since the service was not returning the manifest). This wasn't great for two reasons: 1. We didn't have manifest information, so we couldn't display any of its information (most important the last updated time). 2. This strategy required that the service return all the resources for a stack anytime GetStack was called. While the CLI did not often need this detailed information the fact that we forced the Service to produce it (which in the case of stack managed PPC would require the service to talk to yet another service) creates a bunch of work that we end up ignoring. I've refactored the code such that `backend.Stack`'s `Snapshot()` method now lazily requests the information from the service such that we can construct a `Snapshot()` on demand and only pay the cost when we actually need it. I think making more of this stuff lazy is the long term direction we want to follow. Unfortunately, right now, it means in cases where we do need this data we end up fetching it twice. The service does it once when we call GetStack and then we do it again when we actually need to get at the Snapshot. However, once we land this change, we can update the service to no longer return resources on the apistack.Stack type. The CLI no longer needs this property. We'll likely want to continue in a direction where `apistack.Stack` can be created quickly by the service (without expensive database queries or fetching remote resources) and just add additional endpoints that let us get at the specific information we want in the specific cases when we want it instead of forcing us to return a bunch of data that we often ignore. Fixes pulumi/pulumi-service#371
2018-05-22 22:39:13 +00:00
return snapshot, nil
}
func (b *cloudBackend) getTarget(ctx context.Context, secretsProvider secrets.Provider, stackRef backend.StackReference,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
cfg config.Map, dec config.Decrypter,
) (*deploy.Target, error) {
stackID, err := b.getCloudStackIdentifier(stackRef)
if err != nil {
return nil, err
}
snapshot, err := b.getSnapshot(ctx, secretsProvider, stackRef)
if err != nil {
switch err {
case stack.ErrDeploymentSchemaVersionTooOld:
return nil, fmt.Errorf("the stack '%s' is too old to be used by this version of the Pulumi CLI",
stackRef.Name())
case stack.ErrDeploymentSchemaVersionTooNew:
return nil, fmt.Errorf("the stack '%s' is newer than what this version of the Pulumi CLI understands. "+
"Please update your version of the Pulumi CLI", stackRef.Name())
default:
return nil, fmt.Errorf("could not deserialize deployment: %w", err)
}
}
return &deploy.Target{
Add tokens.StackName (#14487) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new type `tokens.StackName` which is a relatively strongly typed container for a stack name. The only weakly typed aspect of it is Go will always allow the "zero" value to be created for a struct, which for a stack name is the empty string which is invalid. To prevent introducing unexpected empty strings when working with stack names the `String()` method will panic for zero initialized stack names. Apart from the zero value, all other instances of `StackName` are via `ParseStackName` which returns a descriptive error if the string is not valid. This PR only updates "pkg/" to use this type. There are a number of places in "sdk/" which could do with this type as well, but there's no harm in doing a staggered roll out, and some parts of "sdk/" are user facing and will probably have to stay on the current `tokens.Name` and `tokens.QName` types. There are two places in the system where we panic on invalid stack names, both in the http backend. This _should_ be fine as we've had long standing validation that stacks created in the service are valid stack names. Just in case people have managed to introduce invalid stack names, there is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn off the validation _and_ panicing for stack names. Users can use that to temporarily disable the validation and continue working, but it should only be seen as a temporary measure. If they have invalid names they should rename them, or if they think they should be valid raise an issue with us to change the validation code. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-15 07:44:54 +00:00
Name: stackID.Stack,
Organization: tokens.Name(stackID.Owner),
Config: cfg,
Decrypter: dec,
Snapshot: snapshot,
}, nil
}
func isDebugDiagEvent(e engine.Event) bool {
return e.Type == engine.DiagEvent && (e.Payload().(engine.DiagEventPayload)).Severity == diag.Debug
}
type engineEventBatch struct {
sequenceStart int
events []engine.Event
}
// persistEngineEvents reads from a channel of engine events and persists them on the
// Pulumi Service. This is the data that powers the logs display.
func persistEngineEvents(
update *cloudUpdate, persistDebugEvents bool,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
events <-chan engine.Event, done chan<- bool,
) {
// A single update can emit hundreds, if not thousands, or tens of thousands of
// engine events. We transmit engine events in large batches to reduce the overhead
// associated with each HTTP request to the service. We also send multiple HTTP
// requests concurrently, as to not block processing subsequent engine events.
// Maximum number of events to batch up before transmitting.
const maxEventsToTransmit = 50
// Maximum wait time before sending all batched events.
const maxTransmissionDelay = 4 * time.Second
// Maximum number of concurrent requests to the Pulumi Service to persist
// engine events.
const maxConcurrentRequests = 3
// We don't want to indicate that we are done processing every engine event in the
// provided channel until every HTTP request has completed. We use a wait group to
// track all of those requests.
var wg sync.WaitGroup
defer func() {
wg.Wait()
close(done)
}()
Exclude internal events from those sent to the service (#14972) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Following up from https://github.com/pulumi/pulumi/commit/54c956af6d64638983eca50875d1dae704a18240 to also exclude the internal events from being sent to Pulumi Cloud. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-21 00:13:04 +00:00
// Need to filter the engine events here to exclude any internal events.
events = channel.FilterRead(events, func(e engine.Event) bool {
return !e.Internal()
})
var eventBatch []engine.Event
maxDelayTicker := time.NewTicker(maxTransmissionDelay)
// We maintain a sequence counter for each event to ensure that the Pulumi Service can
Exclude internal events from those sent to the service (#14972) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Following up from https://github.com/pulumi/pulumi/commit/54c956af6d64638983eca50875d1dae704a18240 to also exclude the internal events from being sent to Pulumi Cloud. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-21 00:13:04 +00:00
// ensure events can be reconstructed in the same order they were emitted. (And not
// out of order from parallel writes and/or network delays.)
eventIdx := 0
// As we identify batches of engine events to transmit, we put them into a channel.
// This will allow us to issue HTTP requests concurrently, but also limit the maximum
// number of requests in-flight at any one time.
//
// This channel isn't buffered, so adding a new batch of events to persist will block
// until a go-routine is available to send the batch.
batchesToTransmit := make(chan engineEventBatch)
transmitBatchLoop := func() {
defer wg.Done()
for eventBatch := range batchesToTransmit {
err := update.recordEngineEvents(eventBatch.sequenceStart, eventBatch.events)
if err != nil {
logging.V(3).Infof("error recording engine events: %s", err)
}
}
}
// Start N different go-routines which will all pull from the batchesToTransmit channel
// and persist those engine events until the channel is closed.
for i := 0; i < maxConcurrentRequests; i++ {
Run integration tests with race detection (#15895) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Combining #15120 and #15124 and including fixes to allow tests to pass. Enable race detection in the binary we're using for integration tests. This will allow us to catch more data races before they get into any release. This does mean the binary we're using for integration tests is slightly different from the binary we're releasing, however that's already the case as we're running a binary with coverage enabled for them. Later we rebuild the binary we're actually releasing. This requires us to fix the race between snapshot code, display code, and the step executor. I've done that by adding a lock to the State struct. This does not feel great, but it's a quick way to fix this and get race detection running (and unblocks #15871 which was also hitting the race detector because it started pulling snapshot code into unit tests as well). There's probably a more principled overhaul that doesn't require locking at this level. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-11 15:58:42 +00:00
wg.Add(1)
go transmitBatchLoop()
}
// transmitBatch sends off the current batch of engine events (eventIdx, eventBatch) to the
// batchesToTransmit channel. Will mutate eventIdx, eventBatch as a side effect.
transmitBatch := func() {
if len(eventBatch) == 0 {
return
}
batch := engineEventBatch{
sequenceStart: eventIdx,
events: eventBatch,
}
// This will block until one of the spawned go-routines is available to read the data.
// Effectively providing a global rate limit for how quickly we can send data to the
// Pulumi Service, if an update is particularly chatty.
batchesToTransmit <- batch
// With the values of eventIdx and eventBatch copied into engineEventBatch,
// we now modify their values for the next time transmitBatch is called.
eventIdx += len(eventBatch)
eventBatch = nil
}
var sawCancelEvent bool
for {
select {
case e := <-events:
// Ignore debug events unless asked to.
if isDebugDiagEvent(e) && !persistDebugEvents {
break
}
// Stop processing once we see the CancelEvent.
if e.Type == engine.CancelEvent {
sawCancelEvent = true
break
}
eventBatch = append(eventBatch, e)
if len(eventBatch) >= maxEventsToTransmit {
transmitBatch()
}
case <-maxDelayTicker.C:
// If the ticker has fired, send any batched events. This sets an upper bound for
// the delay between the event being observed and persisted.
transmitBatch()
}
if sawCancelEvent {
break
}
}
// Transmit any lingering events.
transmitBatch()
// Closing the batchesToTransmit channel will signal the worker persistence routines to
// terminate, which will trigger the `wg` WaitGroup to be marked as complete, which will
// finally close the `done` channel so the caller knows we are finished processing the
// engine event stream.
close(batchesToTransmit)
}