Commit Graph

207 Commits

Author SHA1 Message Date
Paul C. Roberts b72399bd28
The `--expect-no-changes` flag checks for output diffs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds a pseudo op `OpOutputChange` to the set of changes that are
recorded in `display.ResourceChanges` to count the number of output
changes, this is then included in the check used to evaluate the
`--expect-no-changes` flag

When resource outputs are registered, they are checked against their
previous value using existing functionality, the total count of changes
is then added

The internal capability is validated with an engine test, the cli is
validated using an integration test

This will break user workflows that depend on the previous behavior

Fixes 

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [X] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Paul Roberts <proberts@pulumi.com>
2024-05-01 18:30:49 +00:00
Thomas Gummerer ae8134f5ad
upgrade to latest version of golangci-lint ()
The version we currently have doesn't support Go 1.22 properly, so it
throws a bunch of warnings locally when trying to run it with the latest
Go version installed. Just running the latest version locally also
doesn't quite work, since it throws a bunch of errors from the
perfsprint linter, which seems to have gotten stricter.

Upgrade to the latest version of golangci-lint, and fix all the errors
we're getting from it. Mostly done via `perfsprint -fix`, with some
manual changes that `perfsprint -fix` wouldn't touch.
2024-04-19 06:20:33 +00:00
Thomas Gummerer 99dcdced9d
show all paths when we have multipe local policy packs with the same name ()
It is possible for multiple local policy packs to have the same name.
When there are policy violation, we currently only show one of the paths
for those policy packs (as we consider them the same policy pack because
it has the same name).

Display both of the paths in this case to avoid any confusion. Also sort
the names, so the test we have for this is deterministic.

I think in ideal world it would be better to just error out if there are
multiple policy packs with the same name, but given this behaviour
worked for a long time I wouldn't be surprised if that ends up breaking
some use case, so I'd be a bit worried about doing that now.

A possible alternative would be to not consider policy packs with the
same name as a single policy pack, but if we name it differently I could
imagine people getting confused with that. Happy to hear alternative
opinions on what the right behaviour here is since I'm not quite sure
myself.

Fixes 
2024-03-04 14:02:25 +00:00
Fraser Waters 7ba0541e1c
Rename self-managed to diy-backend ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

We're moving away from referring to filestate as "self managed"
backends, preferring to refer to this as "DIY" backends going forward.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-30 09:00:15 +00:00
Fraser Waters 762fcee66d
Fix some more lint issues ()
Again same as https://github.com/pulumi/pulumi/pull/14644, just fixing
some issues that make lint sees once you enable build tags.
2023-11-28 16:43:48 +00:00
Thomas Gummerer 13371f9865
Start policy packs in parallel ()
Currently we start up and thus compile policy packs one by one. When
multiple policy packs need to be loaded, this increases the start up
time substantially. In previous tests plugins took ~1-3s to start up, so
having multiple of these the time adds up quickly.

Loading them in parallel will help reduce the startup time here.

Fixes 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 14:08:32 +00:00
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Fraser Waters d771acf707
Add tokens.StackName ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new type `tokens.StackName` which is a relatively strongly
typed container for a stack name. The only weakly typed aspect of it is
Go will always allow the "zero" value to be created for a struct, which
for a stack name is the empty string which is invalid. To prevent
introducing unexpected empty strings when working with stack names the
`String()` method will panic for zero initialized stack names.
 
Apart from the zero value, all other instances of `StackName` are via
`ParseStackName` which returns a descriptive error if the string is not
valid.

This PR only updates "pkg/" to use this type. There are a number of
places in "sdk/" which could do with this type as well, but there's no
harm in doing a staggered roll out, and some parts of "sdk/" are user
facing and will probably have to stay on the current `tokens.Name` and
`tokens.QName` types.

There are two places in the system where we panic on invalid stack
names, both in the http backend. This _should_ be fine as we've had long
standing validation that stacks created in the service are valid stack
names.

Just in case people have managed to introduce invalid stack names, there
is the `PULUMI_DISABLE_VALIDATION` environment variable which will turn
off the validation _and_ panicing for stack names. Users can use that to
temporarily disable the validation and continue working, but it should
only be seen as a temporary measure. If they have invalid names they
should rename them, or if they think they should be valid raise an issue
with us to change the validation code.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-15 07:44:54 +00:00
Fraser Waters f34ded39f8
Support 'key.[index]' in property paths ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14439.

Ideally we would treat this as an error, but it seems providers are
building property paths of this format so we need to keep supporting it.

It is at least now tested, and clearly linked back to this issue so we
don't inadvertantly remove support for it in the future.

Yet another element of the system that could be tightened up in a V4
release.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-31 17:37:48 +00:00
Thomas Gummerer c7df3d2513
tighten up parsing of property path to avoid accepting typos ()
The key "my:root.[1]" used to not be accepted, because the `.` before
the index is invalid. However through
https://github.com/pulumi/pulumi/pull/13814/files#r1320306140 and
subsequently https://github.com/pulumi/pulumi/pull/14149, the parsing
was loosened.

Presumably we were never checking this at the right level in the first
place, and we only accidentally refused to parse it. Fix this by
tightening the conditions when parsing the property path.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-17 07:20:28 +00:00
Thomas Gummerer bf76b6c525
ignore unexpected types instead of erroring out ()
Before , when we encountered unexpected types in a yaml file, we
ignored them. However that PR changed that, and we end up panicing, e.g.
in case we have a config like the following:

```
config:
  aws:region: us-east-1
  aws🆔
    -
```

Fix that by getting the old behaviour of ignoring this error back.

Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Kyle Dixler <kyle@pulumi.com>
2023-10-11 16:39:16 +00:00
Fraser Waters baa04160ec Fix `stack rm` removing config files for the wrong project
Fixes https://github.com/pulumi/pulumi/issues/12246
2023-06-21 10:52:25 +01:00
Abhinav Gupta 795a3d07be
fix(cli/state rename): Update parent references on rename
If a resource is renamed with `pulumi state rename`,
in addition to updating references in resources that refer to it
as a dependency,
also update references to the resource as a parent of another resource.

Testing:
Includes an integration test that reproduces the panic,
and a unit test that verifies the new behavior at a more isolated level.
To implement the integration test,
a new RunPulumiCommand method was added to ProgramTester
because we don't have other means of running `pulumi state rename`
with ProgramTest.

Resolves 
2023-06-16 13:46:35 -07:00
bors[bot] e57d82fd5b
Merge
12856: Support 'pulumi:tags' config to set stack tags r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Support a "pulumi:tags" config property to set the stacks tags on operations (e.g. up, refresh, etc).

Fixes https://github.com/pulumi/pulumi/issues/5004

Note that I've gone with a slightly different approach to . The suggestion in that ticket was to add a new section "tags" to the yaml, but given that we probably want to allow setting tags per-project and per-stack with some way to merge them I figured it better to reuse normal config and benefit from https://github.com/pulumi/pulumi/issues/11547 when that's done.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-06-01 17:10:29 +00:00
Fraser Waters 92d4411327 Fix project renames for filestate 2023-05-30 13:54:28 +01:00
Fraser Waters 89a7922b08 Support 'pulumi:tags' config to set stack tags
Fixes https://github.com/pulumi/pulumi/issues/5004
2023-05-24 09:02:22 +01:00
Fraser Waters 1376a13d13 Fix destroy without project file
Fixes https://github.com/pulumi/pulumi/issues/12714.

The empty project created for destroy operations when no Pulumi.yaml is
found still needs to have it's name filled in so that project name
consistency checks work.

Our tests didn't pick this up because for filestate we were still
searching from the working directory for consistency checks rather than
checking the backends current project. I have _not_ changed that in this
changeset as previosly that triggered total breakage of the filestate
backend when not in project mode (see
https://github.com/pulumi/pulumi/issues/12760).
2023-05-02 10:36:16 +01:00
Abhinav Gupta a92b877e5f
sdk/go: Propagate more resource options from provider.Construct
`provider.Construct` is used to build MLCs in Go.
It receives a pulumirpc.ConstructRequest
and turns that into a standard Go SDK ResourceOption.

In , we added a few new fields to pulumirpc.ConstructRequest.
This updates the Go SDK to pick up these new options
and pass them on for the resource to pick up
(and eventually send back to the engine as a RegisterResource call).

Testing:
This change includes unit tests to verify mapping of ConstructRequests
to options, which are then interpreted with NewResourceOptions.

On top of that, it builds upon the integration test added in 
to verify options that are saved in snapshots.
Note that this integration test does not test for the following options:
IgnoreChanges, ReplaceOnChanges, and DeleteBeforeReplace.
These options are not stored in state snapshots,
so we can't validate them just by looking at that.

Refs 
2023-04-21 10:58:59 -07:00
Abhinav Gupta 96dcba18e2
test(go): Integration test for MLC options
Adds a minimal integration test that verifies
options passed to an MLC in Go are propagated properly.

Currently, two options are checked: DependsOn and Protect.
Other options will be added to this test as part of .
2023-04-18 13:22:52 -07:00
Abhinav Gupta 84a7b1ce9c
test: Add failing test case for
Information in the original issue is a bit limited,
but we know that the error occured when the user
passed a `provider=x` to a multi-language component,
which then instantiate a resource handled by that provider
with itself as the parent.
The expected behavior was for the resource to use the explicit provider,
but instead, it used the default global provider.

The reproduction does roughly the same for Go, Python, and Node,
except it instantiates three resources:

1. using the default provider
2. using an explicit provider specified with `provider=x`
3. using an explicit provider specified with `providers=[x]`
3. using an explicit provider specified with `providers={pkg: x}`

Case (2) fails as expected from  for Python.

```
--- PASS: TestConstructProviderPropagationGo (14.66s)
--- PASS: TestConstructProviderPropagationNode (15.03s)
--- FAIL: TestConstructProviderPropagationPython (17.63s)
```
2023-04-13 18:12:18 -07:00
Fraser Waters 035c527bd0 Add regression test for 2023-04-12 15:21:18 +01:00
Fraser Waters 9d0fba3a7e
filestate: Re-add project support
This re-adds project support back to the filestate backend
by implementing a new referenceStore: projectReferenceStore.

We will use this reference store for all new filestate stores.
Existing states will continue to use the legacyReferenceStore.

To accomplish this, and to plan for the future,
we introduce a 'meta.yaml' file inside the .pulumi directory.
This file contains metadata about the storage state.
Currently, this only holds a version number:

    # .pulumi/meta.yaml
    version: 1

Version 1 is the number we've chosen for the initial release
of project support.
If we ever need to make breaking changes to the storage protocol
we can bump the format version.

Notes:

- Stack references produced by filestate will shorten to
  just the stack name if the project name for the stack
  matches the currently selected project.
  This required turning currentProject on localBackend
  into an atomic pointer because otherwise
  SetCurrentProject and localBackendReference.String may race.

Extracted from 

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-03-31 13:21:36 -07:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Fraser Waters cc077b247e Use plugin link and shimless for testprovider 2023-01-19 12:41:19 +00:00
Kyle Dixler a81ad8a2a4
fix : pulumi panics when Pulumi.yaml has a provider with an empty/unspecified `path` key 2023-01-18 10:33:39 -08:00
Fraser Waters f4ad5c5a56 Remove dotnet 2022-12-13 16:13:53 +00:00
Aaron Friel 28b1ae2c37 ci: Move several tests out of smoke test jobs 2022-11-03 18:23:02 -07:00
Zaid Ajaj 12f399225e Revert changing the temp dir TestDestroyStackRef 2022-10-12 15:43:25 +02:00
Zaid Ajaj afeac6e9be simplify test assert even more (attempt ) 2022-09-27 14:33:53 +02:00
Zaid Ajaj 27556385d0 Fix TestDestroyStackRef: don't change CWD and fix TestStackInitValidation (less rigid contains check) 2022-09-27 13:11:37 +02:00
Zaid Ajaj 78ecb2b8fc Merge branch 'master' into minimal-config 2022-09-24 13:50:44 +02:00
Zaid Ajaj 7975498b71 lint 2022-09-22 18:27:08 +02:00
Fraser Waters 9a7ab41347 Merge remote-tracking branch 'origin/master' into useLocalDeps 2022-09-22 15:16:05 +01:00
Aaron Friel 2af74b6469 ci: Use reduced smoke testing on Windows & macOS targets 2022-09-21 09:55:06 -07:00
Aaron Friel a4b1d6b2a7 ci: gofmt 1.18+ clean 2022-09-21 09:48:39 -07:00
Fraser Waters 2461494057 lint 2022-09-20 17:12:43 +01:00
Fraser Waters 1762ed7745 Use plugins.providers rather than setting PATH 2022-09-20 16:45:00 +01:00
bors[bot] 5beffd8986
Merge
10691: Bring back auto-install plugins r=Zaid-Ajaj a=Zaid-Ajaj

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This PR brings back  (cc `@AaronFriel)`

Fixes 
Fixes  

After debugging for ages, it turns out that the [package](https://github.com/checkly/pulumi-checkly/tree/main/sdk/nodejs) that caused the p0/revert of the original PRs were faulty in the sense that they didn't use latest codegen to generate the nodejs package. The generated package.json had
```json
{
    "pulumi": {
        "resource": true,
        "pluginDownloadURL": "https://github.com/checkly/pulumi-checkly/releases/download/v1.1.2"
    }
}
```
And couldn't auto install the third-party plugin because: 
 - Doesn't have a `name` which is now going to be required for third-party packages when the package name is not ``@pulumi/{something}`(see` change in pulumi-language-nodejs/main.go)
 - When `pulumi.name` is missing, `pulumi-language-nodejs/main.go` tried to "derive" the plugin name by simply removing the "`@"` sign from the package name but that is still is wrong because the plugin name was "checkly" and the derived plugin name was "checkly/pulumi" hence why a `packageJson.pulumi.name` is now required
 - Had `pluginDownloadURL` property instead of `server` which is not read anywhere by Pulumi except from the original schema.json itself, not the generated package.json file where it should be transformed to `server`
 - Third-party package used a very old version of tfgen v3.19.0 which is from February this year. 

I've confirmed that after installing the plugin via the install script added to the package _and_ adding a name to the `pulumi` section that the error no longer shows since now

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have updated the [CHANGELOG-PENDING](https://github.com/pulumi/pulumi/blob/master/CHANGELOG_PENDING.md) file with my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


10702: Prep for real 3.40.0 release r=AaronFriel a=AaronFriel

This turns the "safeties off" after validating the bors workflow and ensuring everything up to the dispatch commands succeeds.

It also fixes an error in `get-next-version` that resulted in the "vNext" PR  to calculate the wrong "next version", which was corrected by hand. (The branch name and commit history is evidence of the error.)

10704: ci: Add make changelog shortcut for creating changelog entries r=AaronFriel a=AaronFriel



10713: ci: Fix Windows flake with longer timeout r=AaronFriel a=AaronFriel

This fixes Windows tests failing like this due to very short (1 second) timeouts.

https://github.com/pulumi/pulumi/actions/runs/3054396239/jobs/4926314390


10714: ci: Reduce test duration by avoiding costly subtests in loop r=AaronFriel a=AaronFriel

The loops in this file create millions of subtests, the majority of which do very little work. This reduces the time to execute these tests to a couple seconds.

10715: ci: Remove v0.10.0 compatibility test r=AaronFriel a=AaronFriel

Removes a _very old_ and ostensibly disabled test. The condition in this test was actually incorrect:

```go
func skipIfNotNode610(t *testing.T) {
	nodeVer, err := getNodeVersion()
	if err != nil && nodeVer.Major == 6 && nodeVer.Minor == 10 {
		t.Skip("Skipping 0.10.0 compat tests, because current node version is not 6.10.X")
	}
}
```

This only skips **if** the node version is vMajor = 6 and vMinor == 10.

I discovered this test from bisecting which test was installing a very old Pulumi library and running node-gyp to link an even older gRPC library.

10716: ci: Enable async component builds, synchronous Yarn execution r=AaronFriel a=AaronFriel

The `make test_build` command is blocking for all integration tests, adding roughly 2 minutes of execution time to each. It runs a series of Node, Python, and Go CLI commands to configure providers. These build steps use a file lock to ensure only one runs at a time, and additionally parallelizes the three build steps.

This moves the setup step into an asynchronous action performed on-demand, which enables greater concurrency and allows integration tests that perform _no_ setup to skip most of the costly build process altogether.

The second commit, dependent on the first, adds a global mutex in the sdk library for running `yarn` commands. The Yarn CLI itself has a lock that prohibits concurrent execution, however it times out. On concurrent executions then, we get spurious test failures because `yarn` failed to execute waiting for a separate execution.

These changes improve test reliability and reduce integration test execution time by roughly 2 minutes across all jobs.

10718: ci: Skip refresh on tests that should fail fast r=AaronFriel a=AaronFriel

These tests performed unnecessary refresh steps. Happened to catch these while optimizing build times. The impact on execution times was negligible.

10719: ci: Use yarn link before install, ensure Pulumi latest is used r=AaronFriel a=AaronFriel

These cut execution time on myriad node tests by 5 to 30 seconds each. Linking ``@pulumi/pulumi`` first, before `yarn install`, enables yarn to skip downloading and extracting the node SDK before replacing it with a symlink.

10721: ci: Increase bors timeout r=AaronFriel a=AaronFriel

The default timeout is one hour, which our CI runs can exceed. This should give us a buffer while work is done to reduce execution time to a reasonable threshold.

10722: ci: Improve test parallelism, enabled by  r=AaronFriel a=AaronFriel

These tests no longer mutate environment variables, instead using `e.SetEnvVars` to only set env vars on child processes since .

10723: ci: Reduce test duration with parallel subtests r=AaronFriel a=AaronFriel

This change parallelizes subtests. Although there are only ~200 combinations tested, they are fully parallelizable and this reduces execution time for these tests from 80-100s to 10-15s each, reducing execution time for the package by about a minute or a little more. Each parent test is compute bound and already run in parallel, so the reduction isn't additive.

10724: ci: Enable robust retries on tests r=AaronFriel a=AaronFriel

This adds a retry script to `./scripts/retry` which takes any shell command and retries it up to `PULUMI_TEST_RETRIES` times. On each iteration of the script, parallelism and variation is reduced.


Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-09-14 20:18:59 +00:00
Aaron Friel 5548fa8b28 ci: Improve test parallelism, enabled by 2022-09-14 11:32:30 -07:00
Aaron Friel 7bef1b717e ci: Prevent yarn mutex timeout by single-threading yarn commands 2022-09-14 10:07:21 -07:00
Aaron Friel f00c5374d1 ci: Enable async component builds 2022-09-14 10:06:05 -07:00
Vinícius Vargas 1fc3cebea6
Improve error message about stack name validation ()
* chore: Improve error message about stack name validation
2022-09-12 08:42:38 -07:00
svangordon-fruit 51406fc771
refactor: append to env var list instead of overwriting. () 2022-07-28 08:39:31 +01:00
Fraser Waters d69eea6384
Revert "Use plugin link instead of PATH manipulation in programtest ()" ()
This reverts commit c97aa71443.

Seems that while a load of these worked on my machine, CI is unhappy
with them. I'll revert this and wait till the /run-integration-tests
command is working to confirm it's all good before putting in.
2022-07-25 16:51:40 +01:00
Fraser Waters c97aa71443
Use plugin link instead of PATH manipulation in programtest () 2022-07-25 14:56:50 +01:00
Kyle Dixler 9028a02726
allow `pulumi destroy -s <stack>` if not in a pulumi project dir ()
* pulumi destroy -s <stack> now works as long as you are not in a pulumi project directory
2022-07-11 08:28:53 -07:00
Fraser Waters 06e42a5697
Revert "allow `pulumi destroy -s <stack>` if not in a pulumi project dir ()" ()
* Revert "allow `pulumi destroy -s <stack>` if not in a pulumi project dir ()"

This reverts commit 150aba462e.

Fixes https://github.com/pulumi/pulumi/issues/9815

* Update CHANGELOG

* Update CHANGELOG_PENDING.md

Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
2022-06-09 14:28:52 -04:00
Kyle Dixler 150aba462e
allow `pulumi destroy -s <stack>` if not in a pulumi project dir ()
* pulumi destroy -s <stack> now works as long as you are not in a pulumi project directory
2022-06-01 14:22:32 -07:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Justin Van Patten 6c7a330051
Fix passphrase secrets provider prompting ()
In , caching was introduced inside the passphrase secrets provider to avoid prompting the user for a password multiple times in the same `pulumi` command. However, the cache is not used in all cases, which leads to double prompting and/or erroring if the passphrase is not set in the `PULUMI_CONFIG_PASSPHRASE` envvar. In some cases, the user is prompted for their passphrase and then the command fails because the passphrase wasn't set via the envvar.

This change addresses these issues. The net effect is that when using the passphrase secrets provider, you'll be prompted for your passphrase up-front once (if interactive and the envvar isn't set), and subsequent uses will use the cached value and won't prompt or error if the envvar isn't set.

Implementation note: This moves the prompting code inside the `passphrase` package. I was originally trying to avoid this, to avoid the package depending on `cmdutil.ReadConsoleNoEcho` and `cmdutil.Interactive`, but this lead to unnatural contortions elsewhere. It ended up being much cleaner to have the prompting happen inside the passphrase implementation. Functions that may prompt (when interactive and the envvar isn't set) include "Prompt" in their name.
2022-02-16 13:39:08 -08:00