Commit Graph

840 Commits

Author SHA1 Message Date
Julien Poissonnier 6f37c78262
link plugin install message to progress bar 2024-08-10 16:51:06 +02:00
Julien Poissonnier 7cad2851e3
fix slot free 2024-08-10 16:39:19 +02:00
Julien Poissonnier 11ae3c76c1
Only show one download progress bar at at time 2024-08-10 16:39:04 +02:00
Julien a3c0ea5455
Add NodeJS test for parameterized providers ()
This takes https://github.com/pulumi/pulumi/pull/16392 and updates local
SDK generation for nodejs and adds parametrisation support for invokes

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-08-09 08:34:29 +00:00
Fraser Waters 1416a34e69
Add --local to GeneratePackage and gen-sdk ()
No languages make use of this yet, but this will be needed for some
languages for good local SDK support. E.g. we're planning on adding a
postinstall script for nodejs local packages.
2024-08-07 08:16:37 +00:00
Fraser Waters 765a67effb
Flow PropertyValues through NewPropertyMapFromMap ()
Fixes https://github.com/pulumi/pulumi/issues/16889

Currently `resource.NewPropertyMapFromMap` will drop any
`resource.PropertyValue`s from the input map. This is demonstrated
succinctly by Ian in the linked ticket:

```go
package main

import (
	"fmt"

	"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
)

func main() {
	fmt.Println(resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
		"foo": resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
			"bar": resource.NewNumberProperty(0),
		})),
	})))
	fmt.Println(resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
		"foo": map[string]any{"bar": 0},
	})))
}
```

```
{map[foo:{map[]}]}
{map[foo:{map[bar:{0}]}]}
```

This PR updates `NewPropertyMapFromMap` to instead flow these values
through directly resulting in an output of:
```
{map[foo:{map[bar:{0}]}]}
{map[foo:{map[bar:{0}]}]}
```
2024-08-07 06:50:58 +00:00
Fraser Waters cb4a06b475
Package versions are required for parameterized packages ()
Make this more clear by passing plain value of `semver.Version` instead
of a pointer for these cases.

This also gets rid of some incorrect empty string checks which would
have passed a nil version onwards rather than error'ing (as they should
have).
2024-08-05 16:01:28 +00:00
Julien 1cdd2315de
Don't search upwards for policy packs when determining required plugins ()
We don't need to search upwards for PolicyPack files, we already know
that there aren't any since we don't recurse into policy packs. The
directory traversal is plenty fast on Linux, however it is much slower
on Windows. Node modules often have fairly wide and deep directory
structures, so this can considerably slow down any Pulumi operation that
runs `GetRequiredPlugins`.

Fixes https://github.com/pulumi/pulumi/issues/16863

On an `was-typescript` example, we can see the impact with
`Measure-Command {start-process pulumi -argumentlist "about" -Wait}`,
which goes from 25 seconds to 2 seconds on an Azure Windows VM.
2024-08-02 11:49:16 +00:00
Mikhail Shilkov d4f1cf5c87
URL-based plugin source overrides via env var ()
### Motivation

Pulumi plugin binaries can be downloaded by the CLI from multiple
sources. By default, it's downloaded from Pulumi's GitHub releases or
get.pulumi.com, but plugins can also specify their binary sources via
the `PluginDownloadURL` schema option. They can point to custom GitHub,
Gitlab, or HTTP locations.

Enterprise customers ask for a way to isolate the CLI from downloads
from random locations and to configure the CLI to go to their internal
pre-approved artefact location instead. This way, Pulumi can run in
"air-gapped" environments (which still have access to Cloud APIs, of
course).

Related issues:
- https://github.com/pulumi/pulumi/issues/14459
- https://github.com/pulumi/pulumi/issues/16240

Currently, there is a basic mechanism to do so via the variable
`pluginDownloadURLOverrides`, but it has two major limitations:
- The variable value is set via a compile-time flag, so it requires a
custom build of the CLI
- The overrides are based on the plugin name, so the rules must be
defined without access to the original URL, which makes it hard to
provide universal rules and still distinguish between first-party,
public third-party, or private in-house plugins
- We ignore overrides for all plugins that have `PluginDownloadURL` set
- Overrides can set a plugin replacement redirect only to HTTP(s)
addresses

### Proposal

This PR makes two sets of changes:

1. It allows passing overrides via the
`PULUMI_PLUGIN_DOWNLOAD_URL_OVERRIDES` environment variable. The
compile-time flag is still supported, but the env var takes priority.

More configuration levers could be supported, but it not clear if we
have good ones until [Support .pulumirc file for global
config](https://github.com/pulumi/pulumi/issues/13484) is implemented. I
don't expect users to want to set this via their stack configs, but I'm
curious what others think. In any case, more sources can be added later.

2. The overrides now apply based on the original download URL, not just
on plugin names. Actually, it's the base URL of a download source that
is passed to the regexp matcher. Examples of possible options are:

- `github://api.github.com/pulumi/pulumi-xyz` for a first-party plugin
(note that we don't pass `get.pulumi.com`
- `github://api.github.com/pulumiverse/pulumi-grafana` for a community
plugin that sets `PluginDownloadURL`
- `gitlab://gitlab-host/proj-name` for a community plugin hosted on
Gitlab
    - `https://example.com/downloads/` for HTTP sources

So, the override
`^github://api.github.com/pulumi/pulumi-xyz=https://example.com/downloads/pulumi-xyz/`
will override the single provider URL from our GitHub releases to the
given HTTP location.

On top of that, regular expressions may contain name groups to capture
and use templated values. For example,
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/${org}/${repo}`
captures any GitHub plugin and redirects it to its corresponding HTTP
location. Group indices are also supported: the above override can also
be written as
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/$1/$2`,
with `$0` meaning the full match.

The override URLs have the same semantics as `PluginDownloadURL`, so
they can point to GitHub, Gitlab, HTTP, or anything we introduce in the
future.

### Impact

Technically, this is a breaking change, because name-based overrides
will stop working. However, we are fairly certain that we have a single
customer using the existing compile-time approach, and they indicated
that they don't need the name-based overrides if they have URL-based
overrides. I reviewed this PR with them and made sure they can migrate
immediately after the change is released.

Backwards compatibility is slightly tricky, because we'd need to keep
name-based override _and_ not applying them to third-party plugins. But
we can do it if necessary.

Resolve 
2024-07-26 10:37:09 +00:00
Artur Laksberg 428e9efc37
Add links to StackSummary to for clients like Copilot ()
This data will be filed by the Pulumi service and will be used by
Copilot to provide clickable links to objects
2024-07-25 21:15:48 +00:00
Zaid Ajaj b3546c9fa4
[cli/import] Fix undefined variable errors in code generation when imported resources use a parent or provider ()
Fixes  Fixes 

## Problem Context

When using `pulumi import` we generate code snippets for the resources
that were imported. Sometimes the user specifies `--parent
parentName=URN` or `--provider providerName=URN` which tweak the parent
or provider that the imported resources uses. When using `--parent` or
`--provider` the generated code emits a resource option `parent =
parentName` (in case of using `--parent`) where `parentName` is an
unbound variable.

Usually unbound variables would result in a _bind_ error such as `error:
undefined variable parentName` when type-checking the program however in
the import code generation we specify the bind option
`pcl.AllowMissingVariables` which turns that unbound variable errors
into warnings and code generation can continue to emit code.

This is all good and works as expected. However in the issues linked
above, we do get an _error_ for unbound variables in generated code even
though we specified `AllowMissingVariables`.

The problem as it turns out is when we are trying to generate code via
dynamically loaded `LangaugeRuntime` plugins. Specifically for NodeJS
and Python, we load `pulumi-language-nodejs` or `pulumi-language-python`
and call `GenerateProgram` to get the generated program. That function
`GenerateProgram` takes the text _SOURCE_ of the a bound program (one
that was bound using option `AllowMissingVariables`) and re-binds again
inside the implementation of the language plugin. The second time we
bind the program, we don't pass it the option `AllowMissingVariables`
and so it fails with `unboud variable` error.

I've verified that the issue above don't repro when doing an import for
dotnet (probably same for java/yaml) because we use the statically
linked function `codegen/{lang}/gen_program.go -> GenerateProgram`

## Solution

The problem can be solved by propagating the bind options from the CLI
to the language hosts during import so that they know how to bind the
program. I've extended the gRPC interface in `GenerateProgramRequest`
with a property `Strict` which follows the same logic from `pulumi
convert --strict` and made it such that the import command sends
`strict=false` to the language plugins when doing `GenerateProgram`.
This is consistent with `GenerateProject` that uses the same flag. When
`strict=false` we use `pcl.NonStrictBindOptions()` which includes
`AllowMissingVariables` .

## Repro

Once can test the before and after behaviour by running `pulumi up
--yes` on the following TypeScript program:
```ts
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";

export class MyComponent extends pulumi.ComponentResource {
    public readonly randomPetId: pulumi.Output<string>;
    constructor(name: string, opts?: pulumi.ComponentResourceOptions) {
        super("example:index:MyComponent", name, {}, opts);

        const randomPet = new random.RandomPet("randomPet", {}, { 
            parent: this 
        });

        this.randomPetId = randomPet.id;
        this.registerOutputs({
            randomPetId: randomPet.id,
        });
    }
}

const example = new MyComponent("example");
export const randomPetId = example.randomPetId;
``` 
Then running `pulumi import -f import.json` where `import.json` contains
a resource to be imported under the created component (stack=`dev`,
project=`importerrors`)
```ts
{
    "nameTable": {
        "parentComponent": "urn:pulumi:dev::importerrors::example:index:MyComponent::example"
    },
    "resources": [
        {
            "type": "random:index/randomPassword:RandomPassword",
            "name": "randomPassword",
            "id": "supersecret",
            "parent": "parentComponent"
        }
    ]
}
```
Running this locally I get the following generated code (which
previously failed to generate)
```ts
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";

const randomPassword = new random.RandomPassword("randomPassword", {
    length: 11,
    lower: true,
    number: true,
    numeric: true,
    special: true,
    upper: true,
}, {
    parent: parentComponent,
});
```
2024-07-25 13:53:44 +00:00
Will Jones 403b3d3fa4
Fix panics due to different length `ignoreChanges` arrays ()
The `ignoreChanges` resource option accepts a list of paths into a
resource that should be ignored when computing whether or not something
has changed. For example:

```typescript
const r = new Resource(
  "r",
  {
    a: "a",
    b: [1, 2],
    c: { d: "d" },
  },
  {
    ignoreChanges: [
      "a",
      "b[*]",
    ],
  }
)
```

Here, when diffing `r`, Pulumi will ignore changes to `a` (due to the
path `"a"`) as well as changes to any element of `b` (due to the
_wildcard_ path `"b[*]"`). Under the hood, `ignoreChanges` is
implemented partly by "resetting" pieces of a resource's state to older
values -- that is, rather than ignoring changes that might be reported,
Pulumi will undo the changes before they are used altogether.

In , a panic encountered when resetting children of arrays of
different length was fixed. This commit extends this fix to cover panics
that occur when simply resetting arrays themselves (e.g. resetting
`[1,2]` to `[1,2,3]`). As part of this, some test cases are renamed to
convey that the are actually testing the children of arrays or
wildcard-selected objects, in order to accommodate new test cases for
this panic.

Fixes 
2024-07-22 14:54:37 +00:00
Ian Wahbe a357e7dd89
Set `Parameterize.Value` in `plugin.Provider` () 2024-07-20 09:15:33 +00:00
Thomas Gummerer 73e073efc6
disallow OutputState to be deepcopied ()
We have this deepcopy utility in our public API, but it does not work
properly on structs with unexported values. Make a note of that, and
also disallow OutputState from being copied, since OutputState has
required unexported fields that we can't deepcopy.

Not sure if we should special case `OutputState` here, which I've done
since that has come up in particular, or maybe even disallow
deepcopy'ing any values that have unexported fields? That would be a
breaking change though, so I've shied away from it for now.

Fixes https://github.com/pulumi/pulumi/issues/16634
2024-07-18 12:56:36 +00:00
Fraser Waters a8cd9b9435
Remove unused PropertySet type ()
Doesn't look like anything uses this, and chances are we'll want to add
_actual_ sets of property values at some point which clashes with this
name.
2024-07-17 21:51:54 +00:00
Fraser Waters e0da9cb39a
Adding conformance tests for Go ()
This adds the framework for running conformance tests for Go. Currently
_all_ the tests are skipped, but I'll raise small PRs on top of this one
to start fixing some of the issues.
2024-07-17 09:00:08 +00:00
Julien ff5fe13fe2
Fix incorrect caching of git auth method in error cases ()
urlAuthParser.Parse caches auth methods. To avoid caching the auth
method when running into an error, we first checked that the auth method
is not nil. However we ran into a classic Go issue where a nil value of
a concrete type is assigned to a variable of an interface type, making 
it not equal to nil.

https://go.dev/doc/faq#nil_error

https://go.dev/play/p/AOSdCWd3XC1

Fixes https://github.com/pulumi/pulumi/issues/16637

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-07-16 14:53:32 +00:00
Fraser Waters 0a83daaa08
Fix CopyFile to copy symlink directories ()
Noticed while working on https://github.com/pulumi/pulumi/pull/16662.
CopyFile couldn't handle a symlink'd directory.
2024-07-16 11:57:14 +00:00
Fraser Waters a9947b4e4e
Python parameterized provider test ()
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.

When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.

The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
2024-07-16 10:55:38 +00:00
Will Jones fc69816291
Test provider `Delete` parameter marshalling ()
This commit adds a test to ensure that provider `Delete`s are passed
with the correct set of parameters. This is a regression test for
, as part of the follow-up to  in . Assuming we are
happy with this level/style of test, we should consider adding tests for
the other provider methods also.

Fixes 
2024-07-12 11:19:32 +00:00
Germán Lena 956428f690
Update deployment settings git configuration to support directories that are not a git repo ()
Change how the deployment settings configures git repos to support
directories that are not git repositories

Fix https://github.com/pulumi/pulumi-service/issues/20675

---------

Co-authored-by: Levi Blackstone <levi@pulumi.com>
2024-07-11 14:14:18 +00:00
Fraser Waters a0e0208dd1
Change parameterization to be bytes based ()
Making this change has a couple of benefits.

Firstly (and honestly the main one) is it make codegen much simpler. We
just to emit a base64 string and/or other embedded byte array to the
generated SDK. Currently we need emit a full `proto.Value` expression
for each language, and that's not hard but it's also not trivial and
means more combinations of things per test per language.

Secondly it will allow providers to use more efficient encodings of
their parameter than JSON if there is one. I imagine some providers
might make the parameter value a protobuf message and parse that
(similar to what we do for transform functions), but they can easily
fallback to just treating the bytes as a JSON string if they want.

The only downside to this is the value is obfuscated in the generated
SDK and in the state file. Neither of those are really expected to be
viewed by users, so this feels like a minor loss.
2024-07-10 11:15:35 +00:00
Germán Lena a7d5e238b8
New deployment settings wizards and environment variables management comands ()
- Turns `deployment settings init` command a wizard
- Adds new `deployment settings env` command to manage env variables
(including secrets encryption)
- Adds new `deployment settings set` command to configure individual
settings (including secrets encryption)

https://asciinema.org/a/QhuWHAvkmeAmVJkYqkCP0P6wb

Fix https://github.com/pulumi/pulumi-service/issues/20567
Fix https://github.com/pulumi/pulumi-service/issues/20576
2024-07-03 20:24:26 +00:00
Thomas Gummerer 0dc51aadd9
retry post requests that timeout during handshake ()
We've had quite a few test flakes that happen because of TLS handshake
timeouts to the service backend. These are usually during stack creation
(which happens a lot during tests), which are POST requests.

POST requests are not generally safe to retry, because the request could
already have been sent to the backend and an action taken, but the
client couldn't read the response. However when there is a TLS handshake
failure, the request can never have made it to the backend, so these
errors are safe to retry.

This should hopefully help with
https://github.com/pulumi/pulumi/issues/16529.
2024-07-03 16:22:24 +00:00
Thomas Gummerer 2f4b26b681
rewrite the URN when resources are being moved between projects ()
When a resource is moved between stacks in different projects we also
need to rewrite project part of the URNs. Do that here. Moving between
different projects already works by virtue of the requireStack function
supporting that when providing the fully qualified name of the stack.

There is some awkwardness here as old diy backends may not return a
project name, in which case we error out here. Curious if anyone has
thought about what to do here. Is erroring out the best we can do? What
happens if we don't rewrite the project name in this case?

---------

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-07-01 17:36:11 +00:00
Germán Lena a0b67cda05
Update pu/pu to support deployment run command ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Add new deployment run command

Fixes https://github.com/pulumi/pulumi-service/issues/20500

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-07-01 14:18:44 +00:00
Julien P 68d2236bf5
Add not-found markers to missing executables for packagemanagers ()
During the creation of new python and nodejs projects, flag package
manages that can't be found with a `[not found]` suffix. This allows the
user to see all possible options, but also understand why the
installation will fail (Anoter PR will improve the error message for a
failed installation when the executable could not be found).

<img width="443" alt="Screenshot 2024-06-26 at 13 35 13"
src="https://github.com/pulumi/pulumi/assets/387068/f659277d-1963-4e4e-861d-29166eabb190">

<img width="619" alt="Screenshot 2024-06-26 at 13 43 52"
src="https://github.com/pulumi/pulumi/assets/387068/a423c187-9353-4370-82b5-35cab8db91b1">
2024-06-28 23:21:55 +00:00
Thomas Gummerer 24ce87b566
implement skeleton command for `pulumi state move` ()
Implement a skeleton command for `pulumi state move`. This can so far
only more a single resource from one stack to another, including copying
the provider it needs, and leave valid state files behind.

These state files are not written to the backend yet, nor is there any
kind of UI. Further PRs will build on top of this.

The command is intentionally commented out so this can be merged
independently.

---

Based on discussions in the last retro, I'm trying something new here
and try to split the PR up some more into chunks that can be
individually merged, but are not completely ready yet. The code added
here is essentially dead code at this point, but provides a skeleton to
incrementally add pieces.

To give an idea of where this is headed, the next things I'm planning to
work on here are:
- Writing the state files to the backend
- Make sure this works with stacks from two different projects
- Add a preview phase that asks for user confirmation
- Add output for moved resources and warnings for dependencies that are
being broken
- Implement `--include-parents` flag
- Implement convenience flags (`--yes`, `--skip-preview` etc.)

I'd love some early thoughts on this PR, and also meta thoughts on
splitting PRs up this way.
2024-06-28 09:43:19 +00:00
Julien P a26c010589
Fix TestTerminate_gracefulShutdown/python flake ()
It is unsafe to call `print` in a Python signalhandler. Instead we set a
flag and check this in the main busy loop.

Fixes https://github.com/pulumi/pulumi/issues/16472

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-06-26 05:50:58 +00:00
Mikhail Shilkov cff25c2e89
Set the --continue-on-error flag with PULUMI_CONTINUE_ON_ERROR environment variable ()
If the `PULUMI_CONTINUE_ON_ERROR` environment variable is set to a
truthy value, it will have the effect of `--continue-on-error` for `up`
and `destroy` commands. This helps streamline setting the flag,
including through Automation API and Deployments.

Part of  but only for `--continue-on-error` that was specifically
requested by a customer. If this lands, we can apply the same pattern
elsewhere.
2024-06-25 08:28:37 +00:00
Fraser Waters bf806f5834
Add asset archive test to conformance tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/16092.

Hopefully the really last part of the fixes started by
https://github.com/pulumi/pulumi/pull/16100 and
https://github.com/pulumi/pulumi/pull/16119. This fixes the working
directory handling for asset archives.
2024-06-24 14:23:18 +00:00
Will Jones 283bbd23f6
Fix provider `Delete` ()
 (78c48204e0) introduced a
regression whereby outputs are mixed up with inputs for provider
`Delete`s. This commit fixes the behaviour so that it matches that
before the refactoring.

Fixes 

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-21 15:29:20 +00:00
Julien P 98b90f1902
Add packagemanager prompt to pulumi new for nodejs ()
https://github.com/pulumi/pulumi/pull/16346 introduced the capability to
query the language runtime for additional prompts. We use this to let
the user pick a package manager among npm, yarn and pnpm during `pulumi
new` when using the nodejs runtime.

When there is no explicitly configured package manager, we re-use the
previous behaviour for determining the package manager (check
`PULUMI_PREFER_YARN` env variable, look for lock files).

Defaults to `npm` when running `new` in non-interactive mode.
2024-06-21 11:35:06 +00:00
Germán Lena 64d9266deb
Update pu/pu to support the new settings pull command + new deployment file ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

- Add new deployment settings pull command: this command will pull the
deployment settings from pulumi cloud and generate the new deployment
file. For now it will be hidden until we have completed the whole
feature.
- Add support for the new deployment file, this will contain all the
deployment related information (for now just the settings), example:

```
settings:
  -executorContext: {}
    sourceContext:
      git:
        branch: main
        repoDir: .
    gitHub:
      repository: glena/test-action
      deployCommits: true
      previewPullRequests: false
    operationContext:
      preRunCommands: []
      operation: ""
      environmentVariables: {}
      options:
        skipInstallDependencies: false
        skipIntermediateDeployments: true
        shell: ""
        deleteAfterDestroy: false
        remediateIfDriftDetected: false
    agentPoolID: ...
```

Fixes https://github.com/pulumi/pulumi-service/issues/20306

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Komal <komal@pulumi.com>
Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-06-18 13:24:01 +00:00
Julien P a59b694515
Query language runtime for options during “pulumi new” ()
# Description

Fixes https://github.com/pulumi/pulumi/issues/16309

During `pulumi new` we query the language runtime using the new
`RuntimeOptionsPrompts` RPC call to get additional prompts to ask the
user.

<img width="900" alt="Screenshot 2024-06-07 at 14 28 58"
src="https://github.com/pulumi/pulumi/assets/387068/e68ef702-978b-47f7-9d4b-afdf10409ed8">

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"master","parentHead":"","trunk":"master"}
```
-->

---------

Co-authored-by: Will Jones <will@sacharissa.co.uk>
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-17 17:10:55 +00:00
Zaid Ajaj 2766475bd1
[cli/plugin] Fix plugin install command when plugin type is tool ()
# Description

When trying to install a plugin of type `tool` from GitHub, the Pulumi
convention is to have these plugins available in repositories named
`pulumi-tool-<name>` so that we can install them via the CLI as follows:
```
pulumi plugin install tool <name>
```
However, today this fails because we don't prefix the repository name
correctly with `"pulumi-tool-"`. This PR fixes that. Tested against
[Zaid-Ajaj/pulumi-tool-importer](https://github.com/Zaid-Ajaj/pulumi-tool-importer)

Also removes hardcoded plugin download URL for known converter plugins
of mine. These are moved to the `pulumi` organisation and no longer
require a separate URL.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-17 13:25:57 +00:00
Mikhail Shilkov 8b1882ceb8
Fix a panic when ignoring wildcard values with arrays of different length ()
The panic occurred during pulumi preview while calculating values for
ignored changes with a `*` wildcard, when compared arrays had different
length. We weren't checking the array boundaries carefully, and thus
panicing when going out of bounds.

After the fix, the same repro in the issue leads to a preview error

> cannot ignore changes to the following properties because one or more
elements of the path are missing:
"defaultActions[*].forward.targetGroups[*].weight"

This seems consistent with other code branches and tests, where `reset`
returns false for non-matching shapes. User's expectation may be that we
succeed in this case - curious to get a take from ones who implemented
the wildcard feature in the past.

Fixes 
2024-06-17 08:16:41 +00:00
Paul C. Roberts f0fffe6fb2
Better error messages for schema validation ()
This PR improves the error messages produced during project schema
validation so that, where possible, we suggest valid attribute names
that the user may have meant to type. For instance, if they provide a
"Name" attribute where we wanted "name" (lowercase "n"), we'll now say
so. Where there is not a close match, we'll enumerate the full list of
valid names to try and guide the user.

Matching is implemented using Levenshtein distances and ignores case.
Some examples of the new functionality:

* `{"Name": ...}` yields `project is missing a 'name' attribute; found
'Name' instead`
* `{..., "rutnime": ...}` yields `project is missing a 'runtime'
attribute; found 'rutnime' instead`
* `{..., "template": {"displayNameDisplayName": ...}, ...}` yields
`'displayNameDisplayName' not allowed; the allowed attributes are
'config', 'description', 'displayName', 'important', 'metadata' and
'quickstart'`

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-06-14 09:03:22 +00:00
Germán Lena 14293f7b28
Update pu/pu apitype to support Deployments yaml marshalling ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Add support for deployment settings YAML marshalling. This work is part
of the effort to manage deployment settings through the CLI and source
control.

Fixes https://github.com/pulumi/pulumi-service/issues/20275

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-06-13 12:20:58 +00:00
Luke Hoban f1e4b4ff94
Change `pulumi refresh` to report diff relative to desired state instead of relative to only output changes ()
Presently, the behaviour of diffing during refresh steps is incomplete,
returning only an "output diff" that presents the changes in outputs.
This commit changes refresh steps so that:

* they compute a diff similar to the one that would be computed if a
`preview` were run immediately after the refresh, which is more
typically what users expect and want; and
* `IgnoreChanges` resource options are respected when performing the new
desired-state diffs, so that property additions or changes reported by a
refresh can be ignored.

In particular, `IgnoreChanges` can now be used to acknowledge that part
or all of a resource may change in the provider, but the user is OK with
this and doesn't want to be notified about it during a refresh.
Importantly, this means that the diff won't be reported, but also that
the changes won't be applied to state.

The implementation covers the following:

* A diff is computed using the inputs from the program and then
inverting the result, since in the case of a refresh the diff is being
driven by the provider side and not the program. This doesn't change
what is stored back into the state, but it does produce a diff that is
more aligned with the "true changes to the desired state".
* `IgnoreChanges` resource options are now stored in state, so that this
information can be used in refresh operations that do not have access
to/run the program.
* In the context of a refresh operation, `IgnoreChanges` applies to
*both* input and output properties. This differs from the behaviour of a
normal update operation, where `IgnoreChanges` only considers input
properties.
* The special `"*"` value for `IgnoreChanges` can be used to ignore all
properties. It _also_ ignores the case where the resource cannot be
found in the provider, and instead keeps the resource intact in state
with its existing input and output properties.

Because the program is not run for refresh operations, `IgnoreChanges`
options must be applied separately before a refresh takes place. This
can be accomplished using e.g. a `pulumi up` that applies the options
prior to a refresh. We should investigate perhaps providing a `pulumi
state set ...`-like CLI to make these sorts of changes directly to a
state.

For use cases relying on the legacy refresh diff provider, the
`PULUMI_USE_LEGACY_REFRESH_DIFF` environment variable can be set, which
will disable desired-state diff computation. We only need to perform
checks in `RefreshStep.{ResultOp,Apply}`, since downstream code will
work correctly based on the presence or absence of a `DetailedDiff` in
the step.

### Notes

- https://github.com/pulumi/pulumi/issues/16144 affects some of these
cases - though its technically orthogonal
- https://github.com/pulumi/pulumi/issues/11279 is another technically
orthogonal issue that many providers (at least TFBridge ones) - do not
report back changes to input properties on Read when the input property
(or property path) was missing on the inputs. This is again technically
orthogonal - but leads to cases that appear "wrong" in terms of what is
stored back into the state still - though the same as before this
change.
- Azure Native doesn't seem to handle `ignoreChanges` passed to Diff, so
the ability to ignore changes on refresh doesn't currently work for
Azure Native.

### Fixes

* Fixes 
* Fixes 
* Fixes  
* Not quite , but likely replaces the need for that

Co-authored-by: Will Jones <will@sacharissa.co.uk>
2024-06-12 16:17:05 +00:00
Ian Wahbe 78c48204e0
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) ()
Normalize methods on plugin.Provider to the form:

```go
Method(context.Context, MethodRequest) (MethodResponse, error)
```

This provides a more consistent and forwards compatible interface for
each of our methods.

---

I'm motivated to work on this because the bridge maintains a copy of
this interface: `ProviderWithContext`. This doubles the pain of dealing
with any breaking change and this PR would allow me to remove the extra
interface. I'm willing to fix consumers of `plugin.Provider` in
`pulumi/pulumi`, but I wanted to make sure that we would be willing to
merge this PR if I get it green.

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes # (issue)

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-07 19:47:49 +00:00
Julien P 8476c3f7f5
Pass ProgramInfo through to LanguageRuntime.About ()
# Description

To correctly determine which python executable we are using, we need the
ProgramInfo so we can determine which virtual environment is in use.

This PR updates the `About` rpc call to take `ProgramInfo` as argument.

Fixes https://github.com/pulumi/pulumi/issues/16299
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-06 08:21:46 +00:00
Mikhail Shilkov 642cb5b5c7
Revert "Prefer pluginDownloadURLOverrides over PluginDownloadURL specified in the package" ()
Reverts 
Resolves https://github.com/pulumi/pulumi/issues/16316
2024-06-04 17:37:34 +00:00
Julien P 578e0937a9
[Python] Move existing dependency installation and python command invocation to a Toolchain interface ()
# Description

This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.

Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-03 13:52:27 +00:00
Ian Wahbe eaeaa3fab7
Plugin interface cleanup ()
This PR is best reviewed commit by commit:
- bbe1fc6b2f exchanges `ParseTolerant` for
`Parse` in `Parameterize`.
- 76df18be56 updates the `Parameterize`
interface for forward compatibility.
- 746c057668 requires implementors of
`plugin.Provider` to make a forward compatibility choice explicitly.
This is similar to what we require for gRPC already. Since this will
release with bbe1fc6b2f, consumers will
already need to update their `plugin.Provider` implementor, minimizing
the disturbance.
2024-05-31 00:28:48 +00:00
Ian Wahbe e220f574d5
Abstract plugin.Parameterize away from the gRPC interface ()
`plugin.Provider` exists to provide a low-level abstraction *on top of*
the gRPC interface. It should not expose the gRPC types directly. The
key diff in this commit is:

```patch
-	Parameterize(
-		ctx context.Context, req *pulumirpc.ParameterizeRequest,
-	) (*pulumirpc.ParameterizeResponse, error)
+	Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error)
```

```patch
+type ParameterizeParameters interface {
+	isParameterizeParameters()
+}
+
+type (
+	ParameterizeArgs struct {
+		Args []string
+	}
+
+	ParameterizeValue struct {
+		Name    string
+		Version *semver.Version
+		// Value must be one of:
+		// - nil
+		// - bool
+		// - int, int32, int64
+		// - uint, uint32, uint64
+		// - float32, float64
+		// - string
+		// - []byte
+		// - map[string]interface{}
+		// - []interface{}
+		Value any
+	}
+)
+
+func (ParameterizeArgs) isParameterizeParameters()  {}
+func (ParameterizeValue) isParameterizeParameters() {}
```

This is the new interface exposed in `plugin`. The rest of the PR is
simply complying with the new interface.

While this change is technically breaking (since it was released in
v3.116.0), its a heavily experimental feature and the providers team (my
team) is the biggest consumer of `plugin`. Before this PR,
`Parameterize` was the *only* method that dirrectly exposed the gRPC
interface it was "abstracting".

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-30 03:17:49 +00:00
Pat Gavlin 7273bc02c9
[display] Enable WASM compilation ()
These changes contain some minor refactorings to conditionally disable
the use of packages that are cannot be built for `GOOS=js GOARCH=wasm`.
With these edits, `pkg/display` can be built targeting WASM.

These changes act as a safeguard to ensure that we are not adding
additional code that will _prevent_ building `pkg/display` for WASM
targets. They are not sufficient to produce a version of the display
renderer that is appropriate for actual use in a WASM environment:

- The current renderer API is not well-suited for use outside the
context of the CLI
- The current event stream format has no versioning data
- Actually building this code into a WASM module results in an
unpleasantly large file (70M uncompressed, 13M gzipped)

These changes also add a size gate for the built WASM module. The gate
is set to the 110% of the size of the WASM module as of this commit. Our
goal is to lower the size of the WASM module over time; as we do so we
will tighten this gate.

Part of .
2024-05-24 20:27:56 +00:00
Fraser Waters ab7acdb602
Add Paramaterize to provider interface ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the bare bones changes required to update the provider interface
for parametrization. Nothing in the engine, sdks, cli, or codegen makes
use of these new methods yet.

But providers team can start building on top of this interface. Note
that this is subject to change, while this is _likely_ the right design
for parametrised providers there is a chance that we need to edit this
interface before GA release.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-15 16:22:39 +00:00
Mikhail Shilkov 3e0aedeee2
Prefer pluginDownloadURLOverrides over PluginDownloadURL specified in the package ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Overriding plugin download URLs with compilation flags was originally
added in . Its intent was allowing our customers to override
download locations for all plugins, so that only trusted pre-approved
plugins could be downloaded.

Since then, we've added `PluginDownloadURL` for a package, which is the
default URL for that package's binary if it's shipped outside our Pulumi
org. Currently, `PluginDownloadURL` takes precedence over
`pluginDownloadURLOverrides`, which means it's impossible to override
third-party package binary locations.

This PR changes plugin source resolution to flip the priority of those
two. If an override matches regex, its URL will take priority over the
default `PluginDownloadURL` specified in the package.

I have added tests to verify `pluginDownloadURLOverrides` with and
without `PluginDownloadURL`. The second one fails before my change.

Resolves 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-13 14:35:44 +00:00
Fraser Waters dcf0a1024b
Revert "Revert "Run integration tests and dev builds with race detection" ()" ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This reverts commit 75340dd942.

Fixes https://github.com/pulumi/pulumi/issues/16018.

This re-enables the locking and race detection. The locking is more
finely scoped to not be held over provider methods like Read/Update.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-09 16:15:41 +00:00