This takes https://github.com/pulumi/pulumi/pull/16392 and updates local
SDK generation for nodejs and adds parametrisation support for invokes
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Co-authored-by: Will Jones <will@sacharissa.co.uk>
No languages make use of this yet, but this will be needed for some
languages for good local SDK support. E.g. we're planning on adding a
postinstall script for nodejs local packages.
Fixes https://github.com/pulumi/pulumi/issues/16889
Currently `resource.NewPropertyMapFromMap` will drop any
`resource.PropertyValue`s from the input map. This is demonstrated
succinctly by Ian in the linked ticket:
```go
package main
import (
"fmt"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
)
func main() {
fmt.Println(resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
"foo": resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
"bar": resource.NewNumberProperty(0),
})),
})))
fmt.Println(resource.NewObjectProperty(resource.NewPropertyMapFromMap(map[string]any{
"foo": map[string]any{"bar": 0},
})))
}
```
```
{map[foo:{map[]}]}
{map[foo:{map[bar:{0}]}]}
```
This PR updates `NewPropertyMapFromMap` to instead flow these values
through directly resulting in an output of:
```
{map[foo:{map[bar:{0}]}]}
{map[foo:{map[bar:{0}]}]}
```
Make this more clear by passing plain value of `semver.Version` instead
of a pointer for these cases.
This also gets rid of some incorrect empty string checks which would
have passed a nil version onwards rather than error'ing (as they should
have).
We don't need to search upwards for PolicyPack files, we already know
that there aren't any since we don't recurse into policy packs. The
directory traversal is plenty fast on Linux, however it is much slower
on Windows. Node modules often have fairly wide and deep directory
structures, so this can considerably slow down any Pulumi operation that
runs `GetRequiredPlugins`.
Fixes https://github.com/pulumi/pulumi/issues/16863
On an `was-typescript` example, we can see the impact with
`Measure-Command {start-process pulumi -argumentlist "about" -Wait}`,
which goes from 25 seconds to 2 seconds on an Azure Windows VM.
### Motivation
Pulumi plugin binaries can be downloaded by the CLI from multiple
sources. By default, it's downloaded from Pulumi's GitHub releases or
get.pulumi.com, but plugins can also specify their binary sources via
the `PluginDownloadURL` schema option. They can point to custom GitHub,
Gitlab, or HTTP locations.
Enterprise customers ask for a way to isolate the CLI from downloads
from random locations and to configure the CLI to go to their internal
pre-approved artefact location instead. This way, Pulumi can run in
"air-gapped" environments (which still have access to Cloud APIs, of
course).
Related issues:
- https://github.com/pulumi/pulumi/issues/14459
- https://github.com/pulumi/pulumi/issues/16240
Currently, there is a basic mechanism to do so via the variable
`pluginDownloadURLOverrides`, but it has two major limitations:
- The variable value is set via a compile-time flag, so it requires a
custom build of the CLI
- The overrides are based on the plugin name, so the rules must be
defined without access to the original URL, which makes it hard to
provide universal rules and still distinguish between first-party,
public third-party, or private in-house plugins
- We ignore overrides for all plugins that have `PluginDownloadURL` set
- Overrides can set a plugin replacement redirect only to HTTP(s)
addresses
### Proposal
This PR makes two sets of changes:
1. It allows passing overrides via the
`PULUMI_PLUGIN_DOWNLOAD_URL_OVERRIDES` environment variable. The
compile-time flag is still supported, but the env var takes priority.
More configuration levers could be supported, but it not clear if we
have good ones until [Support .pulumirc file for global
config](https://github.com/pulumi/pulumi/issues/13484) is implemented. I
don't expect users to want to set this via their stack configs, but I'm
curious what others think. In any case, more sources can be added later.
2. The overrides now apply based on the original download URL, not just
on plugin names. Actually, it's the base URL of a download source that
is passed to the regexp matcher. Examples of possible options are:
- `github://api.github.com/pulumi/pulumi-xyz` for a first-party plugin
(note that we don't pass `get.pulumi.com`
- `github://api.github.com/pulumiverse/pulumi-grafana` for a community
plugin that sets `PluginDownloadURL`
- `gitlab://gitlab-host/proj-name` for a community plugin hosted on
Gitlab
- `https://example.com/downloads/` for HTTP sources
So, the override
`^github://api.github.com/pulumi/pulumi-xyz=https://example.com/downloads/pulumi-xyz/`
will override the single provider URL from our GitHub releases to the
given HTTP location.
On top of that, regular expressions may contain name groups to capture
and use templated values. For example,
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/${org}/${repo}`
captures any GitHub plugin and redirects it to its corresponding HTTP
location. Group indices are also supported: the above override can also
be written as
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/$1/$2`,
with `$0` meaning the full match.
The override URLs have the same semantics as `PluginDownloadURL`, so
they can point to GitHub, Gitlab, HTTP, or anything we introduce in the
future.
### Impact
Technically, this is a breaking change, because name-based overrides
will stop working. However, we are fairly certain that we have a single
customer using the existing compile-time approach, and they indicated
that they don't need the name-based overrides if they have URL-based
overrides. I reviewed this PR with them and made sure they can migrate
immediately after the change is released.
Backwards compatibility is slightly tricky, because we'd need to keep
name-based override _and_ not applying them to third-party plugins. But
we can do it if necessary.
Resolve#16240
Fixes#15410Fixes#13339
## Problem Context
When using `pulumi import` we generate code snippets for the resources
that were imported. Sometimes the user specifies `--parent
parentName=URN` or `--provider providerName=URN` which tweak the parent
or provider that the imported resources uses. When using `--parent` or
`--provider` the generated code emits a resource option `parent =
parentName` (in case of using `--parent`) where `parentName` is an
unbound variable.
Usually unbound variables would result in a _bind_ error such as `error:
undefined variable parentName` when type-checking the program however in
the import code generation we specify the bind option
`pcl.AllowMissingVariables` which turns that unbound variable errors
into warnings and code generation can continue to emit code.
This is all good and works as expected. However in the issues linked
above, we do get an _error_ for unbound variables in generated code even
though we specified `AllowMissingVariables`.
The problem as it turns out is when we are trying to generate code via
dynamically loaded `LangaugeRuntime` plugins. Specifically for NodeJS
and Python, we load `pulumi-language-nodejs` or `pulumi-language-python`
and call `GenerateProgram` to get the generated program. That function
`GenerateProgram` takes the text _SOURCE_ of the a bound program (one
that was bound using option `AllowMissingVariables`) and re-binds again
inside the implementation of the language plugin. The second time we
bind the program, we don't pass it the option `AllowMissingVariables`
and so it fails with `unboud variable` error.
I've verified that the issue above don't repro when doing an import for
dotnet (probably same for java/yaml) because we use the statically
linked function `codegen/{lang}/gen_program.go -> GenerateProgram`
## Solution
The problem can be solved by propagating the bind options from the CLI
to the language hosts during import so that they know how to bind the
program. I've extended the gRPC interface in `GenerateProgramRequest`
with a property `Strict` which follows the same logic from `pulumi
convert --strict` and made it such that the import command sends
`strict=false` to the language plugins when doing `GenerateProgram`.
This is consistent with `GenerateProject` that uses the same flag. When
`strict=false` we use `pcl.NonStrictBindOptions()` which includes
`AllowMissingVariables` .
## Repro
Once can test the before and after behaviour by running `pulumi up
--yes` on the following TypeScript program:
```ts
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";
export class MyComponent extends pulumi.ComponentResource {
public readonly randomPetId: pulumi.Output<string>;
constructor(name: string, opts?: pulumi.ComponentResourceOptions) {
super("example:index:MyComponent", name, {}, opts);
const randomPet = new random.RandomPet("randomPet", {}, {
parent: this
});
this.randomPetId = randomPet.id;
this.registerOutputs({
randomPetId: randomPet.id,
});
}
}
const example = new MyComponent("example");
export const randomPetId = example.randomPetId;
```
Then running `pulumi import -f import.json` where `import.json` contains
a resource to be imported under the created component (stack=`dev`,
project=`importerrors`)
```ts
{
"nameTable": {
"parentComponent": "urn:pulumi:dev::importerrors::example:index:MyComponent::example"
},
"resources": [
{
"type": "random:index/randomPassword:RandomPassword",
"name": "randomPassword",
"id": "supersecret",
"parent": "parentComponent"
}
]
}
```
Running this locally I get the following generated code (which
previously failed to generate)
```ts
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";
const randomPassword = new random.RandomPassword("randomPassword", {
length: 11,
lower: true,
number: true,
numeric: true,
special: true,
upper: true,
}, {
parent: parentComponent,
});
```
The `ignoreChanges` resource option accepts a list of paths into a
resource that should be ignored when computing whether or not something
has changed. For example:
```typescript
const r = new Resource(
"r",
{
a: "a",
b: [1, 2],
c: { d: "d" },
},
{
ignoreChanges: [
"a",
"b[*]",
],
}
)
```
Here, when diffing `r`, Pulumi will ignore changes to `a` (due to the
path `"a"`) as well as changes to any element of `b` (due to the
_wildcard_ path `"b[*]"`). Under the hood, `ignoreChanges` is
implemented partly by "resetting" pieces of a resource's state to older
values -- that is, rather than ignoring changes that might be reported,
Pulumi will undo the changes before they are used altogether.
In #16406, a panic encountered when resetting children of arrays of
different length was fixed. This commit extends this fix to cover panics
that occur when simply resetting arrays themselves (e.g. resetting
`[1,2]` to `[1,2,3]`). As part of this, some test cases are renamed to
convey that the are actually testing the children of arrays or
wildcard-selected objects, in order to accommodate new test cases for
this panic.
Fixes#16724
We have this deepcopy utility in our public API, but it does not work
properly on structs with unexported values. Make a note of that, and
also disallow OutputState from being copied, since OutputState has
required unexported fields that we can't deepcopy.
Not sure if we should special case `OutputState` here, which I've done
since that has come up in particular, or maybe even disallow
deepcopy'ing any values that have unexported fields? That would be a
breaking change though, so I've shied away from it for now.
Fixes https://github.com/pulumi/pulumi/issues/16634
This adds the framework for running conformance tests for Go. Currently
_all_ the tests are skipped, but I'll raise small PRs on top of this one
to start fixing some of the issues.
urlAuthParser.Parse caches auth methods. To avoid caching the auth
method when running into an error, we first checked that the auth method
is not nil. However we ran into a classic Go issue where a nil value of
a concrete type is assigned to a variable of an interface type, making
it not equal to nil.
https://go.dev/doc/faq#nil_errorhttps://go.dev/play/p/AOSdCWd3XC1
Fixes https://github.com/pulumi/pulumi/issues/16637
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.
When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.
The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
This commit adds a test to ensure that provider `Delete`s are passed
with the correct set of parameters. This is a regression test for
#16440, as part of the follow-up to #16441 in #16484. Assuming we are
happy with this level/style of test, we should consider adding tests for
the other provider methods also.
Fixes#16484
Change how the deployment settings configures git repos to support
directories that are not git repositories
Fix https://github.com/pulumi/pulumi-service/issues/20675
---------
Co-authored-by: Levi Blackstone <levi@pulumi.com>
Making this change has a couple of benefits.
Firstly (and honestly the main one) is it make codegen much simpler. We
just to emit a base64 string and/or other embedded byte array to the
generated SDK. Currently we need emit a full `proto.Value` expression
for each language, and that's not hard but it's also not trivial and
means more combinations of things per test per language.
Secondly it will allow providers to use more efficient encodings of
their parameter than JSON if there is one. I imagine some providers
might make the parameter value a protobuf message and parse that
(similar to what we do for transform functions), but they can easily
fallback to just treating the bytes as a JSON string if they want.
The only downside to this is the value is obfuscated in the generated
SDK and in the state file. Neither of those are really expected to be
viewed by users, so this feels like a minor loss.
We've had quite a few test flakes that happen because of TLS handshake
timeouts to the service backend. These are usually during stack creation
(which happens a lot during tests), which are POST requests.
POST requests are not generally safe to retry, because the request could
already have been sent to the backend and an action taken, but the
client couldn't read the response. However when there is a TLS handshake
failure, the request can never have made it to the backend, so these
errors are safe to retry.
This should hopefully help with
https://github.com/pulumi/pulumi/issues/16529.
When a resource is moved between stacks in different projects we also
need to rewrite project part of the URNs. Do that here. Moving between
different projects already works by virtue of the requireStack function
supporting that when providing the fully qualified name of the stack.
There is some awkwardness here as old diy backends may not return a
project name, in which case we error out here. Curious if anyone has
thought about what to do here. Is erroring out the best we can do? What
happens if we don't rewrite the project name in this case?
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
Add new deployment run command
Fixes https://github.com/pulumi/pulumi-service/issues/20500
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
Implement a skeleton command for `pulumi state move`. This can so far
only more a single resource from one stack to another, including copying
the provider it needs, and leave valid state files behind.
These state files are not written to the backend yet, nor is there any
kind of UI. Further PRs will build on top of this.
The command is intentionally commented out so this can be merged
independently.
---
Based on discussions in the last retro, I'm trying something new here
and try to split the PR up some more into chunks that can be
individually merged, but are not completely ready yet. The code added
here is essentially dead code at this point, but provides a skeleton to
incrementally add pieces.
To give an idea of where this is headed, the next things I'm planning to
work on here are:
- Writing the state files to the backend
- Make sure this works with stacks from two different projects
- Add a preview phase that asks for user confirmation
- Add output for moved resources and warnings for dependencies that are
being broken
- Implement `--include-parents` flag
- Implement convenience flags (`--yes`, `--skip-preview` etc.)
I'd love some early thoughts on this PR, and also meta thoughts on
splitting PRs up this way.
It is unsafe to call `print` in a Python signalhandler. Instead we set a
flag and check this in the main busy loop.
Fixes https://github.com/pulumi/pulumi/issues/16472
---------
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
If the `PULUMI_CONTINUE_ON_ERROR` environment variable is set to a
truthy value, it will have the effect of `--continue-on-error` for `up`
and `destroy` commands. This helps streamline setting the flag,
including through Automation API and Deployments.
Part of #16405 but only for `--continue-on-error` that was specifically
requested by a customer. If this lands, we can apply the same pattern
elsewhere.
#16302 (78c48204e0) introduced a
regression whereby outputs are mixed up with inputs for provider
`Delete`s. This commit fixes the behaviour so that it matches that
before the refactoring.
Fixes#16440
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
https://github.com/pulumi/pulumi/pull/16346 introduced the capability to
query the language runtime for additional prompts. We use this to let
the user pick a package manager among npm, yarn and pnpm during `pulumi
new` when using the nodejs runtime.
When there is no explicitly configured package manager, we re-use the
previous behaviour for determining the package manager (check
`PULUMI_PREFER_YARN` env variable, look for lock files).
Defaults to `npm` when running `new` in non-interactive mode.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
- Add new deployment settings pull command: this command will pull the
deployment settings from pulumi cloud and generate the new deployment
file. For now it will be hidden until we have completed the whole
feature.
- Add support for the new deployment file, this will contain all the
deployment related information (for now just the settings), example:
```
settings:
-executorContext: {}
sourceContext:
git:
branch: main
repoDir: .
gitHub:
repository: glena/test-action
deployCommits: true
previewPullRequests: false
operationContext:
preRunCommands: []
operation: ""
environmentVariables: {}
options:
skipInstallDependencies: false
skipIntermediateDeployments: true
shell: ""
deleteAfterDestroy: false
remediateIfDriftDetected: false
agentPoolID: ...
```
Fixes https://github.com/pulumi/pulumi-service/issues/20306
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Komal <komal@pulumi.com>
Co-authored-by: Will Jones <will@sacharissa.co.uk>
# Description
Fixes https://github.com/pulumi/pulumi/issues/16309
During `pulumi new` we query the language runtime using the new
`RuntimeOptionsPrompts` RPC call to get additional prompts to ask the
user.
<img width="900" alt="Screenshot 2024-06-07 at 14 28 58"
src="https://github.com/pulumi/pulumi/assets/387068/e68ef702-978b-47f7-9d4b-afdf10409ed8">
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"master","parentHead":"","trunk":"master"}
```
-->
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
# Description
When trying to install a plugin of type `tool` from GitHub, the Pulumi
convention is to have these plugins available in repositories named
`pulumi-tool-<name>` so that we can install them via the CLI as follows:
```
pulumi plugin install tool <name>
```
However, today this fails because we don't prefix the repository name
correctly with `"pulumi-tool-"`. This PR fixes that. Tested against
[Zaid-Ajaj/pulumi-tool-importer](https://github.com/Zaid-Ajaj/pulumi-tool-importer)
Also removes hardcoded plugin download URL for known converter plugins
of mine. These are moved to the `pulumi` organisation and no longer
require a separate URL.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
The panic occurred during pulumi preview while calculating values for
ignored changes with a `*` wildcard, when compared arrays had different
length. We weren't checking the array boundaries carefully, and thus
panicing when going out of bounds.
After the fix, the same repro in the issue leads to a preview error
> cannot ignore changes to the following properties because one or more
elements of the path are missing:
"defaultActions[*].forward.targetGroups[*].weight"
This seems consistent with other code branches and tests, where `reset`
returns false for non-matching shapes. User's expectation may be that we
succeed in this case - curious to get a take from ones who implemented
the wildcard feature in the past.
Fixes#16403
This PR improves the error messages produced during project schema
validation so that, where possible, we suggest valid attribute names
that the user may have meant to type. For instance, if they provide a
"Name" attribute where we wanted "name" (lowercase "n"), we'll now say
so. Where there is not a close match, we'll enumerate the full list of
valid names to try and guide the user.
Matching is implemented using Levenshtein distances and ignores case.
Some examples of the new functionality:
* `{"Name": ...}` yields `project is missing a 'name' attribute; found
'Name' instead`
* `{..., "rutnime": ...}` yields `project is missing a 'runtime'
attribute; found 'rutnime' instead`
* `{..., "template": {"displayNameDisplayName": ...}, ...}` yields
`'displayNameDisplayName' not allowed; the allowed attributes are
'config', 'description', 'displayName', 'important', 'metadata' and
'quickstart'`
Co-authored-by: Will Jones <will@sacharissa.co.uk>
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
Add support for deployment settings YAML marshalling. This work is part
of the effort to manage deployment settings through the CLI and source
control.
Fixes https://github.com/pulumi/pulumi-service/issues/20275
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
Presently, the behaviour of diffing during refresh steps is incomplete,
returning only an "output diff" that presents the changes in outputs.
This commit changes refresh steps so that:
* they compute a diff similar to the one that would be computed if a
`preview` were run immediately after the refresh, which is more
typically what users expect and want; and
* `IgnoreChanges` resource options are respected when performing the new
desired-state diffs, so that property additions or changes reported by a
refresh can be ignored.
In particular, `IgnoreChanges` can now be used to acknowledge that part
or all of a resource may change in the provider, but the user is OK with
this and doesn't want to be notified about it during a refresh.
Importantly, this means that the diff won't be reported, but also that
the changes won't be applied to state.
The implementation covers the following:
* A diff is computed using the inputs from the program and then
inverting the result, since in the case of a refresh the diff is being
driven by the provider side and not the program. This doesn't change
what is stored back into the state, but it does produce a diff that is
more aligned with the "true changes to the desired state".
* `IgnoreChanges` resource options are now stored in state, so that this
information can be used in refresh operations that do not have access
to/run the program.
* In the context of a refresh operation, `IgnoreChanges` applies to
*both* input and output properties. This differs from the behaviour of a
normal update operation, where `IgnoreChanges` only considers input
properties.
* The special `"*"` value for `IgnoreChanges` can be used to ignore all
properties. It _also_ ignores the case where the resource cannot be
found in the provider, and instead keeps the resource intact in state
with its existing input and output properties.
Because the program is not run for refresh operations, `IgnoreChanges`
options must be applied separately before a refresh takes place. This
can be accomplished using e.g. a `pulumi up` that applies the options
prior to a refresh. We should investigate perhaps providing a `pulumi
state set ...`-like CLI to make these sorts of changes directly to a
state.
For use cases relying on the legacy refresh diff provider, the
`PULUMI_USE_LEGACY_REFRESH_DIFF` environment variable can be set, which
will disable desired-state diff computation. We only need to perform
checks in `RefreshStep.{ResultOp,Apply}`, since downstream code will
work correctly based on the presence or absence of a `DetailedDiff` in
the step.
### Notes
- https://github.com/pulumi/pulumi/issues/16144 affects some of these
cases - though its technically orthogonal
- https://github.com/pulumi/pulumi/issues/11279 is another technically
orthogonal issue that many providers (at least TFBridge ones) - do not
report back changes to input properties on Read when the input property
(or property path) was missing on the inputs. This is again technically
orthogonal - but leads to cases that appear "wrong" in terms of what is
stored back into the state still - though the same as before this
change.
- Azure Native doesn't seem to handle `ignoreChanges` passed to Diff, so
the ability to ignore changes on refresh doesn't currently work for
Azure Native.
### Fixes
* Fixes#16072
* Fixes#16278
* Fixes#16334
* Not quite #12346, but likely replaces the need for that
Co-authored-by: Will Jones <will@sacharissa.co.uk>
Normalize methods on plugin.Provider to the form:
```go
Method(context.Context, MethodRequest) (MethodResponse, error)
```
This provides a more consistent and forwards compatible interface for
each of our methods.
---
I'm motivated to work on this because the bridge maintains a copy of
this interface: `ProviderWithContext`. This doubles the pain of dealing
with any breaking change and this PR would allow me to remove the extra
interface. I'm willing to fix consumers of `plugin.Provider` in
`pulumi/pulumi`, but I wanted to make sure that we would be willing to
merge this PR if I get it green.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes # (issue)
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
To correctly determine which python executable we are using, we need the
ProgramInfo so we can determine which virtual environment is in use.
This PR updates the `About` rpc call to take `ProgramInfo` as argument.
Fixes https://github.com/pulumi/pulumi/issues/16299
Ref https://github.com/pulumi/pulumi/issues/15937
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
# Description
This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.
Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
This PR is best reviewed commit by commit:
- bbe1fc6b2f exchanges `ParseTolerant` for
`Parse` in `Parameterize`.
- 76df18be56 updates the `Parameterize`
interface for forward compatibility.
- 746c057668 requires implementors of
`plugin.Provider` to make a forward compatibility choice explicitly.
This is similar to what we require for gRPC already. Since this will
release with bbe1fc6b2f, consumers will
already need to update their `plugin.Provider` implementor, minimizing
the disturbance.
`plugin.Provider` exists to provide a low-level abstraction *on top of*
the gRPC interface. It should not expose the gRPC types directly. The
key diff in this commit is:
```patch
- Parameterize(
- ctx context.Context, req *pulumirpc.ParameterizeRequest,
- ) (*pulumirpc.ParameterizeResponse, error)
+ Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error)
```
```patch
+type ParameterizeParameters interface {
+ isParameterizeParameters()
+}
+
+type (
+ ParameterizeArgs struct {
+ Args []string
+ }
+
+ ParameterizeValue struct {
+ Name string
+ Version *semver.Version
+ // Value must be one of:
+ // - nil
+ // - bool
+ // - int, int32, int64
+ // - uint, uint32, uint64
+ // - float32, float64
+ // - string
+ // - []byte
+ // - map[string]interface{}
+ // - []interface{}
+ Value any
+ }
+)
+
+func (ParameterizeArgs) isParameterizeParameters() {}
+func (ParameterizeValue) isParameterizeParameters() {}
```
This is the new interface exposed in `plugin`. The rest of the PR is
simply complying with the new interface.
While this change is technically breaking (since it was released in
v3.116.0), its a heavily experimental feature and the providers team (my
team) is the biggest consumer of `plugin`. Before this PR,
`Parameterize` was the *only* method that dirrectly exposed the gRPC
interface it was "abstracting".
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
## Checklist
- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
These changes contain some minor refactorings to conditionally disable
the use of packages that are cannot be built for `GOOS=js GOARCH=wasm`.
With these edits, `pkg/display` can be built targeting WASM.
These changes act as a safeguard to ensure that we are not adding
additional code that will _prevent_ building `pkg/display` for WASM
targets. They are not sufficient to produce a version of the display
renderer that is appropriate for actual use in a WASM environment:
- The current renderer API is not well-suited for use outside the
context of the CLI
- The current event stream format has no versioning data
- Actually building this code into a WASM module results in an
unpleasantly large file (70M uncompressed, 13M gzipped)
These changes also add a size gate for the built WASM module. The gate
is set to the 110% of the size of the WASM module as of this commit. Our
goal is to lower the size of the WASM module over time; as we do so we
will tighten this gate.
Part of #13258.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This is the bare bones changes required to update the provider interface
for parametrization. Nothing in the engine, sdks, cli, or codegen makes
use of these new methods yet.
But providers team can start building on top of this interface. Note
that this is subject to change, while this is _likely_ the right design
for parametrised providers there is a chance that we need to edit this
interface before GA release.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
Overriding plugin download URLs with compilation flags was originally
added in #8798. Its intent was allowing our customers to override
download locations for all plugins, so that only trusted pre-approved
plugins could be downloaded.
Since then, we've added `PluginDownloadURL` for a package, which is the
default URL for that package's binary if it's shipped outside our Pulumi
org. Currently, `PluginDownloadURL` takes precedence over
`pluginDownloadURLOverrides`, which means it's impossible to override
third-party package binary locations.
This PR changes plugin source resolution to flip the priority of those
two. If an override matches regex, its URL will take priority over the
default `PluginDownloadURL` specified in the package.
I have added tests to verify `pluginDownloadURLOverrides` with and
without `PluginDownloadURL`. The second one fails before my change.
Resolves#16058
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This reverts commit 75340dd942.
Fixes https://github.com/pulumi/pulumi/issues/16018.
This re-enables the locking and race detection. The locking is more
finely scoped to not be held over provider methods like Read/Update.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->