When a provider with the same URN and ID already exists in the
destination snapshot, we can assume it's the provider that has been
copied over. This is e.g. useful when running the state move command
multiple times with different resources that have the same provider.
This also allows moving resources back and forth between stacks.
This required fixing `__convert` in Go programgen to handle expressions
that needed to result in input-y values. This also fixed the
"single-or-none" codegen test.
The `state move` command is designed to move resources with an
underlying physical resource. Otherwise it's easy enough to just
re-create a resource in the resulting program. Especially for providers,
we usually don't want to move them, but rather want to copy them over.
Since there is not much point in moving a provider, just disallow that
and error out if a user tries to do that.
In `pulumi up --continue-on-error`, we have to make sure to not delete
resources that are still needed in the snapshot because an earlier step
failed. E.g. if B depends on A, and A fails to be deleted, we need to
make sure B is not deleted as well, otherwise we get a snapshot
integrity failure because B is missing A.
We do this by checking the dependencies of B in the *current program*,
e.g. only checking `dep.Res()`. However it is possible that the
dependencies in the new program are not the same.
For example, say resource A depends on provider B. Now we change the
program and resource A depends on provider C. We try to `pulumi up
--continue-on-error` the latter program, but the Update of resource A
fails. But because the new resource A only depends on C, and not on B,
we go ahead and delete B, resulting in a snapshot integrity failure,
because A has not been updated.
Fix this by also checking the `res.Old()` for dependencies, and not
deleting those either if `res` had a failure.
Note that this isn't an issue for creates/updates, because we only need
to check wheter a *new* resource is dependent on the existing resource.
Fixes https://github.com/pulumi/pulumi/issues/16720
Currently when a providers Parent is not moved, we just try to rewrite
the current one using the regular URN rewrite mechanism. This is not
correct.
Instead, we need to make sure to do the same thing as we do for regular
resources. Namely if the parent is not being moved into the destination,
we want the parent to be the root stack of the destination.
Enables the l1-main conformance test for Go. This required fixing up Go
programgen to respect the "main" attribute in the projects Pulumi.yaml
if present.
Change the output for resources to be moved from
```
Planning to move the following resources from dev to dev2:
urn:pulumi:dev::aws-ts::pulumi:providers:aws::Provider_A
urn:pulumi:dev::aws-ts::aws:iam/role:Role::Role_A
Do you want to perform this move? [Use arrows to move, type to filter]
yes
> no
```
to
```
Planning to move the following resources from dev to dev2:
- urn:pulumi:dev::aws-ts::pulumi:providers:aws::Provider_A
- urn:pulumi:dev::aws-ts::aws:iam/role:Role::Role_A
Do you want to perform this move? [Use arrows to move, type to filter]
yes
> no
```
for better readability.
This commit extends PCL so that it knows about the `DeletedWith`
resource option. With this, we can give `pulumi convert` the ability to
preserve `DeletedWith` resource options when converting programs.
Part of https://github.com/pulumi/pulumi-yaml/pull/437
When converting a project from one language to another, `pulumi convert`
will default to producing a project named after the source directory. So
if one converts a project in `some_directory`, the resulting
`Pulumi.yaml` will specify `name: some_directory`. This is generally the
right behaviour, since conversion often implies that the target will
supplant the source (and thus should use the same name). However, there
are cases where it can be useful to override this default. This commit
takes @FonsecaGoncalo's work in #15034 and extends `pulumi convert` so
that it accepts an optional `--name` argument, which if supplied (and
valid) will be used instead. As well as rebasing this work, we add a
couple of tests to make sure that both cases work.
Fixes#14645Closes#15034
Co-authored-by: Gonçalo Fonseca <goncalofnsc@gmail.com>
When printing an error we already prefix it with a `error:` prefix.
Therefore new errors should not include that prefix again, otherwise
we'll end up with a line like:
error: error: no changes were expected but changes occurred
Just a small cleanup I noticed while testing something else.
(From a quick look at `git grep "\"error:"` I couldn't find other
instances of this)
Cobra has its own error reporting when a command returns an error.
However we duplicate that in `main`.
Usually we don't notice this, because we have our own error reporter
that exits the program early. However in some cases, such as when the
command is misspelled, that error reporter doesn't kick in, and we end
up with a duplicated error output.
We can just rely on the cobra one in that case, instead of duplicating
that.
Fixes: https://github.com/pulumi/pulumi/issues/16610
There are a number of use cases for `Delete` steps in a Pulumi
operation. Aside from direct deletions, where a resource has been
removed from a program, they are also a key component of _replacements_,
whereby a resource has changed in a manner where it cannot simply be
updated in the provider but instead must be completely removed and
reconstructed. In such cases, Pulumi offers two options:
* "Delete after replace", in which a new resource is first created
before the old one is deleted.
* "Delete before replace", in which the old resource is first deleted
before a replacement is created.
Delete-after-replace is the default, since where possible it is
typically the preferred option by users, enabling e.g. zero-downtime
deployments. However, there are cases where delete-after-replace is not
possible (e.g. because the new and old resources would clash in some
manner), and so delete-before-replace is an option that can be opted
into.
In cases where the deletion must happen first, we must be careful how we
handle the Pulumi state. Either or both of the delete and create calls
could fail, and we always want a state file that tells us how to resume
a failed call to yield the desired outcome. In the case of
delete-before-replace operations, a key component of resumable state is
the `PendingReplacement` field on a resource. `PendingReplacement`
indicates that a resource has been deleted in the provider, but that
this deletion is part of a replacement (and thus that a create call will
subsequently occur). In this way, the deleted resource can remain in the
state file throughout the operation, meaning that e.g. resources that
depend on the deleted resource won't have their dependencies violated
(causing a snapshot integrity error).
Alas, until this point, `PendingReplacement` was set unconditionally on
the creation of a delete-before-replace step, meaning that if the
provider delete failed, we'd elide the delete on a retry and end up with
a bad state failing snapshot integrity checks. This commit fixes this by
moving the setting of `PendingReplacement` inside `DeleteStep.Apply`, so
that it occurs only if the provider delete call succeeds. It also adds a
lifecycle test to test this case and hopefully guard against
regressions.
Fixes#16597
Part of #16667
While looking at https://github.com/pulumi/pulumi/issues/16469, I
noticed a lot of the missing errors were a bit repetitive for the
not-a-missing-error case and disable-plugin-acquisition case. So this
rejigs the if statements a bit to make them all a bit more standard of
"if not a missing error, or if disable plugin acquisition is set, then
just err immediately, else retry".
Enable the l1-empty test for Go.
This required some changes in reporting versions from the Go language
host (replaced dependencies don't really have a version anymore, because
they've been replaced by a local artefact and that doesn't contain any
version info itself). This then cascaded that the conformance test had
to be less strict about checking versions from GetDependencies because
Go now returns empty versions for these local deps.
Currently when no resources are moved, we just show no resources being
moved, but let the command continue. This can be quite confusing to the
user. Error out in that case instead. Also when an argument doesn't
match any resources in the source snapshot, we now warn the user about
that.
When generating literals for inputs that have TypedDict types, we want
to use the pythonic names (snake_case) for keys.
We also have to take care of tracking the fact that we’re inisde a
TypedDict for nested dicts.
Fixes https://github.com/pulumi/pulumi/issues/16646
Currently when we have stacks with no snapshot, `pulumi state move`
fails because it tries to use a nil pointer. Handle this scenario
correctly by:
- erroring out if there is no snapshot in the source stack. In this case
there are no resources that can be moved, so there's nothing more that
we can do other than showing the error.
- creating a snapshot in the destination stack if necessary. It's valid
to move a resource to a currently empty stack, so we'll make this work.
At some point while moving the code around we lost the output for the
resources to be moved. We would look for them in the source snapshot for
correct ordering, but the code happened after the URN renaming, so the
resources would not match the ones in the source snapshot anymore.
Fix this by using the resourcesToMoveOrdered list to print the
resources, and also moving this code a bit earlier, as we want to show
the resource URNs from the source snapshot, instead of the rewritten
ones.
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.
When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.
The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
This fixes go package generation so users don't _have_ to set the import
base path explicitly in the go info part of the schema. This only really
helps our packages (because we assume the path is "github.com/pulumi")
but it fixes some of the generated test data and should also work for
local SDKs, where they aren't _really_ at "github.com/pulumi" but also
that it doesn't matter what URL is used as long as it's a valid one.
Fixes https://github.com/pulumi/pulumi/issues/13756
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
Clean up the temporary `PULUMI_HOME` directory we create during a
program test.
This is necessary to reclaim the disk space of the plugins that were
downloaded
during the test.
In pulumi-aws we started seeing test failures because the CI runners
started running out of disk space due to plugins in `PULUMI_HOME` not
being cleaned up.
Rather than writing a go.mod with string lerping, this uses the
"golang.org/x/mod/modfile" to construct and render the go.mod files we
generate for programs.
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
The test now checks that destroy works, and also that the parameterized
provider state is what we expect.
This required flipping around some of the state managment around reading
and writing provider inputs.
When running `pulumi up --continue-on-error`, we can't bring up new
resources that have dependencies that have failed, or have been skipped.
Since the resource would have a failed dependency, that dependency would
not be in the snapshot, resulting in a snapshot integrity failure. Also
it simply does not make sense to `up` a resource that has a failed
dependency.
We took care of that for the regular dependency relationship, however at
the time we missed doing the same for other types of dependencies,
namely parent-child relationships, deleted with relationships and
property dependencies.
This can result in snapshot integrity failures for example when a parent
fails to be created, but we still do the resource creation of the child,
such as what happened in https://github.com/pulumi/pulumi/issues/16638.
Fix this by skipping the step when a resource with any type of
dependency relationship fails or is skipped beforehand.
Fixes https://github.com/pulumi/pulumi/issues/16638
When a policy pack is nested within a project, or vice-versa, install
the appropriate dependencies, based on where the command is run from.
If there are both policy pack and project files in the parent
directories of the current working directory, pick the one that is
closest.
Fixes https://github.com/pulumi/pulumi/issues/16605
Correct the usage text, and mark the command not as non-experimental.
The command should be ready for regular usage, so I think the
"EXPERIMENTAL" comment can be dropped now.
We've recently introduced resource transforms, which allow users to
update any resource options and properties for the duration of a program
using a callback. We want to introduce similar functionality for Invokes
(and eventually also StreamInvokes, Read and Calls). This can help users
e.g. set default providers through transforms consistently for all
components.
While this PR only implements the engine parts of invoke transforms, the
API for this will look very similar to what the API for resource
transforms looks like. For example in TypeScript:
```
pulumi.runtime.registerInvokeTransform(args => {
[...]
});
```
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
Part of pulumi/pulumi-yaml#599.
Pulumi YAML uses the schema defined here to type check outputs from the
`pulumi:pulumi:StackReference` resource. This changes the type of
`outputs` from `map[string]string` to `map[string]any`, permitting list,
map, and numeric outputs to be used as inputs to other resources.
To release this fix, we will need to release Pulumi twice and YAML once
if we are to use our ordinary release process:
1. Merge this PR.
2. Release sdk and pkg dependencies with this update applied.
3. Merge https://github.com/pulumi/pulumi-yaml/pull/600
4. Create and merge a PR to update Pulumi YAML's dependency on
`github.com/pulumi/pkg/v3`, as YAML links to the schema loader and will
read the updated schema here:
7f48ca370d/pkg/codegen/schema/loader.go (L136-L140)
5. Release Pulumi YAML
6. Create and merge a PR to update the YAML language plugin shipped with
Pulumi
7. Release Pulumi
Change how the deployment settings configures git repos to support
directories that are not git repositories
Fix https://github.com/pulumi/pulumi-service/issues/20675
---------
Co-authored-by: Levi Blackstone <levi@pulumi.com>
When moving state it can be useful to also include all the parents. Add
a flag to give users that option.
Using this flag, all parents of all the resources being moved will be
included in the move.
When the user tries to move a resource that would end up being a
duplicate in the destination stack, we need to prevent that, to avoid a
broken state file.
Error out in this case.
Depends on: https://github.com/pulumi/pulumi/pull/16543
Making this change has a couple of benefits.
Firstly (and honestly the main one) is it make codegen much simpler. We
just to emit a base64 string and/or other embedded byte array to the
generated SDK. Currently we need emit a full `proto.Value` expression
for each language, and that's not hard but it's also not trivial and
means more combinations of things per test per language.
Secondly it will allow providers to use more efficient encodings of
their parameter than JSON if there is one. I imagine some providers
might make the parameter value a protobuf message and parse that
(similar to what we do for transform functions), but they can easily
fallback to just treating the bytes as a JSON string if they want.
The only downside to this is the value is obfuscated in the generated
SDK and in the state file. Neither of those are really expected to be
viewed by users, so this feels like a minor loss.
Add more tests for moving children, and for breaking
dependencies/property dependencies/deleted with attributes.
This also fixes a couple of bugs that were discovered while testing.
Depends on: https://github.com/pulumi/pulumi/pull/16527
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
The existing overlays (e.g. Chart v3 in Kubernetes, or CallbackFunction
in AWS) are not available in every language Pulumi supports. This often
confuses users because the generated docs include all languages Pulumi
supports (e.g. see
https://github.com/pulumi/pulumi-kubernetes/issues/2181).
To solve that problem, this change adds a new optional parameter to the
schema that allows configuring the languages an overlay (resource or
function) supports.
To support this in docsgen the existing Language Chooser
(`LangChooserLanguages`) of resources is made configurable and extended
to functions.
Note: This doesn't support resource methods right now. They'll need
extra handling because and overlay resource method might not support all
of the languages its resource supports. I'll tackle this in a follow up
PR.
Here's a screenshot of how this will look like for the Helm v3 chart for
example:
<img width="1046" alt="Screenshot 2024-07-01 at 16 11 23"
src="https://github.com/pulumi/pulumi/assets/2453580/b1a1365a-6dee-4099-829a-2859639a4c8c">
The PR contains the following commits. I'd recommend to look at the
first three ones and then check the regenerated golden files in the last
one:
- **Add schema parameter to constrain supported languages for overlays**
- **Update developer docs and changelog**
- **Refactor LanguageChooser and always pass supported languages**
- **Regenerate testdata**
relates to #13231
Currently, when generating docs for the AWSX package, Go constructor
syntax examples were not generated due to a panic. This causes all
constructor examples to not be emitted in the docs.
The panic occurred when trying to get the version of referenced packages
in the PCL program to emit import paths. However, _transitive_ package
references were not resolved in the PCL binder when binding resource
types. This PR fixes the problem such that now we do find the transitive
package references from any input or output property of the resources
being bound.
In the case of the awsx package, the top-level package is awsx itself
and aws is the transitive dependency. Anytime in codegen we call
`program.PackageReferences()` we should get both of them. Added a unit
test for this as well.
Testing this fix locally against the awsx package showed constructor
examples being generated for every language, however there was still a
problem in the _formatting_ of Go code which is also fixed (see change
in `gen_program_expressions.go`)
Resolves part of #16463
For resources that don't have a schema, we emit a best guess for how
they are declared and how imports should be emitted. For these resources
we emit `index.{ResourceName}` if that resource is from the index
module, however this is not how resources are usually generated, instead
we should generate `{packageName}.{ResourceName}` for resources from the
index module which is what this PR resolves.
Fixes#16461
When passing `yes` in interactive mode (or non-interactive mode),
language specific options should use their deafult values without
prompting the user.