By default, `pulumi destroy` removes all resources within a stack but
leaves the stack and its configuration intact. If one passes the
`--remove` option to `destroy`, however, the stack and its configuration
will also be removed once the resources within the stack have been
deleted. This commit updates the work of @Moon1706 in #11080 to add
`remove` as an option to the Go, NodeJS and Python Automation API SDKs'
`destroy` methods, which then perform an analogous clean-up.
Closes#11080
---------
Co-authored-by: Nikita Sharaev <n.p.sharaev@tinkoff.ru>
Currently when no resources are moved, we just show no resources being
moved, but let the command continue. This can be quite confusing to the
user. Error out in that case instead. Also when an argument doesn't
match any resources in the source snapshot, we now warn the user about
that.
When generating literals for inputs that have TypedDict types, we want
to use the pythonic names (snake_case) for keys.
We also have to take care of tracking the fact that we’re inisde a
TypedDict for nested dicts.
Fixes https://github.com/pulumi/pulumi/issues/16646
When converting from a requirements.txt to a pyproject.toml, we were
previously indenting each logical level in the file. Disable the
indentation to match the style of pyproject.toml files as generated
directly by Poetry.
Fixes https://github.com/pulumi/pulumi/issues/16657
urlAuthParser.Parse caches auth methods. To avoid caching the auth
method when running into an error, we first checked that the auth method
is not nil. However we ran into a classic Go issue where a nil value of
a concrete type is assigned to a variable of an interface type, making
it not equal to nil.
https://go.dev/doc/faq#nil_errorhttps://go.dev/play/p/AOSdCWd3XC1
Fixes https://github.com/pulumi/pulumi/issues/16637
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
Currently when we have stacks with no snapshot, `pulumi state move`
fails because it tries to use a nil pointer. Handle this scenario
correctly by:
- erroring out if there is no snapshot in the source stack. In this case
there are no resources that can be moved, so there's nothing more that
we can do other than showing the error.
- creating a snapshot in the destination stack if necessary. It's valid
to move a resource to a currently empty stack, so we'll make this work.
At some point while moving the code around we lost the output for the
resources to be moved. We would look for them in the source snapshot for
correct ordering, but the code happened after the URN renaming, so the
resources would not match the ones in the source snapshot anymore.
Fix this by using the resourcesToMoveOrdered list to print the
resources, and also moving this code a bit earlier, as we want to show
the resource URNs from the source snapshot, instead of the rewritten
ones.
CPython doesn't optimize tail recursion, and because of that has a
fairly low recursion limit set to avoid stack overflows. This also means
that where we rely on recursion we are prone to raising
`RecursionError`s. In this particular case, `_add_dependency` relies on
recursion for finding all the child dependencies.
When deeply nested trees of `ComponentResources` are used,
`_add_dependencies` can end up raising such an error.
Rewrite the function to be iterative instead of recursive to avoid that.
Fixes https://github.com/pulumi/pulumi/issues/16328
(I didn't manage to reproduce the exact stacktrace of the above issue,
so there might be another thing hidden there. But this should help with
it either way)
The Python Automation API SDK serializes project settings (the contents
of `Pulumi.yaml` files) using Python's `pyyaml` package. Project
settings in the Python Automation API SDK are represented as instances
of the `ProjectSettings` class. By default, `pyyaml` will serialize
class instances as YAML objects "tagged" with a string that indicates
their class. This is so that, upon deserialization, it can construct
objects of the appropriate class (as opposed to, just dictionaries). As
an example, the following Python program:
```python
class Person:
def __init(self, name: str, age: int) -> None:
self.name = name
self.age = age
will = Person("will", 37)
yaml.dump(will)
```
will produce the following YAML:
```yaml
!!python/object:__main__.Person
age: 37
name: will
```
The string `!!python/object:__main__.Person` is the _tag_ indicating the
class that Python will need to instantiate if e.g. this YAML is
deserialized with `yaml.load` or similar.
Outside of the various Automation APIs, `Pulumi.yaml` files are
"plain-old YAML files" -- language- or library-specific concepts such as
class tags are nowhere to be seen. We thus don't really want this
behaviour when we serialize project settings to YAML. Fortunately, there
is a relatively simple workaround -- instead of passing instances of
`ProjectSettings` to `yaml.dump`, we can just pass vanilla dictionaries
containing the same data. These will be rendered as YAML objects with no
tags, which is what we want.
<em>Un</em>fortunately, we must turn _all_ objects in a hierarchy into
plain dictionaries, or tags will appear at some point. Presently, this
is not the case in the Python SDK, which just uses
`ProjectSettings.__dict__` to get the dictionary for the top-level
object. If there are nested objects in this dictionary (such as e.g.
`ProjectBackend` or `ProjectRuntimeInfo` objects), these are _not_
converted into dictionaries and `pyyaml` writes them as tagged objects,
which will later fail to deserialize.
This commit fixes this issue, adding explicit `to_dict` and `from_dict`
methods to settings classes for the purposes of recursively converting
objects to and from dictionaries. It also:
* adds a test that confirms serialization/deserialization of
configurations containing nested objects now works.
* in order to write this test, exports some types (`ProjectTemplate` and
friends) that appear to be part of the public API but have until now not
been exported.
This fixes go package generation so users don't _have_ to set the import
base path explicitly in the go info part of the schema. This only really
helps our packages (because we assume the path is "github.com/pulumi")
but it fixes some of the generated test data and should also work for
local SDKs, where they aren't _really_ at "github.com/pulumi" but also
that it doesn't matter what URL is used as long as it's a valid one.
Fixes https://github.com/pulumi/pulumi/issues/13756
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
Clean up the temporary `PULUMI_HOME` directory we create during a
program test.
This is necessary to reclaim the disk space of the plugins that were
downloaded
during the test.
In pulumi-aws we started seeing test failures because the CI runners
started running out of disk space due to plugins in `PULUMI_HOME` not
being cleaned up.
Add support to the Go SDK for invoke transforms. This only adds support
for setting transforms globally, not yet via a resource option, which
resource transforms allow.
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
We run a short script with nodejs to find the path to the nodejs SDK
entrypoint. In https://github.com/pulumi/pulumi/pull/16160 this script
was changed and used newlines.
This broke when using the Volta package manager to manage different
nodejs versions. Volta adds shim programs into the user's path that
routes to the correct nodejs executable. Seemingly Volta does not handle
the newlines in the arguments.
To fix this, we ensure that the script is on a single line.
Fixes https://github.com/pulumi/pulumi/issues/16393
From SDKs, we call invokes in one of two ways:
* In a "non-output" context (e.g. `getX`), which has a result dependent
on the language (e.g. a `Promise` in NodeJS) that _does not_ track
dependencies.
* In an "output" context (e.g. `getXOutput`), which has an `Output` type
and does track dependencies.
In the non-output case, `dependsOn` really doesn't make sense, since
this style of invoke is inherently ignoring dependency tracking/outputs.
This commit thus reverts 492c57c7dd so
that we can rethink the design before people's programs are subtly
broken in this case.
When running `pulumi up --continue-on-error`, we can't bring up new
resources that have dependencies that have failed, or have been skipped.
Since the resource would have a failed dependency, that dependency would
not be in the snapshot, resulting in a snapshot integrity failure. Also
it simply does not make sense to `up` a resource that has a failed
dependency.
We took care of that for the regular dependency relationship, however at
the time we missed doing the same for other types of dependencies,
namely parent-child relationships, deleted with relationships and
property dependencies.
This can result in snapshot integrity failures for example when a parent
fails to be created, but we still do the resource creation of the child,
such as what happened in https://github.com/pulumi/pulumi/issues/16638.
Fix this by skipping the step when a resource with any type of
dependency relationship fails or is skipped beforehand.
Fixes https://github.com/pulumi/pulumi/issues/16638
When a policy pack is nested within a project, or vice-versa, install
the appropriate dependencies, based on where the command is run from.
If there are both policy pack and project files in the parent
directories of the current working directory, pick the one that is
closest.
Fixes https://github.com/pulumi/pulumi/issues/16605
For nodejs we look at the node_modules of all child directories of the
root path to find the required plugins. When a policy pack is nested
within a pulumi project, we were including its dependencies in the list
of required plugins. To avoid this, we stop recursing once we see a
directory that has a PulumiPolicy.yaml.
Fixes https://github.com/pulumi/pulumi/issues/16604
We pass a version like `1.22.x` for go, which results in the setup-go
action using a cached version, even if there is a more recent release
available. Set `check-latest: true` to ensure we're always using the
latest
release, picking up any security fixes.
Unblocks https://github.com/pulumi/pulumi/pull/16608
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
The existing overlays (e.g. Chart v3 in Kubernetes, or CallbackFunction
in AWS) are not available in every language Pulumi supports. This often
confuses users because the generated docs include all languages Pulumi
supports (e.g. see
https://github.com/pulumi/pulumi-kubernetes/issues/2181).
To solve that problem, this change adds a new optional parameter to the
schema that allows configuring the languages an overlay (resource or
function) supports.
To support this in docsgen the existing Language Chooser
(`LangChooserLanguages`) of resources is made configurable and extended
to functions.
Note: This doesn't support resource methods right now. They'll need
extra handling because and overlay resource method might not support all
of the languages its resource supports. I'll tackle this in a follow up
PR.
Here's a screenshot of how this will look like for the Helm v3 chart for
example:
<img width="1046" alt="Screenshot 2024-07-01 at 16 11 23"
src="https://github.com/pulumi/pulumi/assets/2453580/b1a1365a-6dee-4099-829a-2859639a4c8c">
The PR contains the following commits. I'd recommend to look at the
first three ones and then check the regenerated golden files in the last
one:
- **Add schema parameter to constrain supported languages for overlays**
- **Update developer docs and changelog**
- **Refactor LanguageChooser and always pass supported languages**
- **Regenerate testdata**
relates to #13231
Currently, when generating docs for the AWSX package, Go constructor
syntax examples were not generated due to a panic. This causes all
constructor examples to not be emitted in the docs.
The panic occurred when trying to get the version of referenced packages
in the PCL program to emit import paths. However, _transitive_ package
references were not resolved in the PCL binder when binding resource
types. This PR fixes the problem such that now we do find the transitive
package references from any input or output property of the resources
being bound.
In the case of the awsx package, the top-level package is awsx itself
and aws is the transitive dependency. Anytime in codegen we call
`program.PackageReferences()` we should get both of them. Added a unit
test for this as well.
Testing this fix locally against the awsx package showed constructor
examples being generated for every language, however there was still a
problem in the _formatting_ of Go code which is also fixed (see change
in `gen_program_expressions.go`)
Resolves part of #16463
For resources that don't have a schema, we emit a best guess for how
they are declared and how imports should be emitted. For these resources
we emit `index.{ResourceName}` if that resource is from the index
module, however this is not how resources are usually generated, instead
we should generate `{packageName}.{ResourceName}` for resources from the
index module which is what this PR resolves.
Fixes#16461
When passing `yes` in interactive mode (or non-interactive mode),
language specific options should use their deafult values without
prompting the user.
Fixes#16541
Hi,
this is an attempt to fix#16541 , feedback is greatly appreciated!
I was not able to test if the result works as expected, I just adjusted
the code and manually created a test snapshot from the mocha console
output until I thought it looked right.
Could someone tell me...
1. how to actually build the `@pulumi/pulumi` package and how to test
the fixes quickly?
2. how to create a snapshot for a new closure test?
3. if there is a quick way to write the TypeScript code in this project,
set breakpoints and run tests against it?
- I plan to investigate the pnpm bundling bug of the `CallbackFunction`
but working on this and testing the code is extremely tedious right now
as I do not know your secrets (yet 😀)
- I didn't find these infos in the `CONTRIBUTING.md` and other READMEs
- I am very familiar with TypeScript and the nodejs tooling, but no clue
how everything is stitched together in this repo
- feel free to send me a Slack message in the Pulumi Community or
suggesting another communication channel
Thanks!
fyi @julienp
In this PR we fix generating constructor syntax examples for Kubernetes.
The problem initially was due to generating a PCL program with syntax
errors for the kubernetes schema. The syntax error was happening because
we emitted properties `$ref` and `$schema` which are not valid
identifiers in PCL. Fixing this was simple enough, we only needed to
quote these properties that start with dollar signs and we get a valid
PCL program.
Full PCL program for kubernetes:
https://gist.github.com/Zaid-Ajaj/abe899430a0b5f99a428934dcac75d52
However, the valid PCL program wouldn't convert to Go with program-gen
and it would hang without showing any errors or stack traces. I wrote a
script to split the programs for each resource and convert each
separately. This way I narrowed down the problem to this program:
```
resource "def" "kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition" {
apiVersion = "string"
kind = "string"
spec = {
versions = [{}]
}
}
```
Debugging this in Go program-gen didn't show an obvious error and it
kept on hanging. However I noticed that we lowering expressions twice:
once for the entirety of resource inputs when generating resources and
again when generating the expressions separately. Lowering expressions
has been the source of many bugs and it seems to trip up program-gen a
lot and suspected something wrong was going on here, so I simplified it
to lower the expressions once when we generate resource inputs. This
fixed the hang issue as well as a few of our test programs (see diffs
for test programs and tests schemas)
Tested this against the full kubernetes schema and we can now generate
the full Go program:
https://gist.github.com/Zaid-Ajaj/3ac734536969a989edae92219968d51f
> The second time we lower expressions not only is redundant but also
can generate invalid code if the lowered expressions have temporary
variables because program-gen would just emit these variables inside
inline expressions like objects and lists which gives invalid Go code.
Resolves part of #16463
We've had quite a few test flakes that happen because of TLS handshake
timeouts to the service backend. These are usually during stack creation
(which happens a lot during tests), which are POST requests.
POST requests are not generally safe to retry, because the request could
already have been sent to the backend and an action taken, but the
client couldn't read the response. However when there is a TLS handshake
failure, the request can never have made it to the backend, so these
errors are safe to retry.
This should hopefully help with
https://github.com/pulumi/pulumi/issues/16529.
This commit adds support for passing `dependsOn` to invokes (whether
streamed or not) in the NodeJS SDK. This allows programs to ensure that
certain invokes are executed after things they depend on, even if that
dependency is not implicitly captured with an input-output relationship.
Part of #14243
This is a very similar fix to
https://github.com/pulumi/pulumi/pull/16371, but for imports instead of
create and updates.
When using the import resource option with continue-on-error, and there
is a diff in the import, the import fais, but we still also return the
completion function from the `Apply` call. This results in the engine
trying to call `Done` twice, which in turn results in it running
indefinitely, trying to write the result to a channel that's no longer
being read from.
Fix this by not returning the completion function from `Apply` for the
ImportStep when there is an error.
(I went through the other steps as well to double check we don't need a
similar fix there, and it looks like they are all fine).
Fixes: https://github.com/pulumi/pulumi/issues/16570
This commit fixes Python code generation (specifically, program
generation) whereby we were previously emitting calls to the
non-existent `remoteAsset` function/constructor. We now use the
(correct) `RemoteAsset` constructor, in line with other asset types
exposed by the Python SDK.
External resources are resources whose state is tracked but not managed.
That is, they feature in the state file, but Pulumi does not issue
create, read or update operations for them. Typically they arise as the
result of generated `.get` functions and methods in Pulumi programs
(these should not be confused with specific `getX` functions/methods,
which are backed by `Invoke`s and designed to reflect data
sources/arbitrary function calls).
In #16146, `refresh` operations were modified to use actual bonafide
`Diff` calls to compute differences in a manner that would reflect "what
will happen on the next `preview`". In general, this aligns better with
what users expect when running `refresh`, and fixes a number of issues
that have long plagued `refresh`. However, it falls down in the case of
external resources. The existing `Diff` contract takes (somewhat
strangely) _three_ key inputs:
* The old inputs ("what the program used to say")
* The old outputs ("what we got back from the last update")
* The new inputs ("what the program says now")
Intuitively, `Diff`'s job is to return the difference between the old
and new inputs. Aside from old outputs being a bit of a third wheel,
this contract makes sense for an `update` operation, where the inputs
are driving the outputs and changes there will yield changes elsewhere.
In a `refresh` operation, however, we have a different situation -- one
in which _both_ old and new inputs _and_ outputs are available. Alas, we
don't currently have a "four-argument `Diff`" (or a two-argument one we
can call twice with no fear of repercussions), and so the usage in
`refresh` is a little bit of a clever hack that ends up biasing inputs
over outputs (see #16146 or the implementation for details).
For external resources, the old behaviour (where we just compared
outputs) seems like a better fit. This commit therefore adds this
special case and a test to ensure that when we see external resources,
we simply compare outputs and don't call `Diff`.
Fixes#16401Fixes#16502
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
# Description
Following up and extending #16208 such that now we guess references to
_properties_ `name` and `arn` between dependant resources when running
an import based on their literal data retrieved from providers.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
When the typechecker option is set, but the selected typechecker is not
installed in the project's virtual environment, the pulumi operation
will fail with an unhelpful message like `mypy failed: exit status 1`.
To make this more actionable, we check if the typechecker is installed,
and if not, we provide installation instructions to remediate the issue.
The Pulumi CLI provides a customised shell completion command,
`gen-completion`, that can be used to generate scripts that enable tab
completion in shells such as Bash and Zsh. When providing custom
generation commands, Cobra (the library we use to build the Pulumi CLI)
[recommends disabling the default completion
command](https://github.com/spf13/cobra/blob/main/site/content/completions/_index.md#adapting-the-default-completion-command).
This commit rebases @EthanOrlander's work in #7842 on `master` and
disables the default command in the recommended manner.
This is technically a breaking change in the interface of the CLI, but
since we'd (presumably) rather that users were only using the officially
supported completions that Pulumi itself maintains, this feels like us
fixing a bug/the right thing to do.
Fixes#7841Closes#7842
Co-authored-by: Ethan Orlander <ethanorlander@gmail.com>
When the dependency installation fails because we could not find the
packagemanager executable, tell the user how to remedy the error and
finish the dependency installation.
Example:
```
The packagemangager to use for installing dependencies pnpm
Installing dependencies...
error: installing dependencies failed: Could not find `pnpm` executable.
Install pnpm from https://pnpm.io/installation and make sure it is in your PATH.
Run `pulumi install` to complete the installation.
```
This commit rebases @Maradonna90's PR #14066 to add support for
`destroy`'s `--exclude-protected` argument to the Python automation SDK.
This addresses the Python part of #12733.
Closes#14066
Co-authored-by: Jendryczko, Marco <marco.jendryczko@hermesworld.com>
Fixes https://github.com/pulumi/pulumi/issues/16025.
Looks like shutting down the executor and waiting for it to finish
pending tasks should stop callbacks being triggered after we then close
the event loop.
When the `RetainOnDelete` resource option is set, the console should
display `[retain]` next to resources which are being deleted from the
Pulumi program to indicate that, while the Pulumi state for those
resources will be deleted, the provider will not be instructed to
actually `Delete` the resource (effectively leaving the resource
unmanaged).
Currently, Pulumi decides whether or not to add the `[retain]` text by
inspecting resource state attached to a display step. However, there are
two places in which a display step is constructed:
* As part of normal/"in-memory" execution, whereby resource state is
inherently available and attached to the display step.
* As part of deserializing an existing JSON event stream (e.g. from a
Pulumi REST API call), whereby resource state is _not_ available and so
absent (`nil`) from the display step. This is also the codepath taken by
display tests.
The absence of the state in the latter codepath means that `[retain]` is
not displayed. This needn't be the case however. The `RetainOnDelete`
option is explicitly persisted in our JSON events and copied on to the
display step, so we can just use that instead. This commit makes this
change. Additionally, the single test we have for `RetainOnDelete` is
pulled out into its own file and display testing is now enabled for that
test.
repo.FetchContext reads transport.UnsupportedCapabilities, which we
modify, leading to a potential data race in oncurrent calls to
setupGitRepo.
Wrap the call to repo.FetchContext with the global transportMutex.
Fixes https://github.com/pulumi/pulumi/issues/16516
As well as indicating that a resource's state has changes, a diff can
also indicate that those changes require the _replacement_ of the
resource, meaning that it must be recreated and not just updated. In
this scenario, there are two possible ways to replace the resource -- by
first creating another new resource before deleting the old one
("create-before-replace"), or by first deleting the old resource before
creating its replacement ("delete-before-replace").
Create-before-replace is the default since generally, if possible to
implement, it should result in fewer instances of "downtime", where a
desired resource does not exist in the system.
Should delete-before-replace be chosen, Pulumi implements this under the
hood as three steps: delete for replacement, replace, and create
replacement. To track things consistently, as well as enable resumption
of an interrupted operation, Pulumi writes a flag, `PendingReplacement`
to the state of a deleted resource that will later be cleaned up by a
completed replacement.
Should an interrupted operation be resumed, Pulumi does not currently
take `PendingReplacement` into account, and always enqueues a(nother)
delete operation. This is typically fine (albeit wasteful) since deletes
are (should) be idempotent, but unnecessary. This commit adds
@jesse-triplewhale's fix for this behaviour whereby the
`PendingReplacement` flag is simply removed before the remainder of the
required steps (replace, create replacement) are actioned as normal. It
also extends this work with some lifecycle tests for this scenario and a
few others that may arise as a result of an interrupted replacement.
Fixes#16288Closes#16303
Co-authored-by: Jesse Grodman <jesse@triplewhale.com>
In #16428 we parallelized the call&construct calls of the node provider
server. It was wrongly assumed that the state of the settings are
localized to the current call, but that is only done when running in
`asyncLocalStorage.run`.
This change wraps the call&construct calls in `asyncLocalStorage.run` to
correctly localize state.
Addresses
https://github.com/pulumi/pulumi/pull/16428#issuecomment-2190932525
Make `pulumi stack --show-name -Q` print a full stack name including the
org and project names:
```
pulumi stack --show-name -Q
mikhail-pulumi-corp/tmp3/dev
```
The actual change is very simple (see the second commit) but I needed to
refactor the entire command to be able to easily mock inputs and
outputs. Let me know if that makes things worse and I should just do the
one-line change without the refactoring.
Fixes https://github.com/pulumi/pulumi/issues/16445