Commit Graph

245 Commits

Author SHA1 Message Date
Justin Van Patten 472e6bdd50
Allow Python dynamic providers to capture secrets ()
As of https://github.com/pulumi/pulumi/pull/13315 (which shipped in
[v3.75.0](https://github.com/pulumi/pulumi/releases/tag/v3.75.0)),
`__provider` (the serialized provider string) is _always_ set to a
secret in the state. This can lead to poor performance when there are a
lot of dynamic resources and the serialized provider does not actually
have any secrets.

This change does two things:
1. Provides a way to opt-out of always serializing the provider as a
secret.
2. Allows Outputs to be captured during serialization of the provider,
which wasn't previously possible.

## 1. Opt-out of always serializing as secret

A new attribute, `serialize_as_secret_always`, can be set on a subclass
of `ResourceProvider` to opt-out of always serializing the provider as a
secret. If you know you don't have any secrets being serialized into the
provider and want to avoid the encryption overhead, you can set this
attribute to `False`.

```python
class MyProvider(ResourceProvider):
    serialize_as_secret_always = False

    def create(self, props):
        # Doesn't have any secrets that need encrypting
        ...
```

## 2. Allow Outputs to be captured

If you currently try to capture an `Output`, it fails when serializing
the provider with:

```
TypeError: cannot pickle '_asyncio.Future' object
```

This change allows Outputs to be captured/serialized for dynamic
providers, including secret Outputs.

This aligns Python dynamic providers with
[Node.js](https://github.com/pulumi/pulumi/pull/13329).

```python
import pulumi
from pulumi.dynamic import CreateResult, Resource, ResourceProvider

config = pulumi.Config()
password = config.require_secret("password")

class SimpleProvider(ResourceProvider):
    def create(self, props):
        # Need to use `password.get()` to get the underlying value of the secret from within the serialized code.
        # This simulates using this as a credential to talk to an external system.
        return CreateResult("0", { "authenticated": "200" if password.get() == "s3cret" else "401" })

class SimpleResource(Resource):
    authenticated: pulumi.Output[str]

    def __init__(self, name):
        super().__init__(SimpleProvider(), name, { "authenticated": None })


r = SimpleResource("foo")
pulumi.export("out", r.authenticated)
```

Note: In the above example, we didn't have to specify
`serialize_as_secret_always` since the default behavior is to always
serialize the provider as a secret. If we wanted to, we could have
specified `serialize_as_secret_always = False` and it still would have
serialized the provider as a secret, since it captured `password` which
is a secret Output. If `serialize_as_secret_always = False` was
specified and no secrets were captured, then the provider would not be
serialized as a secret.

We plan to recommend this approach to capturing secrets for both Node.js
and Python dynamic providers after this change.

Fixes 
2024-08-02 23:51:52 +00:00
Zaid Ajaj 9c8f8470ba
[python/automation] Implement Stack.import_resources() for batch importing resources into a stack ()
Addressing https://github.com/pulumi/pulumi/issues/8237 for Python. 

Since `import` is a reserved keyword in python, I went with
`import_resources` as the function name. It's a bit unfortunate that it
is not the same name as other SDKs but I wasn't sure which name to use
here. Suggestions are welcome 🙏
2024-07-24 14:58:34 +00:00
Will Jones 01dc95a173
Support `--remove` for `destroy` in the Go, NodeJS and Python Automation API SDKs ()
By default, `pulumi destroy` removes all resources within a stack but
leaves the stack and its configuration intact. If one passes the
`--remove` option to `destroy`, however, the stack and its configuration
will also be removed once the resources within the stack have been
deleted. This commit updates the work of @Moon1706 in  to add
`remove` as an option to the Go, NodeJS and Python Automation API SDKs'
`destroy` methods, which then perform an analogous clean-up.

Closes 

---------

Co-authored-by: Nikita Sharaev <n.p.sharaev@tinkoff.ru>
2024-07-17 09:07:30 +00:00
Will Jones a158b300b9
Fix YAML serialization of project settings in the Python Automation API ()
The Python Automation API SDK serializes project settings (the contents
of `Pulumi.yaml` files) using Python's `pyyaml` package. Project
settings in the Python Automation API SDK are represented as instances
of the `ProjectSettings` class. By default, `pyyaml` will serialize
class instances as YAML objects "tagged" with a string that indicates
their class. This is so that, upon deserialization, it can construct
objects of the appropriate class (as opposed to, just dictionaries). As
an example, the following Python program:

```python
class Person:
  def __init(self, name: str, age: int) -> None:
    self.name = name
    self.age = age

will = Person("will", 37)
yaml.dump(will)
```

will produce the following YAML:

```yaml
!!python/object:__main__.Person
age: 37
name: will
```

The string `!!python/object:__main__.Person` is the _tag_ indicating the
class that Python will need to instantiate if e.g. this YAML is
deserialized with `yaml.load` or similar.

Outside of the various Automation APIs, `Pulumi.yaml` files are
"plain-old YAML files" -- language- or library-specific concepts such as
class tags are nowhere to be seen. We thus don't really want this
behaviour when we serialize project settings to YAML. Fortunately, there
is a relatively simple workaround -- instead of passing instances of
`ProjectSettings` to `yaml.dump`, we can just pass vanilla dictionaries
containing the same data. These will be rendered as YAML objects with no
tags, which is what we want.

<em>Un</em>fortunately, we must turn _all_ objects in a hierarchy into
plain dictionaries, or tags will appear at some point. Presently, this
is not the case in the Python SDK, which just uses
`ProjectSettings.__dict__` to get the dictionary for the top-level
object. If there are nested objects in this dictionary (such as e.g.
`ProjectBackend` or `ProjectRuntimeInfo` objects), these are _not_
converted into dictionaries and `pyyaml` writes them as tagged objects,
which will later fail to deserialize.

This commit fixes this issue, adding explicit `to_dict` and `from_dict`
methods to settings classes for the purposes of recursively converting
objects to and from dictionaries. It also:
* adds a test that confirms serialization/deserialization of
configurations containing nested objects now works.
* in order to write this test, exports some types (`ProjectTemplate` and
friends) that appear to be part of the public API but have until now not
been exported.
2024-07-16 10:00:05 +00:00
Will Jones 7309918554
Support `--exclude-protected` for `destroy` in the Python automation SDK ()
This commit rebases @Maradonna90's PR  to add support for
`destroy`'s `--exclude-protected` argument to the Python automation SDK.
This addresses the Python part of .

Closes 

Co-authored-by: Jendryczko, Marco <marco.jendryczko@hermesworld.com>
2024-06-28 23:21:49 +00:00
Will Jones 79e814fe0f
Don't lift dunder attributes on Python `Output`s ()
`Output`s are a central part of Pulumi programs. As well as tracking
dependencies between resources (e.g. that `A`'s input `x` comes from
`B`'s output `y`), they allow us to specify that some value will only be
available at a future point (e.g. after a resource has been created or
updated). This is typically done in each language by implementing
`Output`s using some asynchronous or future value, such as NodeJS's
`Promise`. In Python, we use `asyncio` `Task`s.

In order to make `Output`s ergonomic to use, we implement "lifting" of
properties in languages that support it. Suppose for instance we have an
object `c` that is an instance of the following class `C`:

```python
class C:
    x: str
    y: int

    def __init__(self, x: str, y: int) -> None:
        self.x = x
        self.y = y

c = C("x", 42)
```

Because `c: C`, we have that `c.x == "x"` and `c.y == 42` as we might
expect. Consider though some output property of a resource that produces
a `C`. This property will be of type `Output[C]`, since the value of
type `C` won't be available until the resource has been set up by Pulumi
as part of program execution. If we want to pass that output's `x` value
to some other resource, we might have to write:

```python
r1 = ... # r1 has a property c: Output[C]

r2 = R("r", RArgs(x=r1.c.apply(lambda cc: cc.x)))
```

Observe that we have to use `apply` to unwrap the output and access the
property inside. This is tedious and ugly, and exactly the problem
lifting solves. Lifting allows us to write `r1.c.x` and have it be
implemented as `r1.c.apply(lambda cc: cc.x)` under the hood. In Python,
this is achieved using Python's `__getattr__` "dunder" method ("dunder"
being short for "double underscore", the convention adopted for
"special" identifiers in Python). `__getattr__` allows us to perform
dynamic property lookup on an object, which in the case of `Output` we
use to delegate property access to the underlying value using `apply`.

This works really well and contributes significantly to Pulumi programs
being easy to read and write in Python. Unfortunately, it has a flaw:
`__getattr__` is also used to check whether attributes exist on an
object (using `hasattr`), and thus has a contract whereby it is
_synchronously_ expected to raise an `AttributeError` if an attribute
does not exist. In returning an _asynchronous_ task (that may _later_
resolve to an `AttributeError`), `Output` is in violation of this
contract. This means that code which calls e.g. `hasattr(r1.c, "z")`
will yield a future task that if resolved, will blow up with an
`AttributeError`.

Historically, this hasn't really been a problem. With the advent of
https://github.com/pulumi/pulumi/pull/15744 and subsequent
improvements/fixes, however, this is now an issue, since we explicitly
await all outstanding outputs before program termination. In the example
above, for instance, the orphaned task will be forced at the end of
program execution. The `AttributeError` will rear its head and Pulumi
will exit with an error and stack trace.

This commit implements a fix proportional to the cases where this
appears to be a problem. Namely, libraries such as Pydantic that use
dunder attributes in their implementation and check for the presence of
these attributes when executing. When `__getattr__` is called with a
dunder attribute, we synchronously `raise AttributeError` rather than
lifting it. There are (we believe) no cases where this would affect a
Pulumi-generated SDK (since dunder names aren't generally used for
public resource properties) and the dunder properties on the `Output`
itself (e.g. `__dict__`, etc.) will continue to work since their
resolution will succeed normally and a call to `__getattr__` will thus
not be made.

Fixes 
2024-06-22 00:04:34 +00:00
Will Jones e65c2aa368
Don't incorrectly emit deprecated warnings ()
Presently, the Python SDK incorrectly emits deprecation warnings for
properties even if user code does not reference them. This is due to
various pieces of code in the SDK that enumerate properties e.g. for the
purpose of serialisation/RPC calls. In enumerating properties, their
getters are invoked, triggering the deprecation warnings.

There isn't currently a way for us to detect which properties may or may
not be deprecated and handle them appropriately, so this commit
introduces one. Deprecation messages are moved into a new decorator,
`@pulumi.deprecated`. This decorator accepts a `Callable` and wraps it,
making two changes:

* Before the decorated `Callable`'s execution, deprecation messages are
printed.
* The returned `Callable` is tagged with a new reserved property,
`_pulumi_deprecated_callable`, which references the wrapped `Callable`
(that is, the original `Callable` whose invocation will _not_ produce
deprecation warnings).

With this in place, we subsequently make the following two changes:

* We modify the SDK enumeration code (specifically that in
`input_type_to_dict` to check for `_pulumi_deprecated_callable`
properties and use them where appropriate to invoke getters without
triggering deprecation warnings.
* We modify Python code generation so that instead of emitting
statements to print deprecation warnings, we simply emit applications of
the `@pulumi.deprecated` decorator instead.

This commit adds some Python unit tests to exercise the decorator and
manual testing has been performed using the AWS Classic SDK.

Fixes 
2024-06-17 16:04:55 +00:00
Mike Seese ec4086650c
Add removeStack options to NodeJS Auto API SDK ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This PR adds an optional `opts` argument to the NodeJS SDK
`Workspace::removeStack` method, which has `force` and `preserveConfig`
optional booleans. Setting these bools to true will add their
corresponding CLI flag to the command.

Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-14 08:35:06 +00:00
Will Jones 486a10e677
Regenerate test `package-lock.json` files ()
Many of our tests test JavaScript/TypeScript behaviour and thus rely on
`package.json` files specifying dependencies and `package-lock.json`
files pinning those dependencies. While the dependencies we use in our
tests are minimal and their bounds haven't changed (nor maybe have much
reason to change), the transitive dependency set is potentially large
and worth keeping up-to-date (e.g. for security reasons, or just keeping
Dependabot happy). This commit thus regenerates all the
`package-lock.json` files associated with tests that Dependabot has
previously recommended we bump.

* Closes 
* Closes 
* Closes  
* Closes  
* Closes  
* Closes  
* Closes  
* Closes 
* Closes  
* Closes 
* Closes  
* Closes 
* Closes  
* Closes  
* Closes 
2024-06-13 10:52:59 +00:00
Zach Buchheit 0ebfde2629
Add support for `--all` parameter of the `stack ls` command to the Automation API ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Adds Go/Nodejs/Python automation API support for pulumi stack ls --all.

Fixes: [](https://github.com/pulumi/pulumi/issues/14226)

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-03 15:53:43 +00:00
kvthr 086a75d18d
More descriptive exception in serialize_property ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-10 11:34:18 +00:00
Fraser Waters f0085de0c7
Handful of mypy fixes in the python test code () 2024-05-02 17:46:52 +00:00
Fraser Waters 3d3729006c
Start mypy linting lib/test ()
This fixes up a load of '[un]expected return statement' errors in the
tests. Mostly property getters which I've filled in with `pulumi.get`
but some of the tests were explictly checking things worked when the
getter was an empty method body.
2024-04-30 12:54:13 +00:00
Justin Van Patten 3b9449325e
[sdk/python] Workaround lazy module regression ()
A regression was introduced in Python 3.11.9 and 3.12.3 that breaks lazy
loaded `config` modules in generated Pulumi provider Python SDKs. It's
unclear if this is going to be addressed in subsequent patch releases of
Python, so we workaround the problem by using a copy of the Python
stdlib's `_LazyModule` and `LazyLoader` classes that don't have the
regression, as suggested by the Python maintainers.

Fixes 
2024-04-24 06:53:59 +00:00
Fraser Waters 48dbd6c596
Use black to format lib/test ()
Test code should be formatted and linted the same as library code. This
is the first step of that, simply including ./lib/test to the folder
that the black formatter runs on.
2024-04-23 08:29:58 +00:00
Thomas Gummerer c91d40ac5f
allow unknowns in apply during update ()
Currently during updates we try to make the values known during apply,
even if they are marked unknown. This results in knows that should
really still be unknown.

Note that this behaviour is currently encoded in tests, which also
needed to change.
2024-04-10 16:48:38 +00:00
Thomas Gummerer 1339f96833
automation: only read complete lines before trying to deserialize ()
When tailing the event log in automation API we currently have nothing
that makes sure we read only complete lines. This means if the OS
happens to flush an incomplete line for whatever reason (or the Go JSON
encoder does, which we're using to write these lines), we might read a
line that is incompletely written, and thus will fail to JSON decode it.

Since the JSON encoder always writes a newline at the end of each
string, we can also make sure that the line we read ends with a newline
and otherwise wait for the rest of the line to be written.

The library we use in Go provides a convenient setting for this, while
in python and nodejs we need to add some code to do this ourselves.

Fixes https://github.com/pulumi/pulumi/issues/15235
Fixes https://github.com/pulumi/pulumi/issues/15652
Fixes https://github.com/pulumi/pulumi/issues/9269 (This is closed
already, but never had a proper resolution afaics)
Fixes https://github.com/pulumi/pulumi/issues/6768

It would be nice to add a typescript test here as well, but I'm not sure
how to do that without marking the readLines function non-private. But I
don't know typescript well, so any hints of how to do that would be
appreciated!

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-26 14:32:56 +00:00
Fraser Waters 50eacf3c2f
Add a test and fix that we don't wait for tasks, just outputs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/6762.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-26 14:07:37 +00:00
Fraser Waters 72a80af706
Type check depends_on is only passed Resource instances ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/13917.

Improves the error message for async type errors to match the error
message users would get for sync type errors.

This also fixes the mock.test function to clear the RPCManager between
tests so an exception in one test doesn't cause other tests to fail.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-25 18:51:21 +00:00
Justin Van Patten 73e2471fee
[sdk/python] Add an `invoke_async` function ()
This commit adds a new `pulumi.runtime.invoke_async` function that
allows calling invokes asynchronously. The initial intended use for this
is inside the Kubernetes Python SDK's `yaml.ConfigFile`,
`yaml.ConfigGroup`, `helm`, `kustomize`, etc. components to avoid stalls
in resource registrations which severely limits parallelism.

I'd love to add some kind of benchmark, perhaps along the lines of the
repro in , but we can add that as a fast-follow.

Part of  and 

After this is merged and released, we can ship an updated version of the
Kubernetes Python SDK that makes use of it.
2024-03-06 00:32:39 +00:00
Fraser Waters 0a0a327ca3
Support async inline programs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This updates the typing annotations and semantics of Python Automation
API inline programs. It's now defined to be a callable that returns an
optional awaitable of `None`. This supports both existing synchronous
functions (should just return `None`) and now also supports async
functions that return an `Awaitable` that eventually resolves to `None`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Julien P <julien@caffeine.lu>
2024-01-29 16:10:13 +00:00
Julien P c26874d411
[auto/python] Add new API to install the Pulumi CLI ()
# Description

Provide a way for the Automation API to install the Pulumi CLI so that
Automation API can be used in a more standalone manner.

https://github.com/pulumi/pulumi/issues/14987

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-26 14:30:19 +00:00
Justin Van Patten c306b13c51
[sdk/python] Install local SDK in automation API test ()
In preparation for supporting Python 3.12...

The test was installing a `requirements.txt` file that depends on the
released version of the `pulumi` package, but that package depends on
the old `grpcio` package that does not work Python 3.12. Instead of
depending on the public `pulumi` package, install the locally built
Python SDK.

Also, fix the test to be able to run on Windows, while making changes
here.
2024-01-24 02:06:38 +00:00
Fraser Waters 72bddd809f
Update github.com/cloudflare/circl to v1.3.7 ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Dependabot updated some references to this in
https://github.com/pulumi/pulumi/pull/15131. But missed a lot,
importantly it didn't update pkg or sdk which are the most important
modules in this repo.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-16 08:59:57 +00:00
Justin Van Patten 37e6ad44d0
Upgrade go-git to v5.11.0 ()
Bumps github.com/go-git/go-git/v5 to 5.11.0 to address
https://github.com/go-git/go-git/security/advisories/GHSA-mw99-9chc-xw7r

Co-authored-by: Roy Reznik <roy@wiz.io>
2024-01-02 18:41:06 +00:00
Komal e5d5f5ab80
Automation API support for listing environments ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Automation API support for `pulumi config env ls`

Fixes https://github.com/pulumi/pulumi/issues/14797

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-22 05:18:14 +00:00
Justin Van Patten 53244f09ae
Bump golang.org/x/crypto to 0.17.0 ()
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) to 0.17.0.

Replaces all the dependabot PRs in the repo with this single PR.

Also bumped `github.com/pulumi/pulumi/sdk/v3` in
`tests/integration/transformations/go/simple/go.mod` from v3.97.0 to
v3.98.0 to use esc v0.6.1, and avoid the appdash issue.
2023-12-20 09:14:29 +00:00
Komal 465c2547f1
[Automation API / Python] - Environment functions ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Add python automation API support for adding and removing environments
for stack configuration.

Fixes https://github.com/pulumi/pulumi/issues/14793

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-11 16:14:22 +00:00
Fraser Waters 6e986f90af
Pass root and main info to language host methods ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is two changes rolled together in a way.

Firstly passing some of the data that we pass on language runtime
startup to also pass it to Run/GetRequiredPlugins/etc. This is needed
for matrix testing, as we only get to start the language runtime up once
for that but want to execute multiple programs with it.
I feel it's also a little more consistent as we use the language
runtimes in other contexts (codegen) where there isn't really a root
directory, and aren't any options (and if we did do options the options
for codegen are not going to be the same as for execution). It also
means we can reuse a language host for shimless and substack programs,
as before they heavily relied on their current working directory to
calculate paths, and obviosly could only take one set of options at
startup. Imagine a shimless python package + a python root program, that
would have needed two startups of the python language host to deal with,
this unblocks it so we can make the engine smarter and only use one.

Secondly renaming some of the fields we pass to
Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very
descriptive and had pretty non-obvious documentation:
```
string pwd = 3;     // the program's working directory.
string program = 4; // the path to the program to execute.
```
`pwd` will remain, although probably rename it to `working_directory` at
some point, because while today we always start programs up with the
working directory equal to the program directory that definitely is
going to change in the future (at least for MLCs and substack programs).
But the name `pwd` doesn't make it clear that this was intended to be
the working directory _and_ the directory which contains the program.

`program` was in fact nearly always ".", and if it wasn't that it was
just a filename. The engine never sent a path for `program` (although we
did have some unit tests to check how that worked for the nodejs and
python hosts).

These are now replaced by a new structure with (I think) more clearly
named and documented fields (see ProgramInfo in langauge.proto).

The engine still sends the old data for now, we need to update
dotnet/yaml/java before we break the old interface and give Virtus Labs
a chance to update [besom](https://github.com/VirtusLab/besom).

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-10 17:30:51 +00:00
Paul C. Roberts 278a8994e5
Sets the `main` project setting to the cwd for inline programs ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Currently if no project settings are provided this is the default value
for `main`, but if the caller providers a project_settings object
without this set it will cause a module load issue if the inline program
depends on other local python files. The current work-around is to put
all the code in one file or set `work_dir`.

Fixes 

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [X] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

Co-authored-by: Paul Roberts <proberts@pulumi.com>
2023-12-01 23:05:08 +00:00
Kyle Pitzen 7b21b5db66
[sdk/python] Adds a default exception when dependency cycles are created ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Currently, when we detect that we've created a cycle in the dependency
graph, we early-exit. This works well enough for simple cycles, but when
early exiting is not sufficient (as when providing a resource output as
an argument to another resource's inputs), we will still fail to resolve
the full dependency graph. This PR introduces an exception-by-default,
as any cycles represent an unsafe/invalid dependency graph and should be
resolved manually. We also provide an escape hatch to fall back to
current behavior, in case users would prefer to retain the ability to
create unsafe dependency graphs (potentially introducing infinite hangs
when resolving those graphs).

Fixes https://github.com/pulumi/pulumi/issues/13551

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-21 16:26:02 +00:00
Justin Van Patten d312b5560a
[sdk/python] Maintain old behavior for empty k8s invoke results ()
The hand-rolled invoke code in the Kubernetes Python SDK is not
expecting empty dicts for empty results. Instead, it has code that
checks for `None`.

In general, we want to keep the behavior of returning empty dicts rather
than `None` for empty results, as this aligns with the behavior of other
SDKs (Node.js and Go) and is actually the original behavior of the
Python SDK (it regressed unintentionally in the v2 timeframe).

We're updating the Kubernetes Python SDK to be resilient to either
getting an empty dict or `None` for invokes with empty results.

Additionally, to maintain compat with older versions of the Kubernetes
Python SDK, we'll special case those invoke tokens. When an invoke's
result is empty and the token is one of the Kubernete's function tokens,
we'll return `None` rather than the empty dict.

Fixes 
2023-11-14 22:26:45 +00:00
Justin Van Patten 5d7204e1c1
[sdk/python] Fix error on empty invoke returns ()
When an empty struct is returned from an invoke, the Python SDK was
returning None, which causes the following error to be raised from
generated provider Python SDKs when calling the invoke:

```
AssertionError: get can only be used with classes decorated with @input_type or @output_type
```

This is because the Python SDK is returning None if the return value
isn't truthy, roughly:

```
ret_obj = getattr(resp, "return")
if ret_obj:
    return ret_obj
return None
```

However, an empty struct isn't truthy, so we're returning None in that
case, and generated provider Python SDKs can't handle that.

Other SDKs like the Node.js and Go SDKs don't have this problem.

This commit fixes the issue by removing the `if ret_obj` check, as it's
not necessary. In practice, `ret_obj` is always going to be an instance
of `struct_pb2.Struct` because all monitors (CLI and mock) return an
instance, though the instance may be empty, which is ok.

Fixes 
2023-11-01 17:47:23 +00:00
Justin Van Patten 7f2555444d
bump google.golang.org/grpc from 1.57.0 to 1.57.1 ()
This PR replaces all the dependabot PRs with a single commit that
updates all relevant go.mod files.

This resolves a high severity dependabot alert.
2023-10-28 15:56:28 +00:00
Justin Van Patten 86eee44bf8
bump golang.org/x/net from 0.10.0 to 0.17.0 ()
This PR replaces all the dependabot PRs with a single commit that
updates all relevant go.mod files.

This resolves 3 Dependabot alerts on golang.org/x/net including a
moderate severity alert.
2023-10-20 18:36:16 +00:00
Justin Van Patten 55abd8f63a
[sdk/python] Add `default` arg to `Config.get_secret` ()
This change adds the missing `default` arg to `Config.get_secret`
method, for consistency with the other `get_secret_<type>` methods that
have a `default` arg.

Fixes 
2023-10-02 13:43:48 +00:00
Fraser Waters cac2e0c2ee
Type check props in the `Resource` initializer ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/13997.

It would be a little nicer if every input type _actually_ implemented
the `Mapping` interface, but that trying to do that turned into a much
deeper fix than this one.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-25 14:54:28 +00:00
Abhinav Gupta 91a079851b
deps: Upgrade google.golang.org/{genproto, grpc}
Updates to the latest versions of
google.golang.org/genproto and google.golang.org/grpc
in all submodules in the repository.

This is necessary because in a recent change,
genproto split out some of its subpackages into independent submodules.
(https://github.com/googleapis/go-genproto/issues/1015)

As a result of this, some users may see the error:

```
google.golang.org/genproto/googleapis/rpc/status: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
    google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 (/home/runner/go/pkg/mod/google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1/googleapis/rpc/status)
    google.golang.org/genproto/googleapis/rpc v0.0.0-20230725213213-b022f6e96895
```

Because pu/pu is using 20230410155749,
which has googleapis/rpc as a subpackage,
but another dependency references the independent submodule (20230725213213),
so the system doesn't know which module to use for the import path,
google.golang.org/genproto/googleapis/rpc/status.

This is a problem for codegen tests and ProgramTest-based tests
for Pulumi Go programs that do not have a go.mod in the test directory.
This issue was encountered by @thomas11 while attempting to upgrade
dependencies in pulumi-docker ().

The grpc upgrade is necessary because the current version of grpc
also pulls the outdated version of genproto.
2023-07-27 16:24:33 -07:00
Fraser Waters bba69e2f38 Make `Output.from_input` recurse into tuples.
Fixes https://github.com/pulumi/pulumi/issues/6635.
2023-07-27 10:13:40 +01:00
Fraser Waters 442924b405 Add the ability to verify an expected log message was written
We used to test that various errors we're written out to the user in
these langhost tests, but at some point those test were removed (I
suspect because we stopped printing them to stderr).

This adds in the ability to check for a log message in the engine
diagnostics stream, and makes use of that in one of the tests.
2023-07-17 13:18:57 +01:00
Pat Gavlin 248f78bafe [sdk/*] Add support for resource source positions
Add support to the core SDKs for reporting resource source positions.

In each SDK, this is implemented by crawling the stack when a resource
is registered in order to determine the position of the user code that
registered the resource.

This is somewhat brittle in that it expects a call stack of the form:
- Resource class constructor
- abstract Resource subclass constructor
- concrete Resource subclass constructor
- user code

This stack reflects the expected class hierarchy of "cloud resource /
component resource < customresource/componentresource < resource".

For example, consider the AWS S3 Bucket resource. When user code
instantiates a Bucket, the stack will look like
this in NodeJS:

    new Resource (/path/to/resource.ts:123:45)
    new CustomResource (/path/to/resource.ts:678:90)
    new Bucket (/path/to/bucket.ts:987:65)
    <user code> (/path/to/index.ts:4:3)

In order to determine the source position, we locate the fourth frame
(the `<user code>` frame).
2023-07-13 16:46:04 -07:00
Justin Van Patten ef41442211 [sdk/python] Add langhost test for packaged component providers
Currently fails without the fix.
2023-07-13 07:16:56 -07:00
bors[bot] 772a057ee1
Merge
13463: Fix links to outputs docs r=cnunciato a=cnunciato

Fixes some broken links pointing to the Inputs & Outputs docs.

Fixes .

Co-authored-by: Christian Nunciato <chris@nunciato.org>
2023-07-13 12:34:13 +00:00
Fraser Waters 8db3087d2b Make pythons RPCManager a context variable 2023-07-13 09:59:12 +01:00
Christian Nunciato 202fbd946f Fix links outputs docs 2023-07-12 14:59:07 -07:00
Abhinav Gupta f59ab49fc5
deps(go): Upgrade to grpc 1.56.1
Upgrades to gRPC Go 1.56.1 to resolve the influx of dependabot PRs.
Supersedes all dependabot PRs created for this.

Fixes CVE-2023-32731
2023-07-06 09:04:16 -07:00
Justin Van Patten 16db89deda Bump semver and @pulumi/pulumi 2023-06-24 13:53:19 -07:00
Fraser Waters b150120506 Allow tuples as sequences in Python serialization
Fixes https://github.com/pulumi/pulumi/issues/7029

For some reason the python RPC serializer explictly disallowed tuple,
despite python codegen outputing types using `Sequence` rather than
`List` suggesting that `tuple` would be acceptable.
2023-06-20 14:26:00 +01:00
Justin Van Patten 6c43845071 [auto/python] Add support for the path option for config operations
Add support for the `--path` flag for config operations in Python automation API.
2023-05-30 04:00:24 -07:00
Abhinav Gupta 52a47d6295
all: cloudflare/circl 1.1.0 => 1.3.3
Upgrade version of cloudflare/circl to pick up important fixes
and supersede a bunch of dependabot PRs.

Addresses CVE-2023-1732
2023-05-11 13:51:01 -07:00