This commit addresses part of #11942, in which we fail to serialise
closures whose code would use reserved identifiers like `exports`. This
is due to a change we made where module imports are hoisted to avoid
importing the same module multiple times. Previously, code adopted a
strategy of passing all dependencies using a `with` statement, viz.:
```typescript
function x() {
return (function () {
with({ fooBarBaz: require("foo/bar/baz"), ... }) {
// Use of fooBarBaz
}
}).apply(...)
}
function y() {
return (function () {
with({ fooBarBaz: require("foo/bar/baz"), ... }) {
// Use of fooBarBaz
}
}).apply(...)
}
```
This was changed to remove the duplicate imports, yielding code like:
```typescript
const fooBarBaz = require("foo/bar/baz")
function x() {
return (function () {
with({ ... }) {
// Use of fooBarBaz
}
}).apply(...)
}
function y() {
return (function () {
with({ ... }) {
// Use of fooBarBaz
}
}).apply(...)
}
```
However, while the previous approach would work with reserved
identifiers such as `exports` (`with({ exports, ... }) { ... }` is
perfectly acceptable), the new one does not (`const exports = ...` is
not acceptable since NodeJS will not allow redeclaration of the
`exports` global).
This commit combines the two approaches. Modules are only imported once,
but if an import would use a reserved identifier, we generate a fresh
non-conflicting identifier and alias this using a `with` statement. For
example:
```typescript
const fooBarBaz = require("foo/bar/baz")
// __pulumi_closure_import_exports is generated to avoid shadowing the reserved "exports"
const __pulumi_closure_import_exports = require("some/other/module")
function x() {
return (function () {
with({ exports: __pulumi_closure_import_exports, ... }) {
// Use of fooBarBaz and exports
}
}).apply(...)
}
```
Note that it is not expected that #11942 will be solved in its entirety.
While this commit fixes code that introduces identifiers like `exports`,
the introduction in question in that issue is caused by the use of
`pulumi.output(...)` in the constructor of a dynamic resource provider.
Since dynamic resource providers are implemented under the hood by
serialising their code, we attempt to serialise `pulumi.output` and its
dependency chain. This commit allows us to make more progress in that
regard, but other things go wrong thereafter. Fixing the deeper issue
that underpins #11942 (and likely other challenges with dynamic
providers) is probably a more involved piece of work.
This reverts commit ebb0e6aaed.
The changes in #7755 introduced a regression tracked in #7795. It is not yet clear how to retain the desired behaviour introduced in #7755 while avoiding this regression, so for now we will revert those changes, and re-open #7453 to track a deeper fix. That fix may require making changes to upstream `dill` to properly support these serialization requirements.
Fixes#7795.
Now there is not possible to change a name of dynamic provider resource without copying a code of the `pulumi.sdk.python.lib.pulumi.dynamic.dynamic.Resource` and changing the hard-coded name `"pulumi-python:dynamic:Resource"`. I successfully uses this proposal to make it possible.
Usage:
```python
class CustomResource(
Resource, name="my-custom-provider:CustomResource"
):
...
```
Co-authored-by: Pat Gavlin <pgavlin@gmail.com>
The underlying library `dill` that we use for serializing dynamic providers into Pulumi state for Python dynamic providers serializes classes differently depending on whether they are in `__main__` or in another module. We need the by-value serialization to be applied in all cases.
https://github.com/uqfoundation/dill/issues/424 is tracking adding the ability into `dill` to specify this by-value serialization explicitly, but until then, we will temporarily re-write the `__module__` of a provder class prior to serialization, so that `dill` behaves as we need for the dynamic provider use case.
Fixes#7453.
* Do not hang but propagate exception when it happens in resolve_outputs
* Add an integration test for the issue
* Better error message
* Add CHANGELOG_PENDING entry
* Update sdk/python/lib/pulumi/runtime/rpc.py
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Address PR feedback and tighten path param typing
* Given Windows builder is failing, allow 2x time for the test
* Give some more time to the Windows runner
* Attempt to solve differently
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Automatically create a virtual environment and install dependencies in it with `pulumi new` and `pulumi policy new` for Python templates.
This will save a new `virtualenv` runtime option in `Pulumi.yaml` (`PulumiPolicy.yaml` for policy packs):
```yaml
runtime:
name: python
options:
virtualenv: venv
```
`virtualenv` is the path to a virtual environment that Pulumi will use when running `python` commands.
Existing projects are unaffected and can opt-in to using this by setting `virtualenv`, otherwise, they'll continue to work as-is.
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.