Commit Graph

599 Commits

Author SHA1 Message Date
Fraser Waters e9730dfc39
Replace dict with typing.Dict in type annotation ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14580.
2023-11-15 18:04:32 +00:00
Justin Van Patten d312b5560a
[sdk/python] Maintain old behavior for empty k8s invoke results ()
The hand-rolled invoke code in the Kubernetes Python SDK is not
expecting empty dicts for empty results. Instead, it has code that
checks for `None`.

In general, we want to keep the behavior of returning empty dicts rather
than `None` for empty results, as this aligns with the behavior of other
SDKs (Node.js and Go) and is actually the original behavior of the
Python SDK (it regressed unintentionally in the v2 timeframe).

We're updating the Kubernetes Python SDK to be resilient to either
getting an empty dict or `None` for invokes with empty results.

Additionally, to maintain compat with older versions of the Kubernetes
Python SDK, we'll special case those invoke tokens. When an invoke's
result is empty and the token is one of the Kubernete's function tokens,
we'll return `None` rather than the empty dict.

Fixes 
2023-11-14 22:26:45 +00:00
Fraser Waters 0e3c5a1d33
Add deserialisation cache to Python dynamic provider ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Provider side fix for https://github.com/pulumi/pulumi/issues/6675,
similar to change made to NodeJS in
https://github.com/pulumi/pulumi/pull/6657.

This might also help with https://github.com/pulumi/pulumi/issues/14159,
as investigation there looks like the issue might be due to memory
pressure.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-03 21:41:30 +00:00
Justin Van Patten 3d78b037f4
Fix traceback diagnostic from being printed when using Python dynamic providers ()
 upgraded grpcio to 1.59.2, to support Python 3.12. Unfortunately,
grpcio >=1.58.0 has a problem where calls to unimplemented gRPC methods
cause a traceback to be emitted to the server's stderr, which affects
Python dynamic providers, which don't implement `DiffConfig`. The result
is a traceback diagnostic shown for the Pulumi program using the dynamic
provider.

This PR fixes this by reverting the change, downgrading grpcio back to
1.56.2.

The existing Python dynamic provider diagnostic test has been extended
to check to make sure there are no unexpected diagnostics, which fails
before the revert, and passes after.

An upstream issue has been opened:
https://github.com/grpc/grpc/issues/34853

The issue to support Python 3.12 has been re-opened:  (currently
blocked on the upstream issue)

Fixes 
2023-11-01 22:19:37 +00:00
Justin Van Patten 5d7204e1c1
[sdk/python] Fix error on empty invoke returns ()
When an empty struct is returned from an invoke, the Python SDK was
returning None, which causes the following error to be raised from
generated provider Python SDKs when calling the invoke:

```
AssertionError: get can only be used with classes decorated with @input_type or @output_type
```

This is because the Python SDK is returning None if the return value
isn't truthy, roughly:

```
ret_obj = getattr(resp, "return")
if ret_obj:
    return ret_obj
return None
```

However, an empty struct isn't truthy, so we're returning None in that
case, and generated provider Python SDKs can't handle that.

Other SDKs like the Node.js and Go SDKs don't have this problem.

This commit fixes the issue by removing the `if ret_obj` check, as it's
not necessary. In practice, `ret_obj` is always going to be an instance
of `struct_pb2.Struct` because all monitors (CLI and mock) return an
instance, though the instance may be empty, which is ok.

Fixes 
2023-11-01 17:47:23 +00:00
Justin Van Patten 7f2555444d
bump google.golang.org/grpc from 1.57.0 to 1.57.1 ()
This PR replaces all the dependabot PRs with a single commit that
updates all relevant go.mod files.

This resolves a high severity dependabot alert.
2023-10-28 15:56:28 +00:00
Thomas Gummerer f278fa9696
stack ls: remove misleading updateInProgress output for file backend ()
File backends currently do not support indicating whether an operation
is currently in progress or not. In the `stack ls --json` output we
always end up with a "updateInProgress = false" field, even when an
update is in progress. This is misleading.

Disable showing this erroneous and misleading output.

Fixes 

Based on Luke's comments in that issue, I believe we should merge this
as bug fix for the issue, and then open a separate feature request
tracking the implementation of tracking whether an update is in progress
in the file backend.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-27 07:33:07 +00:00
Alexander Hungenberg 85b41ecef3
updating grpcio version ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

The PR updates the `grpcio` dependency for the Python SDK.

I also made a change here to [no longer fix the grpcio version to a
specific
one](https://iscinumpy.dev/post/bound-version-constraints/#tldr) - which
makes it very hard to use pulumi in other projects with other
dependencies.

Fixes 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Fraser Waters <frassle@gmail.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-10-22 18:30:21 +00:00
Justin Van Patten 86eee44bf8
bump golang.org/x/net from 0.10.0 to 0.17.0 ()
This PR replaces all the dependabot PRs with a single commit that
updates all relevant go.mod files.

This resolves 3 Dependabot alerts on golang.org/x/net including a
moderate severity alert.
2023-10-20 18:36:16 +00:00
Thomas Gummerer 80acb16a54
pass through property values to RunRequest ()
# Description

This is an alternative to https://github.com/pulumi/pulumi/pull/14244.
Instead of adding type information to the run request, pass the config
through as property values. Property values are properly encoded on the
wire, and can be unmarshalled on the other end including type
information, so this should be a more future proof way to go forward.

Eventually we'll want to parse the config directly into property values,
but that can be left for the future, as it's a bigger change.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-20 10:44:16 +00:00
Fraser Waters 49247554df
Wait on event rather than sleep in dynamic provider ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This might help with https://github.com/pulumi/pulumi/issues/14159,
we've seen some oddities with sleep not being interrupted.
2023-10-17 08:08:45 +00:00
Fraser Waters 1ae75580d4
Support diagnostics in state conversion ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Part of fixing
https://github.com/pulumi/pulumi-converter-terraform/issues/42. Allow
the converter to warn the user which resources have missing mapping
data, and thus are likely to not import.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-17 08:08:39 +00:00
Fraser Waters ca12edfb4e
Send old inputs to Delete ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14115.

This was missed as part of https://github.com/pulumi/pulumi/pull/13139.

Adds a new configure flag (sends_old_inputs_to_delete) which the engine
will now always set to true. If that's set providers can rely on the old
inputs being sent to delete, otherwise they'll get nil.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-13 14:12:26 +00:00
Joe Duffy 96a9a77167
Policy remediations feature ()
This PR implements the new policy transforms feature, which allows
policy packs to not only issue warnings and errors in response to policy
violations, but actually fix them by rewriting resource property state.
This can be used, for instance, to auto-tag resources, remove Internet
access on the fly, or apply encryption to storage, among other use
cases.
2023-10-09 18:31:17 +00:00
Thomas Gummerer b5077dc7d3
fix MockMonitor reporting DeletedWith wasn't supported ()
We should support all features in mocks that we are supporting
otherwise. We did a similar thing in pulumi-dotnet:
https://github.com/pulumi/pulumi-dotnet/pull/93.

However here we're not simply changing hasSupport to true.
1e09626bc7
has a good explanation of why we only support specific values here.
However really we only need to disable support for `outputValues`, and
that only if output values are disabled. Originally this was true if
`PULUMI_ENABLE_OUTPUT_VALUES` was set, but that environment variable was
later inverted: 3027d01f25.

This ended up being a little bit more complicated than just passing
true, but I think it's more correct this way. I'm also happy to change
it back to just passing through `true` if that's preferred. Once we
decided on whether this is the right way to do it vs. just passing
`true` unconditionally I'll also update `java` and `dotnet` (if we're
going this way) to do the same.

Fixes 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-09 14:21:54 +00:00
Justin Van Patten 7c19234e43
[sdk/python] Fix typo in error message ()
Fix a typo in new error message that was just introduced.
2023-10-02 14:40:01 +00:00
Justin Van Patten 55abd8f63a
[sdk/python] Add `default` arg to `Config.get_secret` ()
This change adds the missing `default` arg to `Config.get_secret`
method, for consistency with the other `get_secret_<type>` methods that
have a `default` arg.

Fixes 
2023-10-02 13:43:48 +00:00
Fraser Waters cac2e0c2ee
Type check props in the `Resource` initializer ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/13997.

It would be a little nicer if every input type _actually_ implemented
the `Mapping` interface, but that trying to do that turned into a much
deeper fix than this one.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-25 14:54:28 +00:00
Fraser Waters 183b47bf06
More efficent mapping lookup ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Inspired by a comment from Zaid. This allows providers to return what
providers they have mapping information for without having to marshal
all their mapping data to the engine at the same time, this could save
transmitting a lot of data that the engine might not ever need (for
example if it's not converting code for that specific provider). It also
allows provider to support mulitple mappings.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-21 11:45:07 +00:00
Fraser Waters d2cf01284b
Send args from convert to converter plugin ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This matches up with how we send args to converter plugins from
`import`. Looks like this will be useful in at least two places already.
Firstly the bicep converter needs to know which file to convert.
Secondly the terraform convert needs a --convert-examples (or the like)
flag to turn on it's bulk converter mode for the terraform bridge to
use.

Also fixes that we we're limiting `import` to three args, but a
converter may take more.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-18 11:01:13 +00:00
Fraser Waters c27d768bb1
Add matrix testing ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Adds the first pass of matrix testing.

Matrix testing allows us to define tests once in pulumi/pulumi via PCL
and then run those tests against each language plugin to verify code
generation and runtime correctness.

Rather than packing matrix tests and all the associated data and
machinery into the CLI itself we define a new Go package at
cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment
engine in a unique way for matrix tests but it is running the proper
deployment engine with a proper backend (always filestate, using $TEMP).

Currently only NodeJS is hooked up to run these tests, and all the code
for that currently lives in
sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll
move that helper code to sdk/go/common and use it in each language
plugin to run the tests in the same way.

This first pass includes 3 simple tests:
* l1-empty that runs an empty PCL file and checks just a stack is
created
* l1-output-bool that runs a PCL program that returns two stack outputs
of `true` and `false
* l2-resource-simple that runs a PCL program creating a simple resource
with a single bool property

These tests are themselves tested with a mock language runtime. This
verifies the behavior of the matrix test framework for both correct and
incorrect language hosts (that is some the mock language runtimes
purposefully cause errors or compute the wrong result).

There are a number of things missing from from the core framework still,
but I feel don't block getting this first pass merged and starting to be
used.

1. The tests can not currently run in parallel. That is calling
RunLanguageTest in parallel will break things. This is due to two
separate problems. Firstly is that the SDK snapshot's are not safe to
write in parallel (when PULUMI_ACCEPT is true), this should be fairly
easy to fix by doing a write to dst-{random} and them atomic move to
dst. Secondly is that the deployment engine itself has mutable global
state, short term we should probably just lock around that part
RunLanguageTest, long term it would be good to clean that up.
2. We need a way to verify "preview" behavior, I think this is probably
just a variation of the tests that would call `stack.Preview` and not
pass a snapshot to `assert`.
3. stdout, stderr and log messages are returned in bulk at the end of
the test. Plus there are a couple of calls to the language runtime that
don't correctly thread stdout/stderr to use and so default to the
process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load
of other places as well so I'm thinking of a clean way to handle all of
them together. Log message streaming we can probably do by just turning
RunLanguageTest to a streaming grpc call.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
Fraser Waters a254a53206
Pass provider checksums in requests and save to state ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This extends the resource monitor interface with fields for plugin
checksums (on top of the existing plugin version and download url
fields). These fields are threaded through the engine and are persisted
in resource state. The sent or saved data is then used when installing
plugins to ensure that the checksums match what was recorded at the time
the SDK was built.

Similar to https://github.com/pulumi/pulumi/pull/13776 nothing is using
this yet, but this lays the engine side plumbing for them.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-11 15:54:07 +00:00
Fraser Waters 981be6c458
Add args to state conversions ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

When importing from other ecosystems we need someway in `import` for the
user to tell the converter plugin _what_ to import.

Simply forwarding on the args pass to `import` looks to be the simplest
option to enable this.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-01 18:07:49 +00:00
Fraser Waters 0ade454f11
Allow language plugins to return plugin checksums ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Also fix an issue where if a platform was missing a checksum it would
error with "invalid checksum, expected , actual 01234".

None of the language runtimes yet return anything for this, but it's a
simple plumbing to expose it for the future.

We'll _probably_ start adding checksums to the pulumi-plugin.json files,
and then GetRequiredPlugins can simply return that data.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-08-25 15:26:25 +00:00
Zaid Ajaj 3f54914149
[sdk/python] Update grpc-io from 1.56.0 to 1.56.2 ()
# Description

In  Snyk has updated our reference to grpc-io in the
requirements.txt to 1.56.2 but didn't update `setup.py` which results in
installation conflicts as reported in . This PR updates `setup.py`
to be in sync with `requiremments.txt`

Fixes 

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-08-23 13:52:13 +00:00
Fraser Waters 818caae680 Pass loader to convert plugins 2023-08-09 22:02:57 +01:00
Fraser Waters 182e776660 Add hidden sdk-pack command
We need to be able to "pack" SDKs to refer to them as local dependencies
in matrix testing.

This is for two reasons.
1) We want to test as close as possible to the things we ship.
2) Not every language supports linking to a source tree, some require a
build step to give a linkable artifact.

These commands are going to end up looking _very_ similar to the publish
workflows, but while Providers work on that and while we work on matrix
testing we'll let them evolve in parallel.

The sdk-pack command is hidden unless PULUMI_DEV is set. I've checked
this works with matrix testing for NodeJS. We'll fill in the rest as we
need them for matrix testing.
2023-08-08 17:53:04 +01:00
Fraser Waters 99b736b55b Add localDependencies option to GenerateProject
This isn't currently actually used anywhere. I've just threaded it
through to all the program gen functions where it will be needed.

Matrix testing will be using and testing this.
2023-08-08 12:28:19 +01:00
bors[bot] 58b96975ab
Merge
13605: Add schema loader service r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This moves schema loading out of the language runtimes and over to the engine host.
Language runtimes no longer need to create a plugin host, or diagnostic sink either because of this.

All schema loading is done over grpc. This first pass is _very_ basic, and not expected to be performant but it moves the control of schema loading to the engine which is necessary for matrix testing.

Testing of this is covered by the convert and code generation smoke tests.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


13608: Warn about undefined stack outputs r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/7376

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-07-28 16:38:52 +00:00
Abhinav Gupta 91a079851b
deps: Upgrade google.golang.org/{genproto, grpc}
Updates to the latest versions of
google.golang.org/genproto and google.golang.org/grpc
in all submodules in the repository.

This is necessary because in a recent change,
genproto split out some of its subpackages into independent submodules.
(https://github.com/googleapis/go-genproto/issues/1015)

As a result of this, some users may see the error:

```
google.golang.org/genproto/googleapis/rpc/status: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
    google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 (/home/runner/go/pkg/mod/google.golang.org/genproto@v0.0.0-20230410155749-daa745c078e1/googleapis/rpc/status)
    google.golang.org/genproto/googleapis/rpc v0.0.0-20230725213213-b022f6e96895
```

Because pu/pu is using 20230410155749,
which has googleapis/rpc as a subpackage,
but another dependency references the independent submodule (20230725213213),
so the system doesn't know which module to use for the import path,
google.golang.org/genproto/googleapis/rpc/status.

This is a problem for codegen tests and ProgramTest-based tests
for Pulumi Go programs that do not have a go.mod in the test directory.
This issue was encountered by @thomas11 while attempting to upgrade
dependencies in pulumi-docker ().

The grpc upgrade is necessary because the current version of grpc
also pulls the outdated version of genproto.
2023-07-27 16:24:33 -07:00
Fraser Waters fd5000b32c Add schema loader service
This moves schema loading out of the language runtimes and over to the
engine host.
Language runtimes no longer need to create a plugin host, or diagnostic
sink either because of this.

All schema loading is done over grpc. This first pass is very basic, and
not expected to be performant but it moves the control of schema loading
to the engine which is necessary for matrix testing.

Testing of this is covered by the convert and code generation smoke tests.
2023-07-27 15:03:52 +01:00
bors[bot] e0486a8d98
Merge
13603: Make `Output.from_input` recurse into tuples. r=Frassle a=Frassle



<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/6635.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-07-27 13:57:11 +00:00
Fraser Waters a28066fea6 Make MapperAddress consistent everywhere by calling it MapperTarget 2023-07-27 11:48:28 +01:00
Fraser Waters bba69e2f38 Make `Output.from_input` recurse into tuples.
Fixes https://github.com/pulumi/pulumi/issues/6635.
2023-07-27 10:13:40 +01:00
Fraser Waters 442924b405 Add the ability to verify an expected log message was written
We used to test that various errors we're written out to the user in
these langhost tests, but at some point those test were removed (I
suspect because we stopped printing them to stderr).

This adds in the ability to check for a log message in the engine
diagnostics stream, and makes use of that in one of the tests.
2023-07-17 13:18:57 +01:00
Pat Gavlin 248f78bafe [sdk/*] Add support for resource source positions
Add support to the core SDKs for reporting resource source positions.

In each SDK, this is implemented by crawling the stack when a resource
is registered in order to determine the position of the user code that
registered the resource.

This is somewhat brittle in that it expects a call stack of the form:
- Resource class constructor
- abstract Resource subclass constructor
- concrete Resource subclass constructor
- user code

This stack reflects the expected class hierarchy of "cloud resource /
component resource < customresource/componentresource < resource".

For example, consider the AWS S3 Bucket resource. When user code
instantiates a Bucket, the stack will look like
this in NodeJS:

    new Resource (/path/to/resource.ts:123:45)
    new CustomResource (/path/to/resource.ts:678:90)
    new Bucket (/path/to/bucket.ts:987:65)
    <user code> (/path/to/index.ts:4:3)

In order to determine the source position, we locate the fourth frame
(the `<user code>` frame).
2023-07-13 16:46:04 -07:00
Justin Van Patten 499e381299 [sdk/python] Support explicit providers for packaged components
Updates the Python SDK to support passing an explicit provider.
2023-07-13 07:17:37 -07:00
Justin Van Patten ef41442211 [sdk/python] Add langhost test for packaged component providers
Currently fails without the fix.
2023-07-13 07:16:56 -07:00
bors[bot] 772a057ee1
Merge
13463: Fix links to outputs docs r=cnunciato a=cnunciato

Fixes some broken links pointing to the Inputs & Outputs docs.

Fixes .

Co-authored-by: Christian Nunciato <chris@nunciato.org>
2023-07-13 12:34:13 +00:00
bors[bot] 6daf8622a8
Merge
13458: Make pythons RPCManager a context variable r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Global variables cause issues when running programs in the same process in parallel which we can do with automation api, but also in the test framework. This changes the RPCManager to be a context variable on Settings, along with other "program global" data like the resource monitor and engine logger.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-07-13 09:37:45 +00:00
Fraser Waters 8db3087d2b Make pythons RPCManager a context variable 2023-07-13 09:59:12 +01:00
Christian Nunciato 202fbd946f Fix links outputs docs 2023-07-12 14:59:07 -07:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
bors[bot] acad217564
Merge
13315: Always mark dynamic provider code as secret r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Short term workaround for https://github.com/pulumi/pulumi/issues/8265

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-07-06 18:25:09 +00:00
bors[bot] 95b804911b
Merge
13405: deps(go): Upgrade to grpc 1.56.1 r=abhinav a=abhinav

Upgrades to gRPC Go 1.56.1 to resolve the influx of dependabot PRs.
Supersedes all dependabot PRs created for this.

Fixes CVE-2023-32731

13432: deps(py): upgrade to latest grpcio r=abhinav a=abhinav

Upgrades to the latest release of grpcio for the Python SDK.

Supersedes , 
Fixes CVE-2023-32731

Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-07-06 16:41:37 +00:00
Abhinav Gupta 1447ee55c3
deps(py): upgrade to latest grpcio
Upgrades to the latest release of grpcio for the Python SDK.

Supersedes , 
Fixes CVE-2023-32731
2023-07-06 09:04:59 -07:00
Abhinav Gupta f59ab49fc5
deps(go): Upgrade to grpc 1.56.1
Upgrades to gRPC Go 1.56.1 to resolve the influx of dependabot PRs.
Supersedes all dependabot PRs created for this.

Fixes CVE-2023-32731
2023-07-06 09:04:16 -07:00
Fraser Waters 4baf4a1468 Always mark dynamic provider code as secret 2023-07-06 13:50:59 +01:00
Justin Van Patten 62183ddf11 [sdk/python] Fix comment about `ROOT_STACK_RESOURCE`
Looks like this was copied from the Node.js SDK. It should be `ROOT_STACK_RESOURCE` not `rootStackResource` for Python.
2023-06-29 12:27:16 -07:00
bors[bot] 1756921060
Merge
13276: Bump semver and @pulumi/pulumi r=Frassle a=justinvp

Combines the following dependabot changes into a single PR:
- 
- 
- 
- 
- 
- 
- 

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2023-06-29 12:30:44 +00:00