Commit Graph

30 Commits

Author SHA1 Message Date
Justin Van Patten 40f03fb299
Update dotnet to 3.71.1 ()
Also update the minimum required SDK version for codegen to 3.71.1 as
it's necessary to be able to use parameterized explicit providers
(https://github.com/pulumi/pulumi-dotnet/pull/435).
2024-12-20 12:52:58 +00:00
Julien 6622f8a4e6
Codegen for .NET InvokeOutputOptions ()
Add codegen for https://github.com/pulumi/pulumi-dotnet/pull/412

Generate variants of output form invokes that take `InvokeOutputOptions`
as options, which allow specifying DependsOn. This requires a release of
https://github.com/pulumi/pulumi-dotnet/pull/412 first.
2024-12-11 20:07:57 +00:00
Julien 06886862ff
Update minimum SDK versions for Node.js and Python ()
https://github.com/pulumi/pulumi/pull/17751 &
https://github.com/pulumi/pulumi/pull/17632 require a more recent SDK
with InvokeOutputOptions. This also drops the distinction between the
minimum version for parameterised and non-parameterised SDKs.

In the future, we likely want codegen to set the current version (of the
codegen code) as minimum version.

The first commit contains the actual code changes, other commits updated
the test snapshots.
2024-12-10 18:14:38 +00:00
Julien 42257d8137
[Go] Allow specifying dependencies for output invokes ()
Provider functions that take inputs as arguments, and return an output
(aka output invokes), now allow specifying a `dependsOn` option. This
allows programs to ensure that invokes are executed after things they
depend on, similar to the [`depdendsOn` resource
option](https://www.pulumi.com/docs/iac/concepts/options/dependson/).

https://github.com/pulumi/pulumi/discussions/17710

This PR introduces a new `Context.InvokeOutput` method that the
generated SDK code uses to call the invoke, and pushes secret and
dependency tracking back into the Go SDK instead of the generated SDK
code.

Stacked on https://github.com/pulumi/pulumi/pull/17818

Fixes https://github.com/pulumi/pulumi/issues/17752

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-12-10 13:08:57 +00:00
Julien c3d216beab
Drop Python 3.8 support ()
Python 3.8 became EOL on 2024-10-07 with the release of Python 3.13, see
https://devguide.python.org/versions/.

Note that our docker containers did not support 3.8, and provided 3.9 as
minimum version.

This does not yet make any changes that break compatibility with 3.8,
but going forward we will not test on that version anymore.

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-12-02 16:13:08 +00:00
Julien cfd7ce00ef
[Node.js] Allow specifiying dependencies for output invokes ()
Provider functions that take inputs as arguments, and return an output
(aka output invokes), now allow specifying a `dependsOn` option. This
allows programs to ensure that invokes are executed after things they
depend on, similar to the [`depdendsOn` resource
option](https://www.pulumi.com/docs/iac/concepts/options/dependson/).

This is based on https://github.com/pulumi/pulumi/pull/16560 which was
reverted in https://github.com/pulumi/pulumi/pull/16642. Unlike the
previous PR, this only adds the `dependsOn` option to output invokes
(those that return an output, and use SDK invokeOutput function
internally).

https://github.com/pulumi/pulumi/discussions/17710

Fixes https://github.com/pulumi/pulumi/issues/17748
2024-11-20 16:03:11 +00:00
Julien bd0be8e8ef
[Python] Allow specifiying dependencies for output invokes ()
Provider functions that take inputs as arguments, and return an output
(aka output form invokes), now allow specifying a depends_on option.
This allows programs to ensure that invokes are executed after things
they depend on, similar to the depdends_on resource option.

Fixes https://github.com/pulumi/pulumi/issues/17749
2024-11-20 12:28:41 +00:00
Zaid Ajaj e9bb24c3ea
[sdkgen/python-nodejs] Bump minimum required SDK version to v3.136.0 ()
Requires 

Bumps the emitted minimum SDK version for nodejs and python core sdks to
the version that fixes regressions in output invokes
2024-10-08 02:28:26 +00:00
Julien de17d1a70f
Replace asyncio.ensure_future with create_task ()
`ensure_future` was deprecated in Python 3.10
https://docs.python.org/3/library/asyncio-future.html#asyncio.ensure_future

The preferred way to convert a coroutine into a future is to use
`create_task` instead, which was introduced in Python 3.7, so safe for
us to use everywhere.

Ref https://github.com/pulumi/pulumi/issues/11827
2024-09-30 12:53:15 +00:00
Zaid Ajaj 86eba9bd6a
[sdk-gen/node/python] Bump minimum valid SDK to v3.134.1 so that output-invokes work and maintain secrets ()
### Description

This PR updates SDK-gen for python and nodejs so that they emit latest
published SDK v3.134.1 (at the time of writing) which includes fixes for
output-invokes.

Fixes 
2024-09-26 17:45:27 +00:00
Zaid Ajaj c7ec41f0c5
[sdk/nodejs] Short-circuit output invokes if inputs contains unknowns ()
### Description

 introduced output invokes that don't rely on plain invokes,
however since they now no longer await inputs in the generated provider
sdks and instead await them in the core sdk, we should short-circuit the
output invoke if it contained any unknown values so that the behaviour
matches what was happening before the change.
2024-09-25 20:03:13 +00:00
Zaid Ajaj a7a4870864
[python] First-class output-versioned invokes that maintain secrets ()
Partially addressing https://github.com/pulumi/pulumi/issues/12710 for
Python

This PR extends the python SDK with a new function `invoke_output` which
returns a first class `pulumi.Output[T]` in Python which is then emitted
in the generated code for output-versioned invokes in Python SDK where
these no longer rely on the plain invoke nor they need to use the
lifting utility
2024-09-22 18:23:27 +00:00
Justin Van Patten 597f2145b9
Fully generate TypedDict input types by default ()
https://github.com/pulumi/pulumi/pull/16704 made it so that classes and
TypeDicts are generated for input types by default, but missed some
cases, which lead to referencing some TypedDict types that didn't exist.

This PR fixes that by ensuring that all input types are generated.

A regression test is included that runs pyright in strict mode to ensure
TypedDict types in signatures actually exist.

The second commit regenerates testdata throughout.

Fixes 
2024-09-20 17:16:14 +00:00
Zaid Ajaj a41b5e6762
[sdk-gen/nodejs] Bump valid nodejs version to v3.133 the one that contains invokeOutput and friends ()
### Description

Version v3.133.0 includes  which adds new SDK functions
`invokeOutput` and `invokeSingleOutput`, this PR changes sdk-gen such
that the required `@pulumi/pulumi` dependency is that one that contains
the former primitives.
2024-09-20 12:47:16 +00:00
Zaid Ajaj f74a5dfec1
[nodejs] First-class output-versioned invokes ()
### Description 

Partially addressing https://github.com/pulumi/pulumi/issues/12710

This PR extends the nodejs SDK with functions `invokeOutput` and
`invokeSingleOutput` which are the output-versioned equivalent of the
plain `invoke` and `invokeSingle`. The underlying implementation doesn't
rely on the plain one and properly implements output deserialization
such that secrets are maintained from the invoke response.

Then we extend the SDK-gen part of nodejs such that output-versioned
invokes use the new primitives `invokeOutput` and `invokeSingleOutput`
in their generated function body without wrapping the plain invoke.
2024-09-16 15:39:46 +00:00
Zaid Ajaj ebf838e02e
[go] Allow output-versioned invokes to resolve and maintain secrets ()
Partially addressing 

### Description

This PR extends Go SDK-gen, specifically for output-versioned invokes
such that they no longer rely on the plain invokes for their function
body.

We do this by implementing and using a sdk function `InvokePackageRaw`
which is similar to `InvokePackage` except that it doesn't fail on
secrets and actually returns a boolean indicating whether the invoke
response contained any secrets. This way, the generated output-versioned
invokes can immediately wrap the response as a secret if necessary and
more importantly, not failing immediately if the response contained
secrets.

The sdk-gen test `output-funcs` actually do a runtime test for the newly
generated function body and it passes. ~~However, I don't think it
covers the _if secret then wrap as secret_ path~~ but maybe that's
acceptable because it's a simple one liner
```go
// <invoke function body ommited here>
if secret {
  return pulumi.ToSecret(output).(FuncResultOutput), nil
}
```
Follow-up PRs with less priority:
 - [x] A proper test for invokes with secrets in their response
 - [ ] A conformance test
- [ ] Doing the same for _generic_ output-versioned invokes to let them
maintain secretness

### EDIT
Updated the test in output-funcs such that it now returns a response for
an invoke containing secrets
2024-09-09 12:57:43 +00:00
Zaid Ajaj 377a4b461f
Use Pulumi v3.66.1 in .NET sdk-gen ()
Follow up to https://github.com/pulumi/pulumi/pull/16912 where we no
longer use faulty package v3.66.0 and instead use v3.66.1
2024-08-09 22:12:28 +00:00
Zaid Ajaj 5fedf392e6
[sdk-gen] Support parameterized .NET provider SDKs ()
This PR extends sdk-gen for .NET to support parameterized providers.
When a schema has a parameterization, we emit a new function
`PackageParameterization()` in the `Utilities` class and call this
function for generated resources, invokes and methods.

It requires https://github.com/pulumi/pulumi-dotnet/pull/311 to be
merged first and a new version of Pulumi nuget package to be released,
then we can bump the version we use here.

EDIT: Now using Pulumi nuget package v3.66.0 
2024-08-09 13:16:20 +00:00
Julien 21ed58b719
Link to Python docs about input types ()
In Python object inputs can either be argument classes or dictionary
literals. Link to the [Python input
docs](https://www.pulumi.com/docs/languages-sdks/python/#inputs-and-outputs)
in the input section of the package docs when the current language is
python.

Fixes https://github.com/pulumi/registry/issues/4936

<img width="738" alt="Screenshot 2024-07-30 at 18 23 53"
src="https://github.com/user-attachments/assets/3ad00f45-a81d-4c2d-b3ad-f9ae811735f6">
2024-07-31 08:46:04 +00:00
Julien ca60d337e2
Generate TypedDict input types by default ()
With https://github.com/pulumi/pulumi/pull/15957 we introduced the
schema setting `Languages.Python.InputTypes` to optionally generate
TypedDict based input types side-by-side with Args classes. This setting
defaulted to `classes`, meaning that only Args classes are generated. To
enable the TypedDict types, SDK authors had to explicitly opt in by
setting it to `classes-and-dicts`.

We now flip this setting to generating TypedDict input types
side-by-side with the Args classes, unless explicitly disabled by
setting it to `classes`.

Fixes https://github.com/pulumi/pulumi/issues/16375
2024-07-23 11:26:54 +00:00
Fraser Waters a9947b4e4e
Python parameterized provider test ()
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.

When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.

The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
2024-07-16 10:55:38 +00:00
Florian Stadler 74e8928cff
Add ability to constrain supported languages of resource and function overlays ()
The existing overlays (e.g. Chart v3 in Kubernetes, or CallbackFunction
in AWS) are not available in every language Pulumi supports. This often
confuses users because the generated docs include all languages Pulumi
supports (e.g. see
https://github.com/pulumi/pulumi-kubernetes/issues/2181).

To solve that problem, this change adds a new optional parameter to the
schema that allows configuring the languages an overlay (resource or
function) supports.
To support this in docsgen the existing Language Chooser
(`LangChooserLanguages`) of resources is made configurable and extended
to functions.

Note: This doesn't support resource methods right now. They'll need
extra handling because and overlay resource method might not support all
of the languages its resource supports. I'll tackle this in a follow up
PR.

Here's a screenshot of how this will look like for the Helm v3 chart for
example:
<img width="1046" alt="Screenshot 2024-07-01 at 16 11 23"
src="https://github.com/pulumi/pulumi/assets/2453580/b1a1365a-6dee-4099-829a-2859639a4c8c">

The PR contains the following commits. I'd recommend to look at the
first three ones and then check the regenerated golden files in the last
one:
- **Add schema parameter to constrain supported languages for overlays**
- **Update developer docs and changelog**
- **Refactor LanguageChooser and always pass supported languages**
- **Regenerate testdata**

relates to 
2024-07-09 14:54:50 +00:00
Fraser Waters 969e0b9735
Conformance test for provider pre-release versions ()
Noticed this issue while doing SDK gen for parameterised providers, but
figured it deserved its own conformance test. Check that if a provider
has a pre-release semver that the _exact_ version can be reported by the
generated SDK. This already just works for NodeJS, but Python needed a
fix to write the version to `_utilities.py` rather than trying to
unconvert the pypi version from the package.

Also needed to make the conformance test checks for
`GetProgramDependencies` even weaker (which is fine, they are just a
very basic sanity check) because the provider reports a version of
"3.0.0-alpha.1.internal" while the python version is "3.0.0a1+internal".
2024-06-29 10:07:14 +00:00
Justin Van Patten b29807f2b4
[sdkgen/dotnet] Bump Pulumi version reference ()
Bump the default version range for the referenced `Pulumi` package to a
version that supports `Config.GetDouble` (3.55+;
380536dd85).

Fixes 
2024-06-27 05:16:14 +00:00
Daniel Bradley c6a01328d2
Set range in python codegen default SDK version ()
If the schema doesn't set any specific dependencies, we should be
defaulting to requiring pulumi within the current major version rather
than completely unconstrained.

Discussion:
https://github.com/pulumi/pulumi-kafka/pull/410/files#r1594324390
2024-06-24 13:57:57 +00:00
Will Jones 271553d462
Generate `@deprecated` decorators in Python SDKs ()
The latest version of the core Pulumi SDK contains a decorator,
`@deprecated`, that is used when generating SDK code in order to signify
deprecated properties in a way that can be recognised by other SDK code.
This is useful when writing generic Python code that e.g. traverses
class properties without triggering deprecation warnings for those not
explicitly mentioned in user code. The [original pull
request](https://github.com/pulumi/pulumi/pull/16400) has more details.

Alas, we can't rely on the fact that a user will upgrade _both_ a
particular (generated) provider SDK and the core Pulumi SDK at the same
time. Thus, it's entirely possible that a user bumps their version of
(say) `pulumi_aws`, whilst leaving their `pulumi` library at the same
(compatible, according to specified bounds) version. In doing so they'd
hit errors when the new SDK tried to import the `@deprecated` decorator,
which doesn't exist in the old core SDK.

This commit thus fixes this by altering code generation so that each SDK
receives its own inlined copy of the `@deprecated` decorator, which it
can reference regardless of the version of the core SDK. This decorator
applies the same `_pulumi_deprecated_callable` tag to functions it
decorates, which a sufficiently modern SDK will recognise to avoid
triggering e.g. https://github.com/pulumi/pulumi/issues/15894. Later on,
we can hopefully find a way to avoid doing this and use only a version
of `@deprecated` specified in the core SDK.

Codegen tests have been updated and the inlined decorator has manually
been tested using the AWS Classic SDK.

Addresses
https://github.com/pulumi/pulumi/pull/16400#discussion_r1646562455
2024-06-21 11:34:29 +00:00
Sean Holung c53526661b
Update page layout for new constructor syntax ()
Moves new syntax underneath the same section as original constuctor
definition.

<img width="660" alt="Screen Shot 2024-04-01 at 12 17 03 PM"
src="https://github.com/pulumi/pulumi/assets/16751381/d85969d6-c480-4518-a9ef-7cd6f07e353c">

---------

Co-authored-by: Christian Nunciato <chris@nunciato.org>
2024-04-02 15:47:32 +00:00
Fraser Waters 3043adeeb3
Add SupportPack to schemas to write out in the new style ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->


This adds a new flag to the schema metadata to tell codegen to use the
new proposed style of SDKs where we fill in versions and write go.mods
etc.

I've reworked pack to operate on packages assuming they're in this new
style. That is pack no longer has the responsibility to fill in any
version information.

This updates python and node codegen to write out SDKs in this new
style, and fixes their core libraries to still be buildable via pack.
There are two approaches to fixing those, I've chosen option 1 below but
could pretty easily rework for option 2.

1) Write the version information directly to the SDKs at the same time
as we edit the .version file. To simplify this I've added a new
'set-version.py' script that takes a version string an writes it to all
the relevant places (.version, package.json, etc).

2) Write "pack" in the language host to search up the directory tree for
the ".version" file and then fill in the version information as we we're
doing before with envvar tricks and copying and editing package.json.

I think 1 is simpler long term, but does force some amount of cleanup in
unrelated bits of the system right now (release makefiles need a small
edit). 2 is much more localised but keeps this complexity that
sdk/nodejs sdk/python aren't actually valid source modules.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-22 09:25:46 +00:00
Zaid Ajaj ad56486bf0
[docs] Emit example constructor syntax for resources in typescript, python, go and csharp ()
# Description

This PR is an initial implementation of emitting constructor syntax of
resources into the docs for typescript, python, go and csharp.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-21 13:41:07 +00:00
Anton Tayanovskyy d62c398bfb
Move codegen testdata ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

It appears that Go copies testdata into every GOMODCACHE of a project
that depends on pulumi/pkg; the schemas in codegen testdata add 300MB of
weight to the GOMODCACHE needed for download. What if we moved the
testdata out from under the tree.

The move looks like this:

```
from=pkg/codegen/testing/test/testdata
to=tests/testdata/codegen/
git mv "$from" "$to"
(cd pkg/codegen/testing/test && ln -s ../../../../tests/testdata/codegen ./testdata)
git add "$from"
```

The previous location is symlinked to the new location.

Evidence of `GOMODCACHE` pressure reduction:
https://gist.github.com/t0yv0/05dd8be5880171045aed01e123ae2b09

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-06 20:36:50 +00:00