With https://github.com/pulumi/pulumi/pull/15957 we introduced the
schema setting `Languages.Python.InputTypes` to optionally generate
TypedDict based input types side-by-side with Args classes. This setting
defaulted to `classes`, meaning that only Args classes are generated. To
enable the TypedDict types, SDK authors had to explicitly opt in by
setting it to `classes-and-dicts`.
We now flip this setting to generating TypedDict input types
side-by-side with the Args classes, unless explicitly disabled by
setting it to `classes`.
Fixes https://github.com/pulumi/pulumi/issues/16375
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.
When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.
The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
Noticed this issue while doing SDK gen for parameterised providers, but
figured it deserved its own conformance test. Check that if a provider
has a pre-release semver that the _exact_ version can be reported by the
generated SDK. This already just works for NodeJS, but Python needed a
fix to write the version to `_utilities.py` rather than trying to
unconvert the pypi version from the package.
Also needed to make the conformance test checks for
`GetProgramDependencies` even weaker (which is fine, they are just a
very basic sanity check) because the provider reports a version of
"3.0.0-alpha.1.internal" while the python version is "3.0.0a1+internal".
The latest version of the core Pulumi SDK contains a decorator,
`@deprecated`, that is used when generating SDK code in order to signify
deprecated properties in a way that can be recognised by other SDK code.
This is useful when writing generic Python code that e.g. traverses
class properties without triggering deprecation warnings for those not
explicitly mentioned in user code. The [original pull
request](https://github.com/pulumi/pulumi/pull/16400) has more details.
Alas, we can't rely on the fact that a user will upgrade _both_ a
particular (generated) provider SDK and the core Pulumi SDK at the same
time. Thus, it's entirely possible that a user bumps their version of
(say) `pulumi_aws`, whilst leaving their `pulumi` library at the same
(compatible, according to specified bounds) version. In doing so they'd
hit errors when the new SDK tried to import the `@deprecated` decorator,
which doesn't exist in the old core SDK.
This commit thus fixes this by altering code generation so that each SDK
receives its own inlined copy of the `@deprecated` decorator, which it
can reference regardless of the version of the core SDK. This decorator
applies the same `_pulumi_deprecated_callable` tag to functions it
decorates, which a sufficiently modern SDK will recognise to avoid
triggering e.g. https://github.com/pulumi/pulumi/issues/15894. Later on,
we can hopefully find a way to avoid doing this and use only a version
of `@deprecated` specified in the core SDK.
Codegen tests have been updated and the inlined decorator has manually
been tested using the AWS Classic SDK.
Addresses
https://github.com/pulumi/pulumi/pull/16400#discussion_r1646562455
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds a new flag to the schema metadata to tell codegen to use the
new proposed style of SDKs where we fill in versions and write go.mods
etc.
I've reworked pack to operate on packages assuming they're in this new
style. That is pack no longer has the responsibility to fill in any
version information.
This updates python and node codegen to write out SDKs in this new
style, and fixes their core libraries to still be buildable via pack.
There are two approaches to fixing those, I've chosen option 1 below but
could pretty easily rework for option 2.
1) Write the version information directly to the SDKs at the same time
as we edit the .version file. To simplify this I've added a new
'set-version.py' script that takes a version string an writes it to all
the relevant places (.version, package.json, etc).
2) Write "pack" in the language host to search up the directory tree for
the ".version" file and then fill in the version information as we we're
doing before with envvar tricks and copying and editing package.json.
I think 1 is simpler long term, but does force some amount of cleanup in
unrelated bits of the system right now (release makefiles need a small
edit). 2 is much more localised but keeps this complexity that
sdk/nodejs sdk/python aren't actually valid source modules.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
It appears that Go copies testdata into every GOMODCACHE of a project
that depends on pulumi/pkg; the schemas in codegen testdata add 300MB of
weight to the GOMODCACHE needed for download. What if we moved the
testdata out from under the tree.
The move looks like this:
```
from=pkg/codegen/testing/test/testdata
to=tests/testdata/codegen/
git mv "$from" "$to"
(cd pkg/codegen/testing/test && ln -s ../../../../tests/testdata/codegen ./testdata)
git add "$from"
```
The previous location is symlinked to the new location.
Evidence of `GOMODCACHE` pressure reduction:
https://gist.github.com/t0yv0/05dd8be5880171045aed01e123ae2b09
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->