Part of https://github.com/pulumi/pulumi/issues/17507
For importing resources of parameterized resources we need to be able to
find the parameterized package descriptions without running the users
program. This is the first part of fixing that. We add the parameterized
information about the package into it's pulumi-plugin.json file, so this
file now describes the overall package not just the base provider
plugin.
Once this is merged there are two follow ups;
1) See about fixing sdk-gen to not emit the large byte value twice (in
the source code where we currently have it, and now in the
pulumi-plugin.json). We can probably fix all the sdks to just read from
the embedded plugin file.
2) Update the language hosts to support a "GetAllPackages" instead of
"GetAllPlugins" to read out this information from the pulumi-plugin.json
files.
Currently this doesn't behave exactly how we want, nested secret values
aren't correctly wrapped. We plan on fixing that in the engine though so
as a conformance test for languages this test is still valid.
Provider functions that take inputs as arguments, and return an output
(aka output form invokes), now allow specifying a depends_on option.
This allows programs to ensure that invokes are executed after things
they depend on, similar to the depdends_on resource option.
Fixes https://github.com/pulumi/pulumi/issues/17749
### Description
As part of fixing
https://github.com/pulumi/pulumi-converter-terraform/issues/186 we need
to be able to express resources and packages in PCL that are coming from
parameterized providers.
Today in PCL we will extract the package name from the token of a
resource definition and load that package based on the name (using
`LoadReference`) . For parameterized packages, this will not work
because the schema of the parameterized resource is only available after
we figure out which base provider is being parameterized and what the
parameters are.
In this PR we fix this problem by implementing top-level `package`
blocks that are full package descriptors that contain sufficient
information to load both classic resource plugins and parameterized
packages. The block will looks like:
```tf
package <name> {
baseProviderName = <name>
baseProviderVersion = <version>
baseProviderDownloadUrl = <url>
parameterization {
name = <name>
version = <version>
value = <base64-encoded-value>
}
}
```
Example of a package descriptors for a `terraform-provider` provider
parameterized with `hashicorp/random`:
```tf
package "random" {
baseProviderName = "terraform-provider"
baseProviderVersion = "0.0.27"
parameterization {
name = "random"
version = "3.6.2"
value = "eyJyZW1vdGUiOnsidXJsIjoicmVnaXN0cnkub3BlbnRvZnUub3JnL2hhc2hpY29ycC9yYW5kb20iLCJ2ZXJzaW9uIjoiMy42LjIifX0="
}
}
resource "randomPet" "random:index/pet:Pet" {
length = 10
}
```
Now when we extract the package name from `random:index/pet:Pet =>
random` we check if there is a package descriptor for it and use that
descriptor to load the package, parameterizing the base provider when
necessary. Otherwise if the package name doesn't have a descriptor, we
fallback to old package loader that expects plugin
`pulumi-resource-<packageName>` to be available in the plugin cache or
PATH.
These blocks are not mapped to specific `Program` nodes, they are read
from the AST after the PCL files have been parsed.
This information can be easily made available to `pcl.Program` if needed
since it has access to the source binder.
### Conformance Tests
With the ability to parameterize packages from a PCL program, we can
support conformance testing parameterized providers! I've implemented a
new test `l2-parameterized-resource` which implements a parameterized
provider that has a schema and runtime behaviour dictated by the values
it was parameterized with.
### Test this locally
You can test this locally with different parameterized providers, you
will need their parameter values which you can retrieve from the schema:
```
pulumi package get-schema terraform-provider hashicorp/random > random-schema.json
```
In `random-schema.json` you will find values you need in
`$.parameterization`
### Converted Code
The program above can be successfully converted to TypeScript for
example and generates the following:
```ts
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";
const randomPet = new random.Pet("randomPet", {length: 10});
```
This program loaded the parameterized schema and used it to generate the
resource definition correctly.
However the following still need to be addressed:
- The `pulumi convert` doesn't generate the local SDKs are part of the
conversion. Doing this might make
https://github.com/pulumi/pulumi-converter-terraform/issues/186 a lot
easier than expected. cc @brandonpollack23
- Program generators do not understand parameterized packages so they
reference them like any of our provider sdk `"@pulumi/random": "3.6.2"`
(this is wrong because there is not such sdk on npm). They should expect
the SDK to be available locally and reference it via a relative path
- ~We need a good way to (conformance) test these~ DONE ✅
This adds a conformance test checking we can use map keys like "MY_KEY"
and maintain their casing through program gen. For Python we need to
simplify the type passed to genObjectConsExpression so that we can
reliably determine the target type and check if it's a `TypedDict` or
not, otherwise we might treat it as a plain object and preserve the
keys, instead of `PyName`ing them.
This deletes the "typeddict" test from the codegen tests. Firstly this
is now covered by these conformance tests, secondly it wasn't actually
valid because the schema it was testing against wasn't a valid schema,
and the program snapshot was only how it was because of a bug in
programgen.
That bug is now fixed in this change which fixes
https://github.com/pulumi/pulumi/issues/17294.
---------
Co-authored-by: Julien Poissonnier <julien@caffeine.lu>
Adding language conformance coverage around provider configuration
(Config, CheckConfig) specifically asserting what gRPC payloads the
providers receive in each language. This testing is in preparation for
making changes to remove JSON encoding from Configure for non-YAML SDKs.
Context doc:
https://docs.google.com/document/d/1cnBJzAGWMnjjj2Q5fDdVcq0DcrWd-OqvPPlSxQV50R4/edit?usp=sharing
Testing:
- [x] Every kind of configuration property:
- [x] string
- [x] non-string (bool, integer, number)
- [x] list of T
- [x] map of T
- [x] object-nested T
- [x] First-class secret-ness: whether the user passes a secret-wrapped
value or not
- [ ] Schema secret-ness: whether the schema marks the prop secret or
not
- [ ] Passing unknowns
- [ ] No secret leaks in state for provider configuration
### Description
Resolves#17459
And the Python portion of #17474
When python output invokes serialized their inputs, if any of these
inputs is an output, the shape of the output is maintained in the
serialized form across the wire. However, for invokes we shouldn't
maintain the shape of the output values, instead the values sent across
the wire should be plain. Flipping the option `keep_output_values` to
`False` fixes the problem.
Also here we keep track of input dependencies for output invokes and
merge the full list of dependencies from all inputs into the
dependencies list of the result of the invoke.
Add a conformance test that we can have a type refer to another nested
type.
This will be useful for checking stability of map/object keys across
nested values for fixing https://github.com/pulumi/pulumi/issues/17294.
### Description
Partially addressing https://github.com/pulumi/pulumi/issues/12710
This PR extends the nodejs SDK with functions `invokeOutput` and
`invokeSingleOutput` which are the output-versioned equivalent of the
plain `invoke` and `invokeSingle`. The underlying implementation doesn't
rely on the plain one and properly implements output deserialization
such that secrets are maintained from the invoke response.
Then we extend the SDK-gen part of nodejs such that output-versioned
invokes use the new primitives `invokeOutput` and `invokeSingleOutput`
in their generated function body without wrapping the plain invoke.
This adds a new provider with just a couple of invokes and a single
resource, and two small projects that uses that invoke to fill in a
stack output.
The first just uses one of the invokes to fill stack outputs.
The second tests that output'y inputs work and that no arguments to the
invoke work.
Currently disabled for Go because it doesn't handle the multi-return
invoke expression in the stack output value context.
Provider internal state is now separated from provider user config. This
allows providers to use a config key like "pluginDownloadURL" which
previously would have conflicted with the engines internal state. It
also allows the engine to add new internal state fields without concern
for clashing with existing or future provider config keys.
"version" is maintained as a root level key because providers already
look at this field, and we use it in conformance tests. However it's
been fixed to be the package version, not the plugin version. That's the
same for normal providers, but for a parameterised provider it will be
the version of the parameterised package, not the base plugin version.
As part of this I've made schema more strict that a provider can't
define its own version field. This would have clashed with the use of
version to set the plugin version to download, and is better set by
users by using the version resource option.
Fixes https://github.com/pulumi/pulumi/issues/16757.
---------
Co-authored-by: Will Jones <will@sacharissa.co.uk>
We already had the "simple" provider test that just checked we could
make a resource with a single simple bool field. This adds a new
"primitive' provider and checks we can send strings, numbers, integers,
arrays and maps from the program.
Noticed this issue while doing SDK gen for parameterised providers, but
figured it deserved its own conformance test. Check that if a provider
has a pre-release semver that the _exact_ version can be reported by the
generated SDK. This already just works for NodeJS, but Python needed a
fix to write the version to `_utilities.py` rather than trying to
unconvert the pypi version from the package.
Also needed to make the conformance test checks for
`GetProgramDependencies` even weaker (which is fine, they are just a
very basic sanity check) because the provider reports a version of
"3.0.0-alpha.1.internal" while the python version is "3.0.0a1+internal".
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/16092.
Two main changes in this PR. Firstly we add a test for assets/archives
to the conformance tests, and had to fix up some small issues in the
test runner to enable this (copying test data files, correcting project
directory layout, handling multi-word package names).
The other change is to fix the engine to respect the programs working
directory when serialising/deserialising assets and archives. It does
this by passing the programs working directory as a MarshalOption (n.b
this option is only really needed in the engine, program SDK doesn't
really need it because it doesn't change directories).
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
In the hopes of having a conformance test to trigger
https://github.com/pulumi/pulumi-dotnet/issues/249, this adds a
conformance test that has a provider send back a large (~100mb) string
which is then exported as a stack output.
Similar to destroy --continue-on-error, this flag allows `pulumi up`
to continue if any errors are encountered.
Currently when we encounter an error while creating/updating a
resource, we cancel the context of the deployment executor, and thus
the deployment stops once the resources that are being processed in
parallel with the failed one finish being updated.
For --continue-on-error, we ignore these errors, and let the
deployment executor continue. In order for the deployment executor to
exit eventually we also have to mark these steps as done, as the
deployment executor will otherwise just hang, and callers with open
channels waiting for it to finish/report back will hang indefinitely.
The errors in the step will still be reported back to the user by the
OnResourceStepPost callback.
Fixes https://github.com/pulumi/pulumi/issues/14515
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/15075.
This doubles up the conformance tests to run with `tsc` instead of
`ts-node`. Getting this to work in the conformance framework required a
bit of pushing the envelope for what it means to "install dependencies"
and "run", but it works for testing, and isn't horribly exposed to
users.
It also points out a couple of things that we might want to clean up for
users wanting the same workflow.
1. Adding a `Build` command (which we might want for the type checker
work added in https://github.com/pulumi/pulumi/pull/15725 as well).
2. If you set "main" to ./bin because you're using tsc, you can't then
use InstallDependencies because to start with the "bin" folder is
missing.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->