996afa5388
`pulumi convert` enables users to convert existing programs written in
other languages and tools, such as Terraform, to Pulumi programs which
manage the same infrastructure. In order to support Pulumi's wide array
of target languages and providers, and the possible multitude of source
programs (Terraform, CloudFormation, and so on), conversion is
architected as follows:
* When invoked through `pulumi convert`, the Pulumi CLI boots up a
*converter plugin* responsible for handling the source language. E.g. in
the case of converting `--from terraform`, the
`pulumi-converter-terraform` plugin will be run.
* Converter plugins implement a common gRPC service, which the CLI will
call to ask for a PCL (Pulumi configuration language; Pulumi's internal
representation) program based on the source.
* As part of producing the PCL program, the converter may ask the CLI to
*map* names from the source language (e.g. `aws_s3_bucket` in a
Terraform program) to equivalent Pulumi names (e.g.
`aws:s3/bucket:Bucket`).
* As is often the case, when it comes to mapping names, the CLI does not
do any work itself -- it instead brokers mapping requests to the set of
plugins installed. So in the above example, the CLI might find the
`pulumi-resource-aws` plugin and ask it to service the mapping request,
before returning to the converter plugin so it can continue to produce
PCL.
* When PCL is produced, the CLI can proceed with code generation
("Programgen"), driven by an appropriate language host and its
`GenerateProgram`/`GenerateProject` gRPC methods.
This changeset focuses on the ability for converter plugins to request
mappings from the CLI, specifically in the case of *parameterized
provider plugins*. A parameterized provider plugin is a plugin that can
be sent some data ("parameterization") before any of its other interface
methods (e.g. `GetSchema`, `Configure`, and in this case `GetMappings`
or `GetMapping`) are called. Parameterized providers underpin Pulumi's
"Any Terraform provider" features, in which Terraform providers are
dynamically bridged at runtime by the
`pulumi-resource-terraform-provider` plugin in response to a
parameterization request.
When a converter plugin requests mappings, it can offer a "hint" to the
CLI as to the plugin that it thinks will provide such a mapping. So e.g.
when the Terraform converter sees `aws_s3_bucket`, it will currently
send a hint `"aws"` to the CLI so that the CLI will search for (and
prioritise) asking a plugin named `aws` for its mappings (as opposed to
e.g. enumerating the random set of other plugins it might encounter
first). Here, we modify the `Mapper` interface so that hints can be full
package descriptors. In doing so, we will be able to modify the
converter to ask for the `terraform-provider` *plugin*, but with
*parameterization* that causes it to dynamically bridge a Terraform
provider such that it can return mappings for that provider. These
changes will be made to the `pulumi-converter-terraform` codebase
separately after this work is merged and released (the bridge may also
need to accommodate this change to the interface, but it should be
mechanical).
As part of this change, it should be noted that the way mappings are
*cached* has been changed. Prior to this work, we would aggressively
cache mappings, including those we retrieved "en route" to finding a
correct one. So, for instance, if we were asked for `"gcp"` mappings, in
the absence of a hint we might first ask for such mappings from the
`aws` and `azure` providers (say). Even though these mappings wouldn't
match, we'd cache them so that if someone later came to us asking for
`aws` mappings, we'd just return them from the cache instead of booting
up plugins and making gRPC calls. In the presence of parameterization,
it's not clear that being this aggressive is safe any more. There's no
guarantee, for instance, that a provider which has provided a set of
mappings will never later be able to provide a different set due to e.g.
a different parameterization. This change thus refactors the caching
behaviour to cache providers that have been requested (and succeeded),
but not to cache mappings "found along the way". Hopefully this should
still cover the majority of common cases without significant performance
costs (regression behaviour such as not double installing plugins should
be preserved, for instance).
Part of #18187
|
||
|---|---|---|
| .. | ||
| build-container | ||
| google/protobuf | ||
| pulumi | ||
| .checksum.txt | ||
| README.md | ||
| generate.sh | ||
| grpc_version.txt | ||
README.md
Protobuf and gRPC interfaces
This package contains Protobuf definitions for the various types, messages and interfaces that Pulumi components use to communicate. These definitions serve as the source of truth for several parts of the wider codebase.
Code generation
:::{note}
Code generated from Protobuf files is committed to the repository. If you change
one or more .proto files, you should run make build_proto to regenerate the
necessary stubs and commit the changes as part of the same piece of work.
:::
The gh-file:pulumi#proto/generate.sh script in this directory (called by the
build_proto target in the top-level gh-file:pulumi#Makefile) generates types
and gRPC clients for the languages supported in this repository (Go,
NodeJS/TypeScript and Python). Generated code is committed to the repository,
typically in proto directories at the relevant use sites (e.g.
gh-file:pulumi#sdk/nodejs/proto).
Documentation
We use the protoc-gen-doc
plugin to protoc to generate
Markdown documentation from the Protobuf files. This process is handled by the
gh-file:pulumi#docs/Makefile in the docs directory and uses the
gh-file:pulumi#docs/references/proto.md.tmpl template. Generated documentation ends
up in the docs/_generated directory (which is .gitignored), so e.g. this
index links to files in this folder.
Index
:::{toctree} :maxdepth: 1 :titlesonly:
/docs/_generated/proto/resource /docs/_generated/proto/provider /docs/_generated/proto/plugin /docs/_generated/proto/language /docs/_generated/proto/callback /docs/_generated/proto/loader /docs/_generated/proto/converter /docs/_generated/proto/mapper /docs/_generated/proto/analyzer /docs/_generated/proto/alias /docs/_generated/proto/engine /docs/_generated/proto/errors /docs/_generated/proto/source :::