pulumi/pkg/codegen/pcl
Will Jones 996afa5388
Support parameterized mappings when converting code (#18671)
`pulumi convert` enables users to convert existing programs written in
other languages and tools, such as Terraform, to Pulumi programs which
manage the same infrastructure. In order to support Pulumi's wide array
of target languages and providers, and the possible multitude of source
programs (Terraform, CloudFormation, and so on), conversion is
architected as follows:

* When invoked through `pulumi convert`, the Pulumi CLI boots up a
*converter plugin* responsible for handling the source language. E.g. in
the case of converting `--from terraform`, the
`pulumi-converter-terraform` plugin will be run.

* Converter plugins implement a common gRPC service, which the CLI will
call to ask for a PCL (Pulumi configuration language; Pulumi's internal
representation) program based on the source.

* As part of producing the PCL program, the converter may ask the CLI to
*map* names from the source language (e.g. `aws_s3_bucket` in a
Terraform program) to equivalent Pulumi names (e.g.
`aws:s3/bucket:Bucket`).

* As is often the case, when it comes to mapping names, the CLI does not
do any work itself -- it instead brokers mapping requests to the set of
plugins installed. So in the above example, the CLI might find the
`pulumi-resource-aws` plugin and ask it to service the mapping request,
before returning to the converter plugin so it can continue to produce
PCL.

* When PCL is produced, the CLI can proceed with code generation
("Programgen"), driven by an appropriate language host and its
`GenerateProgram`/`GenerateProject` gRPC methods.

This changeset focuses on the ability for converter plugins to request
mappings from the CLI, specifically in the case of *parameterized
provider plugins*. A parameterized provider plugin is a plugin that can
be sent some data ("parameterization") before any of its other interface
methods (e.g. `GetSchema`, `Configure`, and in this case `GetMappings`
or `GetMapping`) are called. Parameterized providers underpin Pulumi's
"Any Terraform provider" features, in which Terraform providers are
dynamically bridged at runtime by the
`pulumi-resource-terraform-provider` plugin in response to a
parameterization request.

When a converter plugin requests mappings, it can offer a "hint" to the
CLI as to the plugin that it thinks will provide such a mapping. So e.g.
when the Terraform converter sees `aws_s3_bucket`, it will currently
send a hint `"aws"` to the CLI so that the CLI will search for (and
prioritise) asking a plugin named `aws` for its mappings (as opposed to
e.g. enumerating the random set of other plugins it might encounter
first). Here, we modify the `Mapper` interface so that hints can be full
package descriptors. In doing so, we will be able to modify the
converter to ask for the `terraform-provider` *plugin*, but with
*parameterization* that causes it to dynamically bridge a Terraform
provider such that it can return mappings for that provider. These
changes will be made to the `pulumi-converter-terraform` codebase
separately after this work is merged and released (the bridge may also
need to accommodate this change to the interface, but it should be
mechanical).

As part of this change, it should be noted that the way mappings are
*cached* has been changed. Prior to this work, we would aggressively
cache mappings, including those we retrieved "en route" to finding a
correct one. So, for instance, if we were asked for `"gcp"` mappings, in
the absence of a hint we might first ask for such mappings from the
`aws` and `azure` providers (say). Even though these mappings wouldn't
match, we'd cache them so that if someone later came to us asking for
`aws` mappings, we'd just return them from the cache instead of booting
up plugins and making gRPC calls. In the presence of parameterization,
it's not clear that being this aggressive is safe any more. There's no
guarantee, for instance, that a provider which has provided a set of
mappings will never later be able to provide a different set due to e.g.
a different parameterization. This change thus refactors the caching
behaviour to cache providers that have been requested (and succeeded),
but not to cache mappings "found along the way". Hopefully this should
still cover the majority of common cases without significant performance
costs (regression behaviour such as not double installing plugins should
be preserved, for instance).

Part of #18187
2025-02-21 15:02:08 +00:00
..
README.md Add some PCL cheatsheet-style documentation (#18244) 2025-01-15 10:48:20 +00:00
binder.go Add a `call` intrinsic to PCL (#18206) 2025-01-13 15:51:52 +00:00
binder_component.go [program-gen] Emit deferred outputs for mutually dependant components (#17859) 2024-11-27 23:36:31 +00:00
binder_nodes.go [TF circular reference] Allow specifying mutually dependant components in PCL (#17761) 2024-11-14 22:09:55 +00:00
binder_resource.go Allow resource annotations to be publicly queried, for real (#18302) 2025-01-21 17:06:00 +00:00
binder_resource_test.go Support loading parameterized schemas in the schema loader (#17108) 2024-08-30 14:25:29 +00:00
binder_schema.go Support parameterized mappings when converting code (#18671) 2025-02-21 15:02:08 +00:00
binder_schema_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
binder_test.go Add a `call` intrinsic to PCL (#18206) 2025-01-13 15:51:52 +00:00
binding.md Add more documentation on PCL (#18193) 2025-01-09 10:18:08 +00:00
call.go Allow resource annotations to be publicly queried, for real (#18302) 2025-01-21 17:06:00 +00:00
component.go lint 2023-07-27 16:32:06 +02:00
config.go Implement description as comments or docstring for config variables in program-gen 2023-03-21 15:01:16 +01:00
diagnostics.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
functions.go Implement a first version of `can` in PCL, py, and ts (#18571) 2025-02-13 06:34:45 +00:00
functions_test.go Implement a first version of `can` in PCL, py, and ts (#18571) 2025-02-13 06:34:45 +00:00
intrinsics.go
invoke.go Add a `call` intrinsic to PCL (#18206) 2025-01-13 15:51:52 +00:00
local.go Do not panic when the type of PCL local variable isn't known 2023-04-13 20:05:16 +02:00
output.go
program.go Use slice.Prealloc instead of make([]T, 0, ...) 2023-06-29 11:27:50 +01:00
resource.go Add support for `DeletedWith` to `pulumi convert` (#12011) 2024-07-19 14:17:45 +00:00
rewrite_apply.go Vendor the inflector library (#16421) 2024-06-20 09:04:33 +00:00
rewrite_apply_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_convert.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_convert_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_properties.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
type.go
utilities.go Update golangci-lint from v1.62.2 to 1.64.2 (#18628) 2025-02-18 13:43:46 +00:00
utilities_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00

README.md

(pcl)=

Pulumi Configuration Language (PCL)

Pulumi Configuration Language (PCL) is an internal representation of Pulumi programs which supports all core concepts of the Pulumi programming model in a minimal form. Although not exposed directly to users today, this intermediate representation is used to support a variety of program conversion tasks, from and to various supported Pulumi languages.

Pulumi supports a number of operations pertaining to PCL programs:

  • Lexing and parsing, in which PCL source is parsed into an abstract syntax tree (AST).

  • Type checking and binding, in which a PCL AST is verified to be well-formed (describing resources, outputs, and so on), associated with ("bound to") a set of Pulumi schema, and type checked and resolved.

  • Code generation ("programgen"), in which a PCL program is converted into a program written in a supported target language. Much of this underpins the core of tools such as pulumi convert.

(pcl-program)=

PCL programs

PCL programs are comprised of PCL source files which typically have the suffix .pp. This repository contains a large number of examples -- in particular, a large amount of PCL is used to drive language conformance tests. PCL programs support the following concepts:

(pcl-resource)=

Resources

Resources are defined as blocks of type resource. They correspond to stack resource registrations (see resource registration and for more). A resource block must have exactly two labels -- the first is the resource's name, to be used both as a reference to it elsewhere in the program and as its name in the resulting Pulumi stack (see the sections in this documentation on URNs and resource registration). The resource's body contains a set of attributes, which correspond to the resource's input properties. An optional options block may be used to specify resource options (e.g. provider, protect, ignoreChanges, etc.).

resource "r" "pkg:index:Resource" {
    name = "r"

    options {
        protect = true
    }
}

(pcl-component)=

Components

A component in the context of a PCL program is equivalent to the notion of a "local" component resource in a Pulumi program. Components are defined as blocks of type component. A component block must have two labels. The first is the component's name, to be used both as a reference to it elsewhere in the program and as its name in the resulting Pulumi stack. The second is a path to a directory containing a PCL program (set of .pp files) that defines the component's implementation. The component's body contains a set of attributes, which correspond to the component's input properties. An optional options block may be used to specify component options (e.g. provider, protect, etc.).

component "c" "./c" {
    name = "c"
    value = 42
}

Program generation for components results in e.g. the generation of local component types, such as those extending the ComponentResource class in SDKs exposing that type.

(pcl-output)=

Outputs

Outputs are defined as blocks of type output. They correspond exactly to exported stack outputs, such as pulumi.export in Python or ctx.Export in Go, which correspond to on the stack resource under the hood. Output blocks must have exactly one label, which is the output's name. The output's body must contain a single attribute, value, which is the output's value.

resource "r" "pkg:index:Resource" {
    name = "r"
}

output "o" {
    value = r.name
}

(pcl-config)=

Configuration

Configuration variables define the names and schema of values that are expected to be supplied to a program through Pulumi configuration (e.g. from a Pulumi.<stack>.yaml). They are defined as blocks of type config, with each block taking labels for the variable's name and type. The block's body may be used to specify other aspects of the variable, such as a description and default value. config blocks are typically used to generate program code for referring to Pulumi configuration, e.g. by creating and using a pulumi.Config() object in NodeJS.

config "key" "string" {
  description = "The key to use when encrypting data"
  default     = ""
}

(pcl-invoke)=

invoke

The invoke intrinsic is used to express provider function calls and corresponds to the 's method.

result1 = invoke("pkg:index:Function", { x = 3, y = 4 })

result2 = invoke("pkg:index:Function", { z = 5 }, { parent = p })

(pcl-call)=

call

The call intrinsic is used to express a method invocation on a resource and corresponds to the 's method.

resource "c" "pkg:index:Component" {
    ...
}

result = call(c, "method", { x = 3, y = 4 })

(pcl-builtin-function)=

Built-in functions

PCL offers a number of built-in functions for use in expressions. These are captured by the pulumiBuiltins function in .

:::{toctree} :maxdepth: 1 :titlesonly:

Syntax and type checking </pkg/codegen/hcl2/README> Binding </pkg/codegen/pcl/binding> :::