mirrors
/
pulumi
mirror of https://github.com/pulumi/pulumi.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
pulumi/pkg/codegen/pcl
History
Zaid Ajaj fef43d10cf
[program-gen] Emit deferred outputs for mutually dependant components (#17859) 
### Description

This PR extends program-gen to start emitting deferred outputs for
references of mutually dependant components in PCL for nodejs, python
and dotnet.

Addresses the following:
- [[TF circular references] .NET program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17789)
- [[TF circular references] NodeJS program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17790)
- [[TF circular references] python program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17857)


The main idea when extracting references to mutually dependant
components is to replace with variables that are defined as deferred
outputs and later in the program (after the declaration of the dependant
component) we resolve the value of that deferred variable.

The `pcl.ExtractDeferredOutputVariables` utility function contains the
core logic for this implementation and it is what each language
generator uses

Example in PCL:
```tf
component "first" "./first" {
    passwordLength = second.passwordLength
}

component "second" "./second" {
    petName = first.petName
}
```

### Generated TypeScript

```typescript
const [secondPasswordLength, resolveSecondPasswordLength] = pulumi.deferredOutput<number>();
const first = new First("first", {passwordLength: secondPasswordLength});
const second = new Second("second", {petName: first.petName});
resolveSecondPasswordLength(second.passwordLength);
```

### Generated Python

```python
second_password_length, resolve_second_password_length = pulumi.deferred_output()
first = First("first", {
    'passwordLength': second_password_length})
second = Second("second", {
    'petName': first.pet_name})
resolve_second_password_length(second.password_length);
```


### Generated C#

```csharp
var secondPasswordLength = new Pulumi.DeferredOutput<int>();
var first = new Components.First("first", new()
{
    PasswordLength = secondPasswordLength.Output,
});
var second = new Components.Second("second", new()
{
    PetName = first.PetName,
});
secondPasswordLength.Resolve(second.PasswordLength);
```

### Tackling a reference to a _list_ of mutually dependant components

For simple references, the above generated code works. However we also
need to consider when lists of mutually dependant components are being
referenced. Take the following PCL

```tf
component "another" "./first" {
    passwordLength = length([ for _, v in many : v.passwordLength ])
}

component "many" "./second" {
    options { range = 10 }
    petName = another.petName
}
```

In this case the reference `many` is the _collection_ being iterated on
which is a mutually dependant component. This is unfortunately a thing
that happens in the real-world case described in
https://github.com/pulumi/pulumi/issues/13581. What we do here is
extract the entire list comprehension / generator into a variable and
resolve the computation later

<details>
<summary>Generated TypeScript</summary>

```ts
const [loopingOverMany, resolveLoopingOverMany] = pulumi.deferredOutput<Array<number>>();
const another = new First("another", {passwordLength: loopingOverMany.apply(loopingOverMany => loopingOverMany.length)});
const many: Second[] = [];
for (const range = {value: 0}; range.value < 10; range.value++) {
    many.push(new Second(`many-${range.value}`, {petName: another.petName}));
}
resolveLoopingOverMany(pulumi.output(many.map((v, k) => [k, v]).map(([_, v]) => (v.passwordLength))));
```
</details>

<details>
<summary>Generated Python</summary>

```py
looping_over_many, resolve_looping_over_many = pulumi.deferred_output()
another = First("another", {
    'passwordLength': looping_over_many.apply(lambda looping_over_many: len(looping_over_many)})
many = []
for range in [{"value": i} for i in range(0, 10)]:
    many.append(Second(f"many-{range['value']}", {
        'petName': another.pet_name    }))
resolve_looping_over_many(pulumi.Output.from_input([v["passwordLength"] for _, v in many]))
```
</details>

<details>
<summary>Generated C#</summary>

```csharp
var loopingOverMany = new Pulumi.DeferredOutput<List<int>>();
var another = new Components.First("another", new()
{
    PasswordLength = loopingOverMany.Output.Apply(loopingOverMany => loopingOverMany.Length),
});
var many = new List<Components.Second>();
for (var rangeIndex = 0; rangeIndex < 10; rangeIndex++)
{
    var range = new { Value = rangeIndex };
    many.Add(new Components.Second($"many-{range.Value}", new()
    {
        PetName = another.PetName,
    }));
}
loopingOverMany.Resolve(Output.Create(many.Select((value, i) => new { Key = i.ToString(), Value = pair.Value }).Select(v => 
{
    return v.PasswordLength;
}).ToList()));
```
</details>

### Typing and Lifting issues

When extracting the reference expressions and rewriting variables, for
some reason the typing information seem to get lossy and lifting
variables isn't generating the right code 🤔 in the previous examples
with lists of components (ts) the expression `loopingOverMany.length`
should have been _lifted_ into `loopingOverMany.apply(many =>
many.length)` similarly python and C# examples should have been lifted.
Currently this is why I've skipped the compilation step in the program
test options.

EDIT: fixed some of the lifting issues, now we correctly use `.apply`
when necessary

For the sake of not making this PR even bigger, I will open another
issue to tackle these related typing and lifting problems.
- [ ] ~TODO link to issue about lifting replaced variables~ Lifting
deferred output variables is working
 		2024-11-27 23:36:31 +00:00
..
README.md Document code generation concepts (#17162) 2024-09-05 13:12:59 +00:00
binder.go [PCL] Implement package descriptor blocks to support parameterized packages (#17589) 2024-11-05 00:58:48 +00:00
binder_component.go [program-gen] Emit deferred outputs for mutually dependant components (#17859) 2024-11-27 23:36:31 +00:00
binder_nodes.go [TF circular reference] Allow specifying mutually dependant components in PCL (#17761) 2024-11-14 22:09:55 +00:00
binder_resource.go [PCL] Implement package descriptor blocks to support parameterized packages (#17589) 2024-11-05 00:58:48 +00:00
binder_resource_test.go Support loading parameterized schemas in the schema loader (#17108) 2024-08-30 14:25:29 +00:00
binder_schema.go [PCL] Implement package descriptor blocks to support parameterized packages (#17589) 2024-11-05 00:58:48 +00:00
binder_schema_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
binder_test.go [program-gen] Emit deferred outputs for mutually dependant components (#17859) 2024-11-27 23:36:31 +00:00
component.go lint 2023-07-27 16:32:06 +02:00
config.go Implement description as comments or docstring for config variables in program-gen 2023-03-21 15:01:16 +01:00
diagnostics.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
functions.go [pcl] Allow PCL function element to take a dynamic expression as input in non-strict mode (#17587) 2024-11-11 12:15:54 +00:00
functions_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
intrinsics.go all: Assert => Assertf 2023-03-03 14:37:43 -08:00
invoke.go [Python] Allow specifiying dependencies for output invokes (#17751) 2024-11-20 12:28:41 +00:00
local.go Do not panic when the type of PCL local variable isn't known 2023-04-13 20:05:16 +02:00
output.go codegen: preserve externally visible names of a resources and outputs (#9464) 2022-04-25 15:07:25 -07:00
program.go Use slice.Prealloc instead of make([]T, 0, ...) 2023-06-29 11:27:50 +01:00
resource.go Add support for `DeletedWith` to `pulumi convert` (#12011) 2024-07-19 14:17:45 +00:00
rewrite_apply.go Vendor the inflector library (#16421) 2024-06-20 09:04:33 +00:00
rewrite_apply_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_convert.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_convert_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
rewrite_properties.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00
type.go [codegen] simplify opaque types to string newtype (#9770) 2022-06-13 11:13:03 -07:00
utilities.go [program-gen] Emit deferred outputs for mutually dependant components (#17859) 2024-11-27 23:36:31 +00:00
utilities_test.go Enable goheader rule and add missing license headers (#15473) 2024-09-09 12:05:45 +00:00

README.md

(pcl)=

Pulumi Configuration Language (PCL)

Pulumi Configuration Language (PCL) is an internal representation of Pulumi programs which supports all core concepts of the Pulumi programming model in a minimal form. Although not exposed directly to users today, this intermediate representation is used to support a variety of program conversion tasks, from and to various supported Pulumi languages.