### Description
This PR extends program-gen to start emitting deferred outputs for
references of mutually dependant components in PCL for nodejs, python
and dotnet.
Addresses the following:
- [[TF circular references] .NET program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17789)
- [[TF circular references] NodeJS program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17790)
- [[TF circular references] python program-gem emitting DeferredOutput
from mutually dependant
components](https://github.com/pulumi/pulumi/issues/17857)
The main idea when extracting references to mutually dependant
components is to replace with variables that are defined as deferred
outputs and later in the program (after the declaration of the dependant
component) we resolve the value of that deferred variable.
The `pcl.ExtractDeferredOutputVariables` utility function contains the
core logic for this implementation and it is what each language
generator uses
Example in PCL:
```tf
component "first" "./first" {
passwordLength = second.passwordLength
}
component "second" "./second" {
petName = first.petName
}
```
### Generated TypeScript
```typescript
const [secondPasswordLength, resolveSecondPasswordLength] = pulumi.deferredOutput<number>();
const first = new First("first", {passwordLength: secondPasswordLength});
const second = new Second("second", {petName: first.petName});
resolveSecondPasswordLength(second.passwordLength);
```
### Generated Python
```python
second_password_length, resolve_second_password_length = pulumi.deferred_output()
first = First("first", {
'passwordLength': second_password_length})
second = Second("second", {
'petName': first.pet_name})
resolve_second_password_length(second.password_length);
```
### Generated C#
```csharp
var secondPasswordLength = new Pulumi.DeferredOutput<int>();
var first = new Components.First("first", new()
{
PasswordLength = secondPasswordLength.Output,
});
var second = new Components.Second("second", new()
{
PetName = first.PetName,
});
secondPasswordLength.Resolve(second.PasswordLength);
```
### Tackling a reference to a _list_ of mutually dependant components
For simple references, the above generated code works. However we also
need to consider when lists of mutually dependant components are being
referenced. Take the following PCL
```tf
component "another" "./first" {
passwordLength = length([ for _, v in many : v.passwordLength ])
}
component "many" "./second" {
options { range = 10 }
petName = another.petName
}
```
In this case the reference `many` is the _collection_ being iterated on
which is a mutually dependant component. This is unfortunately a thing
that happens in the real-world case described in
https://github.com/pulumi/pulumi/issues/13581. What we do here is
extract the entire list comprehension / generator into a variable and
resolve the computation later
<details>
<summary>Generated TypeScript</summary>
```ts
const [loopingOverMany, resolveLoopingOverMany] = pulumi.deferredOutput<Array<number>>();
const another = new First("another", {passwordLength: loopingOverMany.apply(loopingOverMany => loopingOverMany.length)});
const many: Second[] = [];
for (const range = {value: 0}; range.value < 10; range.value++) {
many.push(new Second(`many-${range.value}`, {petName: another.petName}));
}
resolveLoopingOverMany(pulumi.output(many.map((v, k) => [k, v]).map(([_, v]) => (v.passwordLength))));
```
</details>
<details>
<summary>Generated Python</summary>
```py
looping_over_many, resolve_looping_over_many = pulumi.deferred_output()
another = First("another", {
'passwordLength': looping_over_many.apply(lambda looping_over_many: len(looping_over_many)})
many = []
for range in [{"value": i} for i in range(0, 10)]:
many.append(Second(f"many-{range['value']}", {
'petName': another.pet_name }))
resolve_looping_over_many(pulumi.Output.from_input([v["passwordLength"] for _, v in many]))
```
</details>
<details>
<summary>Generated C#</summary>
```csharp
var loopingOverMany = new Pulumi.DeferredOutput<List<int>>();
var another = new Components.First("another", new()
{
PasswordLength = loopingOverMany.Output.Apply(loopingOverMany => loopingOverMany.Length),
});
var many = new List<Components.Second>();
for (var rangeIndex = 0; rangeIndex < 10; rangeIndex++)
{
var range = new { Value = rangeIndex };
many.Add(new Components.Second($"many-{range.Value}", new()
{
PetName = another.PetName,
}));
}
loopingOverMany.Resolve(Output.Create(many.Select((value, i) => new { Key = i.ToString(), Value = pair.Value }).Select(v =>
{
return v.PasswordLength;
}).ToList()));
```
</details>
### Typing and Lifting issues
When extracting the reference expressions and rewriting variables, for
some reason the typing information seem to get lossy and lifting
variables isn't generating the right code 🤔 in the previous examples
with lists of components (ts) the expression `loopingOverMany.length`
should have been _lifted_ into `loopingOverMany.apply(many =>
many.length)` similarly python and C# examples should have been lifted.
Currently this is why I've skipped the compilation step in the program
test options.
EDIT: fixed some of the lifting issues, now we correctly use `.apply`
when necessary
For the sake of not making this PR even bigger, I will open another
issue to tackle these related typing and lifting problems.
- [ ] ~TODO link to issue about lifting replaced variables~ Lifting
deferred output variables is working
fef43d10cf