10 KiB

Raw Permalink Blame History

[RFC] DependsOn for Provider Functions

Summary

Add a new DependsOn option for provider functions, which allows specifying additional dependencies that are not captured as part of the function’s inputs.

Background

Outputs are the core mechanism for tracking dependencies in Pulumi. If the Output of one resource is passed to the input of another, Pulumi will recognise the dependency and appropriately order operations on those resources, ensuring that a dependency is not created after resources which depend on it. There are occasions when there is some user-known dependency between two resources, but where there is no natural Output property that can be used to link the two. For these situations, Pulumi provides DependsOn, a resource option that programs can use to explicitly encode known dependencies and enforce ordering constraints on resource operations.

However, while resources typically make up the majority of a Pulumi program, they are not the only interface to Pulumi’s engine and provider set. Provider functions, or “invokes”, allow providers to expose arbitrary functions to Pulumi programs. A common use case is looking up provider resources: getAmi is an example from the AWS provider which gives programs a means to lookup an Amazon Machine Image (AMI), e.g. for use in constructing an EC2 instance later on. Presently, every invoke is offered in two “invocation forms”:

A direct form, which accepts plain arguments and either blocks until a result is available, or acts asynchronously using a language-native concept of asynchronicity (e.g. Promise in JavaScript, Task in .Net, etc.)
An Output form, which accepts any valid Pulumi Input (i.e. plain or Output values) and returns a Pulumi Output-wrapped result.

While direct-form invocations do not have any means to track dependencies, Output form invocations do, and will respect ordering according to the inputs they are passed. Unlike resources, however, Output form invokes do not currently offer a facility like DependsOn, whereby users can explicitly control the ordering of invocations where a dependency is not visible to Pulumi. This RFC proposes adding a similar option to invokes.

Problem

Both direct form and Output form invokes take an optional argument to pass in the invoke options. These options allow configuring a parent or provider for the invoke.

As an example, in TypeScript we use InvokeOptions:

function getAmi(args: GetAmiArgs, opts?: InvokeOptions): Promise<GetAmiResult> { ... }

function getAmiOutput(args: GetAmiOutputArgs, opts?: InvokeOptions): Output<GetAmiResult> { ... }

For Go, the invokes take a variadic ...InvokeOption argument:

func LookupAmi(ctx *Context, args *LookupAmiArgs, opts ...InvokeOption) (*LookupAmiResult, error) { ... }

func LookupAmiOutput(ctx *Context, args *LookupAmiOutputArgs, opts ...InvokeOption) LookupAmiResultOutput { ... }

Pulumi's type system represents the fact of potentially waiting for a dependency as an Output, however direct form invokes do not return an Output, and thus the DependsOn option does not make sense for these invokes. The option should be limited to Output form invokes, which do return an Output and fully take part in Pulumi’s dependency tracking system.

This means we cannot add the new DependsOn option to the existing InvokeOptions. We initially attempted this in https://github.com/pulumi/pulumi/pull/16560, and reverted it again in https://github.com/pulumi/pulumi/pull/16642.

Note that you can use apply to work around the limitation. This is described in the dependencies and ordering section of the provider functions documentation.

API Proposal

This RFC proposes to add a new InvokeOutputOptions type for languages other than Go to disambiguate the options for direct and Output form invokes. For Go we propose a slightly different approach.

This new argument type has to be introduced in a backwards compatibility preserving way, with different approaches for our supported languages.

TypeScript

The InvokeOptions type in the TypeScript SDK is an interface, which the new InvokeOutputOptions interface can extend:

export interface InvokeOutputOptions extends InvokeOptions {
    dependsOn?: Input<Input<Resource>[]> | Input<Resource>;
}

Optionality in TypeScript means T | undefined, and looking up an property on object that has not been set returns undefined, thus any value satisfying the InvokeOptions interface also satisfies the InvokeOptionsOutput interface.

Output form invokes can take an argument of this type while preserving backwards compatibility.

function getAmiOutput(args: GetAmiOutputArgs, opts?: InvokeOutputOptions): Output<GetAmiResult> { ... }

Python

The InvokeOptions type in the Python SDK is a class. The new InvokeOutputOptions extends this class and adds the optional depends_on property:

class InvokeOutputOptions(InvokeOptions):
  depends_on: Optional[Input[Union[Sequence[Input[Resource]], Resource]]]

In Python, optionality means T | None, and attempting to access a property that has not been defined on an object results in an AttributeError. This means that an instance of type InvokeOptions does not conform to InvokeOutputOptions. To allow passing both types, we can make the argument type the union of the types:

def get_ami_output(..., opts: Optional[Union[InvokeOptions, InvokeOutputOptions]] = None) -> Output[GetAmiResult]
  ...

Go

In Go, the type for invoke options is a variadic ...InvokeOption, for example:

func LookupAmiOutput(ctx *Context, args *LookupAmiOutputArgs, opts ...InvokeOption) LookupAmiResultOutput

We propose to update the existing DependsOn for resources to also apply to invokes:

func DependsOn(o []Resource) ResourceOption

becomes

func DependsOn(o []Resource) ResourceOrInvokeOption

ResourceOrInvokeOption is composed of both ResourceOption and InvokeOption and a value of this type can always be assigned to its narrower constituent types.

type ResourceOrInvokeOption interface {
	ResourceOption
	InvokeOption
}

Since there are no union types in Go, we cannot change type of existing Output form invokes without breaking backwards compatibility. Instead, we propose to allow passing this option to both forms of invokes, and to return an error if the option is passed to a direct invoke.

ami, err := ec2.LookupAmi(ctx,
	&ec2.LookupAmiArgs{...},
	pulumi.DependsOn(someResource),
)
// err: Can not pass DependsOn to direct form invoke, use Output form instead

Ideally the error message can point to the matching Output form invoke.

Alternative

Alternatively, we could add a 3rd invoke variant to Go SDKs, besides the direct and Output form variants, which takes an additional dependsOn argument:

func LookupAmi(ctx *Context, args *LookupAmiArgs, opts ...InvokeOption) (*LookupAmiResult, error) { ... }
func LookupAmiOutput(ctx *Context, args *LookupAmiOutputArgs, opts ...InvokeOption) LookupAmiResultOutput { ... }
// New variant
func LookupAmiOutputWithDependsOn(ctx *Context, args *LookupAmiOutputArgs, dependsOn: []Resource, opts ...InvokeOption) LookupAmiResultOutput { ... }

Besides increasing the code size of SDKs, we feel that this would make Go SDKs more difficult for users to navigate and pick the function they need. The general use case does not require specifying additional dependencies, and exposing the variant at the level of functions feels like it would lead to more confusion.

Java

Options are represented by the InvokeOptions class in the Java SDK. We propose adding a new class InvokeOutputOptions with a dependsOn field, and to add a new overload for the Output form invokes:

public static Output<GetAmiResult> getAmi(GetAmiArgs args)
public static Output<GetAmiResult> getAmi(GetAmiArgs args, InvokeOptions options)
// new overload
public static Output<GetAmiResult> getAmi(GetAmiArgs args, InvokeOutputOptions options)

Dotnet

Options are represented by the InvokeOptions class in the Dotnet SDK. Similar to the proposed solution for Java, we propose adding new class InvokeOutputOptions with a dependsOn field, and to add a new overload for the Output form invokes:

public static class GetAmi {
  public static Output<GetAmiResult> Invoke(GetAmiInvokeArgs? args = null, InvokeOptions? options = null)
  // new overload
  public static Output<GetAmiResult> Invoke(GetAmiInvokeArgs? args = null, InvokeOutputOptions? options = null)
}

YAML

In the YAML SDK, the fn::invoke built-in function takes an options property of type InvokeOptions. This type can be extended to include a newdependsOn field.

Changelog

2024-11-06

Initial version

10 KiB Raw Permalink Blame History Unescape Escape