pulumi/pkg/engine/lifecycletest/transformation_test.go

689 lines
24 KiB
Go
Raw Normal View History

Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
// Copyright 2016-2022, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package lifecycletest
import (
2024-04-02 14:39:58 +00:00
"errors"
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
"fmt"
"testing"
"github.com/blang/semver"
"google.golang.org/protobuf/proto"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
. "github.com/pulumi/pulumi/pkg/v3/engine" //nolint:revive
"github.com/pulumi/pulumi/pkg/v3/resource/deploy/deploytest"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
pulumirpc "github.com/pulumi/pulumi/sdk/v3/proto/go"
)
func TransformFunction(
f func(
name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error),
) func([]byte) (proto.Message, error) {
return func(request []byte) (proto.Message, error) {
var transformationRequest pulumirpc.TransformRequest
err := proto.Unmarshal(request, &transformationRequest)
if err != nil {
return nil, fmt.Errorf("unmarshaling request: %w", err)
}
mprops, err := plugin.UnmarshalProperties(transformationRequest.Properties, plugin.MarshalOptions{
KeepUnknowns: true,
KeepSecrets: true,
KeepResources: true,
KeepOutputValues: true,
})
if err != nil {
return nil, fmt.Errorf("unmarshaling properties: %w", err)
}
ret, opts, err := f(
transformationRequest.Name, transformationRequest.Type, transformationRequest.Custom, transformationRequest.Parent,
mprops, transformationRequest.Options)
if err != nil {
return nil, err
}
mret, err := plugin.MarshalProperties(ret, plugin.MarshalOptions{
KeepUnknowns: true,
KeepSecrets: true,
KeepResources: true,
KeepOutputValues: true,
})
if err != nil {
return nil, err
}
return &pulumirpc.TransformResponse{
Properties: mret,
Options: opts,
}, nil
}
}
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
func pvApply(pv resource.PropertyValue, f func(resource.PropertyValue) resource.PropertyValue) resource.PropertyValue {
if pv.IsOutput() {
o := pv.OutputValue()
if !o.Known {
return pv
}
return resource.NewOutputProperty(resource.Output{
Element: f(o.Element),
Known: true,
Secret: o.Secret,
Dependencies: o.Dependencies,
})
}
return f(pv)
}
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
// Test that the engine invokes all transformation functions in the correct order.
func TestRemoteTransforms(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
callback1, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
return resource.NewNumberProperty(v.NumberValue() + 1)
})
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
// callback 2 should run before this one so "bar" should exist at this point
props["bar"] = resource.NewStringProperty(props["bar"].StringValue() + "baz")
return props, opts, nil
}))
require.NoError(t, err)
callback2, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
return resource.NewNumberProperty(v.NumberValue() + 1)
})
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
props["bar"] = resource.NewStringProperty("bar")
// if this is for resB then callback 3 will have run before this one
if prop, has := props["frob"]; has {
props["frob"] = resource.MakeSecret(prop)
} else {
props["frob"] = resource.NewStringProperty("nofrob")
}
return props, opts, nil
}))
require.NoError(t, err)
callback3, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
return resource.NewNumberProperty(v.NumberValue() + 1)
})
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
props["frob"] = resource.NewStringProperty("frob")
return props, opts, nil
}))
require.NoError(t, err)
err = monitor.RegisterStackTransform(callback1)
require.NoError(t, err)
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
Transforms: []*pulumirpc.Callback{
callback2,
},
})
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typA", "resB", true, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(10),
},
Transforms: []*pulumirpc.Callback{
callback3,
},
Parent: respA.URN,
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
})
require.NoError(t, err)
return nil
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
// Skip display tests because secrets are serialized with the blinding crypter and can't be restored
Options: TestUpdateOptions{T: t, HostF: hostF, SkipDisplayTests: true},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.NoError(t, err)
assert.Len(t, snap.Resources, 3)
// Check Resources[1] is the resA resource
res := snap.Resources[1]
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA::resA"), res.URN)
// Check it's final input properties match what we expected from the transformations
assert.Equal(t, resource.PropertyMap{
"foo": resource.NewNumberProperty(3),
"bar": resource.NewStringProperty("barbaz"),
"frob": resource.NewStringProperty("nofrob"),
}, res.Inputs)
// Check Resources[2] is the resB resource
res = snap.Resources[2]
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA$pkgA:m:typA::resB"), res.URN)
// Check it's final input properties match what we expected from the transformations
assert.Equal(t, resource.PropertyMap{
"foo": resource.NewNumberProperty(13),
"bar": resource.NewStringProperty("barbaz"),
"frob": resource.MakeSecret(resource.NewStringProperty("frob")),
}, res.Inputs)
}
// Test that the engine errors if a transformation function returns an unexpected response.
func TestRemoteTransformBadResponse(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
callback1, err := callbacks.Allocate(func(args []byte) (proto.Message, error) {
// return the wrong message type
return &pulumirpc.RegisterResourceResponse{
Urn: "boom",
}, nil
})
require.NoError(t, err)
err = monitor.RegisterStackTransform(callback1)
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
})
assert.ErrorContains(t, err, "unmarshaling response: proto:")
assert.ErrorContains(t, err, "cannot parse invalid wire-format data")
return err
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.ErrorContains(t, err, "unmarshaling response: proto:")
assert.ErrorContains(t, err, "cannot parse invalid wire-format data")
assert.Len(t, snap.Resources, 0)
}
2024-04-02 14:39:58 +00:00
// Test that the engine errors if a transformation function returns an error.
func TestRemoteTransformErrorResponse(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
callback1, err := callbacks.Allocate(func(args []byte) (proto.Message, error) {
return nil, errors.New("bad transform")
})
require.NoError(t, err)
err = monitor.RegisterStackTransform(callback1)
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
2024-04-02 14:39:58 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
})
assert.ErrorContains(t, err, "Unknown desc = bad transform")
return err
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
2024-04-02 14:39:58 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.ErrorContains(t, err, "Unknown desc = bad transform")
assert.Len(t, snap.Resources, 0)
}
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
// Test that a remote transform applies to a resource inside a component construct.
func TestRemoteTransformationsConstruct(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{
ConstructF: func(
monitor *deploytest.ResourceMonitor, typ string, name string, parent resource.URN,
inputs resource.PropertyMap, info plugin.ConstructInfo, options plugin.ConstructOptions,
) (plugin.ConstructResult, error) {
assert.Equal(t, "pkgA:m:typC", typ)
resp, err := monitor.RegisterResource(tokens.Type(typ), name, false, deploytest.ResourceOptions{})
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
Parent: resp.URN,
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
})
require.NoError(t, err)
return plugin.ConstructResult{
URN: resp.URN,
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}, nil
},
}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
callback1, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
if typ == "pkgA:m:typA" {
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
return resource.NewNumberProperty(v.NumberValue() + 1)
})
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}
return props, opts, nil
}))
require.NoError(t, err)
err = monitor.RegisterStackTransform(callback1)
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typC", "resC", false, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Remote: true,
})
require.NoError(t, err)
return nil
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.NoError(t, err)
assert.Len(t, snap.Resources, 3)
// Check Resources[2] is the resA resource
res := snap.Resources[2]
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typC$pkgA:m:typA::resA"), res.URN)
// Check it's final input properties match what we expected from the transformations
assert.Equal(t, resource.PropertyMap{
"foo": resource.NewNumberProperty(2),
}, res.Inputs)
}
// Test that all options are passed and can be modified by a transformation function.
func TestRemoteTransformsOptions(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{}, nil
}),
}
urnB := "urn:pulumi:test::test::pkgA:m:typA::resB"
urnC := "urn:pulumi:test::test::pkgA:m:typA::resC"
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
urnD := "urn:pulumi:test::test::pkgA:m:typA::resD"
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{})
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
require.NoError(t, err)
respC, err := monitor.RegisterResource("pkgA:m:typA", "resC", true, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
Version: "1.0.0",
})
require.NoError(t, err)
callback1, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
// Check that the options are passed through correctly
assert.Equal(t, []string{"foo"}, opts.AdditionalSecretOutputs)
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
assert.Equal(t, urnB, opts.Aliases[0].Alias.(*pulumirpc.Alias_Urn).Urn)
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
assert.Equal(t, "16m40s", opts.CustomTimeouts.Create)
assert.Equal(t, "33m20s", opts.CustomTimeouts.Update)
assert.Equal(t, "50m0s", opts.CustomTimeouts.Delete)
assert.True(t, *opts.DeleteBeforeReplace)
assert.Equal(t, string(respA.URN), opts.DeletedWith)
assert.Equal(t, []string{string(respA.URN)}, opts.DependsOn)
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
assert.Equal(t, []string{"foo"}, opts.IgnoreChanges)
assert.Equal(t, "http://server", opts.PluginDownloadUrl)
assert.Equal(t, false, opts.Protect)
assert.Equal(t, []string{"foo"}, opts.ReplaceOnChanges)
assert.Equal(t, "2.0.0", opts.Version)
// Modify all the options
opts = &pulumirpc.TransformResourceOptions{
AdditionalSecretOutputs: []string{"bar"},
Aliases: []*pulumirpc.Alias{
{Alias: &pulumirpc.Alias_Urn{Urn: urnB}},
},
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: "1s",
Update: "2s",
Delete: "3s",
},
DeleteBeforeReplace: nil,
DeletedWith: string(respC.URN),
DependsOn: []string{string(respC.URN)},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
IgnoreChanges: []string{"bar"},
PluginDownloadUrl: "",
Protect: true,
ReplaceOnChanges: []string{"bar"},
Version: "1.0.0",
}
return props, opts, nil
}))
require.NoError(t, err)
err = monitor.RegisterStackTransform(callback1)
require.NoError(t, err)
dbr := true
_, err = monitor.RegisterResource("pkgA:m:typA", "resD", true, deploytest.ResourceOptions{
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
AdditionalSecretOutputs: []resource.PropertyKey{"foo"},
Aliases: []*pulumirpc.Alias{
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
{Alias: &pulumirpc.Alias_Urn{Urn: urnB}},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
},
CustomTimeouts: &resource.CustomTimeouts{
Create: 1000,
Update: 2000,
Delete: 3000,
},
DeleteBeforeReplace: &dbr,
DeletedWith: respA.URN,
Dependencies: []resource.URN{respA.URN},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
IgnoreChanges: []string{"foo"},
PluginDownloadURL: "http://server",
ReplaceOnChanges: []string{"foo"},
Version: "2.0.0",
})
require.NoError(t, err)
return nil
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
require.NoError(t, err)
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
assert.Len(t, snap.Resources, 5)
// Check Resources[4] is the resD resource
res := snap.Resources[4]
require.Equal(t, resource.URN(urnD), res.URN)
Engine support for remote transforms (#15290) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds support to the engine for "remote transformations". A transform is "remote" because it is being invoked via the engine on receiving a resource registration, rather than being ran locally in process before sending a resource registration. These transforms can also span multiple process boundaries, e.g. a transform function in a user program, then a transform function in a component library, both running for a resource registered by another component library. The underlying new feature here is the idea of a `Callback`. The expectation is we're going to use callbacks for multiple features so these are _not_ defined in terms of transformations. A callback is an untyped byte array (usually will be a protobuf message), plus an address to define which server should be invoked to do the callback, and a token to identify it. A language sdk can start up and serve a `Callbacks` service, keep a mapping of tokens to in-process functions (currently just using UUID's for this), and then pass that service address and token to the engine to be invoked later on. The engine uses these callbacks to track transformations callbacks per resource, and on a new resource registrations invokes each relevant callback with the resource properties and options, having new properties and options returned that are then passed to the next relevant transform callback until all have been called and the engine has the final resource state and options to use. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-21 16:30:46 +00:00
assert.Equal(t, []resource.PropertyKey{"bar"}, res.AdditionalSecretOutputs)
assert.Equal(t, resource.CustomTimeouts{
Create: 1,
Update: 2,
Delete: 3,
}, res.CustomTimeouts)
assert.Equal(t, resource.URN(urnC), res.DeletedWith)
assert.Equal(t, true, res.Protect)
}
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
// Test that a transform can change the dependencies of a resource.
func TestRemoteTransformsDependencies(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{
CreateF: func(
urn resource.URN, props resource.PropertyMap, timeout float64, preview bool,
) (resource.ID, resource.PropertyMap, resource.Status, error) {
return "some-id", props, resource.StatusOK, nil
},
}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
})
require.NoError(t, err)
assert.True(t, respA.Outputs["foo"].IsNumber())
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
// Register a separate resource that
respB, err := monitor.RegisterResource("pkgA:m:typA", "resB", true, deploytest.ResourceOptions{
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(10),
},
})
require.NoError(t, err)
callback, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
// props should be tracking that it depends on resB
assert.True(t, props["foo"].IsOutput())
assert.Equal(t, []resource.URN{respB.URN}, props["foo"].OutputValue().Dependencies)
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
// Add a dependency on resA
props["foo"] = resource.NewOutputProperty(resource.Output{
Element: respA.Outputs["foo"],
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
Known: true,
Dependencies: []resource.URN{respA.URN},
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
})
return props, opts, nil
}))
require.NoError(t, err)
// Register a resource that initially depends on resB but the transform will turn to depend on resA
respC, err := monitor.RegisterResource(
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
"pkgA:m:typA", "resC", true, deploytest.ResourceOptions{
Inputs: resource.PropertyMap{
"foo": respB.Outputs["foo"],
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
},
PropertyDeps: map[resource.PropertyKey][]resource.URN{
"foo": {respB.URN},
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
},
Transforms: []*pulumirpc.Callback{
callback,
},
})
require.NoError(t, err)
assert.True(t, respC.Outputs["foo"].IsNumber())
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
// This is a custom resource so no output dependencies
assert.Empty(t, respC.Dependencies)
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
return nil
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
Send output values to transforms for dependency tracking (#15637) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This engine change is necessary for correct dependency tracking of properties through transform functions. Unlike other parts of the runtime/provider/engine interface transform functions do not have a property dependencies map sent or returned. Transforms rely entirely on the dependency arrays in output values for their dependency tracking. This change takes the property dependency map and upgrades all the input properties to be output values tracking those same dependencies (if a property doesn't have any dependencies it doesn't need to be upgraded to an output value). This upgrade is only done if we're using transform functions, there's currently no need to do this unless we're using transforms. After running the transforms we downgrade the output values back to plain/secret/computed values if we're registering a custom resource. If we're registering a component resource we just leave all the properties upgraded as output values. We also rebuild the resources dependencies slice and propertyDependency map with the dependencies recorded in the transformed properties. If the transform didn't change the properties this will just be the same data we encoded into the output values in the properties structure. There are a couple of things to keep in mind with this change. Firstly using a transform can cause a property that was being sent to a component provider as just a `resource.NumberProperty` to start being sent as an `OutputProperty` with the `NumberProperty` as its element. Component providers _should_ handle that, but it's feasible that it could break some component code. This is a fairly limited blast radius as it will only happen when that user starts using a transform function. Secondly, we can't know in the engine if a property dependencies are from a top level output value or a nested output. That is SDKs send both of the following property shapes the same way to the engine for custom resources: ``` A) Output(Array([Number(1)]), ["dep"]) --- B) Array([Output(Number(1), ["dep"]) ``` Currently both would get upgraded to `Output(Array([Number(1)]), ["dep"])`. For a component resource the SDK will send output values, and as long as they define a superset of the dependencies in the property map then we don't add the top-level output tracking the same dependency set. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-20 09:53:33 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.NoError(t, err)
assert.Len(t, snap.Resources, 4)
// Check Resources[3] is the resC resource
res := snap.Resources[3]
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA::resC"), res.URN)
// Check it's final input properties match what we expected from the transformations
assert.Equal(t, resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
}, res.Inputs)
// Check the dependencies are as expected
assert.Equal(t, map[resource.PropertyKey][]resource.URN{
"foo": {resource.URN("urn:pulumi:test::test::pkgA:m:typA::resA")},
}, res.PropertyDependencies)
assert.Equal(t, []resource.URN{
"urn:pulumi:test::test::pkgA:m:typA::resA",
}, res.Dependencies)
}
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
// Regression test for https://github.com/pulumi/pulumi/issues/15843. Ensure that if a component resource has a
// transform that's saved and looked up by it's children.
func TestRemoteComponentTransforms(t *testing.T) {
t.Parallel()
loaders := []*deploytest.ProviderLoader{
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
return &deploytest.Provider{
ConstructF: func(
monitor *deploytest.ResourceMonitor, typ string, name string, parent resource.URN,
inputs resource.PropertyMap, info plugin.ConstructInfo, options plugin.ConstructOptions,
) (plugin.ConstructResult, error) {
assert.Equal(t, "pkgA:m:typC", typ)
resp, err := monitor.RegisterResource(tokens.Type(typ), name, false, deploytest.ResourceOptions{})
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
Parent: resp.URN,
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
Inputs: resource.PropertyMap{
"foo": resource.NewNumberProperty(1),
},
})
require.NoError(t, err)
return plugin.ConstructResult{
URN: resp.URN,
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
}, nil
},
}, nil
}),
}
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
callbacks, err := deploytest.NewCallbacksServer()
require.NoError(t, err)
defer func() { require.NoError(t, callbacks.Close()) }()
callback1, err := callbacks.Allocate(
TransformFunction(func(name, typ string, custom bool, parent string,
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
if typ == "pkgA:m:typA" {
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
return resource.NewNumberProperty(v.NumberValue() + 1)
})
}
return props, opts, nil
}))
require.NoError(t, err)
_, err = monitor.RegisterResource("pkgA:m:typC", "resC", false, deploytest.ResourceOptions{
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
Remote: true,
Transforms: []*pulumirpc.Callback{
callback1,
},
})
require.NoError(t, err)
return nil
})
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
p := &TestPlan{
Add display to the engine tests (#16050) We want to add more test coverage to the display code. The best way to do that is to add it to the engine tests, that already cover most of the pulumi functionality. It's probably not really possible to review all of the output, but at least it gives us a baseline, which we can work with. There's a couple of tests that are flaky for reasons I don't quite understand yet. I marked them as to skip and we can look at them later. I'd rather get in the baseline tests sooner, rather than spending a bunch of time looking at that. The output differences also seem very minor, so not super concerning. The biggest remaining issue is that this doesn't interact well with the Chdir we're doing in the engine. We could either pass the CWD through, or just try to get rid of that Chdir. So this should only be merged after https://github.com/pulumi/pulumi/pull/15607. I've tried to split this into a few commits, separating out adding the testdata, so it's hopefully a little easier to review, even though the PR is still quite large. One other thing to note is that we're comparing that the output has all the same lines, and not that it is exactly the same. Because of how the engine is implemented, there's a bunch of race conditions otherwise, that would make us have to skip a bunch of tests, just because e.g. resource A is sometimes deleted before resource B and sometimes it's the other way around. The biggest downside of that is that running with `PULUMI_ACCEPT` will produce a diff even when there are no changes. Hopefully we won't have to run that way too often though, so it might not be a huge issue? --------- Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
Options: TestUpdateOptions{T: t, HostF: hostF},
Save component resources as parents (#15846) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/15843. This tracks parent information separate from Goal information (as that was only generated and saved for custom, not component, resources). We also need to keep track of the transform list between the first RegisterResource starting a remote component construction and the remote component _actually_ registering itself. This means we save the transform list correctly once the component is registered inside the remote process allowing other resource registrations in the remote process to see that saved transform list. If we just wait till the Construct call is done then none of the inner children pick up the transform list. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-04-15 12:07:45 +00:00
}
project := p.GetProject()
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
assert.NoError(t, err)
assert.Len(t, snap.Resources, 3)
// Check Resources[2] is the resA resource
res := snap.Resources[2]
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typC$pkgA:m:typA::resA"), res.URN)
// Check it's final input properties match what we expected from the transformations
assert.Equal(t, resource.PropertyMap{
"foo": resource.NewNumberProperty(2),
}, res.Inputs)
}