Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
// Copyright 2016-2022, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
|
|
|
package lifecycletest
|
|
|
|
|
|
|
|
import (
|
2024-04-02 14:39:58 +00:00
|
|
|
"errors"
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
"fmt"
|
|
|
|
"testing"
|
|
|
|
|
|
|
|
"github.com/blang/semver"
|
|
|
|
"google.golang.org/protobuf/proto"
|
|
|
|
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
|
|
"github.com/stretchr/testify/require"
|
|
|
|
|
|
|
|
. "github.com/pulumi/pulumi/pkg/v3/engine" //nolint:revive
|
|
|
|
"github.com/pulumi/pulumi/pkg/v3/resource/deploy/deploytest"
|
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
|
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
|
|
|
|
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
|
|
|
|
pulumirpc "github.com/pulumi/pulumi/sdk/v3/proto/go"
|
|
|
|
)
|
|
|
|
|
|
|
|
func TransformFunction(
|
|
|
|
f func(
|
|
|
|
name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error),
|
|
|
|
) func([]byte) (proto.Message, error) {
|
|
|
|
return func(request []byte) (proto.Message, error) {
|
|
|
|
var transformationRequest pulumirpc.TransformRequest
|
|
|
|
err := proto.Unmarshal(request, &transformationRequest)
|
|
|
|
if err != nil {
|
|
|
|
return nil, fmt.Errorf("unmarshaling request: %w", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
mprops, err := plugin.UnmarshalProperties(transformationRequest.Properties, plugin.MarshalOptions{
|
|
|
|
KeepUnknowns: true,
|
|
|
|
KeepSecrets: true,
|
|
|
|
KeepResources: true,
|
|
|
|
KeepOutputValues: true,
|
|
|
|
})
|
|
|
|
if err != nil {
|
|
|
|
return nil, fmt.Errorf("unmarshaling properties: %w", err)
|
|
|
|
}
|
|
|
|
|
|
|
|
ret, opts, err := f(
|
|
|
|
transformationRequest.Name, transformationRequest.Type, transformationRequest.Custom, transformationRequest.Parent,
|
|
|
|
mprops, transformationRequest.Options)
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
mret, err := plugin.MarshalProperties(ret, plugin.MarshalOptions{
|
|
|
|
KeepUnknowns: true,
|
|
|
|
KeepSecrets: true,
|
|
|
|
KeepResources: true,
|
|
|
|
KeepOutputValues: true,
|
|
|
|
})
|
|
|
|
if err != nil {
|
|
|
|
return nil, err
|
|
|
|
}
|
|
|
|
|
|
|
|
return &pulumirpc.TransformResponse{
|
|
|
|
Properties: mret,
|
|
|
|
Options: opts,
|
|
|
|
}, nil
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
func pvApply(pv resource.PropertyValue, f func(resource.PropertyValue) resource.PropertyValue) resource.PropertyValue {
|
|
|
|
if pv.IsOutput() {
|
|
|
|
o := pv.OutputValue()
|
|
|
|
if !o.Known {
|
|
|
|
return pv
|
|
|
|
}
|
|
|
|
return resource.NewOutputProperty(resource.Output{
|
|
|
|
Element: f(o.Element),
|
|
|
|
Known: true,
|
|
|
|
Secret: o.Secret,
|
|
|
|
Dependencies: o.Dependencies,
|
|
|
|
})
|
|
|
|
}
|
|
|
|
return f(pv)
|
|
|
|
}
|
|
|
|
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
// Test that the engine invokes all transformation functions in the correct order.
|
|
|
|
func TestRemoteTransforms(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
|
|
|
|
return resource.NewNumberProperty(v.NumberValue() + 1)
|
|
|
|
})
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
// callback 2 should run before this one so "bar" should exist at this point
|
|
|
|
props["bar"] = resource.NewStringProperty(props["bar"].StringValue() + "baz")
|
|
|
|
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
callback2, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
|
|
|
|
return resource.NewNumberProperty(v.NumberValue() + 1)
|
|
|
|
})
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
props["bar"] = resource.NewStringProperty("bar")
|
|
|
|
// if this is for resB then callback 3 will have run before this one
|
|
|
|
if prop, has := props["frob"]; has {
|
|
|
|
props["frob"] = resource.MakeSecret(prop)
|
|
|
|
} else {
|
|
|
|
props["frob"] = resource.NewStringProperty("nofrob")
|
|
|
|
}
|
|
|
|
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
callback3, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
|
|
|
|
return resource.NewNumberProperty(v.NumberValue() + 1)
|
|
|
|
})
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
props["frob"] = resource.NewStringProperty("frob")
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
err = monitor.RegisterStackTransform(callback1)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
Transforms: []*pulumirpc.Callback{
|
|
|
|
callback2,
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resB", true, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(10),
|
|
|
|
},
|
|
|
|
Transforms: []*pulumirpc.Callback{
|
|
|
|
callback3,
|
|
|
|
},
|
2024-04-19 11:08:56 +00:00
|
|
|
Parent: respA.URN,
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
return nil
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
// Skip display tests because secrets are serialized with the blinding crypter and can't be restored
|
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF, SkipDisplayTests: true},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
|
|
|
|
assert.Len(t, snap.Resources, 3)
|
|
|
|
// Check Resources[1] is the resA resource
|
|
|
|
res := snap.Resources[1]
|
|
|
|
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA::resA"), res.URN)
|
|
|
|
// Check it's final input properties match what we expected from the transformations
|
|
|
|
assert.Equal(t, resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(3),
|
|
|
|
"bar": resource.NewStringProperty("barbaz"),
|
|
|
|
"frob": resource.NewStringProperty("nofrob"),
|
|
|
|
}, res.Inputs)
|
|
|
|
|
|
|
|
// Check Resources[2] is the resB resource
|
|
|
|
res = snap.Resources[2]
|
|
|
|
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA$pkgA:m:typA::resB"), res.URN)
|
|
|
|
// Check it's final input properties match what we expected from the transformations
|
|
|
|
assert.Equal(t, resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(13),
|
|
|
|
"bar": resource.NewStringProperty("barbaz"),
|
|
|
|
"frob": resource.MakeSecret(resource.NewStringProperty("frob")),
|
|
|
|
}, res.Inputs)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Test that the engine errors if a transformation function returns an unexpected response.
|
|
|
|
func TestRemoteTransformBadResponse(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(func(args []byte) (proto.Message, error) {
|
|
|
|
// return the wrong message type
|
|
|
|
return &pulumirpc.RegisterResourceResponse{
|
|
|
|
Urn: "boom",
|
|
|
|
}, nil
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
err = monitor.RegisterStackTransform(callback1)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
assert.ErrorContains(t, err, "unmarshaling response: proto:")
|
|
|
|
assert.ErrorContains(t, err, "cannot parse invalid wire-format data")
|
|
|
|
return err
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.ErrorContains(t, err, "unmarshaling response: proto:")
|
|
|
|
assert.ErrorContains(t, err, "cannot parse invalid wire-format data")
|
|
|
|
assert.Len(t, snap.Resources, 0)
|
|
|
|
}
|
|
|
|
|
2024-04-02 14:39:58 +00:00
|
|
|
// Test that the engine errors if a transformation function returns an error.
|
|
|
|
func TestRemoteTransformErrorResponse(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(func(args []byte) (proto.Message, error) {
|
|
|
|
return nil, errors.New("bad transform")
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
err = monitor.RegisterStackTransform(callback1)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
2024-04-02 14:39:58 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
assert.ErrorContains(t, err, "Unknown desc = bad transform")
|
|
|
|
return err
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
2024-04-02 14:39:58 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.ErrorContains(t, err, "Unknown desc = bad transform")
|
|
|
|
assert.Len(t, snap.Resources, 0)
|
|
|
|
}
|
|
|
|
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
// Test that a remote transform applies to a resource inside a component construct.
|
|
|
|
func TestRemoteTransformationsConstruct(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{
|
|
|
|
ConstructF: func(
|
|
|
|
monitor *deploytest.ResourceMonitor, typ string, name string, parent resource.URN,
|
|
|
|
inputs resource.PropertyMap, info plugin.ConstructInfo, options plugin.ConstructOptions,
|
|
|
|
) (plugin.ConstructResult, error) {
|
|
|
|
assert.Equal(t, "pkgA:m:typC", typ)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
resp, err := monitor.RegisterResource(tokens.Type(typ), name, false, deploytest.ResourceOptions{})
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
|
|
|
Parent: resp.URN,
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
return plugin.ConstructResult{
|
2024-04-19 11:08:56 +00:00
|
|
|
URN: resp.URN,
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}, nil
|
|
|
|
},
|
|
|
|
}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
|
|
|
if typ == "pkgA:m:typA" {
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
|
|
|
|
return resource.NewNumberProperty(v.NumberValue() + 1)
|
|
|
|
})
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
err = monitor.RegisterStackTransform(callback1)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typC", "resC", false, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Remote: true,
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
return nil
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
|
|
|
|
assert.Len(t, snap.Resources, 3)
|
|
|
|
// Check Resources[2] is the resA resource
|
|
|
|
res := snap.Resources[2]
|
|
|
|
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typC$pkgA:m:typA::resA"), res.URN)
|
|
|
|
// Check it's final input properties match what we expected from the transformations
|
|
|
|
assert.Equal(t, resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(2),
|
|
|
|
}, res.Inputs)
|
|
|
|
}
|
|
|
|
|
|
|
|
// Test that all options are passed and can be modified by a transformation function.
|
|
|
|
func TestRemoteTransformsOptions(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
urnB := "urn:pulumi:test::test::pkgA:m:typA::resB"
|
|
|
|
urnC := "urn:pulumi:test::test::pkgA:m:typA::resC"
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
urnD := "urn:pulumi:test::test::pkgA:m:typA::resD"
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{})
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
respC, err := monitor.RegisterResource("pkgA:m:typA", "resC", true, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
Version: "1.0.0",
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
|
|
|
// Check that the options are passed through correctly
|
|
|
|
assert.Equal(t, []string{"foo"}, opts.AdditionalSecretOutputs)
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
assert.Equal(t, urnB, opts.Aliases[0].Alias.(*pulumirpc.Alias_Urn).Urn)
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
assert.Equal(t, "16m40s", opts.CustomTimeouts.Create)
|
|
|
|
assert.Equal(t, "33m20s", opts.CustomTimeouts.Update)
|
|
|
|
assert.Equal(t, "50m0s", opts.CustomTimeouts.Delete)
|
|
|
|
assert.True(t, *opts.DeleteBeforeReplace)
|
2024-04-19 11:08:56 +00:00
|
|
|
assert.Equal(t, string(respA.URN), opts.DeletedWith)
|
|
|
|
assert.Equal(t, []string{string(respA.URN)}, opts.DependsOn)
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
assert.Equal(t, []string{"foo"}, opts.IgnoreChanges)
|
|
|
|
assert.Equal(t, "http://server", opts.PluginDownloadUrl)
|
|
|
|
assert.Equal(t, false, opts.Protect)
|
|
|
|
assert.Equal(t, []string{"foo"}, opts.ReplaceOnChanges)
|
|
|
|
assert.Equal(t, "2.0.0", opts.Version)
|
|
|
|
|
|
|
|
// Modify all the options
|
|
|
|
opts = &pulumirpc.TransformResourceOptions{
|
|
|
|
AdditionalSecretOutputs: []string{"bar"},
|
|
|
|
Aliases: []*pulumirpc.Alias{
|
|
|
|
{Alias: &pulumirpc.Alias_Urn{Urn: urnB}},
|
|
|
|
},
|
|
|
|
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
|
|
|
|
Create: "1s",
|
|
|
|
Update: "2s",
|
|
|
|
Delete: "3s",
|
|
|
|
},
|
|
|
|
DeleteBeforeReplace: nil,
|
2024-04-19 11:08:56 +00:00
|
|
|
DeletedWith: string(respC.URN),
|
|
|
|
DependsOn: []string{string(respC.URN)},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
IgnoreChanges: []string{"bar"},
|
|
|
|
PluginDownloadUrl: "",
|
|
|
|
Protect: true,
|
|
|
|
ReplaceOnChanges: []string{"bar"},
|
|
|
|
Version: "1.0.0",
|
|
|
|
}
|
|
|
|
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
err = monitor.RegisterStackTransform(callback1)
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
dbr := true
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resD", true, deploytest.ResourceOptions{
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
AdditionalSecretOutputs: []resource.PropertyKey{"foo"},
|
|
|
|
Aliases: []*pulumirpc.Alias{
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
{Alias: &pulumirpc.Alias_Urn{Urn: urnB}},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
},
|
|
|
|
CustomTimeouts: &resource.CustomTimeouts{
|
|
|
|
Create: 1000,
|
|
|
|
Update: 2000,
|
|
|
|
Delete: 3000,
|
|
|
|
},
|
|
|
|
DeleteBeforeReplace: &dbr,
|
2024-04-19 11:08:56 +00:00
|
|
|
DeletedWith: respA.URN,
|
|
|
|
Dependencies: []resource.URN{respA.URN},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
IgnoreChanges: []string{"foo"},
|
|
|
|
PluginDownloadURL: "http://server",
|
|
|
|
ReplaceOnChanges: []string{"foo"},
|
|
|
|
Version: "2.0.0",
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
return nil
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
require.NoError(t, err)
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
assert.Len(t, snap.Resources, 5)
|
|
|
|
// Check Resources[4] is the resD resource
|
|
|
|
res := snap.Resources[4]
|
|
|
|
require.Equal(t, resource.URN(urnD), res.URN)
|
Engine support for remote transforms (#15290)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This adds support to the engine for "remote transformations".
A transform is "remote" because it is being invoked via the engine on
receiving a resource registration, rather than being ran locally in
process before sending a resource registration. These transforms can
also span multiple process boundaries, e.g. a transform function in a
user program, then a transform function in a component library, both
running for a resource registered by another component library.
The underlying new feature here is the idea of a `Callback`. The
expectation is we're going to use callbacks for multiple features so
these are _not_ defined in terms of transformations. A callback is an
untyped byte array (usually will be a protobuf message), plus an address
to define which server should be invoked to do the callback, and a token
to identify it.
A language sdk can start up and serve a `Callbacks` service, keep a
mapping of tokens to in-process functions (currently just using UUID's
for this), and then pass that service address and token to the engine to
be invoked later on.
The engine uses these callbacks to track transformations callbacks per
resource, and on a new resource registrations invokes each relevant
callback with the resource properties and options, having new properties
and options returned that are then passed to the next relevant transform
callback until all have been called and the engine has the final
resource state and options to use.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-21 16:30:46 +00:00
|
|
|
assert.Equal(t, []resource.PropertyKey{"bar"}, res.AdditionalSecretOutputs)
|
|
|
|
assert.Equal(t, resource.CustomTimeouts{
|
|
|
|
Create: 1,
|
|
|
|
Update: 2,
|
|
|
|
Delete: 3,
|
|
|
|
}, res.CustomTimeouts)
|
|
|
|
assert.Equal(t, resource.URN(urnC), res.DeletedWith)
|
|
|
|
assert.Equal(t, true, res.Protect)
|
|
|
|
}
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
|
|
|
|
// Test that a transform can change the dependencies of a resource.
|
|
|
|
func TestRemoteTransformsDependencies(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{
|
|
|
|
CreateF: func(
|
|
|
|
urn resource.URN, props resource.PropertyMap, timeout float64, preview bool,
|
|
|
|
) (resource.ID, resource.PropertyMap, resource.Status, error) {
|
|
|
|
return "some-id", props, resource.StatusOK, nil
|
|
|
|
},
|
|
|
|
}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
respA, err := monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
2024-04-19 11:08:56 +00:00
|
|
|
assert.True(t, respA.Outputs["foo"].IsNumber())
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
|
|
|
|
// Register a separate resource that
|
2024-04-19 11:08:56 +00:00
|
|
|
respB, err := monitor.RegisterResource("pkgA:m:typA", "resB", true, deploytest.ResourceOptions{
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(10),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
callback, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
|
|
|
// props should be tracking that it depends on resB
|
|
|
|
assert.True(t, props["foo"].IsOutput())
|
2024-04-19 11:08:56 +00:00
|
|
|
assert.Equal(t, []resource.URN{respB.URN}, props["foo"].OutputValue().Dependencies)
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
|
|
|
|
// Add a dependency on resA
|
|
|
|
props["foo"] = resource.NewOutputProperty(resource.Output{
|
2024-04-19 11:08:56 +00:00
|
|
|
Element: respA.Outputs["foo"],
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
Known: true,
|
2024-04-19 11:08:56 +00:00
|
|
|
Dependencies: []resource.URN{respA.URN},
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
})
|
|
|
|
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
// Register a resource that initially depends on resB but the transform will turn to depend on resA
|
2024-04-19 11:08:56 +00:00
|
|
|
respC, err := monitor.RegisterResource(
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
"pkgA:m:typA", "resC", true, deploytest.ResourceOptions{
|
|
|
|
Inputs: resource.PropertyMap{
|
2024-04-19 11:08:56 +00:00
|
|
|
"foo": respB.Outputs["foo"],
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
},
|
|
|
|
PropertyDeps: map[resource.PropertyKey][]resource.URN{
|
2024-04-19 11:08:56 +00:00
|
|
|
"foo": {respB.URN},
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
},
|
|
|
|
Transforms: []*pulumirpc.Callback{
|
|
|
|
callback,
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
2024-04-19 11:08:56 +00:00
|
|
|
assert.True(t, respC.Outputs["foo"].IsNumber())
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
// This is a custom resource so no output dependencies
|
2024-04-19 11:08:56 +00:00
|
|
|
assert.Empty(t, respC.Dependencies)
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
return nil
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
Send output values to transforms for dependency tracking (#15637)
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This engine change is necessary for correct dependency tracking of
properties through transform functions.
Unlike other parts of the runtime/provider/engine interface transform
functions do not have a property dependencies map sent or returned.
Transforms rely entirely on the dependency arrays in output values for
their dependency tracking.
This change takes the property dependency map and upgrades all the input
properties to be output values tracking those same dependencies (if a
property doesn't have any dependencies it doesn't need to be upgraded to
an output value).
This upgrade is only done if we're using transform functions, there's
currently no need to do this unless we're using transforms.
After running the transforms we downgrade the output values back to
plain/secret/computed values if we're registering a custom resource. If
we're registering a component resource we just leave all the properties
upgraded as output values.
We also rebuild the resources dependencies slice and propertyDependency
map with the dependencies recorded in the transformed properties. If the
transform didn't change the properties this will just be the same data
we encoded into the output values in the properties structure.
There are a couple of things to keep in mind with this change.
Firstly using a transform can cause a property that was being sent to a
component provider as just a `resource.NumberProperty` to start being
sent as an `OutputProperty` with the `NumberProperty` as its element.
Component providers _should_ handle that, but it's feasible that it
could break some component code. This is a fairly limited blast radius
as it will only happen when that user starts using a transform function.
Secondly, we can't know in the engine if a property dependencies are
from a top level output value or a nested output. That is SDKs send both
of the following property shapes the same way to the engine for custom
resources:
```
A) Output(Array([Number(1)]), ["dep"])
---
B) Array([Output(Number(1), ["dep"])
```
Currently both would get upgraded to `Output(Array([Number(1)]),
["dep"])`. For a component resource the SDK will send output values, and
as long as they define a superset of the dependencies in the property
map then we don't add the top-level output tracking the same dependency
set.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-20 09:53:33 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
|
|
|
|
assert.Len(t, snap.Resources, 4)
|
|
|
|
// Check Resources[3] is the resC resource
|
|
|
|
res := snap.Resources[3]
|
|
|
|
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typA::resC"), res.URN)
|
|
|
|
// Check it's final input properties match what we expected from the transformations
|
|
|
|
assert.Equal(t, resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
}, res.Inputs)
|
|
|
|
// Check the dependencies are as expected
|
|
|
|
assert.Equal(t, map[resource.PropertyKey][]resource.URN{
|
|
|
|
"foo": {resource.URN("urn:pulumi:test::test::pkgA:m:typA::resA")},
|
|
|
|
}, res.PropertyDependencies)
|
|
|
|
assert.Equal(t, []resource.URN{
|
|
|
|
"urn:pulumi:test::test::pkgA:m:typA::resA",
|
|
|
|
}, res.Dependencies)
|
|
|
|
}
|
2024-04-15 12:07:45 +00:00
|
|
|
|
|
|
|
// Regression test for https://github.com/pulumi/pulumi/issues/15843. Ensure that if a component resource has a
|
|
|
|
// transform that's saved and looked up by it's children.
|
|
|
|
func TestRemoteComponentTransforms(t *testing.T) {
|
|
|
|
t.Parallel()
|
|
|
|
|
|
|
|
loaders := []*deploytest.ProviderLoader{
|
|
|
|
deploytest.NewProviderLoader("pkgA", semver.MustParse("1.0.0"), func() (plugin.Provider, error) {
|
|
|
|
return &deploytest.Provider{
|
|
|
|
ConstructF: func(
|
|
|
|
monitor *deploytest.ResourceMonitor, typ string, name string, parent resource.URN,
|
|
|
|
inputs resource.PropertyMap, info plugin.ConstructInfo, options plugin.ConstructOptions,
|
|
|
|
) (plugin.ConstructResult, error) {
|
|
|
|
assert.Equal(t, "pkgA:m:typC", typ)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
resp, err := monitor.RegisterResource(tokens.Type(typ), name, false, deploytest.ResourceOptions{})
|
2024-04-15 12:07:45 +00:00
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typA", "resA", true, deploytest.ResourceOptions{
|
|
|
|
Parent: resp.URN,
|
2024-04-15 12:07:45 +00:00
|
|
|
Inputs: resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(1),
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
return plugin.ConstructResult{
|
2024-04-19 11:08:56 +00:00
|
|
|
URN: resp.URN,
|
2024-04-15 12:07:45 +00:00
|
|
|
}, nil
|
|
|
|
},
|
|
|
|
}, nil
|
|
|
|
}),
|
|
|
|
}
|
|
|
|
|
|
|
|
programF := deploytest.NewLanguageRuntimeF(func(_ plugin.RunInfo, monitor *deploytest.ResourceMonitor) error {
|
|
|
|
callbacks, err := deploytest.NewCallbacksServer()
|
|
|
|
require.NoError(t, err)
|
|
|
|
defer func() { require.NoError(t, callbacks.Close()) }()
|
|
|
|
|
|
|
|
callback1, err := callbacks.Allocate(
|
|
|
|
TransformFunction(func(name, typ string, custom bool, parent string,
|
|
|
|
props resource.PropertyMap, opts *pulumirpc.TransformResourceOptions,
|
|
|
|
) (resource.PropertyMap, *pulumirpc.TransformResourceOptions, error) {
|
|
|
|
if typ == "pkgA:m:typA" {
|
|
|
|
props["foo"] = pvApply(props["foo"], func(v resource.PropertyValue) resource.PropertyValue {
|
|
|
|
return resource.NewNumberProperty(v.NumberValue() + 1)
|
|
|
|
})
|
|
|
|
}
|
|
|
|
return props, opts, nil
|
|
|
|
}))
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
2024-04-19 11:08:56 +00:00
|
|
|
_, err = monitor.RegisterResource("pkgA:m:typC", "resC", false, deploytest.ResourceOptions{
|
2024-04-15 12:07:45 +00:00
|
|
|
Remote: true,
|
|
|
|
Transforms: []*pulumirpc.Callback{
|
|
|
|
callback1,
|
|
|
|
},
|
|
|
|
})
|
|
|
|
require.NoError(t, err)
|
|
|
|
|
|
|
|
return nil
|
|
|
|
})
|
|
|
|
hostF := deploytest.NewPluginHostF(nil, nil, programF, loaders...)
|
|
|
|
|
|
|
|
p := &TestPlan{
|
Add display to the engine tests (#16050)
We want to add more test coverage to the display code. The best way to
do that is to add it to the engine tests, that already cover most of the
pulumi functionality.
It's probably not really possible to review all of the output, but at
least it gives us a baseline, which we can work with.
There's a couple of tests that are flaky for reasons I don't quite
understand yet. I marked them as to skip and we can look at them later.
I'd rather get in the baseline tests sooner, rather than spending a
bunch of time looking at that. The output differences also seem very
minor, so not super concerning.
The biggest remaining issue is that this doesn't interact well with the
Chdir we're doing in the engine. We could either pass the CWD through,
or just try to get rid of that Chdir. So this should only be merged
after https://github.com/pulumi/pulumi/pull/15607.
I've tried to split this into a few commits, separating out adding the
testdata, so it's hopefully a little easier to review, even though the
PR is still quite large.
One other thing to note is that we're comparing that the output has all
the same lines, and not that it is exactly the same. Because of how the
engine is implemented, there's a bunch of race conditions otherwise,
that would make us have to skip a bunch of tests, just because e.g.
resource A is sometimes deleted before resource B and sometimes it's the
other way around.
The biggest downside of that is that running with `PULUMI_ACCEPT` will
produce a diff even when there are no changes. Hopefully we won't have
to run that way too often though, so it might not be a huge issue?
---------
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-05-13 07:18:25 +00:00
|
|
|
Options: TestUpdateOptions{T: t, HostF: hostF},
|
2024-04-15 12:07:45 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
project := p.GetProject()
|
|
|
|
snap, err := TestOp(Update).Run(project, p.GetTarget(t, nil), p.Options, false, p.BackendClient, nil)
|
|
|
|
assert.NoError(t, err)
|
|
|
|
|
|
|
|
assert.Len(t, snap.Resources, 3)
|
|
|
|
// Check Resources[2] is the resA resource
|
|
|
|
res := snap.Resources[2]
|
|
|
|
assert.Equal(t, resource.URN("urn:pulumi:test::test::pkgA:m:typC$pkgA:m:typA::resA"), res.URN)
|
|
|
|
// Check it's final input properties match what we expected from the transformations
|
|
|
|
assert.Equal(t, resource.PropertyMap{
|
|
|
|
"foo": resource.NewNumberProperty(2),
|
|
|
|
}, res.Inputs)
|
|
|
|
}
|