2018-05-22 19:43:36 +00:00
|
|
|
// Copyright 2016-2018, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2018-05-16 22:37:34 +00:00
|
|
|
import * as grpc from "grpc";
|
2017-10-08 19:10:46 +00:00
|
|
|
import * as log from "../log";
|
2018-08-10 23:18:21 +00:00
|
|
|
import { CustomResourceOptions, ID, Input, Inputs, Output, Resource, ResourceOptions, URN } from "../resource";
|
2017-10-18 22:03:56 +00:00
|
|
|
import { debuggablePromise, errorString } from "./debuggable";
|
2018-04-05 16:48:09 +00:00
|
|
|
import {
|
|
|
|
deserializeProperties,
|
2018-05-23 21:47:40 +00:00
|
|
|
deserializeProperty,
|
2018-04-05 16:48:09 +00:00
|
|
|
OutputResolvers,
|
|
|
|
resolveProperties,
|
|
|
|
serializeProperties,
|
|
|
|
serializeProperty,
|
2018-05-01 22:05:42 +00:00
|
|
|
serializeResourceProperties,
|
2018-04-05 16:48:09 +00:00
|
|
|
transferProperties,
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
unknownValue,
|
2018-04-05 16:48:09 +00:00
|
|
|
} from "./rpc";
|
2018-04-07 15:02:59 +00:00
|
|
|
import { excessiveDebugOutput, getMonitor, rpcKeepAlive, serialize } from "./settings";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2018-01-25 21:34:21 +00:00
|
|
|
const gstruct = require("google-protobuf/google/protobuf/struct_pb.js");
|
2017-11-17 02:21:41 +00:00
|
|
|
const resproto = require("../proto/resource_pb.js");
|
2017-09-09 20:49:50 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
interface ResourceResolverOperation {
|
|
|
|
// A resolver for a resource's URN.
|
|
|
|
resolveURN: (urn: URN) => void;
|
|
|
|
// A resolver for a resource's ID (for custom resources only).
|
2018-04-07 17:15:58 +00:00
|
|
|
resolveID: ((v: ID, performApply: boolean) => void) | undefined;
|
2018-04-05 16:48:09 +00:00
|
|
|
// A collection of resolvers for a resource's properties.
|
|
|
|
resolvers: OutputResolvers;
|
|
|
|
// A parent URN, fully resolved, if any.
|
|
|
|
parentURN: URN | undefined;
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
// A provider reference, fully resolved, if any.
|
|
|
|
providerRef: string | undefined;
|
2018-04-05 16:48:09 +00:00
|
|
|
// All serialized properties, fully awaited, serialized, and ready to go.
|
|
|
|
serializedProps: Record<string, any>;
|
|
|
|
// A set of dependency URNs that this resource is dependent upon (both implicitly and explicitly).
|
|
|
|
dependencies: Set<URN>;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Reads an existing custom resource's state from the resource monitor. Note that resources read in this way
|
|
|
|
* will not be part of the resulting stack's state, as they are presumed to belong to another.
|
|
|
|
*/
|
2018-04-07 17:15:58 +00:00
|
|
|
export function readResource(res: Resource, t: string, name: string, props: Inputs, opts: ResourceOptions): void {
|
|
|
|
const id: Input<ID> | undefined = opts.id;
|
|
|
|
if (!id) {
|
|
|
|
throw new Error("Cannot read resource whose options are lacking an ID value");
|
|
|
|
}
|
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
|
|
|
log.debug(`Reading resource: id=${id}, t=${t}, name=${name}`);
|
|
|
|
|
|
|
|
const monitor: any = getMonitor();
|
|
|
|
const resopAsync = prepareResource(label, res, true, props, opts);
|
|
|
|
debuggablePromise(resopAsync.then(async (resop) => {
|
|
|
|
const resolvedID = await serializeProperty(label, id, []);
|
|
|
|
log.debug(`ReadResource RPC prepared: id=${resolvedID}, t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
|
|
|
|
|
2018-04-07 14:52:10 +00:00
|
|
|
// Create a resource request and do the RPC.
|
2018-04-05 16:48:09 +00:00
|
|
|
const req = new resproto.ReadResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
|
|
|
req.setId(resolvedID);
|
|
|
|
req.setParent(resop.parentURN);
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
req.setProvider(resop.providerRef);
|
2018-04-05 16:48:09 +00:00
|
|
|
req.setProperties(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
2018-08-03 21:06:00 +00:00
|
|
|
req.setDependenciesList(Array.from(resop.dependencies));
|
2018-04-05 16:48:09 +00:00
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.readResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
const resp: any = await debuggablePromise(new Promise((resolve, reject) =>
|
|
|
|
monitor.readResource(req, (err: Error, innerResponse: any) => {
|
|
|
|
log.debug(`ReadResource RPC finished: ${label}; err: ${err}, resp: ${innerResponse}`);
|
|
|
|
if (err) {
|
|
|
|
log.error(`Failed to read resource #${resolvedID} '${name}' [${t}]: ${err.stack}`);
|
|
|
|
reject(err);
|
|
|
|
}
|
|
|
|
else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
})), opLabel);
|
|
|
|
|
|
|
|
// Now resolve everything: the URN, the ID (supplied as input), and the output properties.
|
|
|
|
resop.resolveURN(resp.getUrn());
|
2018-04-07 17:15:58 +00:00
|
|
|
resop.resolveID!(resolvedID, resolvedID !== undefined);
|
2018-04-05 16:48:09 +00:00
|
|
|
await resolveOutputs(res, t, name, props, resp.getProperties(), resop.resolvers);
|
|
|
|
});
|
|
|
|
}));
|
|
|
|
}
|
|
|
|
|
2017-09-22 01:15:29 +00:00
|
|
|
/**
|
2017-11-29 19:27:32 +00:00
|
|
|
* registerResource registers a new resource object with a given type t and name. It returns the auto-generated
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
* URN and the ID that will resolve after the deployment has completed. All properties will be initialized to property
|
2017-09-22 01:15:29 +00:00
|
|
|
* objects that the registration operation will resolve at the right time (or remain unresolved for deployments).
|
|
|
|
*/
|
2017-11-29 19:27:32 +00:00
|
|
|
export function registerResource(res: Resource, t: string, name: string, custom: boolean,
|
2018-04-05 16:48:09 +00:00
|
|
|
props: Inputs, opts: ResourceOptions): void {
|
2017-10-31 14:48:59 +00:00
|
|
|
const label = `resource:${name}[${t}]`;
|
2018-04-05 16:48:09 +00:00
|
|
|
log.debug(`Registering resource: t=${t}, name=${name}, custom=${custom}`);
|
|
|
|
|
|
|
|
const monitor: any = getMonitor();
|
|
|
|
const resopAsync = prepareResource(label, res, custom, props, opts);
|
|
|
|
debuggablePromise(resopAsync.then(async (resop) => {
|
|
|
|
log.debug(`RegisterResource RPC prepared: t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
|
|
|
|
|
|
|
|
const req = new resproto.RegisterResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
|
|
|
req.setParent(resop.parentURN);
|
|
|
|
req.setCustom(custom);
|
|
|
|
req.setObject(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
|
|
|
req.setProtect(opts.protect);
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
req.setProvider(resop.providerRef);
|
2018-04-05 16:48:09 +00:00
|
|
|
req.setDependenciesList(Array.from(resop.dependencies));
|
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.registerResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
const resp: any = await debuggablePromise(new Promise((resolve, reject) =>
|
2018-05-16 22:37:34 +00:00
|
|
|
monitor.registerResource(req, (err: grpc.ServiceError, innerResponse: any) => {
|
2018-04-05 16:48:09 +00:00
|
|
|
log.debug(`RegisterResource RPC finished: ${label}; err: ${err}, resp: ${innerResponse}`);
|
|
|
|
if (err) {
|
2018-05-16 22:37:34 +00:00
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs.
|
|
|
|
if (err.code === grpc.status.UNAVAILABLE) {
|
|
|
|
log.debug("Resource monitor is terminating");
|
|
|
|
waitForDeath();
|
|
|
|
}
|
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
log.error(`Failed to register new resource '${name}' [${t}]: ${err.stack}`);
|
|
|
|
reject(err);
|
|
|
|
}
|
|
|
|
else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
})), opLabel);
|
|
|
|
|
|
|
|
resop.resolveURN(resp.getUrn());
|
|
|
|
|
|
|
|
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
|
|
|
|
// undefined so that later parts of our system don't have to deal with values like 'null'.
|
|
|
|
if (resop.resolveID) {
|
|
|
|
const id = resp.getId() || undefined;
|
|
|
|
resop.resolveID(id, id !== undefined);
|
|
|
|
}
|
2017-10-31 14:48:59 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
// Now resolve the output properties.
|
|
|
|
await resolveOutputs(res, t, name, props, resp.getObject(), resop.resolvers);
|
|
|
|
});
|
|
|
|
}));
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Prepares for an RPC that will manufacture a resource, and hence deals with input and output properties.
|
|
|
|
*/
|
|
|
|
async function prepareResource(label: string, res: Resource, custom: boolean,
|
|
|
|
props: Inputs, opts: ResourceOptions): Promise<ResourceResolverOperation> {
|
2017-11-17 02:21:41 +00:00
|
|
|
// Simply initialize the URN property and get prepared to resolve it later on.
|
2018-03-18 07:15:22 +00:00
|
|
|
// Note: a resource urn will always get a value, and thus the output property
|
|
|
|
// for it can always run .apply calls.
|
2018-02-05 22:44:23 +00:00
|
|
|
let resolveURN: (urn: URN) => void;
|
|
|
|
(res as any).urn = Output.create(
|
|
|
|
res,
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<URN>(resolve => resolveURN = resolve),
|
2018-03-18 07:15:22 +00:00
|
|
|
`resolveURN(${label})`),
|
|
|
|
/*performApply:*/ Promise.resolve(true));
|
2017-11-17 02:21:41 +00:00
|
|
|
|
2017-10-15 10:52:04 +00:00
|
|
|
// If a custom resource, make room for the ID property.
|
2018-04-05 16:48:09 +00:00
|
|
|
let resolveID: ((v: any, performApply: boolean) => void) | undefined;
|
2017-10-15 10:52:04 +00:00
|
|
|
if (custom) {
|
2018-03-18 07:15:22 +00:00
|
|
|
let resolveValue: (v: ID) => void;
|
|
|
|
let resolvePerformApply: (v: boolean) => void;
|
2018-02-05 22:44:23 +00:00
|
|
|
(res as any).id = Output.create(
|
|
|
|
res,
|
2018-03-18 07:15:22 +00:00
|
|
|
debuggablePromise(new Promise<ID>(resolve => resolveValue = resolve), `resolveID(${label})`),
|
|
|
|
debuggablePromise(new Promise<boolean>(
|
|
|
|
resolve => resolvePerformApply = resolve), `resolveIDPerformApply(${label})`));
|
2018-04-05 16:48:09 +00:00
|
|
|
|
|
|
|
resolveID = (v, performApply) => {
|
|
|
|
resolveValue(v);
|
|
|
|
resolvePerformApply(performApply);
|
|
|
|
};
|
Implement components
This change implements core support for "components" in the Pulumi
Fabric. This work is described further in pulumi/pulumi#340, where
we are still discussing some of the finer points.
In a nutshell, resources no longer imply external providers. It's
entirely possible to have a resource that logically represents
something but without having a physical manifestation that needs to
be tracked and managed by our typical CRUD operations.
For example, the aws/serverless/Function helper is one such type.
It aggregates Lambda-related resources and exposes a nice interface.
All of the Pulumi Cloud Framework resources are also examples.
To indicate that a resource does participate in the usual CRUD resource
provider, it simply derives from ExternalResource instead of Resource.
All resources now have the ability to adopt children. This is purely
a metadata/tagging thing, and will help us roll up displays, provide
attribution to the developer, and even hide aspects of the resource
graph as appropriate (e.g., when they are implementation details).
Our use of this capability is ultra limited right now; in fact, the
only place we display children is in the CLI output. For instance:
+ aws:serverless:Function: (create)
[urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda]
=> urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole
=> urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0
=> urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda
The bit indicating whether a resource is external or not is tracked
in the resulting checkpoint file, along with any of its children.
2017-10-14 21:18:43 +00:00
|
|
|
}
|
|
|
|
|
2018-01-25 21:34:21 +00:00
|
|
|
// Now "transfer" all input properties into unresolved Promises on res. This way,
|
|
|
|
// this resource will look like it has all its output properties to anyone it is
|
|
|
|
// passed to. However, those promises won't actually resolve until the registerResource
|
|
|
|
// RPC returns
|
2018-04-05 16:48:09 +00:00
|
|
|
const resolvers = transferProperties(res, label, props);
|
|
|
|
|
|
|
|
/** IMPORTANT! We should never await prior to this line, otherwise the Resource will be partly uninitialized. */
|
2017-11-21 01:38:09 +00:00
|
|
|
|
2018-07-12 01:33:53 +00:00
|
|
|
// Before we can proceed, all our dependencies must be finished.
|
2018-08-02 20:13:33 +00:00
|
|
|
let dependsOn: Resource[] = [];
|
|
|
|
if (Array.isArray(opts.dependsOn)) {
|
|
|
|
dependsOn = opts.dependsOn;
|
|
|
|
} else if (opts.dependsOn) {
|
|
|
|
dependsOn = [opts.dependsOn];
|
|
|
|
}
|
2018-07-12 01:33:53 +00:00
|
|
|
const explicitURNDeps = await debuggablePromise(
|
2018-02-21 17:43:17 +00:00
|
|
|
Promise.all(dependsOn.map(d => d.urn.promise())), `dependsOn(${label})`);
|
2018-02-05 22:44:23 +00:00
|
|
|
|
2018-07-12 01:33:53 +00:00
|
|
|
// Serialize out all our props to their final values. In doing so, we'll also collect all
|
|
|
|
// the Resources pointed to by any Dependency objects we encounter, adding them to 'propertyDependencies'.
|
|
|
|
const implicitDependencies: Resource[] = [];
|
|
|
|
const serializedProps = await serializeResourceProperties(label, props, implicitDependencies);
|
2018-01-25 23:26:39 +00:00
|
|
|
|
2018-07-12 01:33:53 +00:00
|
|
|
let parentURN: URN | undefined;
|
|
|
|
if (opts.parent) {
|
|
|
|
parentURN = await opts.parent.urn.promise();
|
|
|
|
}
|
2018-07-10 23:41:56 +00:00
|
|
|
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
let providerRef: string | undefined;
|
2018-08-10 23:18:21 +00:00
|
|
|
if (custom && (<CustomResourceOptions>opts).provider) {
|
|
|
|
const provider = (<CustomResourceOptions>opts).provider!;
|
|
|
|
const providerURN = await provider.urn.promise();
|
|
|
|
const providerID = await provider.id.promise() || unknownValue;
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
providerRef = `${providerURN}::${providerID}`;
|
|
|
|
}
|
|
|
|
|
2018-07-12 01:33:53 +00:00
|
|
|
const dependencies: Set<URN> = new Set<URN>(explicitURNDeps);
|
|
|
|
for (const implicitDep of implicitDependencies) {
|
|
|
|
dependencies.add(await implicitDep.urn.promise());
|
|
|
|
}
|
2018-02-21 17:43:17 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
return {
|
|
|
|
resolveURN: resolveURN!,
|
|
|
|
resolveID: resolveID,
|
|
|
|
resolvers: resolvers,
|
|
|
|
serializedProps: serializedProps,
|
|
|
|
parentURN: parentURN,
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
providerRef: providerRef,
|
2018-04-05 16:48:09 +00:00
|
|
|
dependencies: dependencies,
|
|
|
|
};
|
|
|
|
}
|
2018-01-25 23:26:39 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
/**
|
|
|
|
* Finishes a resource creation RPC operation by resolving its outputs to the resulting RPC payload.
|
|
|
|
*/
|
|
|
|
async function resolveOutputs(res: Resource, t: string, name: string,
|
|
|
|
props: Inputs, outputs: any, resolvers: OutputResolvers): Promise<void> {
|
|
|
|
// Produce a combined set of property states, starting with inputs and then applying
|
|
|
|
// outputs. If the same property exists in the inputs and outputs states, the output wins.
|
|
|
|
const allProps: Record<string, any> = {};
|
|
|
|
if (outputs) {
|
|
|
|
Object.assign(allProps, deserializeProperties(outputs));
|
|
|
|
}
|
2018-02-05 22:44:23 +00:00
|
|
|
|
2018-05-23 21:47:40 +00:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
2018-04-05 16:48:09 +00:00
|
|
|
for (const key of Object.keys(props)) {
|
|
|
|
if (!allProps.hasOwnProperty(key)) {
|
2018-05-23 21:47:40 +00:00
|
|
|
// input prop the engine didn't give us a final value for. Just use the value passed into the resource
|
|
|
|
// after round-tripping it through serialization. We do the round-tripping primarily s.t. we ensure that
|
|
|
|
// Output values are handled properly w.r.t. unknowns.
|
|
|
|
const inputProp = await serializeProperty(label, props[key], []);
|
|
|
|
if (inputProp === undefined) {
|
|
|
|
continue;
|
2018-02-05 22:44:23 +00:00
|
|
|
}
|
2018-05-23 21:47:40 +00:00
|
|
|
allProps[key] = deserializeProperty(inputProp);
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
|
|
|
}
|
2018-02-05 22:44:23 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
resolveProperties(res, resolvers, t, name, allProps);
|
2017-11-29 19:27:32 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* registerResourceOutputs completes the resource registration, attaching an optional set of computed outputs.
|
|
|
|
*/
|
2018-02-05 22:44:23 +00:00
|
|
|
export function registerResourceOutputs(res: Resource, outputs: Inputs) {
|
2018-01-25 21:34:21 +00:00
|
|
|
// Now run the operation. Note that we explicitly do not serialize output registration with
|
|
|
|
// respect to other resource operations, as outputs may depend on properties of other resources
|
|
|
|
// that will not resolve until later turns. This would create a circular promise chain that can
|
|
|
|
// never resolve.
|
2017-11-29 19:27:32 +00:00
|
|
|
const opLabel = `monitor.registerResourceOutputs(...)`;
|
2017-12-14 23:22:01 +00:00
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
2018-02-05 22:44:23 +00:00
|
|
|
// The registration could very well still be taking place, so we will need to wait for its
|
|
|
|
// URN. Additionally, the output properties might have come from other resources, so we
|
|
|
|
// must await those too.
|
|
|
|
const urn = await res.urn.promise();
|
2018-01-25 21:34:21 +00:00
|
|
|
const outputsObj = gstruct.Struct.fromJavaScript(
|
|
|
|
await serializeProperties(`completeResource`, outputs));
|
2017-11-29 19:27:32 +00:00
|
|
|
log.debug(`RegisterResourceOutputs RPC prepared: urn=${urn}` +
|
|
|
|
(excessiveDebugOutput ? `, outputs=${JSON.stringify(outputsObj)}` : ``));
|
|
|
|
|
|
|
|
// Fetch the monitor and make an RPC request.
|
|
|
|
const monitor: any = getMonitor();
|
2018-01-25 21:34:21 +00:00
|
|
|
|
|
|
|
const req = new resproto.RegisterResourceOutputsRequest();
|
|
|
|
req.setUrn(urn);
|
|
|
|
req.setOutputs(outputsObj);
|
|
|
|
|
|
|
|
await debuggablePromise(new Promise((resolve, reject) =>
|
2018-05-16 22:37:34 +00:00
|
|
|
monitor.registerResourceOutputs(req, (err: grpc.ServiceError, innerResponse: any) => {
|
2018-01-25 21:34:21 +00:00
|
|
|
log.debug(`RegisterResourceOutputs RPC finished: urn=${urn}; `+
|
|
|
|
`err: ${err}, resp: ${innerResponse}`);
|
|
|
|
if (err) {
|
2018-05-16 22:37:34 +00:00
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs.
|
|
|
|
if (err.code === grpc.status.UNAVAILABLE) {
|
|
|
|
log.debug("Resource monitor is terminating");
|
|
|
|
waitForDeath();
|
|
|
|
}
|
|
|
|
|
2018-01-25 21:34:21 +00:00
|
|
|
log.error(`Failed to end new resource registration '${urn}': ${err.stack}`);
|
|
|
|
reject(err);
|
|
|
|
}
|
|
|
|
else {
|
|
|
|
resolve();
|
|
|
|
}
|
|
|
|
})), opLabel);
|
2017-12-15 00:24:08 +00:00
|
|
|
}, false);
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* resourceChain is used to serialize all resource requests. If we don't do this, all resource operations will be
|
|
|
|
* entirely asynchronous, meaning the dataflow graph that results will determine ordering of operations. This
|
|
|
|
* causes problems with some resource providers, so for now we will serialize all of them. The issue
|
|
|
|
* pulumi/pulumi#335 tracks coming up with a long-term solution here.
|
|
|
|
*/
|
|
|
|
let resourceChain: Promise<void> = Promise.resolve();
|
|
|
|
let resourceChainLabel: string | undefined = undefined;
|
|
|
|
|
|
|
|
// runAsyncResourceOp runs an asynchronous resource operation, possibly serializing it as necessary.
|
2017-12-15 00:24:08 +00:00
|
|
|
function runAsyncResourceOp(label: string, callback: () => Promise<void>, serial?: boolean): void {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
// Serialize the invocation if necessary.
|
2017-12-15 00:24:08 +00:00
|
|
|
if (serial === undefined) {
|
|
|
|
serial = serialize();
|
|
|
|
}
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
const resourceOp: Promise<void> = debuggablePromise(resourceChain.then(async () => {
|
|
|
|
if (serial) {
|
|
|
|
resourceChainLabel = label;
|
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is current`);
|
|
|
|
}
|
|
|
|
return callback();
|
2017-09-09 17:32:25 +00:00
|
|
|
}));
|
2017-09-15 23:38:52 +00:00
|
|
|
|
2017-12-14 23:22:01 +00:00
|
|
|
// Ensure the process won't exit until this RPC call finishes and resolve it when appropriate.
|
2017-10-18 22:03:56 +00:00
|
|
|
const done: () => void = rpcKeepAlive();
|
2017-12-14 23:22:01 +00:00
|
|
|
const finalOp: Promise<void> = debuggablePromise(resourceOp.then(() => { done(); }, () => { done(); }));
|
|
|
|
|
|
|
|
// Set up another promise that propagates the error, if any, so that it triggers unhandled rejection logic.
|
|
|
|
resourceOp.catch((err) => Promise.reject(err));
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 14:40:09 +00:00
|
|
|
|
2017-09-15 23:38:52 +00:00
|
|
|
// If serialization is requested, wait for the prior resource operation to finish before we proceed, serializing
|
|
|
|
// them, and make this the current resource operation so that everybody piles up on it.
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
if (serial) {
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 14:40:09 +00:00
|
|
|
resourceChain = finalOp;
|
2017-10-31 13:52:42 +00:00
|
|
|
if (resourceChainLabel) {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is behind ${resourceChainLabel}`);
|
2017-10-31 13:52:42 +00:00
|
|
|
}
|
2017-09-15 23:38:52 +00:00
|
|
|
}
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
}
|
2018-05-16 22:37:34 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* waitForDeath loops forever. This is a hack.
|
|
|
|
*
|
|
|
|
* The purpose of this hack is to deal with graceful termination of the resource monitor.
|
|
|
|
* When the engine decides that it needs to terminate, it shuts down the Log and ResourceMonitor RPC
|
|
|
|
* endpoints. Shutting down RPC endpoints involves draining all outstanding RPCs and denying new connections.
|
|
|
|
*
|
|
|
|
* This is all fine for us as the language host, but we need to 1) not let the RPC that just failed due to
|
|
|
|
* the ResourceMonitor server shutdown get displayed as an error and 2) not do any more RPCs, since they'll fail.
|
|
|
|
*
|
|
|
|
* We can accomplish both by just doing nothing until the engine kills us. It's ugly, but it works.
|
|
|
|
*/
|
|
|
|
function waitForDeath(): never {
|
|
|
|
// tslint:disable-next-line
|
|
|
|
while (true) {}
|
|
|
|
}
|