2021-09-16 01:25:26 +00:00
|
|
|
// Copyright 2016-2021, Pulumi Corporation.
|
2018-05-22 19:43:36 +00:00
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2020-04-14 08:30:25 +00:00
|
|
|
import * as grpc from "@grpc/grpc-js";
|
2019-04-18 22:25:15 +00:00
|
|
|
import * as query from "@pulumi/query";
|
2017-10-08 19:10:46 +00:00
|
|
|
import * as log from "../log";
|
2019-08-08 19:11:46 +00:00
|
|
|
import * as utils from "../utils";
|
|
|
|
|
2019-12-18 03:04:09 +00:00
|
|
|
import { getAllResources, Input, Inputs, Output, output } from "../output";
|
2019-04-18 22:25:15 +00:00
|
|
|
import { ResolvedResource } from "../queryable";
|
2019-04-17 05:20:01 +00:00
|
|
|
import {
|
2022-10-31 18:25:24 +00:00
|
|
|
Alias,
|
|
|
|
allAliases,
|
2019-04-17 05:20:01 +00:00
|
|
|
ComponentResource,
|
2021-04-19 22:41:53 +00:00
|
|
|
ComponentResourceOptions,
|
2019-06-01 06:01:01 +00:00
|
|
|
createUrn,
|
2019-04-17 05:20:01 +00:00
|
|
|
CustomResource,
|
|
|
|
CustomResourceOptions,
|
2022-09-22 17:13:55 +00:00
|
|
|
expandProviders,
|
2019-04-17 05:20:01 +00:00
|
|
|
ID,
|
2019-10-15 05:08:06 +00:00
|
|
|
ProviderResource,
|
2019-04-17 05:20:01 +00:00
|
|
|
Resource,
|
|
|
|
ResourceOptions,
|
|
|
|
URN,
|
|
|
|
} from "../resource";
|
2023-06-15 14:43:05 +00:00
|
|
|
import { debuggablePromise, debugPromiseLeaks } from "./debuggable";
|
2019-04-18 22:25:15 +00:00
|
|
|
import { invoke } from "./invoke";
|
2023-12-19 14:35:23 +00:00
|
|
|
import { getStore } from "./state";
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2023-12-04 15:22:44 +00:00
|
|
|
import { isGrpcError } from "../errors";
|
2018-04-05 16:48:09 +00:00
|
|
|
import {
|
|
|
|
deserializeProperties,
|
2018-05-23 21:47:40 +00:00
|
|
|
deserializeProperty,
|
2018-04-05 16:48:09 +00:00
|
|
|
OutputResolvers,
|
|
|
|
resolveProperties,
|
|
|
|
serializeProperties,
|
|
|
|
serializeProperty,
|
2018-05-01 22:05:42 +00:00
|
|
|
serializeResourceProperties,
|
2020-09-24 02:06:26 +00:00
|
|
|
suppressUnhandledGrpcRejections,
|
2018-04-05 16:48:09 +00:00
|
|
|
transferProperties,
|
|
|
|
} from "./rpc";
|
2019-04-17 05:20:01 +00:00
|
|
|
import {
|
2024-03-07 08:52:34 +00:00
|
|
|
awaitStackRegistrations,
|
2019-04-17 05:20:01 +00:00
|
|
|
excessiveDebugOutput,
|
2024-03-07 08:52:34 +00:00
|
|
|
getCallbacks,
|
2019-04-17 05:20:01 +00:00
|
|
|
getMonitor,
|
|
|
|
getStack,
|
2019-08-05 19:44:04 +00:00
|
|
|
isDryRun,
|
|
|
|
isLegacyApplyEnabled,
|
2019-04-17 05:20:01 +00:00
|
|
|
rpcKeepAlive,
|
|
|
|
serialize,
|
2020-09-24 02:06:26 +00:00
|
|
|
terminateRpcs,
|
2019-04-17 05:20:01 +00:00
|
|
|
} from "./settings";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2023-12-04 15:22:44 +00:00
|
|
|
import * as gempty from "google-protobuf/google/protobuf/empty_pb";
|
|
|
|
import * as gstruct from "google-protobuf/google/protobuf/struct_pb";
|
|
|
|
import * as aliasproto from "../proto/alias_pb";
|
2024-03-07 08:52:34 +00:00
|
|
|
import { Callback } from "../proto/callback_pb";
|
2023-12-04 15:22:44 +00:00
|
|
|
import * as provproto from "../proto/provider_pb";
|
|
|
|
import * as resproto from "../proto/resource_pb";
|
|
|
|
import * as sourceproto from "../proto/source_pb";
|
2023-07-10 17:59:15 +00:00
|
|
|
|
|
|
|
export interface SourcePosition {
|
|
|
|
uri: string;
|
|
|
|
line: number;
|
|
|
|
column: number;
|
|
|
|
}
|
|
|
|
|
|
|
|
function marshalSourcePosition(sourcePosition?: SourcePosition) {
|
|
|
|
if (sourcePosition === undefined) {
|
|
|
|
return undefined;
|
|
|
|
}
|
|
|
|
const pos = new sourceproto.SourcePosition();
|
|
|
|
pos.setUri(sourcePosition.uri);
|
|
|
|
pos.setLine(sourcePosition.line);
|
|
|
|
pos.setColumn(sourcePosition.column);
|
|
|
|
return pos;
|
|
|
|
}
|
2017-09-09 20:49:50 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
interface ResourceResolverOperation {
|
|
|
|
// A resolver for a resource's URN.
|
2020-09-24 02:06:26 +00:00
|
|
|
resolveURN: (urn: URN, err?: Error) => void;
|
2018-04-05 16:48:09 +00:00
|
|
|
// A resolver for a resource's ID (for custom resources only).
|
2020-09-24 02:06:26 +00:00
|
|
|
resolveID: ((v: ID, performApply: boolean, err?: Error) => void) | undefined;
|
2018-04-05 16:48:09 +00:00
|
|
|
// A collection of resolvers for a resource's properties.
|
|
|
|
resolvers: OutputResolvers;
|
|
|
|
// A parent URN, fully resolved, if any.
|
|
|
|
parentURN: URN | undefined;
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
// A provider reference, fully resolved, if any.
|
|
|
|
providerRef: string | undefined;
|
2021-04-19 22:41:53 +00:00
|
|
|
// A map of provider references, fully resolved, if any.
|
|
|
|
providerRefs: Map<string, string>;
|
2018-04-05 16:48:09 +00:00
|
|
|
// All serialized properties, fully awaited, serialized, and ready to go.
|
|
|
|
serializedProps: Record<string, any>;
|
2019-03-06 01:06:57 +00:00
|
|
|
// A set of URNs that this resource is directly dependent upon. These will all be URNs of
|
|
|
|
// custom resources, not component resources.
|
2019-02-22 04:18:29 +00:00
|
|
|
allDirectDependencyURNs: Set<URN>;
|
2019-03-06 01:06:57 +00:00
|
|
|
// Set of URNs that this resource is directly dependent upon, keyed by the property that causes
|
|
|
|
// the dependency. All urns in this map must exist in [allDirectDependencyURNs]. These will
|
|
|
|
// all be URNs of custom resources, not component resources.
|
2019-02-22 04:18:29 +00:00
|
|
|
propertyToDirectDependencyURNs: Map<string, Set<URN>>;
|
2019-06-01 06:01:01 +00:00
|
|
|
// A list of aliases applied to this resource.
|
2022-10-31 18:25:24 +00:00
|
|
|
aliases: (Alias | URN)[];
|
2019-07-12 18:12:01 +00:00
|
|
|
// An ID to import, if any.
|
|
|
|
import: ID | undefined;
|
2022-10-31 18:25:24 +00:00
|
|
|
// Any important feature support from the monitor.
|
|
|
|
monitorSupportsStructuredAliases: boolean;
|
[sdk/{go,nodejs,python}] Fix DeletedWith resource option
This change fixes the `DeletedWith` resource option in the Go, Node.js,
and Python SDKs and adds tests.
This feature was a community contribution and while there were engine
tests included with the original PR, there weren't any tests confirming
the functionality worked correctly from each SDK.
Here's a summary of the fixes:
* Go: The `DeletedWith` resource option was never usable as it accepted
a URN instead of a Resource. We discussed this internally a while back
and decided to go ahead and fix this. (Note: While changing the
signature is technically a breaking change, the feature is currently
unusable, so the change would not break anyone, so there's no need to
wait for a major version bump.)
* Node.js: The `deletedWith` resource option did not work at all from
the Node.js SDK because it was incorrectly passing the resource object
itself in the RegisterResource request, rather than the resource's
URN.
* Python: The `deleted_with` resource option did not work at all from
the Python SDK because it was incorrectly passing the resource object
itself in the RegisterResource request, rather than the resource's
URN.
A `FailsOnDelete` resource has been added to the testprovider, which
will fail when its `Delete` gRPC is called. The tests use this to ensure
`Delete` is not called for resources of this type with the `DeletedWith`
option specified.
2023-01-16 00:19:26 +00:00
|
|
|
// If set, the providers Delete method will not be called for this resource
|
|
|
|
// if specified is being deleted as well.
|
|
|
|
deletedWithURN: URN | undefined;
|
2019-04-17 05:20:01 +00:00
|
|
|
}
|
|
|
|
|
2020-12-01 18:58:15 +00:00
|
|
|
/**
|
|
|
|
* Get an existing resource's state from the engine.
|
|
|
|
*/
|
2023-04-28 22:27:10 +00:00
|
|
|
export function getResource(
|
|
|
|
res: Resource,
|
|
|
|
parent: Resource | undefined,
|
|
|
|
props: Inputs,
|
|
|
|
custom: boolean,
|
|
|
|
urn: string,
|
|
|
|
): void {
|
2020-12-01 18:58:15 +00:00
|
|
|
// Extract the resource type from the URN.
|
|
|
|
const urnParts = urn.split("::");
|
|
|
|
const qualifiedType = urnParts[2];
|
|
|
|
const urnName = urnParts[3];
|
|
|
|
const type = qualifiedType.split("$").pop()!;
|
|
|
|
|
|
|
|
const label = `resource:urn=${urn}`;
|
|
|
|
log.debug(`Getting resource: urn=${urn}`);
|
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Wait for all values to be available, and then perform the RPC.
|
|
|
|
const done = rpcKeepAlive();
|
|
|
|
|
2023-12-04 15:22:44 +00:00
|
|
|
const monitor = getMonitor();
|
2022-06-30 10:04:49 +00:00
|
|
|
const resopAsync = prepareResource(label, res, parent, custom, false, props, {});
|
2020-12-01 18:58:15 +00:00
|
|
|
|
|
|
|
const preallocError = new Error();
|
2023-04-28 22:27:10 +00:00
|
|
|
debuggablePromise(
|
|
|
|
resopAsync.then(async (resop) => {
|
|
|
|
const inputs = await serializeProperties(label, { urn });
|
|
|
|
|
|
|
|
const req = new resproto.ResourceInvokeRequest();
|
|
|
|
req.setTok("pulumi:pulumi:getResource");
|
|
|
|
req.setArgs(gstruct.Struct.fromJavaScript(inputs));
|
|
|
|
req.setProvider("");
|
|
|
|
req.setVersion("");
|
|
|
|
req.setAcceptresources(!utils.disableResourceReferences);
|
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.getResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
let resp: any = {};
|
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
resp = await debuggablePromise(
|
|
|
|
new Promise((resolve, reject) =>
|
2023-12-04 15:22:44 +00:00
|
|
|
monitor.invoke(
|
|
|
|
req,
|
|
|
|
(
|
|
|
|
rpcError: grpc.ServiceError | null,
|
|
|
|
innerResponse: provproto.InvokeResponse | undefined,
|
|
|
|
) => {
|
|
|
|
log.debug(
|
|
|
|
`getResource Invoke RPC finished: err: ${rpcError}, resp: ${innerResponse}`,
|
|
|
|
);
|
|
|
|
if (rpcError) {
|
|
|
|
if (
|
|
|
|
rpcError.code === grpc.status.UNAVAILABLE ||
|
|
|
|
rpcError.code === grpc.status.CANCELLED
|
|
|
|
) {
|
|
|
|
err = rpcError;
|
|
|
|
terminateRpcs();
|
|
|
|
rpcError.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcError.code;
|
|
|
|
}
|
2023-04-28 22:27:10 +00:00
|
|
|
|
2023-12-04 15:22:44 +00:00
|
|
|
preallocError.message = `failed to get resource:urn=${urn}: ${rpcError.message}`;
|
|
|
|
reject(new Error(rpcError.details));
|
|
|
|
} else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
},
|
|
|
|
),
|
2023-04-28 22:27:10 +00:00
|
|
|
),
|
|
|
|
opLabel,
|
|
|
|
);
|
|
|
|
|
|
|
|
// If the invoke failed, raise an error
|
2023-12-04 15:22:44 +00:00
|
|
|
const failures = resp.getFailuresList();
|
2023-04-29 02:16:01 +00:00
|
|
|
if (failures?.length) {
|
2023-04-28 22:27:10 +00:00
|
|
|
let reasons = "";
|
|
|
|
for (let i = 0; i < failures.length; i++) {
|
|
|
|
if (reasons !== "") {
|
|
|
|
reasons += "; ";
|
2020-12-01 18:58:15 +00:00
|
|
|
}
|
2023-04-28 22:27:10 +00:00
|
|
|
reasons += `${failures[i].getReason()} (${failures[i].getProperty()})`;
|
2020-12-01 18:58:15 +00:00
|
|
|
}
|
2023-04-28 22:27:10 +00:00
|
|
|
throw new Error(`getResource Invoke failed: ${reasons}`);
|
2020-12-01 18:58:15 +00:00
|
|
|
}
|
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Otherwise, return the response.
|
|
|
|
const m = resp.getReturn().getFieldsMap();
|
|
|
|
resp = {
|
|
|
|
urn: m.get("urn").toJavaScript(),
|
|
|
|
id: m.get("id").toJavaScript() || undefined,
|
|
|
|
state: m.get("state").getStructValue(),
|
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
2020-12-01 18:58:15 +00:00
|
|
|
resp = {
|
2023-04-28 22:27:10 +00:00
|
|
|
urn: "",
|
|
|
|
id: undefined,
|
|
|
|
state: undefined,
|
2020-12-01 18:58:15 +00:00
|
|
|
};
|
|
|
|
}
|
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
resop.resolveURN(resp.urn, err);
|
2020-12-01 18:58:15 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
|
|
|
|
// undefined so that later parts of our system don't have to deal with values like 'null'.
|
|
|
|
if (resop.resolveID) {
|
|
|
|
const id = resp.id || undefined;
|
|
|
|
resop.resolveID(id, id !== undefined, err);
|
|
|
|
}
|
2020-12-01 18:58:15 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
await resolveOutputs(res, type, urnName, props, resp.state, {}, resop.resolvers, err);
|
2024-04-23 12:57:04 +00:00
|
|
|
done();
|
2023-04-28 22:27:10 +00:00
|
|
|
});
|
|
|
|
}),
|
|
|
|
label,
|
|
|
|
);
|
2020-12-01 18:58:15 +00:00
|
|
|
}
|
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
/**
|
|
|
|
* Reads an existing custom resource's state from the resource monitor. Note that resources read in this way
|
|
|
|
* will not be part of the resulting stack's state, as they are presumed to belong to another.
|
|
|
|
*/
|
2023-04-28 22:27:10 +00:00
|
|
|
export function readResource(
|
|
|
|
res: Resource,
|
|
|
|
parent: Resource | undefined,
|
|
|
|
t: string,
|
|
|
|
name: string,
|
|
|
|
props: Inputs,
|
|
|
|
opts: ResourceOptions,
|
2023-07-10 17:59:15 +00:00
|
|
|
sourcePosition?: SourcePosition,
|
2023-04-28 22:27:10 +00:00
|
|
|
): void {
|
2023-10-19 16:00:10 +00:00
|
|
|
if (!opts.id) {
|
2018-04-07 17:15:58 +00:00
|
|
|
throw new Error("Cannot read resource whose options are lacking an ID value");
|
|
|
|
}
|
2023-10-19 16:00:10 +00:00
|
|
|
const id: Promise<Input<ID>> = output(opts.id).promise(true);
|
2018-04-07 17:15:58 +00:00
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
2023-10-19 16:00:10 +00:00
|
|
|
log.debug(`Reading resource: t=${t}, name=${name}`);
|
2018-04-05 16:48:09 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Wait for all values to be available, and then perform the RPC.
|
|
|
|
const done = rpcKeepAlive();
|
|
|
|
|
2019-04-17 05:20:01 +00:00
|
|
|
const monitor = getMonitor();
|
2022-06-30 10:04:49 +00:00
|
|
|
const resopAsync = prepareResource(label, res, parent, true, false, props, opts);
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2018-09-07 22:19:18 +00:00
|
|
|
const preallocError = new Error();
|
2023-04-28 22:27:10 +00:00
|
|
|
debuggablePromise(
|
|
|
|
resopAsync.then(async (resop) => {
|
2023-10-19 16:00:10 +00:00
|
|
|
const resolvedID = await serializeProperty(label, await id, new Set(), { keepOutputValues: false });
|
2023-04-28 22:27:10 +00:00
|
|
|
log.debug(
|
|
|
|
`ReadResource RPC prepared: id=${resolvedID}, t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``),
|
|
|
|
);
|
|
|
|
|
|
|
|
// Create a resource request and do the RPC.
|
|
|
|
const req = new resproto.ReadResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
|
|
|
req.setId(resolvedID);
|
2023-12-04 15:22:44 +00:00
|
|
|
req.setParent(resop.parentURN || "");
|
|
|
|
req.setProvider(resop.providerRef || "");
|
2023-04-28 22:27:10 +00:00
|
|
|
req.setProperties(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
|
|
|
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
|
|
|
|
req.setVersion(opts.version || "");
|
|
|
|
req.setPlugindownloadurl(opts.pluginDownloadURL || "");
|
|
|
|
req.setAcceptsecrets(true);
|
|
|
|
req.setAcceptresources(!utils.disableResourceReferences);
|
|
|
|
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
|
2023-07-10 17:59:15 +00:00
|
|
|
req.setSourceposition(marshalSourcePosition(sourcePosition));
|
2023-04-28 22:27:10 +00:00
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.readResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
let resp: any = {};
|
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
// If we're attached to the engine, make an RPC call and wait for it to resolve.
|
|
|
|
resp = await debuggablePromise(
|
|
|
|
new Promise((resolve, reject) =>
|
2023-12-04 15:22:44 +00:00
|
|
|
monitor.readResource(
|
2023-04-28 22:27:10 +00:00
|
|
|
req,
|
2023-12-04 15:22:44 +00:00
|
|
|
(
|
|
|
|
rpcError: grpc.ServiceError | null,
|
|
|
|
innerResponse: resproto.ReadResourceResponse | undefined,
|
|
|
|
) => {
|
2023-04-28 22:27:10 +00:00
|
|
|
log.debug(
|
|
|
|
`ReadResource RPC finished: ${label}; err: ${rpcError}, resp: ${innerResponse}`,
|
|
|
|
);
|
|
|
|
if (rpcError) {
|
|
|
|
if (
|
|
|
|
rpcError.code === grpc.status.UNAVAILABLE ||
|
|
|
|
rpcError.code === grpc.status.CANCELLED
|
|
|
|
) {
|
|
|
|
err = rpcError;
|
|
|
|
terminateRpcs();
|
|
|
|
rpcError.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcError.code;
|
|
|
|
}
|
|
|
|
|
|
|
|
preallocError.message = `failed to read resource #${resolvedID} '${name}' [${t}]: ${rpcError.message}`;
|
|
|
|
reject(preallocError);
|
|
|
|
} else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
},
|
|
|
|
),
|
|
|
|
),
|
|
|
|
opLabel,
|
|
|
|
);
|
|
|
|
} else {
|
|
|
|
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
|
|
|
|
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
|
|
|
|
resp = {
|
|
|
|
getUrn: () => mockurn,
|
|
|
|
getProperties: () => req.getProperties(),
|
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
2020-09-24 02:06:26 +00:00
|
|
|
resp = {
|
2023-04-28 22:27:10 +00:00
|
|
|
getUrn: () => "",
|
|
|
|
getProperties: () => undefined,
|
2020-09-24 02:06:26 +00:00
|
|
|
};
|
|
|
|
}
|
2018-04-05 16:48:09 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Now resolve everything: the URN, the ID (supplied as input), and the output properties.
|
|
|
|
resop.resolveURN(resp.getUrn(), err);
|
|
|
|
resop.resolveID!(resolvedID, resolvedID !== undefined, err);
|
|
|
|
await resolveOutputs(res, t, name, props, resp.getProperties(), {}, resop.resolvers, err);
|
2024-04-23 12:57:04 +00:00
|
|
|
done();
|
2023-04-28 22:27:10 +00:00
|
|
|
});
|
|
|
|
}),
|
|
|
|
label,
|
|
|
|
);
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
|
|
|
|
2022-10-31 18:25:24 +00:00
|
|
|
function getParentURN(parent?: Resource | Input<string>) {
|
|
|
|
if (Resource.isInstance(parent)) {
|
|
|
|
return parent.urn;
|
|
|
|
}
|
|
|
|
return output(parent);
|
|
|
|
}
|
|
|
|
|
2024-03-07 08:52:34 +00:00
|
|
|
export function mapAliasesForRequest(
|
|
|
|
aliases: (URN | Alias)[] | undefined,
|
|
|
|
parentURN?: URN,
|
|
|
|
): Promise<aliasproto.Alias[]> {
|
2022-10-31 18:25:24 +00:00
|
|
|
if (aliases === undefined) {
|
2024-03-07 08:52:34 +00:00
|
|
|
return Promise.resolve([]);
|
2022-10-31 18:25:24 +00:00
|
|
|
}
|
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
return Promise.all(
|
|
|
|
aliases.map(async (a) => {
|
|
|
|
const newAlias = new aliasproto.Alias();
|
|
|
|
if (typeof a === "string") {
|
|
|
|
newAlias.setUrn(a);
|
2022-10-31 18:25:24 +00:00
|
|
|
} else {
|
2023-04-28 22:27:10 +00:00
|
|
|
const newAliasSpec = new aliasproto.Alias.Spec();
|
2023-12-04 15:22:44 +00:00
|
|
|
const name = a.name === undefined ? undefined : await output(a.name).promise();
|
|
|
|
const type = a.type === undefined ? undefined : await output(a.type).promise();
|
|
|
|
const stack = a.stack === undefined ? undefined : await output(a.stack).promise();
|
|
|
|
const project = a.project === undefined ? undefined : await output(a.project).promise();
|
|
|
|
|
|
|
|
newAliasSpec.setName(name || "");
|
|
|
|
newAliasSpec.setType(type || "");
|
|
|
|
newAliasSpec.setStack(stack || "");
|
|
|
|
newAliasSpec.setProject(project || "");
|
2023-05-05 23:45:34 +00:00
|
|
|
if (a.hasOwnProperty("parent")) {
|
|
|
|
if (a.parent === undefined) {
|
|
|
|
newAliasSpec.setNoparent(true);
|
|
|
|
} else {
|
|
|
|
const aliasParentUrn = getParentURN(a.parent);
|
|
|
|
const urn = await aliasParentUrn.promise();
|
2023-12-04 15:22:44 +00:00
|
|
|
if (urn !== undefined) {
|
|
|
|
newAliasSpec.setParenturn(urn);
|
|
|
|
}
|
2023-05-05 23:45:34 +00:00
|
|
|
}
|
|
|
|
} else if (parentURN) {
|
|
|
|
// If a parent isn't specified for the alias and the resource has a parent,
|
|
|
|
// pass along the resource's parent in the alias spec.
|
|
|
|
// It shouldn't be necessary to do this because the engine should fill-in the
|
|
|
|
// resource's parent if one wasn't specified for the alias.
|
|
|
|
// However, some older versions of the CLI don't do this correctly, and this
|
|
|
|
// SDK has always passed along the parent in this way, so we continue doing it
|
|
|
|
// to maintain compatibility with these versions of the CLI.
|
|
|
|
newAliasSpec.setParenturn(parentURN);
|
2023-04-28 22:27:10 +00:00
|
|
|
}
|
|
|
|
newAlias.setSpec(newAliasSpec);
|
2022-10-31 18:25:24 +00:00
|
|
|
}
|
2023-04-28 22:27:10 +00:00
|
|
|
return newAlias;
|
|
|
|
}),
|
|
|
|
);
|
2022-10-31 18:25:24 +00:00
|
|
|
}
|
|
|
|
|
2017-09-22 01:15:29 +00:00
|
|
|
/**
|
2017-11-29 19:27:32 +00:00
|
|
|
* registerResource registers a new resource object with a given type t and name. It returns the auto-generated
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
* URN and the ID that will resolve after the deployment has completed. All properties will be initialized to property
|
2017-09-22 01:15:29 +00:00
|
|
|
* objects that the registration operation will resolve at the right time (or remain unresolved for deployments).
|
|
|
|
*/
|
2023-04-28 22:27:10 +00:00
|
|
|
export function registerResource(
|
|
|
|
res: Resource,
|
|
|
|
parent: Resource | undefined,
|
|
|
|
t: string,
|
|
|
|
name: string,
|
|
|
|
custom: boolean,
|
|
|
|
remote: boolean,
|
|
|
|
newDependency: (urn: URN) => Resource,
|
|
|
|
props: Inputs,
|
|
|
|
opts: ResourceOptions,
|
2023-07-10 17:59:15 +00:00
|
|
|
sourcePosition?: SourcePosition,
|
2023-04-28 22:27:10 +00:00
|
|
|
): void {
|
2017-10-31 14:48:59 +00:00
|
|
|
const label = `resource:${name}[${t}]`;
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 02:33:55 +00:00
|
|
|
log.debug(`Registering resource: t=${t}, name=${name}, custom=${custom}, remote=${remote}`);
|
2018-04-05 16:48:09 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Wait for all values to be available, and then perform the RPC.
|
|
|
|
const done = rpcKeepAlive();
|
|
|
|
|
2019-04-17 05:20:01 +00:00
|
|
|
const monitor = getMonitor();
|
2022-10-31 18:25:24 +00:00
|
|
|
const resopAsync = prepareResource(label, res, parent, custom, remote, props, opts, t, name);
|
2018-09-07 22:19:18 +00:00
|
|
|
|
|
|
|
// In order to present a useful stack trace if an error does occur, we preallocate potential
|
|
|
|
// errors here. V8 captures a stack trace at the moment an Error is created and this stack
|
|
|
|
// trace will lead directly to user code. Throwing in `runAsyncResourceOp` results in an Error
|
|
|
|
// with a non-useful stack trace.
|
|
|
|
const preallocError = new Error();
|
2023-04-28 22:27:10 +00:00
|
|
|
debuggablePromise(
|
|
|
|
resopAsync.then(async (resop) => {
|
|
|
|
log.debug(
|
|
|
|
`RegisterResource RPC prepared: t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``),
|
|
|
|
);
|
|
|
|
|
2024-03-07 08:52:34 +00:00
|
|
|
await awaitStackRegistrations();
|
|
|
|
|
|
|
|
const callbacks: Callback[] = [];
|
2024-04-29 10:54:41 +00:00
|
|
|
if (opts.transforms !== undefined && opts.transforms.length > 0) {
|
2024-03-07 08:52:34 +00:00
|
|
|
if (!getStore().supportsTransforms) {
|
|
|
|
throw new Error("The Pulumi CLI does not support transforms. Please update the Pulumi CLI");
|
|
|
|
}
|
|
|
|
|
|
|
|
const callbackServer = getCallbacks();
|
|
|
|
if (callbackServer === undefined) {
|
|
|
|
throw new Error("Callback server could not initialize");
|
|
|
|
}
|
|
|
|
|
2024-04-29 10:54:41 +00:00
|
|
|
for (const transform of opts.transforms) {
|
2024-03-07 08:52:34 +00:00
|
|
|
callbacks.push(await callbackServer.registerTransform(transform));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
const req = new resproto.RegisterResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
2023-12-04 15:22:44 +00:00
|
|
|
req.setParent(resop.parentURN || "");
|
2023-04-28 22:27:10 +00:00
|
|
|
req.setCustom(custom);
|
|
|
|
req.setObject(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
2023-12-04 15:22:44 +00:00
|
|
|
req.setProtect(opts.protect || false);
|
|
|
|
req.setProvider(resop.providerRef || "");
|
2023-04-28 22:27:10 +00:00
|
|
|
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
|
|
|
|
req.setDeletebeforereplace((<any>opts).deleteBeforeReplace || false);
|
|
|
|
req.setDeletebeforereplacedefined((<any>opts).deleteBeforeReplace !== undefined);
|
|
|
|
req.setIgnorechangesList(opts.ignoreChanges || []);
|
|
|
|
req.setVersion(opts.version || "");
|
|
|
|
req.setAcceptsecrets(true);
|
|
|
|
req.setAcceptresources(!utils.disableResourceReferences);
|
|
|
|
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
|
|
|
|
if (resop.monitorSupportsStructuredAliases) {
|
|
|
|
const aliasesList = await mapAliasesForRequest(resop.aliases, resop.parentURN);
|
|
|
|
req.setAliasesList(aliasesList);
|
|
|
|
} else {
|
2023-12-04 15:22:44 +00:00
|
|
|
const urns = new Array<string>();
|
|
|
|
resop.aliases.forEach((v) => {
|
|
|
|
if (typeof v === "string") {
|
|
|
|
urns.push(v);
|
|
|
|
}
|
|
|
|
});
|
|
|
|
req.setAliasurnsList(urns);
|
2023-04-28 22:27:10 +00:00
|
|
|
}
|
|
|
|
req.setImportid(resop.import || "");
|
|
|
|
req.setSupportspartialvalues(true);
|
|
|
|
req.setRemote(remote);
|
|
|
|
req.setReplaceonchangesList(opts.replaceOnChanges || []);
|
|
|
|
req.setPlugindownloadurl(opts.pluginDownloadURL || "");
|
|
|
|
req.setRetainondelete(opts.retainOnDelete || false);
|
|
|
|
req.setDeletedwith(resop.deletedWithURN || "");
|
Maintain alias compat for older Node.js SDKs on new CLIs
This change updates the engine to detect if a `RegisterResource` request
is coming from an older Node.js SDK that is using incorrect alias specs
and, if so, transforms the aliases to be correct. This allows us to
maintain compatibility for users who have upgraded their CLI but are
still using an older version of the Node.js SDK with incorrect alias
specs.
We detect if the request is from a Node.js SDK by looking at the gRPC
request's metadata headers, specifically looking at the "pulumi-runtime"
and "user-agent" headers.
First, if the request has a "pulumi-runtime" header with a value of
"nodejs", we know it's coming from the Node.js plugin. The Node.js
language plugin proxies gRPC calls from the Node.js SDK to the resource
monitor and the proxy now sets the "pulumi-runtime" header to "nodejs"
for `RegisterResource` calls.
Second, if the request has a "user-agent" header that starts with
"grpc-node-js/", we know it's coming from the Node.js SDK. This is the
case for inline programs in the automation API, which connects directly
to the resource monitor, rather than going through the language plugin's
proxy.
We can't just look at "user-agent", because in the proxy case it will
have a Go-specific "user-agent".
Updated Node.js SDKs set a new `aliasSpecs` field on the
`RegisterResource` request, which indicates that the alias specs are
correct, and no transforms are needed.
2023-05-31 22:37:59 +00:00
|
|
|
req.setAliasspecs(true);
|
2023-07-10 17:59:15 +00:00
|
|
|
req.setSourceposition(marshalSourcePosition(sourcePosition));
|
2024-03-07 08:52:34 +00:00
|
|
|
req.setTransformsList(callbacks);
|
2024-04-22 11:12:45 +00:00
|
|
|
req.setSupportsresultreporting(true);
|
2023-04-28 22:27:10 +00:00
|
|
|
|
2023-12-19 14:35:23 +00:00
|
|
|
if (resop.deletedWithURN && !getStore().supportsDeletedWith) {
|
2023-04-28 22:27:10 +00:00
|
|
|
throw new Error(
|
|
|
|
"The Pulumi CLI does not support the DeletedWith option. Please update the Pulumi CLI.",
|
|
|
|
);
|
|
|
|
}
|
Addition of Custom Timeouts (#2885)
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-15 21:26:28 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
const customTimeouts = new resproto.RegisterResourceRequest.CustomTimeouts();
|
|
|
|
if (opts.customTimeouts != null) {
|
2023-12-04 15:22:44 +00:00
|
|
|
customTimeouts.setCreate(opts.customTimeouts.create || "");
|
|
|
|
customTimeouts.setUpdate(opts.customTimeouts.update || "");
|
|
|
|
customTimeouts.setDelete(opts.customTimeouts.delete || "");
|
2023-04-28 22:27:10 +00:00
|
|
|
}
|
|
|
|
req.setCustomtimeouts(customTimeouts);
|
Implement more precise delete-before-replace semantics. (#2369)
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
2019-01-28 17:46:30 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
const propertyDependencies = req.getPropertydependenciesMap();
|
|
|
|
for (const [key, resourceURNs] of resop.propertyToDirectDependencyURNs) {
|
|
|
|
const deps = new resproto.RegisterResourceRequest.PropertyDependencies();
|
|
|
|
deps.setUrnsList(Array.from(resourceURNs));
|
|
|
|
propertyDependencies.set(key, deps);
|
|
|
|
}
|
2021-04-19 22:41:53 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
const providerRefs = req.getProvidersMap();
|
|
|
|
for (const [key, ref] of resop.providerRefs) {
|
|
|
|
providerRefs.set(key, ref);
|
|
|
|
}
|
2020-09-24 02:06:26 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.registerResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
let resp: any = {};
|
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
// If we're running with an attachment to the engine, perform the operation.
|
|
|
|
resp = await debuggablePromise(
|
|
|
|
new Promise((resolve, reject) =>
|
2023-12-04 15:22:44 +00:00
|
|
|
monitor.registerResource(
|
2023-04-28 22:27:10 +00:00
|
|
|
req,
|
2023-12-04 15:22:44 +00:00
|
|
|
(
|
|
|
|
rpcErr: grpc.ServiceError | null,
|
|
|
|
innerResponse: resproto.RegisterResourceResponse | undefined,
|
|
|
|
) => {
|
2023-04-28 22:27:10 +00:00
|
|
|
if (rpcErr) {
|
|
|
|
err = rpcErr;
|
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs, just exit.
|
|
|
|
if (
|
|
|
|
rpcErr.code === grpc.status.UNAVAILABLE ||
|
|
|
|
rpcErr.code === grpc.status.CANCELLED
|
|
|
|
) {
|
|
|
|
// Re-emit the message
|
|
|
|
terminateRpcs();
|
|
|
|
rpcErr.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcErr.code;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Node lets us hack the message as long as we do it before accessing the `stack` property.
|
|
|
|
log.debug(
|
|
|
|
`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`,
|
|
|
|
);
|
|
|
|
preallocError.message = `failed to register new resource ${name} [${t}]: ${rpcErr.message}`;
|
|
|
|
reject(preallocError);
|
|
|
|
} else {
|
|
|
|
log.debug(
|
|
|
|
`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`,
|
|
|
|
);
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
},
|
|
|
|
),
|
|
|
|
),
|
|
|
|
opLabel,
|
|
|
|
);
|
|
|
|
} else {
|
|
|
|
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
|
|
|
|
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
|
|
|
|
resp = {
|
|
|
|
getUrn: () => mockurn,
|
|
|
|
getId: () => undefined,
|
|
|
|
getObject: () => req.getObject(),
|
|
|
|
getPropertydependenciesMap: () => undefined,
|
2024-04-22 11:12:45 +00:00
|
|
|
getResult: () => 0,
|
2023-04-28 22:27:10 +00:00
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
2020-09-24 02:06:26 +00:00
|
|
|
resp = {
|
2023-04-28 22:27:10 +00:00
|
|
|
getUrn: () => "",
|
2020-09-24 02:06:26 +00:00
|
|
|
getId: () => undefined,
|
|
|
|
getObject: () => req.getObject(),
|
2020-10-01 19:31:52 +00:00
|
|
|
getPropertydependenciesMap: () => undefined,
|
2024-04-22 11:12:45 +00:00
|
|
|
getResult: () => 0,
|
2020-09-24 02:06:26 +00:00
|
|
|
};
|
|
|
|
}
|
2018-04-05 16:48:09 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
resop.resolveURN(resp.getUrn(), err);
|
2018-04-05 16:48:09 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
|
|
|
|
// undefined so that later parts of our system don't have to deal with values like 'null'.
|
|
|
|
if (resop.resolveID) {
|
|
|
|
const id = resp.getId() || undefined;
|
|
|
|
resop.resolveID(id, id !== undefined, err);
|
|
|
|
}
|
2017-10-31 14:48:59 +00:00
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
const deps: Record<string, Resource[]> = {};
|
|
|
|
const rpcDeps = resp.getPropertydependenciesMap();
|
|
|
|
if (rpcDeps) {
|
|
|
|
for (const [k, propertyDeps] of resp.getPropertydependenciesMap().entries()) {
|
|
|
|
const urns = <URN[]>propertyDeps.getUrnsList();
|
|
|
|
deps[k] = urns.map((urn) => newDependency(urn));
|
|
|
|
}
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 02:33:55 +00:00
|
|
|
}
|
|
|
|
|
2023-04-28 22:27:10 +00:00
|
|
|
// Now resolve the output properties.
|
2024-04-22 11:12:45 +00:00
|
|
|
const keepUnknowns = resp.getResult() !== resproto.Result.SUCCESS;
|
|
|
|
await resolveOutputs(res, t, name, props, resp.getObject(), deps, resop.resolvers, err, keepUnknowns);
|
2024-04-23 12:57:04 +00:00
|
|
|
done();
|
2023-04-28 22:27:10 +00:00
|
|
|
});
|
|
|
|
}),
|
|
|
|
label,
|
|
|
|
);
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
|
|
|
|
2023-06-09 19:04:35 +00:00
|
|
|
/** @internal
|
2019-02-22 04:18:29 +00:00
|
|
|
* Prepares for an RPC that will manufacture a resource, and hence deals with input and output
|
|
|
|
* properties.
|
2018-04-05 16:48:09 +00:00
|
|
|
*/
|
2023-06-09 19:04:35 +00:00
|
|
|
export async function prepareResource(
|
2023-04-28 22:27:10 +00:00
|
|
|
label: string,
|
|
|
|
res: Resource,
|
|
|
|
parent: Resource | undefined,
|
|
|
|
custom: boolean,
|
|
|
|
remote: boolean,
|
|
|
|
props: Inputs,
|
|
|
|
opts: ResourceOptions,
|
|
|
|
type?: string,
|
|
|
|
name?: string,
|
|
|
|
): Promise<ResourceResolverOperation> {
|
2024-04-23 12:57:04 +00:00
|
|
|
// Simply initialize the URN property and get prepared to resolve it later on.
|
|
|
|
// Note: a resource urn will always get a value, and thus the output property
|
|
|
|
// for it can always run .apply calls.
|
|
|
|
let resolveURN: (urn: URN, err?: Error) => void;
|
|
|
|
{
|
|
|
|
let resolveValue: (urn: URN) => void;
|
|
|
|
let rejectValue: (err: Error) => void;
|
|
|
|
let resolveIsKnown: (isKnown: boolean) => void;
|
|
|
|
let rejectIsKnown: (err: Error) => void;
|
|
|
|
(res as any).urn = new Output(
|
|
|
|
res,
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<URN>((resolve, reject) => {
|
|
|
|
resolveValue = resolve;
|
|
|
|
rejectValue = reject;
|
|
|
|
}),
|
|
|
|
`resolveURN(${label})`,
|
|
|
|
),
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<boolean>((resolve, reject) => {
|
|
|
|
resolveIsKnown = resolve;
|
|
|
|
rejectIsKnown = reject;
|
|
|
|
}),
|
|
|
|
`resolveURNIsKnown(${label})`,
|
|
|
|
),
|
|
|
|
/*isSecret:*/ Promise.resolve(false),
|
|
|
|
Promise.resolve(res),
|
|
|
|
);
|
2021-08-04 20:19:43 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
resolveURN = (v, err) => {
|
|
|
|
if (err) {
|
|
|
|
if (isGrpcError(err)) {
|
|
|
|
if (debugPromiseLeaks) {
|
|
|
|
console.error("info: skipped rejection in resolveURN");
|
2023-06-09 19:04:35 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
return;
|
2021-08-04 20:19:43 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
rejectValue(err);
|
|
|
|
rejectIsKnown(err);
|
|
|
|
} else {
|
|
|
|
resolveValue(v);
|
|
|
|
resolveIsKnown(true);
|
|
|
|
}
|
|
|
|
};
|
|
|
|
}
|
2021-08-04 20:19:43 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// If a custom resource, make room for the ID property.
|
|
|
|
let resolveID: ((v: any, performApply: boolean, err?: Error) => void) | undefined;
|
|
|
|
if (custom) {
|
|
|
|
let resolveValue: (v: ID) => void;
|
|
|
|
let rejectValue: (err: Error) => void;
|
|
|
|
let resolveIsKnown: (v: boolean) => void;
|
|
|
|
let rejectIsKnown: (err: Error) => void;
|
|
|
|
|
|
|
|
(res as any).id = new Output(
|
|
|
|
res,
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<ID>((resolve, reject) => {
|
|
|
|
resolveValue = resolve;
|
|
|
|
rejectValue = reject;
|
|
|
|
}),
|
|
|
|
`resolveID(${label})`,
|
|
|
|
),
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<boolean>((resolve, reject) => {
|
|
|
|
resolveIsKnown = resolve;
|
|
|
|
rejectIsKnown = reject;
|
|
|
|
}),
|
|
|
|
`resolveIDIsKnown(${label})`,
|
|
|
|
),
|
|
|
|
Promise.resolve(false),
|
|
|
|
Promise.resolve(res),
|
|
|
|
);
|
2021-08-04 20:19:43 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
resolveID = (v, isKnown, err) => {
|
|
|
|
if (err) {
|
|
|
|
if (isGrpcError(err)) {
|
|
|
|
if (debugPromiseLeaks) {
|
|
|
|
console.error("info: skipped rejection in resolveID");
|
2023-06-09 19:04:35 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
return;
|
2021-08-04 20:19:43 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
rejectValue(err);
|
|
|
|
rejectIsKnown(err);
|
|
|
|
} else {
|
|
|
|
resolveValue(v);
|
|
|
|
resolveIsKnown(isKnown);
|
|
|
|
}
|
|
|
|
};
|
|
|
|
}
|
Implement components
This change implements core support for "components" in the Pulumi
Fabric. This work is described further in pulumi/pulumi#340, where
we are still discussing some of the finer points.
In a nutshell, resources no longer imply external providers. It's
entirely possible to have a resource that logically represents
something but without having a physical manifestation that needs to
be tracked and managed by our typical CRUD operations.
For example, the aws/serverless/Function helper is one such type.
It aggregates Lambda-related resources and exposes a nice interface.
All of the Pulumi Cloud Framework resources are also examples.
To indicate that a resource does participate in the usual CRUD resource
provider, it simply derives from ExternalResource instead of Resource.
All resources now have the ability to adopt children. This is purely
a metadata/tagging thing, and will help us roll up displays, provide
attribution to the developer, and even hide aspects of the resource
graph as appropriate (e.g., when they are implementation details).
Our use of this capability is ultra limited right now; in fact, the
only place we display children is in the CLI output. For instance:
+ aws:serverless:Function: (create)
[urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda]
=> urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole
=> urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0
=> urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda
The bit indicating whether a resource is external or not is tracked
in the resulting checkpoint file, along with any of its children.
2017-10-14 21:18:43 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Now "transfer" all input properties into unresolved Promises on res. This way,
|
|
|
|
// this resource will look like it has all its output properties to anyone it is
|
|
|
|
// passed to. However, those promises won't actually resolve until the registerResource
|
|
|
|
// RPC returns
|
|
|
|
const resolvers = transferProperties(res, label, props);
|
|
|
|
|
|
|
|
/** IMPORTANT! We should never await prior to this line, otherwise the Resource will be partly uninitialized. */
|
|
|
|
|
|
|
|
// Before we can proceed, all our dependencies must be finished.
|
|
|
|
const explicitDirectDependencies = new Set(await gatherExplicitDependencies(opts.dependsOn));
|
|
|
|
|
|
|
|
// Serialize out all our props to their final values. In doing so, we'll also collect all
|
|
|
|
// the Resources pointed to by any Dependency objects we encounter, adding them to 'propertyDependencies'.
|
|
|
|
const [serializedProps, propertyToDirectDependencies] = await serializeResourceProperties(label, props, {
|
|
|
|
// To initially scope the use of this new feature, we only keep output values when
|
|
|
|
// remote is true (for multi-lang components).
|
|
|
|
keepOutputValues: remote,
|
|
|
|
});
|
|
|
|
|
|
|
|
// Wait for the parent to complete.
|
|
|
|
// If no parent was provided, parent to the root resource.
|
|
|
|
const parentURN = parent ? await parent.urn.promise() : undefined;
|
|
|
|
|
|
|
|
let importID: ID | undefined;
|
|
|
|
if (custom) {
|
|
|
|
const customOpts = <CustomResourceOptions>opts;
|
|
|
|
importID = customOpts.import;
|
|
|
|
}
|
2023-06-26 06:25:18 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
let providerRef: string | undefined;
|
|
|
|
let sendProvider = custom;
|
|
|
|
if (remote && opts.provider) {
|
|
|
|
// If it's a remote component and a provider was specified, only
|
|
|
|
// send the provider in the request if the provider's package is
|
|
|
|
// the same as the component's package. Otherwise, don't send it
|
|
|
|
// because the user specified `provider: someProvider` as shorthand
|
|
|
|
// for `providers: [someProvider]`.
|
|
|
|
const pkg = pkgFromType(type!);
|
|
|
|
if (pkg && pkg === opts.provider.getPackage()) {
|
|
|
|
sendProvider = true;
|
2021-08-04 20:19:43 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
}
|
|
|
|
if (sendProvider) {
|
|
|
|
providerRef = await ProviderResource.register(opts.provider);
|
|
|
|
}
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 00:50:29 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
const providerRefs: Map<string, string> = new Map<string, string>();
|
|
|
|
if (remote || !custom) {
|
|
|
|
const componentOpts = <ComponentResourceOptions>opts;
|
|
|
|
expandProviders(componentOpts);
|
|
|
|
// the <ProviderResource[]> casts are safe because expandProviders
|
|
|
|
// /always/ leaves providers as an array.
|
|
|
|
if (componentOpts.provider !== undefined) {
|
|
|
|
if (componentOpts.providers === undefined) {
|
|
|
|
// We still want to do the promotion, so we define providers
|
|
|
|
componentOpts.providers = [componentOpts.provider];
|
|
|
|
} else if ((<ProviderResource[]>componentOpts.providers)?.indexOf(componentOpts.provider) !== -1) {
|
|
|
|
const pkg = componentOpts.provider.getPackage();
|
|
|
|
const message = `There is a conflit between the 'provider' field (${pkg}) and a member of the 'providers' map'. `;
|
|
|
|
const deprecationd =
|
|
|
|
"This will become an error in a future version. See https://github.com/pulumi/pulumi/issues/8799 for more details";
|
|
|
|
log.warn(message + deprecationd);
|
|
|
|
} else {
|
|
|
|
(<ProviderResource[]>componentOpts.providers).push(componentOpts.provider);
|
2022-02-04 11:38:34 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
}
|
|
|
|
if (componentOpts.providers) {
|
|
|
|
for (const provider of componentOpts.providers as ProviderResource[]) {
|
|
|
|
const pref = await ProviderResource.register(provider);
|
|
|
|
if (pref) {
|
|
|
|
providerRefs.set(provider.getPackage(), pref);
|
2021-04-19 22:41:53 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
}
|
2021-04-19 22:41:53 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Collect the URNs for explicit/implicit dependencies for the engine so that it can understand
|
|
|
|
// the dependency graph and optimize operations accordingly.
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// The list of all dependencies (implicit or explicit).
|
|
|
|
const allDirectDependencies = new Set<Resource>(explicitDirectDependencies);
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
const exclude = new Set<Resource>([res]);
|
|
|
|
const allDirectDependencyURNs = await getAllTransitivelyReferencedResourceURNs(explicitDirectDependencies, exclude);
|
|
|
|
const propertyToDirectDependencyURNs = new Map<string, Set<URN>>();
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
for (const [propertyName, directDependencies] of propertyToDirectDependencies) {
|
|
|
|
addAll(allDirectDependencies, directDependencies);
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
const urns = await getAllTransitivelyReferencedResourceURNs(directDependencies, exclude);
|
|
|
|
addAll(allDirectDependencyURNs, urns);
|
|
|
|
propertyToDirectDependencyURNs.set(propertyName, urns);
|
|
|
|
}
|
2018-02-21 17:43:17 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
const monitorSupportsStructuredAliases = getStore().supportsAliasSpecs;
|
|
|
|
let computedAliases;
|
|
|
|
if (!monitorSupportsStructuredAliases && parent) {
|
|
|
|
computedAliases = allAliases(opts.aliases || [], name!, type!, parent, parent.__name!);
|
|
|
|
} else {
|
|
|
|
computedAliases = opts.aliases || [];
|
|
|
|
}
|
2022-10-31 18:25:24 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
// Wait for all aliases.
|
|
|
|
const aliases = [];
|
|
|
|
const uniqueAliases = new Set<Alias | URN>();
|
|
|
|
for (const alias of computedAliases || []) {
|
|
|
|
const aliasVal = await output(alias).promise();
|
|
|
|
if (!uniqueAliases.has(aliasVal)) {
|
|
|
|
uniqueAliases.add(aliasVal);
|
|
|
|
aliases.push(aliasVal);
|
2019-06-23 16:49:04 +00:00
|
|
|
}
|
2021-08-04 20:19:43 +00:00
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
|
|
|
|
const deletedWithURN = opts?.deletedWith ? await opts.deletedWith.urn.promise() : undefined;
|
|
|
|
|
|
|
|
return {
|
|
|
|
resolveURN: resolveURN,
|
|
|
|
resolveID: resolveID,
|
|
|
|
resolvers: resolvers,
|
|
|
|
serializedProps: serializedProps,
|
|
|
|
parentURN: parentURN,
|
|
|
|
providerRef: providerRef,
|
|
|
|
providerRefs: providerRefs,
|
|
|
|
allDirectDependencyURNs: allDirectDependencyURNs,
|
|
|
|
propertyToDirectDependencyURNs: propertyToDirectDependencyURNs,
|
|
|
|
aliases: aliases,
|
|
|
|
import: importID,
|
|
|
|
monitorSupportsStructuredAliases,
|
|
|
|
deletedWithURN,
|
|
|
|
};
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
2018-01-25 23:26:39 +00:00
|
|
|
|
2019-02-22 04:18:29 +00:00
|
|
|
function addAll<T>(to: Set<T>, from: Set<T>) {
|
|
|
|
for (const val of from) {
|
|
|
|
to.add(val);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-09-16 01:25:26 +00:00
|
|
|
/** @internal */
|
2023-04-28 22:27:10 +00:00
|
|
|
export async function getAllTransitivelyReferencedResourceURNs(
|
|
|
|
resources: Set<Resource>,
|
|
|
|
exclude: Set<Resource>,
|
|
|
|
): Promise<Set<string>> {
|
2019-03-06 01:06:57 +00:00
|
|
|
// Go through 'resources', but transitively walk through **Component** resources, collecting any
|
|
|
|
// of their child resources. This way, a Component acts as an aggregation really of all the
|
2021-07-16 23:11:34 +00:00
|
|
|
// reachable resources it parents. This walking will stop when it hits custom resources.
|
2019-03-06 01:06:57 +00:00
|
|
|
//
|
2021-07-16 23:11:34 +00:00
|
|
|
// This function also terminates at remote components, whose children are not known to the Node SDK directly.
|
|
|
|
// Remote components will always wait on all of their children, so ensuring we return the remote component
|
|
|
|
// itself here and waiting on it will accomplish waiting on all of it's children regardless of whether they
|
|
|
|
// are returned explicitly here.
|
2019-03-06 01:06:57 +00:00
|
|
|
//
|
2021-07-16 23:11:34 +00:00
|
|
|
// In other words, if we had:
|
|
|
|
//
|
|
|
|
// Comp1
|
|
|
|
// / | \
|
|
|
|
// Cust1 Comp2 Remote1
|
|
|
|
// / \ \
|
|
|
|
// Cust2 Cust3 Comp3
|
|
|
|
// / \
|
|
|
|
// Cust4 Cust5
|
|
|
|
//
|
|
|
|
// Then the transitively reachable resources of Comp1 will be [Cust1, Cust2, Cust3, Remote1].
|
|
|
|
// It will *not* include:
|
|
|
|
// * Cust4 because it is a child of a custom resource
|
2021-11-01 18:10:27 +00:00
|
|
|
// * Comp2 because it is a non-remote component resource
|
2021-07-16 23:11:34 +00:00
|
|
|
// * Comp3 and Cust5 because Comp3 is a child of a remote component resource
|
2019-03-06 01:06:57 +00:00
|
|
|
|
|
|
|
// To do this, first we just get the transitively reachable set of resources (not diving
|
|
|
|
// into custom resources). In the above picture, if we start with 'Comp1', this will be
|
|
|
|
// [Comp1, Cust1, Comp2, Cust2, Cust3]
|
2023-04-28 22:27:10 +00:00
|
|
|
const transitivelyReachableResources = await getTransitivelyReferencedChildResourcesOfComponentResources(
|
|
|
|
resources,
|
|
|
|
exclude,
|
|
|
|
);
|
2019-03-06 01:06:57 +00:00
|
|
|
|
2021-07-16 23:11:34 +00:00
|
|
|
// Then we filter to only include Custom and Remote resources.
|
2023-04-28 22:27:10 +00:00
|
|
|
const transitivelyReachableCustomResources = [...transitivelyReachableResources].filter(
|
|
|
|
(r) => (CustomResource.isInstance(r) || (r as ComponentResource).__remote) && !exclude.has(r),
|
|
|
|
);
|
|
|
|
const promises = transitivelyReachableCustomResources.map((r) => r.urn.promise());
|
2019-03-06 01:06:57 +00:00
|
|
|
const urns = await Promise.all(promises);
|
|
|
|
return new Set<string>(urns);
|
|
|
|
}
|
2019-02-22 04:18:29 +00:00
|
|
|
|
2019-03-06 01:06:57 +00:00
|
|
|
/**
|
|
|
|
* Recursively walk the resources passed in, returning them and all resources reachable from
|
|
|
|
* [Resource.__childResources] through any **Component** resources we encounter.
|
|
|
|
*/
|
[sdk/nodejs] Fix hang due to component children cycles
When a resource depends on a local component resource, rather than setting the component resource itself as a dependency, each of the component's descendants is added as a dependency. This can lead to hangs when cycles are introduced.
For example, consider the following parent/child hierarchy, where `ComponentA` is the parent of `ComponentB` and `ComponentB` is the parent of `CustomC`:
```
ComponentA
|
ComponentB
|
CustomC
```
If `ComponentB` specifies it has a dependency on `ComponentA`, the following takes place as part determining the full set of transitive dependencies:
1. `ComponentA` is a component resource so it isn't added as a dependency, its children are.
2. `ComponentA` has one child: `ComponentB`
3. `ComponentB` is a component resource so it isn't added as a dependency, its children are.
4. `ComponentB` has one child: `CustomC`, a custom resource.
5. Since `CustomC` is a custom resource, it is added to the set of dependencies.
6. We try to await its URN, but we'll never get it because `RegisterResource` hasn't yet been called for it. And we hang waiting.
To address this, skip looking at a component's children if it is the component from which the dependency is being added.
In the case of the example, at step 3 the dependency expansion will stop: we won't look at `ComponentB`'s children because we're adding the dependency from `ComponentB`.
fix
2023-03-27 01:50:24 +00:00
|
|
|
async function getTransitivelyReferencedChildResourcesOfComponentResources(
|
2023-04-28 22:27:10 +00:00
|
|
|
resources: Set<Resource>,
|
|
|
|
exclude: Set<Resource>,
|
|
|
|
) {
|
2019-03-06 01:06:57 +00:00
|
|
|
// Recursively walk the dependent resources through their children, adding them to the result set.
|
|
|
|
const result = new Set<Resource>();
|
[sdk/nodejs] Fix hang due to component children cycles
When a resource depends on a local component resource, rather than setting the component resource itself as a dependency, each of the component's descendants is added as a dependency. This can lead to hangs when cycles are introduced.
For example, consider the following parent/child hierarchy, where `ComponentA` is the parent of `ComponentB` and `ComponentB` is the parent of `CustomC`:
```
ComponentA
|
ComponentB
|
CustomC
```
If `ComponentB` specifies it has a dependency on `ComponentA`, the following takes place as part determining the full set of transitive dependencies:
1. `ComponentA` is a component resource so it isn't added as a dependency, its children are.
2. `ComponentA` has one child: `ComponentB`
3. `ComponentB` is a component resource so it isn't added as a dependency, its children are.
4. `ComponentB` has one child: `CustomC`, a custom resource.
5. Since `CustomC` is a custom resource, it is added to the set of dependencies.
6. We try to await its URN, but we'll never get it because `RegisterResource` hasn't yet been called for it. And we hang waiting.
To address this, skip looking at a component's children if it is the component from which the dependency is being added.
In the case of the example, at step 3 the dependency expansion will stop: we won't look at `ComponentB`'s children because we're adding the dependency from `ComponentB`.
fix
2023-03-27 01:50:24 +00:00
|
|
|
await addTransitivelyReferencedChildResourcesOfComponentResources(resources, exclude, result);
|
2019-02-22 04:18:29 +00:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
[sdk/nodejs] Fix hang due to component children cycles
When a resource depends on a local component resource, rather than setting the component resource itself as a dependency, each of the component's descendants is added as a dependency. This can lead to hangs when cycles are introduced.
For example, consider the following parent/child hierarchy, where `ComponentA` is the parent of `ComponentB` and `ComponentB` is the parent of `CustomC`:
```
ComponentA
|
ComponentB
|
CustomC
```
If `ComponentB` specifies it has a dependency on `ComponentA`, the following takes place as part determining the full set of transitive dependencies:
1. `ComponentA` is a component resource so it isn't added as a dependency, its children are.
2. `ComponentA` has one child: `ComponentB`
3. `ComponentB` is a component resource so it isn't added as a dependency, its children are.
4. `ComponentB` has one child: `CustomC`, a custom resource.
5. Since `CustomC` is a custom resource, it is added to the set of dependencies.
6. We try to await its URN, but we'll never get it because `RegisterResource` hasn't yet been called for it. And we hang waiting.
To address this, skip looking at a component's children if it is the component from which the dependency is being added.
In the case of the example, at step 3 the dependency expansion will stop: we won't look at `ComponentB`'s children because we're adding the dependency from `ComponentB`.
fix
2023-03-27 01:50:24 +00:00
|
|
|
async function addTransitivelyReferencedChildResourcesOfComponentResources(
|
2023-04-28 22:27:10 +00:00
|
|
|
resources: Set<Resource> | undefined,
|
|
|
|
exclude: Set<Resource>,
|
|
|
|
result: Set<Resource>,
|
|
|
|
) {
|
2019-03-06 01:06:57 +00:00
|
|
|
if (resources) {
|
|
|
|
for (const resource of resources) {
|
|
|
|
if (!result.has(resource)) {
|
|
|
|
result.add(resource);
|
|
|
|
|
|
|
|
if (ComponentResource.isInstance(resource)) {
|
[sdk/nodejs] Fix hang due to component children cycles
When a resource depends on a local component resource, rather than setting the component resource itself as a dependency, each of the component's descendants is added as a dependency. This can lead to hangs when cycles are introduced.
For example, consider the following parent/child hierarchy, where `ComponentA` is the parent of `ComponentB` and `ComponentB` is the parent of `CustomC`:
```
ComponentA
|
ComponentB
|
CustomC
```
If `ComponentB` specifies it has a dependency on `ComponentA`, the following takes place as part determining the full set of transitive dependencies:
1. `ComponentA` is a component resource so it isn't added as a dependency, its children are.
2. `ComponentA` has one child: `ComponentB`
3. `ComponentB` is a component resource so it isn't added as a dependency, its children are.
4. `ComponentB` has one child: `CustomC`, a custom resource.
5. Since `CustomC` is a custom resource, it is added to the set of dependencies.
6. We try to await its URN, but we'll never get it because `RegisterResource` hasn't yet been called for it. And we hang waiting.
To address this, skip looking at a component's children if it is the component from which the dependency is being added.
In the case of the example, at step 3 the dependency expansion will stop: we won't look at `ComponentB`'s children because we're adding the dependency from `ComponentB`.
fix
2023-03-27 01:50:24 +00:00
|
|
|
// Skip including children of a resource in the excluded set to avoid depending on
|
|
|
|
// children that haven't been registered yet.
|
|
|
|
if (exclude.has(resource)) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2019-12-17 23:34:30 +00:00
|
|
|
// This await is safe even if __isConstructed is undefined. Ensure that the
|
|
|
|
// resource has completely finished construction. That way all parent/child
|
|
|
|
// relationships will have been setup.
|
|
|
|
await resource.__data;
|
[sdk/nodejs] Fix hang due to component children cycles
When a resource depends on a local component resource, rather than setting the component resource itself as a dependency, each of the component's descendants is added as a dependency. This can lead to hangs when cycles are introduced.
For example, consider the following parent/child hierarchy, where `ComponentA` is the parent of `ComponentB` and `ComponentB` is the parent of `CustomC`:
```
ComponentA
|
ComponentB
|
CustomC
```
If `ComponentB` specifies it has a dependency on `ComponentA`, the following takes place as part determining the full set of transitive dependencies:
1. `ComponentA` is a component resource so it isn't added as a dependency, its children are.
2. `ComponentA` has one child: `ComponentB`
3. `ComponentB` is a component resource so it isn't added as a dependency, its children are.
4. `ComponentB` has one child: `CustomC`, a custom resource.
5. Since `CustomC` is a custom resource, it is added to the set of dependencies.
6. We try to await its URN, but we'll never get it because `RegisterResource` hasn't yet been called for it. And we hang waiting.
To address this, skip looking at a component's children if it is the component from which the dependency is being added.
In the case of the example, at step 3 the dependency expansion will stop: we won't look at `ComponentB`'s children because we're adding the dependency from `ComponentB`.
fix
2023-03-27 01:50:24 +00:00
|
|
|
const children = resource.__childResources;
|
|
|
|
addTransitivelyReferencedChildResourcesOfComponentResources(children, exclude, result);
|
2019-03-06 01:06:57 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-11-19 16:22:55 +00:00
|
|
|
/**
|
2019-02-22 04:18:29 +00:00
|
|
|
* Gathers explicit dependent Resources from a list of Resources (possibly Promises and/or Outputs).
|
2018-11-19 16:22:55 +00:00
|
|
|
*/
|
|
|
|
async function gatherExplicitDependencies(
|
2023-04-28 22:27:10 +00:00
|
|
|
dependsOn: Input<Input<Resource>[]> | Input<Resource> | undefined,
|
|
|
|
): Promise<Resource[]> {
|
2018-11-19 16:22:55 +00:00
|
|
|
if (dependsOn) {
|
|
|
|
if (Array.isArray(dependsOn)) {
|
2019-02-22 04:18:29 +00:00
|
|
|
const dos: Resource[] = [];
|
2018-11-19 16:22:55 +00:00
|
|
|
for (const d of dependsOn) {
|
|
|
|
dos.push(...(await gatherExplicitDependencies(d)));
|
|
|
|
}
|
|
|
|
return dos;
|
|
|
|
} else if (dependsOn instanceof Promise) {
|
|
|
|
return gatherExplicitDependencies(await dependsOn);
|
|
|
|
} else if (Output.isInstance(dependsOn)) {
|
|
|
|
// Recursively gather dependencies, await the promise, and append the output's dependencies.
|
2023-04-28 22:27:10 +00:00
|
|
|
const dos = (dependsOn as Output<Input<Resource>[] | Input<Resource>>).apply((v) =>
|
|
|
|
gatherExplicitDependencies(v),
|
|
|
|
);
|
2018-11-19 16:22:55 +00:00
|
|
|
const urns = await dos.promise();
|
2019-12-18 03:04:09 +00:00
|
|
|
const dosResources = await getAllResources(dos);
|
|
|
|
const implicits = await gatherExplicitDependencies([...dosResources]);
|
2020-05-09 00:57:08 +00:00
|
|
|
return (urns ?? []).concat(implicits);
|
2018-11-19 16:22:55 +00:00
|
|
|
} else {
|
2019-03-29 00:27:51 +00:00
|
|
|
if (!Resource.isInstance(dependsOn)) {
|
|
|
|
throw new Error("'dependsOn' was passed a value that was not a Resource.");
|
|
|
|
}
|
|
|
|
|
|
|
|
return [dependsOn];
|
2018-11-19 16:22:55 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return [];
|
|
|
|
}
|
|
|
|
|
2018-04-05 16:48:09 +00:00
|
|
|
/**
|
|
|
|
* Finishes a resource creation RPC operation by resolving its outputs to the resulting RPC payload.
|
|
|
|
*/
|
2023-04-28 22:27:10 +00:00
|
|
|
async function resolveOutputs(
|
|
|
|
res: Resource,
|
|
|
|
t: string,
|
|
|
|
name: string,
|
|
|
|
props: Inputs,
|
2024-03-02 00:00:57 +00:00
|
|
|
outputs: gstruct.Struct | undefined,
|
2023-04-28 22:27:10 +00:00
|
|
|
deps: Record<string, Resource[]>,
|
|
|
|
resolvers: OutputResolvers,
|
|
|
|
err?: Error,
|
2024-04-22 11:12:45 +00:00
|
|
|
keepUnknowns?: boolean,
|
2023-04-28 22:27:10 +00:00
|
|
|
): Promise<void> {
|
2018-04-05 16:48:09 +00:00
|
|
|
// Produce a combined set of property states, starting with inputs and then applying
|
|
|
|
// outputs. If the same property exists in the inputs and outputs states, the output wins.
|
|
|
|
const allProps: Record<string, any> = {};
|
|
|
|
if (outputs) {
|
2024-04-22 11:12:45 +00:00
|
|
|
Object.assign(allProps, deserializeProperties(outputs, keepUnknowns));
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
2018-02-05 22:44:23 +00:00
|
|
|
|
2018-05-23 21:47:40 +00:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
2019-08-05 19:44:04 +00:00
|
|
|
if (!isDryRun() || isLegacyApplyEnabled()) {
|
|
|
|
for (const key of Object.keys(props)) {
|
|
|
|
if (!allProps.hasOwnProperty(key)) {
|
|
|
|
// input prop the engine didn't give us a final value for. Just use the value passed into the resource
|
|
|
|
// after round-tripping it through serialization. We do the round-tripping primarily s.t. we ensure that
|
|
|
|
// Output values are handled properly w.r.t. unknowns.
|
2021-09-16 01:25:26 +00:00
|
|
|
const inputProp = await serializeProperty(label, props[key], new Set(), { keepOutputValues: false });
|
2019-08-05 19:44:04 +00:00
|
|
|
if (inputProp === undefined) {
|
|
|
|
continue;
|
|
|
|
}
|
2024-04-22 11:12:45 +00:00
|
|
|
allProps[key] = deserializeProperty(inputProp, keepUnknowns);
|
2018-02-05 22:44:23 +00:00
|
|
|
}
|
2018-04-05 16:48:09 +00:00
|
|
|
}
|
|
|
|
}
|
2018-02-05 22:44:23 +00:00
|
|
|
|
2024-04-22 11:12:45 +00:00
|
|
|
resolveProperties(res, resolvers, t, name, allProps, deps, err, keepUnknowns);
|
2017-11-29 19:27:32 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* registerResourceOutputs completes the resource registration, attaching an optional set of computed outputs.
|
|
|
|
*/
|
2018-09-02 19:15:58 +00:00
|
|
|
export function registerResourceOutputs(res: Resource, outputs: Inputs | Promise<Inputs> | Output<Inputs>) {
|
2018-01-25 21:34:21 +00:00
|
|
|
// Now run the operation. Note that we explicitly do not serialize output registration with
|
|
|
|
// respect to other resource operations, as outputs may depend on properties of other resources
|
|
|
|
// that will not resolve until later turns. This would create a circular promise chain that can
|
|
|
|
// never resolve.
|
2017-11-29 19:27:32 +00:00
|
|
|
const opLabel = `monitor.registerResourceOutputs(...)`;
|
2024-04-23 12:57:04 +00:00
|
|
|
const done = rpcKeepAlive();
|
2023-04-28 22:27:10 +00:00
|
|
|
runAsyncResourceOp(
|
|
|
|
opLabel,
|
|
|
|
async () => {
|
|
|
|
// The registration could very well still be taking place, so we will need to wait for its URN.
|
|
|
|
// Additionally, the output properties might have come from other resources, so we must await those too.
|
|
|
|
const urn = await res.urn.promise();
|
|
|
|
const resolved = await serializeProperties(opLabel, { outputs });
|
|
|
|
const outputsObj = gstruct.Struct.fromJavaScript(resolved.outputs);
|
|
|
|
log.debug(
|
|
|
|
`RegisterResourceOutputs RPC prepared: urn=${urn}` +
|
|
|
|
(excessiveDebugOutput ? `, outputs=${JSON.stringify(outputsObj)}` : ``),
|
|
|
|
);
|
|
|
|
|
|
|
|
// Fetch the monitor and make an RPC request.
|
|
|
|
const monitor = getMonitor();
|
|
|
|
if (monitor) {
|
|
|
|
const req = new resproto.RegisterResourceOutputsRequest();
|
|
|
|
req.setUrn(urn);
|
|
|
|
req.setOutputs(outputsObj);
|
|
|
|
|
|
|
|
const label = `monitor.registerResourceOutputs(${urn}, ...)`;
|
|
|
|
await debuggablePromise(
|
|
|
|
new Promise<void>((resolve, reject) =>
|
2023-12-04 15:22:44 +00:00
|
|
|
monitor.registerResourceOutputs(
|
|
|
|
req,
|
|
|
|
(err: grpc.ServiceError | null, innerResponse: gempty.Empty | undefined) => {
|
2023-04-28 22:27:10 +00:00
|
|
|
log.debug(
|
|
|
|
`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
|
|
|
|
`err: ${err}, resp: ${innerResponse}`,
|
|
|
|
);
|
2023-12-04 15:22:44 +00:00
|
|
|
if (err) {
|
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs, just exit.
|
|
|
|
if (err.code === grpc.status.UNAVAILABLE || err.code === grpc.status.CANCELLED) {
|
|
|
|
terminateRpcs();
|
|
|
|
err.message = "Resource monitor is terminating";
|
|
|
|
}
|
|
|
|
|
|
|
|
reject(err);
|
|
|
|
} else {
|
|
|
|
log.debug(
|
|
|
|
`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
|
|
|
|
`err: ${err}, resp: ${innerResponse}`,
|
|
|
|
);
|
|
|
|
resolve();
|
|
|
|
}
|
|
|
|
},
|
|
|
|
),
|
2023-04-28 22:27:10 +00:00
|
|
|
),
|
|
|
|
label,
|
|
|
|
);
|
|
|
|
}
|
2024-04-23 12:57:04 +00:00
|
|
|
done();
|
2023-04-28 22:27:10 +00:00
|
|
|
},
|
|
|
|
false,
|
|
|
|
);
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
}
|
|
|
|
|
2019-04-18 22:25:15 +00:00
|
|
|
function isAny(o: any): o is any {
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* listResourceOutputs returns the resource outputs (if any) for a stack, or an error if the stack
|
|
|
|
* cannot be found. Resources are retrieved from the latest stack snapshot, which may include
|
|
|
|
* ongoing updates.
|
|
|
|
*
|
|
|
|
* @param stackName Name of stack to retrieve resource outputs for. Defaults to the current stack.
|
|
|
|
* @param typeFilter A [type
|
|
|
|
* guard](https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defined-type-guards)
|
|
|
|
* that specifies which resource types to list outputs of.
|
|
|
|
*
|
|
|
|
* @example
|
|
|
|
* const buckets = pulumi.runtime.listResourceOutput(aws.s3.Bucket.isInstance);
|
|
|
|
*/
|
|
|
|
export function listResourceOutputs<U extends Resource>(
|
|
|
|
typeFilter?: (o: any) => o is U,
|
|
|
|
stackName?: string,
|
|
|
|
): query.AsyncQueryable<ResolvedResource<U>> {
|
|
|
|
if (typeFilter === undefined) {
|
|
|
|
typeFilter = isAny;
|
|
|
|
}
|
|
|
|
|
|
|
|
return query
|
|
|
|
.from(
|
|
|
|
invoke("pulumi:pulumi:readStackResourceOutputs", {
|
|
|
|
stackName: stackName || getStack(),
|
2019-08-08 19:11:46 +00:00
|
|
|
}).then<any[]>(({ outputs }) => utils.values(outputs)),
|
2019-04-18 22:25:15 +00:00
|
|
|
)
|
|
|
|
.map<ResolvedResource<U>>(({ type: typ, outputs }) => {
|
2023-04-28 22:27:10 +00:00
|
|
|
return { ...outputs, __pulumiType: typ };
|
|
|
|
})
|
2019-04-18 22:25:15 +00:00
|
|
|
.filter(typeFilter);
|
|
|
|
}
|
|
|
|
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
/**
|
|
|
|
* resourceChain is used to serialize all resource requests. If we don't do this, all resource operations will be
|
|
|
|
* entirely asynchronous, meaning the dataflow graph that results will determine ordering of operations. This
|
|
|
|
* causes problems with some resource providers, so for now we will serialize all of them. The issue
|
|
|
|
* pulumi/pulumi#335 tracks coming up with a long-term solution here.
|
|
|
|
*/
|
|
|
|
let resourceChain: Promise<void> = Promise.resolve();
|
|
|
|
let resourceChainLabel: string | undefined = undefined;
|
|
|
|
|
|
|
|
// runAsyncResourceOp runs an asynchronous resource operation, possibly serializing it as necessary.
|
2017-12-15 00:24:08 +00:00
|
|
|
function runAsyncResourceOp(label: string, callback: () => Promise<void>, serial?: boolean): void {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
// Serialize the invocation if necessary.
|
2017-12-15 00:24:08 +00:00
|
|
|
if (serial === undefined) {
|
|
|
|
serial = serialize();
|
|
|
|
}
|
2023-04-28 22:27:10 +00:00
|
|
|
const resourceOp: Promise<void> = suppressUnhandledGrpcRejections(
|
|
|
|
debuggablePromise(
|
|
|
|
resourceChain.then(async () => {
|
|
|
|
if (serial) {
|
|
|
|
resourceChainLabel = label;
|
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is current`);
|
|
|
|
}
|
|
|
|
return callback();
|
|
|
|
}),
|
|
|
|
label + "-initial",
|
|
|
|
),
|
|
|
|
);
|
2017-09-15 23:38:52 +00:00
|
|
|
|
2024-04-23 12:57:04 +00:00
|
|
|
const finalOp: Promise<void> = debuggablePromise(resourceOp, label + "-final");
|
2017-12-14 23:22:01 +00:00
|
|
|
|
|
|
|
// Set up another promise that propagates the error, if any, so that it triggers unhandled rejection logic.
|
|
|
|
resourceOp.catch((err) => Promise.reject(err));
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 14:40:09 +00:00
|
|
|
|
2017-09-15 23:38:52 +00:00
|
|
|
// If serialization is requested, wait for the prior resource operation to finish before we proceed, serializing
|
|
|
|
// them, and make this the current resource operation so that everybody piles up on it.
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
if (serial) {
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 14:40:09 +00:00
|
|
|
resourceChain = finalOp;
|
2017-10-31 13:52:42 +00:00
|
|
|
if (resourceChainLabel) {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 15:42:14 +00:00
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is behind ${resourceChainLabel}`);
|
2017-10-31 13:52:42 +00:00
|
|
|
}
|
2017-09-15 23:38:52 +00:00
|
|
|
}
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
}
|
2023-06-26 06:25:18 +00:00
|
|
|
|
|
|
|
/**
|
|
|
|
* Extract the pkg from the type token of the form "pkg:module:member".
|
|
|
|
* @internal
|
|
|
|
*/
|
|
|
|
export function pkgFromType(type: string): string | undefined {
|
|
|
|
const parts = type.split(":");
|
|
|
|
if (parts.length === 3) {
|
|
|
|
return parts[0];
|
|
|
|
}
|
|
|
|
return undefined;
|
|
|
|
}
|