2018-05-22 19:43:36 +00:00
|
|
|
// Copyright 2016-2018, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
|
|
|
// This is the entrypoint for running a Node.js program with minimal scaffolding.
|
|
|
|
|
2018-02-26 18:54:56 +00:00
|
|
|
import * as fs from "fs";
|
2017-09-04 15:30:39 +00:00
|
|
|
import * as minimist from "minimist";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
import * as path from "path";
|
2018-04-10 19:03:11 +00:00
|
|
|
import * as util from "util";
|
2017-11-17 23:22:41 +00:00
|
|
|
import * as pulumi from "../../";
|
Improve output formatting
This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
2017-09-23 12:20:11 +00:00
|
|
|
import { RunError } from "../../errors";
|
2017-10-08 19:10:46 +00:00
|
|
|
import * as log from "../../log";
|
2017-09-01 23:00:16 +00:00
|
|
|
import * as runtime from "../../runtime";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2017-09-04 15:30:39 +00:00
|
|
|
function usage(): void {
|
|
|
|
console.error(`usage: RUN <flags> [program] <[arg]...>`);
|
|
|
|
console.error(``);
|
|
|
|
console.error(` where [flags] may include`);
|
2017-10-19 15:26:57 +00:00
|
|
|
console.error(` --project=p set the project name to p`);
|
|
|
|
console.error(` --stack=s set the stack name to s`);
|
2017-09-04 15:30:39 +00:00
|
|
|
console.error(` --config.k=v... set runtime config key k to value v`);
|
2017-09-17 15:10:46 +00:00
|
|
|
console.error(` --parallel=p run up to p resource operations in parallel (default is serial)`);
|
2017-09-04 15:30:39 +00:00
|
|
|
console.error(` --dry-run true to simulate resource changes, but without making them`);
|
|
|
|
console.error(` --pwd=pwd change the working directory before running the program`);
|
2018-01-25 23:26:39 +00:00
|
|
|
console.error(` --monitor=addr [required] the RPC address for a resource monitor to connect to`);
|
2017-09-04 15:30:39 +00:00
|
|
|
console.error(` --engine=addr the RPC address for a resource engine to connect to`);
|
2017-11-09 01:08:51 +00:00
|
|
|
console.error(` --tracing=url a Zipkin-compatible endpoint to send tracing data to`);
|
2017-09-04 15:30:39 +00:00
|
|
|
console.error(``);
|
|
|
|
console.error(` and [program] is a JavaScript program to run in Node.js, and [arg]... optional args to it.`);
|
|
|
|
}
|
|
|
|
|
2018-01-25 23:26:39 +00:00
|
|
|
function printErrorUsageAndExit(message: string): never {
|
|
|
|
console.error(message);
|
|
|
|
usage();
|
|
|
|
return process.exit(-1);
|
|
|
|
}
|
|
|
|
|
2018-02-26 18:54:56 +00:00
|
|
|
/**
|
|
|
|
* Attempts to provide a detailed error message for module load failure if the
|
|
|
|
* module that failed to load is the top-level module.
|
|
|
|
* @param program The name of the program given to `run`, i.e. the top level module
|
|
|
|
* @param error The error that occured. Must be a module load error.
|
|
|
|
*/
|
|
|
|
function reportModuleLoadFailure(program: string, error: Error): never {
|
|
|
|
// error is guaranteed to be a Node module load error. Node emits a very
|
|
|
|
// specific string in its error message for module load errors, which includes
|
|
|
|
// the module it was trying to load.
|
|
|
|
const errorRegex = /Cannot find module '(.*)'/;
|
|
|
|
|
|
|
|
// If there's no match, who knows what this exception is; it's not something
|
|
|
|
// we can provide an intelligent diagnostic for.
|
|
|
|
const moduleNameMatches = errorRegex.exec(error.message);
|
|
|
|
if (moduleNameMatches === null) {
|
|
|
|
throw error;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Is the module that failed to load exactly the one that this script considered to
|
|
|
|
// be the top-level module for this program?
|
|
|
|
//
|
|
|
|
// We are only interested in producing good diagnostics for top-level module loads,
|
|
|
|
// since anything else are probably user code issues.
|
|
|
|
const moduleName = moduleNameMatches[1];
|
|
|
|
if (moduleName !== program) {
|
|
|
|
throw error;
|
|
|
|
}
|
|
|
|
|
|
|
|
console.error(`We failed to locate the entry point for your program: ${program}`);
|
|
|
|
|
|
|
|
// From here on out, we're going to try to inspect the program we're being asked to run
|
|
|
|
// a little to see what sort of details we can glean from it, in the hopes of producing
|
|
|
|
// a better error message.
|
|
|
|
//
|
|
|
|
// The first step of this is trying to slurp up a package.json for this program, if
|
|
|
|
// one exists.
|
|
|
|
const stat = fs.lstatSync(program);
|
|
|
|
let projectRoot: string;
|
|
|
|
if (stat.isDirectory()) {
|
|
|
|
projectRoot = program;
|
|
|
|
} else {
|
|
|
|
projectRoot = path.dirname(program);
|
|
|
|
}
|
|
|
|
|
|
|
|
let packageObject: Record<string, any>;
|
|
|
|
try {
|
|
|
|
const packageJson = path.join(projectRoot, "package.json");
|
|
|
|
packageObject = require(packageJson);
|
|
|
|
} catch {
|
|
|
|
// This is all best-effort so if we can't load the package.json file, that's
|
|
|
|
// fine.
|
|
|
|
return process.exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
console.error("Here's what we think went wrong:");
|
|
|
|
|
|
|
|
// The objective here is to emit the best diagnostic we can, starting from the
|
|
|
|
// most specific to the least specific.
|
|
|
|
const deps = packageObject["dependencies"] || {};
|
|
|
|
const devDeps = packageObject["devDependencies"] || {};
|
|
|
|
const scripts = packageObject["scripts"] || {};
|
|
|
|
const mainProperty = packageObject["main"] || "index.js";
|
|
|
|
|
|
|
|
// Is there a build script associated with this program? It's a little confusing that the
|
|
|
|
// Pulumi CLI doesn't run build scripts before running the program so call that out
|
|
|
|
// explicitly.
|
|
|
|
if ("build" in scripts) {
|
|
|
|
const command = scripts["build"];
|
|
|
|
console.error(` * Your program looks like it has a build script associated with it ('${command}').\n`);
|
|
|
|
console.error("Pulumi does not run build scripts before running your program. " +
|
2018-06-04 22:27:29 +00:00
|
|
|
`Please run '${command}', 'yarn build', or 'npm run build' and try again.`);
|
2018-02-26 18:54:56 +00:00
|
|
|
return process.exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Not all typescript programs have build scripts. If we think it's a typescript program,
|
|
|
|
// tell the user to run tsc.
|
|
|
|
if ("typescript" in deps || "typescript" in devDeps) {
|
|
|
|
console.error(" * Your program looks like a TypeScript program. Have you run 'tsc'?");
|
|
|
|
return process.exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Not all projects are typescript. If there's a main property, check that the file exists.
|
|
|
|
if (mainProperty !== undefined && typeof mainProperty === "string") {
|
|
|
|
const mainFile = path.join(projectRoot, mainProperty);
|
|
|
|
if (!fs.existsSync(mainFile)) {
|
|
|
|
console.error(` * Your program's 'main' file (${mainFile}) does not exist.`);
|
|
|
|
return process.exit(1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
console.error(" * Yowzas, our sincere apologies, we haven't seen this before!");
|
|
|
|
console.error(` Here is the raw exception message we received: ${error.message}`);
|
|
|
|
return process.exit(1);
|
|
|
|
}
|
|
|
|
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
export function main(args: string[]): void {
|
2017-09-04 15:30:39 +00:00
|
|
|
// See usage above for the intended usage of this program, including flags and required args.
|
2017-10-10 21:50:55 +00:00
|
|
|
const config: {[key: string]: string} = {};
|
|
|
|
const argv: minimist.ParsedArgs = minimist(args, {
|
2017-09-17 15:10:46 +00:00
|
|
|
boolean: [ "dry-run" ],
|
2017-11-09 01:08:51 +00:00
|
|
|
string: [ "project", "stack", "parallel", "pwd", "monitor", "engine", "tracing" ],
|
2017-09-04 15:30:39 +00:00
|
|
|
unknown: (arg: string) => {
|
2017-10-12 01:41:52 +00:00
|
|
|
if (arg.indexOf("-") === 0) {
|
2018-01-25 23:26:39 +00:00
|
|
|
return printErrorUsageAndExit(`error: Unrecognized flag ${arg}`);
|
2017-09-04 15:30:39 +00:00
|
|
|
}
|
|
|
|
return true;
|
|
|
|
},
|
|
|
|
stopEarly: true,
|
|
|
|
});
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
2018-03-02 19:36:07 +00:00
|
|
|
// Load configuration passed from the language plugin
|
2018-04-13 18:26:01 +00:00
|
|
|
runtime.ensureConfig();
|
2017-08-27 18:52:04 +00:00
|
|
|
|
2017-10-19 15:26:57 +00:00
|
|
|
// If there is a --project=p, and/or a --stack=s, use them in the options.
|
|
|
|
const project: string | undefined = argv["project"];
|
|
|
|
const stack: string | undefined = argv["stack"];
|
|
|
|
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
// If there is a --pwd directive, switch directories.
|
2017-10-10 21:50:55 +00:00
|
|
|
const pwd: string | undefined = argv["pwd"];
|
2017-09-04 15:30:39 +00:00
|
|
|
if (pwd) {
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
process.chdir(pwd);
|
|
|
|
}
|
|
|
|
|
2017-09-17 15:10:46 +00:00
|
|
|
// If resource parallelism was requested, turn it on.
|
|
|
|
let parallel: number | undefined;
|
|
|
|
if (argv["parallel"]) {
|
|
|
|
parallel = parseInt(argv["parallel"], 10);
|
|
|
|
if (isNaN(parallel)) {
|
2018-01-25 23:26:39 +00:00
|
|
|
return printErrorUsageAndExit(
|
|
|
|
`error: --parallel flag must specify a number: ${argv["parallel"]} is not a number`);
|
2017-09-17 15:10:46 +00:00
|
|
|
}
|
|
|
|
}
|
2017-09-15 23:38:52 +00:00
|
|
|
|
2017-09-04 15:30:39 +00:00
|
|
|
// If ther is a --dry-run directive, flip the switch. This controls whether we are planning vs. really doing it.
|
2017-10-10 21:50:55 +00:00
|
|
|
const dryRun: boolean = !!(argv["dry-run"]);
|
2017-09-04 15:30:39 +00:00
|
|
|
|
2017-09-08 22:11:09 +00:00
|
|
|
// If there is a monitor argument, connect to it.
|
2018-01-25 23:26:39 +00:00
|
|
|
const monitorAddr = argv["monitor"];
|
|
|
|
if (!monitorAddr) {
|
|
|
|
return printErrorUsageAndExit(`error: --monitor=addr must be provided.`);
|
2017-09-04 15:30:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
// If there is an engine argument, connect to it too.
|
2017-10-10 21:50:55 +00:00
|
|
|
const engineAddr: string | undefined = argv["engine"];
|
2017-09-04 15:30:39 +00:00
|
|
|
|
|
|
|
// Now configure the runtime and get it ready to run the program.
|
2018-04-07 15:02:59 +00:00
|
|
|
runtime.setOptions({
|
2017-10-19 15:26:57 +00:00
|
|
|
project: project,
|
|
|
|
stack: stack,
|
2017-09-15 23:38:52 +00:00
|
|
|
dryRun: dryRun,
|
2017-09-17 15:10:46 +00:00
|
|
|
parallel: parallel,
|
2018-04-07 15:02:59 +00:00
|
|
|
monitorAddr: monitorAddr,
|
|
|
|
engineAddr: engineAddr,
|
2017-09-15 23:38:52 +00:00
|
|
|
});
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
|
|
|
|
// Pluck out the program and arguments.
|
2017-09-04 15:30:39 +00:00
|
|
|
if (argv._.length === 0) {
|
2018-01-25 23:26:39 +00:00
|
|
|
return printErrorUsageAndExit("error: Missing program to execute");
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
}
|
2017-09-04 15:30:39 +00:00
|
|
|
let program: string = argv._[0];
|
2017-11-16 15:49:07 +00:00
|
|
|
if (program.indexOf("/") !== 0) {
|
|
|
|
// If this isn't an absolute path, make it relative to the working directory.
|
|
|
|
program = path.join(process.cwd(), program);
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
}
|
|
|
|
|
2017-09-04 15:30:39 +00:00
|
|
|
// Now fake out the process-wide argv, to make the program think it was run normally.
|
2017-10-10 21:50:55 +00:00
|
|
|
const programArgs: string[] = argv._.slice(1);
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
process.argv = [ process.argv[0], process.argv[1], ...programArgs ];
|
|
|
|
|
2017-12-14 01:24:47 +00:00
|
|
|
// Set up the process uncaught exception, unhandled rejection, and program exit handlers.
|
2017-10-30 18:48:54 +00:00
|
|
|
let uncaught: Error | undefined;
|
2017-12-14 01:24:47 +00:00
|
|
|
const uncaughtHandler = (err: Error) => {
|
2017-10-29 00:05:05 +00:00
|
|
|
// First, log the error.
|
Don't use `instanceof` for RTTI
This change moves us away from using JavaScript RTTI, by way of
`instanceof`, for built-in Pulumi types. If we use `instanceof`,
then the same logical type loaded from separate copies of the
SDK package -- as will happen in SxS scenarios -- are considered
different. This isn't actually what we want. The solution is
simple: implement our own quasi-RTTI solution, using __pulumi*
properties and manual as* and is* functions. Note that we could
have skipped the as* and is* functions, but I found that they led
to slightly easier to read code.
There is one strange thing in here, which I spoke to
@CyrusNajmabadi about: SerializedOutput<T>, because it implements
Output<T> as an _interface_, did not previously masquerade as an
actual Output<T>. In other words, `instanceof` would have returned
false, and indeed a few important properties (like promise) are
missing. This change preserves that behavior, although I'll admit
that this is slightly odd. I suspect we'll want to revisit this as
part of https://github.com/pulumi/pulumi/issues/1074.
Fixes https://github.com/pulumi/pulumi/issues/1203.
2018-04-16 21:03:37 +00:00
|
|
|
if (RunError.isInstance(err)) {
|
Improve output formatting
This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
2017-09-23 12:20:11 +00:00
|
|
|
// For errors that are subtypes of RunError, we will print the message without hitting the unhandled error
|
|
|
|
// logic, which will dump all sorts of verbose spew like the origin source and stack trace.
|
2017-10-08 19:10:46 +00:00
|
|
|
log.error(err.message);
|
Improve output formatting
This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
2017-09-23 12:20:11 +00:00
|
|
|
}
|
|
|
|
else {
|
2017-10-30 18:48:54 +00:00
|
|
|
log.error(`Running program '${program}' failed with an unhandled exception:`);
|
2018-04-10 19:03:11 +00:00
|
|
|
log.error(util.format(err));
|
Improve output formatting
This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
2017-09-23 12:20:11 +00:00
|
|
|
}
|
2017-10-30 18:48:54 +00:00
|
|
|
|
|
|
|
// Remember that we failed with an error. Don't quit just yet so we have a chance to drain the message loop.
|
|
|
|
uncaught = err;
|
2017-12-14 01:24:47 +00:00
|
|
|
};
|
|
|
|
process.on("uncaughtException", uncaughtHandler);
|
|
|
|
process.on("unhandledRejection", uncaughtHandler);
|
2017-10-26 18:30:25 +00:00
|
|
|
|
2017-10-30 18:48:54 +00:00
|
|
|
process.on("exit", (code: number) => {
|
|
|
|
runtime.disconnectSync();
|
|
|
|
|
|
|
|
// If we don't already have an exit code, and we had an unhandled error, exit with a non-success.
|
|
|
|
if (code === 0 && uncaught) {
|
|
|
|
process.exit(1);
|
|
|
|
}
|
|
|
|
});
|
2017-10-26 18:30:25 +00:00
|
|
|
|
2017-11-17 23:22:41 +00:00
|
|
|
// Construct a `Stack` resource to represent the outputs of the program.
|
2017-11-20 18:08:59 +00:00
|
|
|
runtime.runInPulumiStack(() => {
|
2017-11-17 23:22:41 +00:00
|
|
|
// We run the program inside this context so that it adopts all resources.
|
|
|
|
//
|
|
|
|
// IDEA: This will miss any resources created on other turns of the event loop. I think that's a fundamental
|
|
|
|
// problem with the current Component design though - not sure what else we could do here.
|
2017-11-20 18:08:59 +00:00
|
|
|
//
|
|
|
|
// Now go ahead and execute the code. The process will remain alive until the message loop empties.
|
|
|
|
log.debug(`Running program '${program}' in pwd '${process.cwd()}' w/ args: ${programArgs}`);
|
2018-02-26 18:54:56 +00:00
|
|
|
try {
|
|
|
|
return require(program);
|
|
|
|
} catch (e) {
|
|
|
|
// User JavaScript can throw anything, so if it's not an Error it's definitely
|
|
|
|
// not something we want to catch up here.
|
|
|
|
if (!(e instanceof Error)) {
|
|
|
|
throw e;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Give a better error message, if we can.
|
|
|
|
const errorCode = (<any>e).code;
|
|
|
|
if (errorCode === "MODULE_NOT_FOUND") {
|
|
|
|
reportModuleLoadFailure(program, e);
|
|
|
|
}
|
|
|
|
|
|
|
|
throw e;
|
|
|
|
}
|
2017-11-20 18:08:59 +00:00
|
|
|
});
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 19:07:54 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
main(process.argv.slice(2));
|