pulumi/sdk/go/common/resource/plugin/langruntime_plugin.go

607 lines
19 KiB
Go
Raw Permalink Normal View History

2018-05-22 19:43:36 +00:00
// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package plugin
import (
"context"
"fmt"
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
"io"
"os"
"path/filepath"
"strings"
"github.com/blang/semver"
"github.com/hashicorp/hcl/v2"
2022-11-01 15:15:09 +00:00
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
"google.golang.org/protobuf/types/known/emptypb"
"github.com/pulumi/pulumi/sdk/v3/go/common/apitype"
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/diag/colors"
"github.com/pulumi/pulumi/sdk/v3/go/common/slice"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/cmdutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/logging"
2022-11-01 15:15:09 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/util/rpcutil"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/rpcutil/rpcerror"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
pulumirpc "github.com/pulumi/pulumi/sdk/v3/proto/go"
)
// langhost reflects a language host plugin, loaded dynamically for a single language/runtime pair.
type langhost struct {
ctx *Context
runtime string
plug *plugin
client pulumirpc.LanguageRuntimeClient
}
// NewLanguageRuntime binds to a language's runtime plugin and then creates a gRPC connection to it. If the
// plugin could not be found, or an error occurs while creating the child process, an error is returned.
func NewLanguageRuntime(host Host, ctx *Context, runtime, workingDirectory string, info ProgramInfo,
all: Reformat with gofumpt Per team discussion, switching to gofumpt. [gofumpt][1] is an alternative, stricter alternative to gofmt. It addresses other stylistic concerns that gofmt doesn't yet cover. [1]: https://github.com/mvdan/gofumpt See the full list of [Added rules][2], but it includes: - Dropping empty lines around function bodies - Dropping unnecessary variable grouping when there's only one variable - Ensuring an empty line between multi-line functions - simplification (`-s` in gofmt) is always enabled - Ensuring multi-line function signatures end with `) {` on a separate line. [2]: https://github.com/mvdan/gofumpt#Added-rules gofumpt is stricter, but there's no lock-in. All gofumpt output is valid gofmt output, so if we decide we don't like it, it's easy to switch back without any code changes. gofumpt support is built into the tooling we use for development so this won't change development workflows. - golangci-lint includes a gofumpt check (enabled in this PR) - gopls, the LSP for Go, includes a gofumpt option (see [installation instrutions][3]) [3]: https://github.com/mvdan/gofumpt#installation This change was generated by running: ```bash gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error) ``` The following files were manually tweaked afterwards: - pkg/cmd/pulumi/stack_change_secrets_provider.go: one of the lines overflowed and had comments in an inconvenient place - pkg/cmd/pulumi/destroy.go: `var x T = y` where `T` wasn't necessary - pkg/cmd/pulumi/policy_new.go: long line because of error message - pkg/backend/snapshot_test.go: long line trying to assign three variables in the same assignment I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 16:36:39 +00:00
) (LanguageRuntime, error) {
path, err := workspace.GetPluginPath(ctx.Diag,
apitype.LanguagePlugin, strings.ReplaceAll(runtime, tokens.QNameDelimiter, "_"), nil, host.GetProjectPlugins())
if err != nil {
return nil, err
}
contract.Assertf(path != "", "unexpected empty path for language plugin %s", runtime)
args, err := buildArgsForNewPlugin(host, info.RootDirectory(), info.Options())
if err != nil {
return nil, err
}
plug, err := newPlugin(ctx, workingDirectory, path, runtime,
apitype.LanguagePlugin, args, nil /*env*/, langRuntimePluginDialOptions(ctx, runtime))
if err != nil {
return nil, err
}
contract.Assertf(plug != nil, "unexpected nil language plugin for %s", runtime)
return &langhost{
ctx: ctx,
runtime: runtime,
plug: plug,
client: pulumirpc.NewLanguageRuntimeClient(plug.Conn),
}, nil
}
2022-11-01 15:15:09 +00:00
func langRuntimePluginDialOptions(ctx *Context, runtime string) []grpc.DialOption {
dialOpts := append(
rpcutil.OpenTracingInterceptorDialOptions(),
grpc.WithTransportCredentials(insecure.NewCredentials()),
2022-11-01 15:15:09 +00:00
rpcutil.GrpcChannelOptions(),
)
if ctx.DialOptions != nil {
metadata := map[string]interface{}{
"mode": "client",
"kind": "language",
}
if runtime != "" {
metadata["runtime"] = runtime
}
dialOpts = append(dialOpts, ctx.DialOptions(metadata)...)
}
return dialOpts
}
func buildArgsForNewPlugin(host Host, root string, options map[string]interface{}) ([]string, error) {
root, err := filepath.Abs(root)
if err != nil {
return nil, err
}
args := slice.Prealloc[string](len(options))
for k, v := range options {
args = append(args, fmt.Sprintf("-%s=%v", k, v))
}
Enable perfsprint linter (#14813) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Prompted by a comment in another review: https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945 This lints that we don't use `fmt.Errorf` when `errors.New` will suffice, it also covers a load of other cases where `Sprintf` is sub-optimal. Most of these edits were made by running `perfsprint --fix`. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-12 12:19:42 +00:00
args = append(args, "-root="+filepath.Clean(root))
// NOTE: positional argument for the server addresss must come last
args = append(args, host.ServerAddr())
return args, nil
}
func NewLanguageRuntimeClient(ctx *Context, runtime string, client pulumirpc.LanguageRuntimeClient) LanguageRuntime {
return &langhost{
ctx: ctx,
runtime: runtime,
client: client,
}
}
// GetRequiredPlugins computes the complete set of anticipated plugins required by a program.
func (h *langhost) GetRequiredPlugins(info ProgramInfo) ([]workspace.PluginSpec, error) {
logging.V(7).Infof("langhost[%v].GetRequiredPlugins(%s) executing",
h.runtime, info)
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
minfo, err := info.Marshal()
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
if err != nil {
return nil, err
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
}
resp, err := h.client.GetRequiredPlugins(h.ctx.Request(), &pulumirpc.GetRequiredPluginsRequest{
Clean up project usage (#15154) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> The project field on `GetProgramDependenciesRequest` and `GetRequiredPluginsRequest` was marked deprecated at the start of December. None of the language runtimes are using this, so this cleans up the engine side code so we don't need to thread a `*workspace.Project` down to the plugin layer to fill in these fields anymore. I haven't fully removed them from the Protobuf structs yet, we probably could but just to give a little more time for people to get a clear usage error if still using it. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-16 17:06:14 +00:00
Project: "deprecated",
Pwd: info.ProgramDirectory(),
Program: info.EntryPoint(),
Info: minfo,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].GetRequiredPlugins(%s) failed: err=%v",
h.runtime, info, rpcError)
// It's possible this is just an older language host, prior to the emergence of the GetRequiredPlugins
// method. In such cases, we will silently error (with the above log left behind).
if rpcError.Code() == codes.Unimplemented {
return nil, nil
}
return nil, rpcError
}
results := slice.Prealloc[workspace.PluginSpec](len(resp.GetPlugins()))
for _, info := range resp.GetPlugins() {
var version *semver.Version
if v := info.GetVersion(); v != "" {
sv, err := semver.ParseTolerant(v)
if err != nil {
return nil, fmt.Errorf("illegal semver returned by language host: %s@%s: %w", info.GetName(), v, err)
}
version = &sv
}
if !apitype.IsPluginKind(info.Kind) {
return nil, fmt.Errorf("unrecognized plugin kind: %s", info.Kind)
}
results = append(results, workspace.PluginSpec{
Allow language plugins to return plugin checksums (#13776) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Also fix an issue where if a platform was missing a checksum it would error with "invalid checksum, expected , actual 01234". None of the language runtimes yet return anything for this, but it's a simple plumbing to expose it for the future. We'll _probably_ start adding checksums to the pulumi-plugin.json files, and then GetRequiredPlugins can simply return that data. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-08-25 15:26:25 +00:00
Name: info.Name,
Kind: apitype.PluginKind(info.Kind),
Version: version,
Allow language plugins to return plugin checksums (#13776) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Also fix an issue where if a platform was missing a checksum it would error with "invalid checksum, expected , actual 01234". None of the language runtimes yet return anything for this, but it's a simple plumbing to expose it for the future. We'll _probably_ start adding checksums to the pulumi-plugin.json files, and then GetRequiredPlugins can simply return that data. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-08-25 15:26:25 +00:00
PluginDownloadURL: info.Server,
Checksums: info.Checksums,
})
}
logging.V(7).Infof("langhost[%v].GetRequiredPlugins(%s) success: #versions=%d",
h.runtime, info, len(results))
return results, nil
}
// Run executes a program in the language runtime for planning or deployment purposes. If
// info.DryRun is true, the code must not assume that side-effects or final values resulting from
// resource deployments are actually available. If it is false, on the other hand, a real
// deployment is occurring and it may safely depend on these.
func (h *langhost) Run(info RunInfo) (string, bool, error) {
logging.V(7).Infof("langhost[%v].Run(pwd=%v,%s,#args=%v,proj=%s,stack=%v,#config=%v,dryrun=%v) executing",
h.runtime, info.Pwd, info.Info, len(info.Args), info.Project, info.Stack, len(info.Config), info.DryRun)
config := make(map[string]string, len(info.Config))
for k, v := range info.Config {
config[k.String()] = v
}
configSecretKeys := make([]string, len(info.ConfigSecretKeys))
for i, k := range info.ConfigSecretKeys {
configSecretKeys[i] = k.String()
}
pass through property values to RunRequest (#14273) # Description This is an alternative to https://github.com/pulumi/pulumi/pull/14244. Instead of adding type information to the run request, pass the config through as property values. Property values are properly encoded on the wire, and can be unmarshalled on the other end including type information, so this should be a more future proof way to go forward. Eventually we'll want to parse the config directly into property values, but that can be left for the future, as it's a bigger change. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-20 10:44:16 +00:00
configPropertyMap, err := MarshalProperties(info.ConfigPropertyMap,
MarshalOptions{RejectUnknowns: true, KeepSecrets: true, SkipInternalKeys: true})
if err != nil {
return "", false, err
}
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
minfo, err := info.Info.Marshal()
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
if err != nil {
return "", false, err
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
}
resp, err := h.client.Run(h.ctx.Request(), &pulumirpc.RunRequest{
pass through property values to RunRequest (#14273) # Description This is an alternative to https://github.com/pulumi/pulumi/pull/14244. Instead of adding type information to the run request, pass the config through as property values. Property values are properly encoded on the wire, and can be unmarshalled on the other end including type information, so this should be a more future proof way to go forward. Eventually we'll want to parse the config directly into property values, but that can be left for the future, as it's a bigger change. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-20 10:44:16 +00:00
MonitorAddress: info.MonitorAddress,
Pwd: info.Pwd,
Program: info.Info.EntryPoint(),
pass through property values to RunRequest (#14273) # Description This is an alternative to https://github.com/pulumi/pulumi/pull/14244. Instead of adding type information to the run request, pass the config through as property values. Property values are properly encoded on the wire, and can be unmarshalled on the other end including type information, so this should be a more future proof way to go forward. Eventually we'll want to parse the config directly into property values, but that can be left for the future, as it's a bigger change. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-20 10:44:16 +00:00
Args: info.Args,
Project: info.Project,
Stack: info.Stack,
Config: config,
ConfigSecretKeys: configSecretKeys,
ConfigPropertyMap: configPropertyMap,
DryRun: info.DryRun,
QueryMode: info.QueryMode,
Parallel: info.Parallel,
pass through property values to RunRequest (#14273) # Description This is an alternative to https://github.com/pulumi/pulumi/pull/14244. Instead of adding type information to the run request, pass the config through as property values. Property values are properly encoded on the wire, and can be unmarshalled on the other end including type information, so this should be a more future proof way to go forward. Eventually we'll want to parse the config directly into property values, but that can be left for the future, as it's a bigger change. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-20 10:44:16 +00:00
Organization: info.Organization,
Info: minfo,
LoaderTarget: info.LoaderAddress,
AttachDebugger: info.AttachDebugger,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].Run(pwd=%v,%s,...,dryrun=%v) failed: err=%v",
h.runtime, info.Pwd, info.Info, info.DryRun, rpcError)
return "", false, rpcError
}
progerr := resp.GetError()
bail := resp.GetBail()
logging.V(7).Infof("langhost[%v].Run(pwd=%v,%s,...,dryrun=%v) success: progerr=%v, bail=%v",
h.runtime, info.Pwd, info.Info, info.DryRun, progerr, bail)
return progerr, bail, nil
}
// GetPluginInfo returns this plugin's information.
func (h *langhost) GetPluginInfo() (workspace.PluginInfo, error) {
logging.V(7).Infof("langhost[%v].GetPluginInfo() executing", h.runtime)
2020-07-24 08:31:54 +00:00
plugInfo := workspace.PluginInfo{
Name: h.runtime,
Kind: apitype.LanguagePlugin,
}
if h.plug != nil {
plugInfo.Path = h.plug.Bin
}
2020-07-24 08:31:54 +00:00
Remove deprecated Protobufs imports (#15158) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> github.com/golang/protobuf is marked deprecated and I was getting increasingly triggered by the inconsistency of importing the `Empty` type from "github.com/golang/protobuf/ptypes/empty" or "google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty" or "emptypb". Similar for the struct type. So this replaces all the Protobufs imports with ones from "google.golang.org/protobuf", normalises the import name to always just be the module name (emptypb), and adds the depguard linter to ensure we don't use the deprecated package anymore. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-01-17 09:35:20 +00:00
resp, err := h.client.GetPluginInfo(h.ctx.Request(), &emptypb.Empty{})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].GetPluginInfo() failed: err=%v", h.runtime, rpcError)
return workspace.PluginInfo{}, rpcError
2020-07-24 08:31:54 +00:00
}
vers := resp.Version
2020-07-24 08:31:54 +00:00
if vers != "" {
sv, err := semver.ParseTolerant(vers)
if err != nil {
return workspace.PluginInfo{}, err
}
2020-07-24 08:31:54 +00:00
plugInfo.Version = &sv
}
2020-07-24 08:31:54 +00:00
return plugInfo, nil
}
// Close tears down the underlying plugin RPC connection and process.
func (h *langhost) Close() error {
2020-07-24 08:31:54 +00:00
if h.plug != nil {
return h.plug.Close()
}
return nil
}
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
func (h *langhost) InstallDependencies(request InstallDependenciesRequest) error {
logging.V(7).Infof("langhost[%v].InstallDependencies(%s) executing",
h.runtime, request)
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
minfo, err := request.Info.Marshal()
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
if err != nil {
return err
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
}
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
resp, err := h.client.InstallDependencies(h.ctx.Request(), &pulumirpc.InstallDependenciesRequest{
Directory: request.Info.ProgramDirectory(),
IsTerminal: cmdutil.GetGlobalColorization() != colors.Never,
Info: minfo,
UseLanguageVersionTools: request.UseLanguageVersionTools,
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].InstallDependencies(%s) failed: err=%v",
h.runtime, request, rpcError)
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
// It's possible this is just an older language host, prior to the emergence of the InstallDependencies
// method. In such cases, we will silently error (with the above log left behind).
if rpcError.Code() == codes.Unimplemented {
return nil
}
return rpcError
}
for {
output, err := resp.Recv()
if err != nil {
if err == io.EOF {
break
}
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].InstallDependencies(%s) failed: err=%v",
h.runtime, request, rpcError)
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
return rpcError
}
if len(output.Stdout) != 0 {
os.Stdout.Write(output.Stdout)
}
if len(output.Stderr) != 0 {
os.Stderr.Write(output.Stderr)
}
}
logging.V(7).Infof("langhost[%v].InstallDependencies(%s) success",
h.runtime, request)
Move InstallDependencies to the language plugin (#9294) * Move InstallDependencies to the language plugin This changes `pulumi new` and `pulumi up <template>` to invoke the language plugin to install dependencies, rather than having the code to install dependencies hardcoded into the cli itself. This does not change the way policypacks or plugin dependencies are installed. In theory we can make pretty much the same change to just invoke the language plugin, but baby steps we don't need to make that change at the same time as this. We used to feed the result of these install commands (dotnet build, npm install, etc) directly through to the CLI stdout/stderr. To mostly maintain that behaviour the InstallDependencies gRCP method streams back bytes to be written to stdout/stderr, those bytes are either read from pipes or a pty that we run the install commands with. The use of a pty is controlled by the global colorisation option in the cli. An alternative designs was to use the Engine interface to Log the results of install commands. This renders very differently to just writing directly to the standard outputs and I don't think would support control codes so well. The design as is means that `npm install` for example is still able to display a progress bar and colors even though we're running it in a separate process and streaming its output back via gRPC. The only "oddity" I feel that's fallen out of this work is that InstallDependencies for python used to overwrite the virtualenv runtime option. It looks like this was because our templates don't bother setting that. Because InstallDependencies doesn't have the project file, and at any rate will be used for policy pack projects in the future, I've moved that logic into `pulumi new` when it mutates the other project file settings. I think we should at some point cleanup so the templates correctly indicate to use a venv, or maybe change python to assume a virtual env of "venv" if none is given? * Just warn if pty fails to open * Add tests and return real tty files * Add to CHANGELOG * lint * format * Test strings * Log pty opening for trace debugging * s/Hack/Workaround * Use termios * Tweak terminal test * lint * Fix windows build
2022-04-03 14:54:59 +00:00
return nil
}
func (h *langhost) RuntimeOptionsPrompts(info ProgramInfo) ([]RuntimeOptionPrompt, error) {
logging.V(7).Infof("langhost[%v].RuntimeOptionsPrompts() executing", h.runtime)
minfo, err := info.Marshal()
if err != nil {
return []RuntimeOptionPrompt{}, err
}
resp, err := h.client.RuntimeOptionsPrompts(h.ctx.Request(), &pulumirpc.RuntimeOptionsRequest{
Info: minfo,
})
if err != nil {
if status.Code(err) == codes.Unimplemented {
logging.V(7).Infof("langhost[%v].RuntimeOptionsPrompts() not implemented, returning no prompts", h.runtime)
return []RuntimeOptionPrompt{}, nil
}
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].RuntimeOptionsPrompts() failed: err=%v", h.runtime, rpcError)
return []RuntimeOptionPrompt{}, rpcError
}
prompts := []RuntimeOptionPrompt{}
for _, prompt := range resp.Prompts {
newPrompt, err := UnmarshallRuntimeOptionPrompt(prompt)
if err != nil {
return []RuntimeOptionPrompt{}, err
}
prompts = append(prompts, newPrompt)
}
logging.V(7).Infof("langhost[%v].RuntimeOptionsPrompts() success", h.runtime)
return prompts, nil
}
func (h *langhost) About(info ProgramInfo) (AboutInfo, error) {
logging.V(7).Infof("langhost[%v].About() executing", h.runtime)
minfo, err := info.Marshal()
if err != nil {
return AboutInfo{}, err
}
resp, err := h.client.About(h.ctx.Request(), &pulumirpc.AboutRequest{
Info: minfo,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].About() failed: err=%v", h.runtime, rpcError)
return AboutInfo{}, rpcError
}
result := AboutInfo{
Executable: resp.Executable,
Version: resp.Version,
Metadata: resp.Metadata,
}
logging.V(7).Infof("langhost[%v].About() success", h.runtime)
return result, nil
}
func (h *langhost) GetProgramDependencies(info ProgramInfo, transitiveDependencies bool) ([]DependencyInfo, error) {
prefix := fmt.Sprintf("langhost[%v].GetProgramDependencies(%s)", h.runtime, info.String())
minfo, err := info.Marshal()
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
if err != nil {
return nil, err
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
}
logging.V(7).Infof("%s executing", prefix)
resp, err := h.client.GetProgramDependencies(h.ctx.Request(), &pulumirpc.GetProgramDependenciesRequest{
TransitiveDependencies: transitiveDependencies,
Info: minfo,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", prefix, rpcError)
return nil, rpcError
}
results := slice.Prealloc[DependencyInfo](len(resp.GetDependencies()))
for _, dep := range resp.GetDependencies() {
results = append(results, DependencyInfo{
Name: dep.Name,
Program dependencies aren't necessarily semantic versions (#15409) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This weakens the contract of GetProgramDependencies that the versions returned are just "strings". The format of those strings is defined by the language plugin. This is especially useful for python which uses PEP440 not semantic versions for it's pip dependencies. There wasn't anywhere in the system that took a strong dependency on these being semvers. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-02-09 22:18:42 +00:00
Version: dep.Version,
})
}
logging.V(7).Infof("%s success: #versions=%d", prefix, len(results))
return results, nil
}
func (h *langhost) RunPlugin(info RunPluginInfo) (io.Reader, io.Reader, context.CancelFunc, error) {
logging.V(7).Infof("langhost[%v].RunPlugin(%s) executing",
h.runtime, info.Info.String())
minfo, err := info.Info.Marshal()
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
if err != nil {
return nil, nil, nil, err
Pass root and main info to language host methods (#14654) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is two changes rolled together in a way. Firstly passing some of the data that we pass on language runtime startup to also pass it to Run/GetRequiredPlugins/etc. This is needed for matrix testing, as we only get to start the language runtime up once for that but want to execute multiple programs with it. I feel it's also a little more consistent as we use the language runtimes in other contexts (codegen) where there isn't really a root directory, and aren't any options (and if we did do options the options for codegen are not going to be the same as for execution). It also means we can reuse a language host for shimless and substack programs, as before they heavily relied on their current working directory to calculate paths, and obviosly could only take one set of options at startup. Imagine a shimless python package + a python root program, that would have needed two startups of the python language host to deal with, this unblocks it so we can make the engine smarter and only use one. Secondly renaming some of the fields we pass to Run/GetRequiredPlugins/etc today. `Pwd` and `Program` were not very descriptive and had pretty non-obvious documentation: ``` string pwd = 3; // the program's working directory. string program = 4; // the path to the program to execute. ``` `pwd` will remain, although probably rename it to `working_directory` at some point, because while today we always start programs up with the working directory equal to the program directory that definitely is going to change in the future (at least for MLCs and substack programs). But the name `pwd` doesn't make it clear that this was intended to be the working directory _and_ the directory which contains the program. `program` was in fact nearly always ".", and if it wasn't that it was just a filename. The engine never sent a path for `program` (although we did have some unit tests to check how that worked for the nodejs and python hosts). These are now replaced by a new structure with (I think) more clearly named and documented fields (see ProgramInfo in langauge.proto). The engine still sends the old data for now, we need to update dotnet/yaml/java before we break the old interface and give Virtus Labs a chance to update [besom](https://github.com/VirtusLab/besom). ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-10 17:30:51 +00:00
}
ctx, kill := context.WithCancel(h.ctx.Request())
resp, err := h.client.RunPlugin(ctx, &pulumirpc.RunPluginRequest{
Pwd: minfo.GetProgramDirectory(),
Args: info.Args,
Env: info.Env,
Info: minfo,
})
if err != nil {
// If there was an error starting the plugin kill the context for this request to ensure any lingering
// connection terminates.
kill()
return nil, nil, nil, err
}
outr, outw := io.Pipe()
errr, errw := io.Pipe()
go func() {
for {
logging.V(10).Infoln("Waiting for plugin message")
msg, err := resp.Recv()
if err != nil {
contract.IgnoreError(outw.CloseWithError(err))
contract.IgnoreError(errw.CloseWithError(err))
break
}
logging.V(10).Infoln("Got plugin response: ", msg)
if value, ok := msg.Output.(*pulumirpc.RunPluginResponse_Stdout); ok {
n, err := outw.Write(value.Stdout)
contract.AssertNoErrorf(err, "failed to write to stdout pipe: %v", err)
contract.Assertf(n == len(value.Stdout), "wrote fewer bytes (%d) than expected (%d)", n, len(value.Stdout))
} else if value, ok := msg.Output.(*pulumirpc.RunPluginResponse_Stderr); ok {
n, err := errw.Write(value.Stderr)
contract.AssertNoErrorf(err, "failed to write to stderr pipe: %v", err)
contract.Assertf(n == len(value.Stderr), "wrote fewer bytes (%d) than expected (%d)", n, len(value.Stderr))
} else if _, ok := msg.Output.(*pulumirpc.RunPluginResponse_Exitcode); ok {
// If stdout and stderr are empty we've flushed and are returning the exit code
outw.Close()
errw.Close()
break
}
}
}()
return outr, errr, kill, nil
}
func (h *langhost) GenerateProject(
sourceDirectory, targetDirectory, project string, strict bool,
loaderTarget string, localDependencies map[string]string,
2023-05-26 10:32:19 +00:00
) (hcl.Diagnostics, error) {
logging.V(7).Infof("langhost[%v].GenerateProject() executing", h.runtime)
2023-05-26 10:32:19 +00:00
resp, err := h.client.GenerateProject(h.ctx.Request(), &pulumirpc.GenerateProjectRequest{
SourceDirectory: sourceDirectory,
TargetDirectory: targetDirectory,
Project: project,
Strict: strict,
LoaderTarget: loaderTarget,
LocalDependencies: localDependencies,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].GenerateProject() failed: err=%v", h.runtime, rpcError)
2023-05-26 10:32:19 +00:00
return nil, rpcError
}
// Translate the rpc diagnostics into hcl.Diagnostics.
var diags hcl.Diagnostics
for _, rpcDiag := range resp.Diagnostics {
diags = append(diags, RPCDiagnosticToHclDiagnostic(rpcDiag))
}
logging.V(7).Infof("langhost[%v].GenerateProject() success", h.runtime)
2023-05-26 10:32:19 +00:00
return diags, nil
}
func (h *langhost) GeneratePackage(
Use the local artifacts, not released artifacts in conformance tests (#15777) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This threads the "local_dependencies" property through to GeneratePackage, following exactly the same semantics as for "GenerateProgram". Fixes https://github.com/pulumi/pulumi/issues/15074. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 13:10:34 +00:00
directory string, schema string, extraFiles map[string][]byte,
loaderTarget string, localDependencies map[string]string,
local bool,
Return diagnostics from GeneratePackage (#14661) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14660. Fairly simple change to bring the GeneratePackage RPC method into alignment with the other codegen methods and use returned diagnostics rather than just error values. `gen-sdk` is updated to print those diagnostics, and the python/node/go runtimes updated to return the diagnostics from schema binding as diagnostics rather than just an error value. Might be worth at some point seeing if the rest of package generation could use diagnostics rather than error values, but that's a larger lift. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-05 17:47:52 +00:00
) (hcl.Diagnostics, error) {
logging.V(7).Infof("langhost[%v].GeneratePackage() executing", h.runtime)
Return diagnostics from GeneratePackage (#14661) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14660. Fairly simple change to bring the GeneratePackage RPC method into alignment with the other codegen methods and use returned diagnostics rather than just error values. `gen-sdk` is updated to print those diagnostics, and the python/node/go runtimes updated to return the diagnostics from schema binding as diagnostics rather than just an error value. Might be worth at some point seeing if the rest of package generation could use diagnostics rather than error values, but that's a larger lift. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-05 17:47:52 +00:00
resp, err := h.client.GeneratePackage(h.ctx.Request(), &pulumirpc.GeneratePackageRequest{
Use the local artifacts, not released artifacts in conformance tests (#15777) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This threads the "local_dependencies" property through to GeneratePackage, following exactly the same semantics as for "GenerateProgram". Fixes https://github.com/pulumi/pulumi/issues/15074. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-26 13:10:34 +00:00
Directory: directory,
Schema: schema,
ExtraFiles: extraFiles,
LoaderTarget: loaderTarget,
LocalDependencies: localDependencies,
Local: local,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].GeneratePackage() failed: err=%v", h.runtime, rpcError)
Return diagnostics from GeneratePackage (#14661) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14660. Fairly simple change to bring the GeneratePackage RPC method into alignment with the other codegen methods and use returned diagnostics rather than just error values. `gen-sdk` is updated to print those diagnostics, and the python/node/go runtimes updated to return the diagnostics from schema binding as diagnostics rather than just an error value. Might be worth at some point seeing if the rest of package generation could use diagnostics rather than error values, but that's a larger lift. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-05 17:47:52 +00:00
return nil, rpcError
}
logging.V(7).Infof("langhost[%v].GeneratePackage() success", h.runtime)
Return diagnostics from GeneratePackage (#14661) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi/issues/14660. Fairly simple change to bring the GeneratePackage RPC method into alignment with the other codegen methods and use returned diagnostics rather than just error values. `gen-sdk` is updated to print those diagnostics, and the python/node/go runtimes updated to return the diagnostics from schema binding as diagnostics rather than just an error value. Might be worth at some point seeing if the rest of package generation could use diagnostics rather than error values, but that's a larger lift. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-12-05 17:47:52 +00:00
// Translate the rpc diagnostics into hcl.Diagnostics.
var diags hcl.Diagnostics
for _, rpcDiag := range resp.Diagnostics {
diags = append(diags, RPCDiagnosticToHclDiagnostic(rpcDiag))
}
return diags, nil
}
[cli/import] Fix undefined variable errors in code generation when imported resources use a parent or provider (#16786) Fixes #15410 Fixes #13339 ## Problem Context When using `pulumi import` we generate code snippets for the resources that were imported. Sometimes the user specifies `--parent parentName=URN` or `--provider providerName=URN` which tweak the parent or provider that the imported resources uses. When using `--parent` or `--provider` the generated code emits a resource option `parent = parentName` (in case of using `--parent`) where `parentName` is an unbound variable. Usually unbound variables would result in a _bind_ error such as `error: undefined variable parentName` when type-checking the program however in the import code generation we specify the bind option `pcl.AllowMissingVariables` which turns that unbound variable errors into warnings and code generation can continue to emit code. This is all good and works as expected. However in the issues linked above, we do get an _error_ for unbound variables in generated code even though we specified `AllowMissingVariables`. The problem as it turns out is when we are trying to generate code via dynamically loaded `LangaugeRuntime` plugins. Specifically for NodeJS and Python, we load `pulumi-language-nodejs` or `pulumi-language-python` and call `GenerateProgram` to get the generated program. That function `GenerateProgram` takes the text _SOURCE_ of the a bound program (one that was bound using option `AllowMissingVariables`) and re-binds again inside the implementation of the language plugin. The second time we bind the program, we don't pass it the option `AllowMissingVariables` and so it fails with `unboud variable` error. I've verified that the issue above don't repro when doing an import for dotnet (probably same for java/yaml) because we use the statically linked function `codegen/{lang}/gen_program.go -> GenerateProgram` ## Solution The problem can be solved by propagating the bind options from the CLI to the language hosts during import so that they know how to bind the program. I've extended the gRPC interface in `GenerateProgramRequest` with a property `Strict` which follows the same logic from `pulumi convert --strict` and made it such that the import command sends `strict=false` to the language plugins when doing `GenerateProgram`. This is consistent with `GenerateProject` that uses the same flag. When `strict=false` we use `pcl.NonStrictBindOptions()` which includes `AllowMissingVariables` . ## Repro Once can test the before and after behaviour by running `pulumi up --yes` on the following TypeScript program: ```ts import * as pulumi from "@pulumi/pulumi"; import * as random from "@pulumi/random"; export class MyComponent extends pulumi.ComponentResource { public readonly randomPetId: pulumi.Output<string>; constructor(name: string, opts?: pulumi.ComponentResourceOptions) { super("example:index:MyComponent", name, {}, opts); const randomPet = new random.RandomPet("randomPet", {}, { parent: this }); this.randomPetId = randomPet.id; this.registerOutputs({ randomPetId: randomPet.id, }); } } const example = new MyComponent("example"); export const randomPetId = example.randomPetId; ``` Then running `pulumi import -f import.json` where `import.json` contains a resource to be imported under the created component (stack=`dev`, project=`importerrors`) ```ts { "nameTable": { "parentComponent": "urn:pulumi:dev::importerrors::example:index:MyComponent::example" }, "resources": [ { "type": "random:index/randomPassword:RandomPassword", "name": "randomPassword", "id": "supersecret", "parent": "parentComponent" } ] } ``` Running this locally I get the following generated code (which previously failed to generate) ```ts import * as pulumi from "@pulumi/pulumi"; import * as random from "@pulumi/random"; const randomPassword = new random.RandomPassword("randomPassword", { length: 11, lower: true, number: true, numeric: true, special: true, upper: true, }, { parent: parentComponent, }); ```
2024-07-25 13:53:44 +00:00
func (h *langhost) GenerateProgram(program map[string]string, loaderTarget string, strict bool,
) (map[string][]byte, hcl.Diagnostics, error) {
logging.V(7).Infof("langhost[%v].GenerateProgram() executing", h.runtime)
resp, err := h.client.GenerateProgram(h.ctx.Request(), &pulumirpc.GenerateProgramRequest{
Source: program,
LoaderTarget: loaderTarget,
[cli/import] Fix undefined variable errors in code generation when imported resources use a parent or provider (#16786) Fixes #15410 Fixes #13339 ## Problem Context When using `pulumi import` we generate code snippets for the resources that were imported. Sometimes the user specifies `--parent parentName=URN` or `--provider providerName=URN` which tweak the parent or provider that the imported resources uses. When using `--parent` or `--provider` the generated code emits a resource option `parent = parentName` (in case of using `--parent`) where `parentName` is an unbound variable. Usually unbound variables would result in a _bind_ error such as `error: undefined variable parentName` when type-checking the program however in the import code generation we specify the bind option `pcl.AllowMissingVariables` which turns that unbound variable errors into warnings and code generation can continue to emit code. This is all good and works as expected. However in the issues linked above, we do get an _error_ for unbound variables in generated code even though we specified `AllowMissingVariables`. The problem as it turns out is when we are trying to generate code via dynamically loaded `LangaugeRuntime` plugins. Specifically for NodeJS and Python, we load `pulumi-language-nodejs` or `pulumi-language-python` and call `GenerateProgram` to get the generated program. That function `GenerateProgram` takes the text _SOURCE_ of the a bound program (one that was bound using option `AllowMissingVariables`) and re-binds again inside the implementation of the language plugin. The second time we bind the program, we don't pass it the option `AllowMissingVariables` and so it fails with `unboud variable` error. I've verified that the issue above don't repro when doing an import for dotnet (probably same for java/yaml) because we use the statically linked function `codegen/{lang}/gen_program.go -> GenerateProgram` ## Solution The problem can be solved by propagating the bind options from the CLI to the language hosts during import so that they know how to bind the program. I've extended the gRPC interface in `GenerateProgramRequest` with a property `Strict` which follows the same logic from `pulumi convert --strict` and made it such that the import command sends `strict=false` to the language plugins when doing `GenerateProgram`. This is consistent with `GenerateProject` that uses the same flag. When `strict=false` we use `pcl.NonStrictBindOptions()` which includes `AllowMissingVariables` . ## Repro Once can test the before and after behaviour by running `pulumi up --yes` on the following TypeScript program: ```ts import * as pulumi from "@pulumi/pulumi"; import * as random from "@pulumi/random"; export class MyComponent extends pulumi.ComponentResource { public readonly randomPetId: pulumi.Output<string>; constructor(name: string, opts?: pulumi.ComponentResourceOptions) { super("example:index:MyComponent", name, {}, opts); const randomPet = new random.RandomPet("randomPet", {}, { parent: this }); this.randomPetId = randomPet.id; this.registerOutputs({ randomPetId: randomPet.id, }); } } const example = new MyComponent("example"); export const randomPetId = example.randomPetId; ``` Then running `pulumi import -f import.json` where `import.json` contains a resource to be imported under the created component (stack=`dev`, project=`importerrors`) ```ts { "nameTable": { "parentComponent": "urn:pulumi:dev::importerrors::example:index:MyComponent::example" }, "resources": [ { "type": "random:index/randomPassword:RandomPassword", "name": "randomPassword", "id": "supersecret", "parent": "parentComponent" } ] } ``` Running this locally I get the following generated code (which previously failed to generate) ```ts import * as pulumi from "@pulumi/pulumi"; import * as random from "@pulumi/random"; const randomPassword = new random.RandomPassword("randomPassword", { length: 11, lower: true, number: true, numeric: true, special: true, upper: true, }, { parent: parentComponent, }); ```
2024-07-25 13:53:44 +00:00
Strict: strict,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("langhost[%v].GenerateProgram() failed: err=%v", h.runtime, rpcError)
return nil, nil, rpcError
}
// Translate the rpc diagnostics into hcl.Diagnostics.
var diags hcl.Diagnostics
for _, rpcDiag := range resp.Diagnostics {
diags = append(diags, RPCDiagnosticToHclDiagnostic(rpcDiag))
}
logging.V(7).Infof("langhost[%v].GenerateProgram() success", h.runtime)
return resp.Source, diags, nil
}
func (h *langhost) Pack(
Add SupportPack to schemas to write out in the new style (#15713) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to the schema metadata to tell codegen to use the new proposed style of SDKs where we fill in versions and write go.mods etc. I've reworked pack to operate on packages assuming they're in this new style. That is pack no longer has the responsibility to fill in any version information. This updates python and node codegen to write out SDKs in this new style, and fixes their core libraries to still be buildable via pack. There are two approaches to fixing those, I've chosen option 1 below but could pretty easily rework for option 2. 1) Write the version information directly to the SDKs at the same time as we edit the .version file. To simplify this I've added a new 'set-version.py' script that takes a version string an writes it to all the relevant places (.version, package.json, etc). 2) Write "pack" in the language host to search up the directory tree for the ".version" file and then fill in the version information as we we're doing before with envvar tricks and copying and editing package.json. I think 1 is simpler long term, but does force some amount of cleanup in unrelated bits of the system right now (release makefiles need a small edit). 2 is much more localised but keeps this complexity that sdk/nodejs sdk/python aren't actually valid source modules. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-22 09:25:46 +00:00
packageDirectory string, destinationDirectory string,
) (string, error) {
Add SupportPack to schemas to write out in the new style (#15713) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This adds a new flag to the schema metadata to tell codegen to use the new proposed style of SDKs where we fill in versions and write go.mods etc. I've reworked pack to operate on packages assuming they're in this new style. That is pack no longer has the responsibility to fill in any version information. This updates python and node codegen to write out SDKs in this new style, and fixes their core libraries to still be buildable via pack. There are two approaches to fixing those, I've chosen option 1 below but could pretty easily rework for option 2. 1) Write the version information directly to the SDKs at the same time as we edit the .version file. To simplify this I've added a new 'set-version.py' script that takes a version string an writes it to all the relevant places (.version, package.json, etc). 2) Write "pack" in the language host to search up the directory tree for the ".version" file and then fill in the version information as we we're doing before with envvar tricks and copying and editing package.json. I think 1 is simpler long term, but does force some amount of cleanup in unrelated bits of the system right now (release makefiles need a small edit). 2 is much more localised but keeps this complexity that sdk/nodejs sdk/python aren't actually valid source modules. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-03-22 09:25:46 +00:00
label := fmt.Sprintf("langhost[%v].Pack(%s, %s)", h.runtime, packageDirectory, destinationDirectory)
logging.V(7).Infof("%s executing", label)
// Always send absolute paths to the plugin, as it may be running in a different working directory.
packageDirectory, err := filepath.Abs(packageDirectory)
if err != nil {
return "", err
}
destinationDirectory, err = filepath.Abs(destinationDirectory)
if err != nil {
return "", err
}
req, err := h.client.Pack(h.ctx.Request(), &pulumirpc.PackRequest{
PackageDirectory: packageDirectory,
DestinationDirectory: destinationDirectory,
})
if err != nil {
rpcError := rpcerror.Convert(err)
logging.V(7).Infof("%s failed: err=%v", label, rpcError)
return "", rpcError
}
logging.V(7).Infof("%s success: artifactPath=%s", label, req.ArtifactPath)
return req.ArtifactPath, nil
}