pulumi/pkg/codegen/schema/loader.go

397 lines
12 KiB
Go
Raw Permalink Normal View History

// Copyright 2016-2022, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package schema
import (
"bytes"
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
"context"
"errors"
"fmt"
"github.com/natefinch/atomic"
"github.com/blang/semver"
"github.com/segmentio/encoding/json"
pkgWorkspace "github.com/pulumi/pulumi/pkg/v3/workspace"
"github.com/pulumi/pulumi/sdk/v3/go/common/apitype"
"github.com/pulumi/pulumi/sdk/v3/go/common/diag"
Add an envar to disable automatic provider installation (#14083) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is primarily for the providers team to enable during builds so they can have more confidence about reproducibility of builds (especially examples conversion), but I imagine some customers would enable this as well. Fixes #14086 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-03 15:35:23 +00:00
"github.com/pulumi/pulumi/sdk/v3/go/common/env"
"github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin"
"github.com/pulumi/pulumi/sdk/v3/go/common/tokens"
"github.com/pulumi/pulumi/sdk/v3/go/common/util/contract"
"github.com/pulumi/pulumi/sdk/v3/go/common/workspace"
)
type ParameterizationDescriptor struct {
Name string // the name of the package.
Version semver.Version // the version of the package.
Value []byte // the parameter value of the package.
}
// PackageDescriptor is a descriptor for a package, this is similar to a plugin spec but also contains parameterization
// info.
type PackageDescriptor struct {
Name string // the simple name of the plugin.
Version *semver.Version // the plugin's semantic version, if present.
DownloadURL string // an optional server to use when downloading this plugin.
Parameterization *ParameterizationDescriptor // the optional parameterization of the package.
}
type Loader interface {
// deprecated: use LoadPackageV2
LoadPackage(pkg string, version *semver.Version) (*Package, error)
LoadPackageV2(ctx context.Context, descriptor *PackageDescriptor) (*Package, error)
}
type ReferenceLoader interface {
Loader
// deprecated: use LoadPackageReferenceV2
LoadPackageReference(pkg string, version *semver.Version) (PackageReference, error)
LoadPackageReferenceV2(ctx context.Context, descriptor *PackageDescriptor) (PackageReference, error)
}
type pluginLoader struct {
host plugin.Host
cacheOptions pluginLoaderCacheOptions
}
// Caching options intended for benchmarking or debugging:
type pluginLoaderCacheOptions struct {
// useEntriesCache enables in-memory re-use of packages
disableEntryCache bool
// useFileCache enables skipping plugin loading when possible and caching JSON schemas to files
disableFileCache bool
// useMmap enables the use of memory mapped IO to avoid copying the JSON schema
disableMmap bool
}
func NewPluginLoader(host plugin.Host) ReferenceLoader {
return newPluginLoaderWithOptions(host, pluginLoaderCacheOptions{})
}
func newPluginLoaderWithOptions(host plugin.Host, cacheOptions pluginLoaderCacheOptions) ReferenceLoader {
var l ReferenceLoader
l = &pluginLoader{
host: host,
cacheOptions: cacheOptions,
}
if !cacheOptions.disableEntryCache {
l = NewCachedLoader(l)
}
return l
}
func (l *pluginLoader) LoadPackage(pkg string, version *semver.Version) (*Package, error) {
ref, err := l.LoadPackageReference(pkg, version)
if err != nil {
return nil, err
}
return ref.Definition()
}
func (l *pluginLoader) LoadPackageV2(ctx context.Context, descriptor *PackageDescriptor) (*Package, error) {
ref, err := l.LoadPackageReferenceV2(ctx, descriptor)
if err != nil {
return nil, err
}
return ref.Definition()
}
var ErrGetSchemaNotImplemented = getSchemaNotImplemented{}
type getSchemaNotImplemented struct{}
func (f getSchemaNotImplemented) Error() string {
return "it looks like GetSchema is not implemented"
}
func schemaIsEmpty(schemaBytes []byte) bool {
// A non-empty schema is any that contains non-whitespace, non brace characters.
//
// Some providers implemented GetSchema initially by returning text matching the regular
// expression: "\s*\{\s*\}\s*". This handles those cases while not strictly checking that braces
// match or reading the whole document.
for _, v := range schemaBytes {
if v != ' ' && v != '\t' && v != '\r' && v != '\n' && v != '{' && v != '}' {
return false
}
}
return true
}
func (l *pluginLoader) LoadPackageReference(pkg string, version *semver.Version) (PackageReference, error) {
return l.LoadPackageReferenceV2(
context.TODO(),
&PackageDescriptor{
Name: pkg,
Version: version,
})
}
func (l *pluginLoader) LoadPackageReferenceV2(
ctx context.Context, descriptor *PackageDescriptor,
) (PackageReference, error) {
if descriptor.Name == "pulumi" {
return DefaultPulumiPackage.Reference(), nil
2022-07-19 02:47:31 +00:00
}
schemaBytes, version, err := l.loadSchemaBytes(ctx, descriptor)
if err != nil {
return nil, err
}
if schemaIsEmpty(schemaBytes) {
return nil, getSchemaNotImplemented{}
}
var spec PartialPackageSpec
if _, err := json.Parse(schemaBytes, &spec, json.ZeroCopy); err != nil {
return nil, err
}
// Insert a version into the spec if the package does not provide one or if the
// existing version is less than the provided one
if version != nil {
setVersion := true
if spec.PackageInfoSpec.Version != "" {
vSemver, err := semver.Make(spec.PackageInfoSpec.Version)
if err == nil {
if vSemver.Compare(*version) == 1 {
setVersion = false
}
}
}
if setVersion {
spec.PackageInfoSpec.Version = version.String()
}
}
Add matrix testing (#13705) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Adds the first pass of matrix testing. Matrix testing allows us to define tests once in pulumi/pulumi via PCL and then run those tests against each language plugin to verify code generation and runtime correctness. Rather than packing matrix tests and all the associated data and machinery into the CLI itself we define a new Go package at cmd/pulumi-test-lanaguage. This depends on pkg and runs the deployment engine in a unique way for matrix tests but it is running the proper deployment engine with a proper backend (always filestate, using $TEMP). Currently only NodeJS is hooked up to run these tests, and all the code for that currently lives in sdk/nodejs/cmd/pulumi-language-nodejs/language_test.go. I expect we'll move that helper code to sdk/go/common and use it in each language plugin to run the tests in the same way. This first pass includes 3 simple tests: * l1-empty that runs an empty PCL file and checks just a stack is created * l1-output-bool that runs a PCL program that returns two stack outputs of `true` and `false * l2-resource-simple that runs a PCL program creating a simple resource with a single bool property These tests are themselves tested with a mock language runtime. This verifies the behavior of the matrix test framework for both correct and incorrect language hosts (that is some the mock language runtimes purposefully cause errors or compute the wrong result). There are a number of things missing from from the core framework still, but I feel don't block getting this first pass merged and starting to be used. 1. The tests can not currently run in parallel. That is calling RunLanguageTest in parallel will break things. This is due to two separate problems. Firstly is that the SDK snapshot's are not safe to write in parallel (when PULUMI_ACCEPT is true), this should be fairly easy to fix by doing a write to dst-{random} and them atomic move to dst. Secondly is that the deployment engine itself has mutable global state, short term we should probably just lock around that part RunLanguageTest, long term it would be good to clean that up. 2. We need a way to verify "preview" behavior, I think this is probably just a variation of the tests that would call `stack.Preview` and not pass a snapshot to `assert`. 3. stdout, stderr and log messages are returned in bulk at the end of the test. Plus there are a couple of calls to the language runtime that don't correctly thread stdout/stderr to use and so default to the process `os.Stdout/Stderr`. stdout/stderr streaming shows up in a load of other places as well so I'm thinking of a clean way to handle all of them together. Log message streaming we can probably do by just turning RunLanguageTest to a streaming grpc call. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-09-13 15:17:46 +00:00
p, err := ImportPartialSpec(spec, nil, l)
if err != nil {
return nil, err
}
return p, nil
}
// deprecated: use LoadPackageReferenceV2
func LoadPackageReference(loader Loader, pkg string, version *semver.Version) (PackageReference, error) {
return LoadPackageReferenceV2(
context.TODO(),
loader,
&PackageDescriptor{
Name: pkg,
Version: version,
})
}
func LoadPackageReferenceV2(
ctx context.Context, loader Loader, descriptor *PackageDescriptor,
) (PackageReference, error) {
var ref PackageReference
var err error
if refLoader, ok := loader.(ReferenceLoader); ok {
ref, err = refLoader.LoadPackageReferenceV2(ctx, descriptor)
} else {
p, pErr := loader.LoadPackageV2(ctx, descriptor)
err = pErr
if err == nil {
ref = p.Reference()
}
}
if err != nil {
return nil, err
}
name := descriptor.Name
if descriptor.Parameterization != nil {
name = descriptor.Parameterization.Name
}
version := descriptor.Version
if descriptor.Parameterization != nil {
version = &descriptor.Parameterization.Version
}
if name != ref.Name() ||
version != nil &&
ref.Version() != nil &&
!ref.Version().Equals(*version) {
if l, ok := loader.(*cachedLoader); ok {
return nil, fmt.Errorf("req: %s@%v: entries: %v (returned %s@%v)", name, version,
l.entries, ref.Name(), ref.Version())
}
return nil, fmt.Errorf(
"loader returned %s@%v: expected %s@%v", ref.Name(), ref.Version(), name, version)
}
return ref, nil
}
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
func (l *pluginLoader) loadSchemaBytes(
ctx context.Context, descriptor *PackageDescriptor,
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
) ([]byte, *semver.Version, error) {
attachPort, err := plugin.GetProviderAttachPort(tokens.Package(descriptor.Name))
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
if err != nil {
return nil, nil, err
}
version := descriptor.Version
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
// If PULUMI_DEBUG_PROVIDERS requested an attach port, skip caching and workspace
// interaction and load the schema directly from the given port.
if attachPort != nil {
schemaBytes, provider, err := l.loadPluginSchemaBytes(ctx, descriptor)
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
if err != nil {
return nil, nil, fmt.Errorf("Error loading schema from plugin: %w", err)
}
if version == nil {
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
info, err := provider.GetPluginInfo(ctx)
Schema loader made to respect PULUMI_DEBUG_PROVIDERS (#15526) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> With this change pulumi-yaml can pick up local provider and attach to it from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which enables using non-existent test-only providers. Before the change it would fail hard trying to download it. ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
contract.IgnoreError(err) // nonfatal error
version = info.Version
}
return schemaBytes, version, nil
}
pluginInfo, err := l.host.ResolvePlugin(apitype.ResourcePlugin, descriptor.Name, version)
if err != nil {
Add an envar to disable automatic provider installation (#14083) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is primarily for the providers team to enable during builds so they can have more confidence about reproducibility of builds (especially examples conversion), but I imagine some customers would enable this as well. Fixes #14086 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-03 15:35:23 +00:00
// Try and install the plugin if it was missing and try again, unless auto plugin installs are turned off.
var missingError *workspace.MissingError
if !errors.As(err, &missingError) || env.DisableAutomaticPluginAcquisition.Value() {
Add an envar to disable automatic provider installation (#14083) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is primarily for the providers team to enable during builds so they can have more confidence about reproducibility of builds (especially examples conversion), but I imagine some customers would enable this as well. Fixes #14086 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-03 15:35:23 +00:00
return nil, nil, err
}
spec := workspace.PluginSpec{
Kind: apitype.ResourcePlugin,
Name: descriptor.Name,
Version: version,
PluginDownloadURL: descriptor.DownloadURL,
}
log := func(sev diag.Severity, msg string) {
l.host.Log(sev, "", msg, 0)
}
_, err = pkgWorkspace.InstallPlugin(spec, log)
if err != nil {
return nil, nil, err
}
pluginInfo, err = l.host.ResolvePlugin(apitype.ResourcePlugin, descriptor.Name, version)
if err != nil {
return nil, version, err
}
}
contract.Assertf(pluginInfo != nil, "loading pkg %q: pluginInfo was unexpectedly nil", descriptor.Name)
if version == nil {
version = pluginInfo.Version
}
if pluginInfo.SchemaPath != "" && version != nil && descriptor.Parameterization == nil {
schemaBytes, ok := l.loadCachedSchemaBytes(descriptor.Name, pluginInfo.SchemaPath, pluginInfo.SchemaTime)
if ok {
return schemaBytes, nil, nil
}
}
schemaBytes, provider, err := l.loadPluginSchemaBytes(ctx, descriptor)
if err != nil {
return nil, nil, fmt.Errorf("Error loading schema from plugin: %w", err)
}
if pluginInfo.SchemaPath != "" {
err = atomic.WriteFile(pluginInfo.SchemaPath, bytes.NewReader(schemaBytes))
if err != nil {
return nil, nil, fmt.Errorf("Error writing schema from plugin to cache: %w", err)
}
}
if version == nil {
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
info, _ := provider.GetPluginInfo(ctx) // nonfatal error
version = info.Version
}
return schemaBytes, version, nil
}
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
func (l *pluginLoader) loadPluginSchemaBytes(
ctx context.Context, descriptor *PackageDescriptor,
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
) ([]byte, plugin.Provider, error) {
wsDescriptor := workspace.PackageDescriptor{
PluginSpec: workspace.PluginSpec{
Name: descriptor.Name,
Version: descriptor.Version,
PluginDownloadURL: descriptor.DownloadURL,
},
}
if descriptor.Parameterization != nil {
wsDescriptor.Parameterization = &workspace.Parameterization{
Name: descriptor.Parameterization.Name,
Version: descriptor.Parameterization.Version,
Value: descriptor.Parameterization.Value,
}
}
provider, err := l.host.Provider(wsDescriptor)
if err != nil {
return nil, nil, err
}
contract.Assertf(provider != nil, "unexpected nil provider for %s@%v", descriptor.Name, descriptor.Version)
var schemaFormatVersion int32
getSchemaRequest := plugin.GetSchemaRequest{
Add Paramaterize to provider interface (#16174) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> This is the bare bones changes required to update the provider interface for parametrization. Nothing in the engine, sdks, cli, or codegen makes use of these new methods yet. But providers team can start building on top of this interface. Note that this is subject to change, while this is _likely_ the right design for parametrised providers there is a chance that we need to edit this interface before GA release. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-05-15 16:22:39 +00:00
Version: schemaFormatVersion,
}
// If this is a parameterized package, we need to pass the parameter value to the provider.
if descriptor.Parameterization != nil {
parameterization := plugin.ParameterizeRequest{
Parameters: &plugin.ParameterizeValue{
Name: descriptor.Parameterization.Name,
Version: descriptor.Parameterization.Version,
Value: descriptor.Parameterization.Value,
},
}
resp, err := provider.Parameterize(ctx, parameterization)
if err != nil {
return nil, nil, err
}
if resp.Name != descriptor.Parameterization.Name {
return nil, nil, fmt.Errorf(
"unexpected parameterization response: %s != %s", resp.Name, descriptor.Parameterization.Name)
}
if !resp.Version.EQ(descriptor.Parameterization.Version) {
return nil, nil, fmt.Errorf(
"unexpected parameterization response: %s != %s", resp.Version, descriptor.Parameterization.Version)
}
getSchemaRequest.SubpackageName = resp.Name
getSchemaRequest.SubpackageVersion = &resp.Version
}
schema, err := provider.GetSchema(ctx, getSchemaRequest)
if err != nil {
return nil, nil, err
}
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) (#16302) Normalize methods on plugin.Provider to the form: ```go Method(context.Context, MethodRequest) (MethodResponse, error) ``` This provides a more consistent and forwards compatible interface for each of our methods. --- I'm motivated to work on this because the bridge maintains a copy of this interface: `ProviderWithContext`. This doubles the pain of dealing with any breaking change and this PR would allow me to remove the extra interface. I'm willing to fix consumers of `plugin.Provider` in `pulumi/pulumi`, but I wanted to make sure that we would be willing to merge this PR if I get it green. <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes # (issue) ## Checklist - [ ] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2024-06-07 19:47:49 +00:00
return schema.Schema, provider, nil
}