Commit Graph

86 Commits

Author SHA1 Message Date
Fraser Waters cb4a06b475
Package versions are required for parameterized packages ()
Make this more clear by passing plain value of `semver.Version` instead
of a pointer for these cases.

This also gets rid of some incorrect empty string checks which would
have passed a nil version onwards rather than error'ing (as they should
have).
2024-08-05 16:01:28 +00:00
Fraser Waters a9947b4e4e
Python parameterized provider test ()
This adds support for replacement parameterised providers to Python and
a small integration test to check it works e2e.

When using parameterised providers we need to use the new (currently
unstable) RegisterPackage system, instead of sending
Version/DownloadURL/etc via RegisterResourceRequest. Once
RegisterPackage is stable the intention is to change _all_ packages to
use it and for normal packages to fall back to the
RegisterResourceRequest options, while parameterised packages will
error.

The actual parameter value is embedded in the python SDK as a base64
string that we decode before sending to the gRPC endpoint as bytes.
2024-07-16 10:55:38 +00:00
Fraser Waters a0e0208dd1
Change parameterization to be bytes based ()
Making this change has a couple of benefits.

Firstly (and honestly the main one) is it make codegen much simpler. We
just to emit a base64 string and/or other embedded byte array to the
generated SDK. Currently we need emit a full `proto.Value` expression
for each language, and that's not hard but it's also not trivial and
means more combinations of things per test per language.

Secondly it will allow providers to use more efficient encodings of
their parameter than JSON if there is one. I imagine some providers
might make the parameter value a protobuf message and parse that
(similar to what we do for transform functions), but they can easily
fallback to just treating the bytes as a JSON string if they want.

The only downside to this is the value is obfuscated in the generated
SDK and in the state file. Neither of those are really expected to be
viewed by users, so this feels like a minor loss.
2024-07-10 11:15:35 +00:00
Will Jones 283bbd23f6
Fix provider `Delete` ()
 (78c48204e0) introduced a
regression whereby outputs are mixed up with inputs for provider
`Delete`s. This commit fixes the behaviour so that it matches that
before the refactoring.

Fixes 

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-21 15:29:20 +00:00
Ian Wahbe 78c48204e0
Normalize plugin.Provider methods to (Context, Request) -> (Response, error) ()
Normalize methods on plugin.Provider to the form:

```go
Method(context.Context, MethodRequest) (MethodResponse, error)
```

This provides a more consistent and forwards compatible interface for
each of our methods.

---

I'm motivated to work on this because the bridge maintains a copy of
this interface: `ProviderWithContext`. This doubles the pain of dealing
with any breaking change and this PR would allow me to remove the extra
interface. I'm willing to fix consumers of `plugin.Provider` in
`pulumi/pulumi`, but I wanted to make sure that we would be willing to
merge this PR if I get it green.

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes # (issue)

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-07 19:47:49 +00:00
Ian Wahbe eaeaa3fab7
Plugin interface cleanup ()
This PR is best reviewed commit by commit:
- bbe1fc6b2f exchanges `ParseTolerant` for
`Parse` in `Parameterize`.
- 76df18be56 updates the `Parameterize`
interface for forward compatibility.
- 746c057668 requires implementors of
`plugin.Provider` to make a forward compatibility choice explicitly.
This is similar to what we require for gRPC already. Since this will
release with bbe1fc6b2f, consumers will
already need to update their `plugin.Provider` implementor, minimizing
the disturbance.
2024-05-31 00:28:48 +00:00
Ian Wahbe e220f574d5
Abstract plugin.Parameterize away from the gRPC interface ()
`plugin.Provider` exists to provide a low-level abstraction *on top of*
the gRPC interface. It should not expose the gRPC types directly. The
key diff in this commit is:

```patch
-	Parameterize(
-		ctx context.Context, req *pulumirpc.ParameterizeRequest,
-	) (*pulumirpc.ParameterizeResponse, error)
+	Parameterize(parameters ParameterizeParameters) (string, *semver.Version, error)
```

```patch
+type ParameterizeParameters interface {
+	isParameterizeParameters()
+}
+
+type (
+	ParameterizeArgs struct {
+		Args []string
+	}
+
+	ParameterizeValue struct {
+		Name    string
+		Version *semver.Version
+		// Value must be one of:
+		// - nil
+		// - bool
+		// - int, int32, int64
+		// - uint, uint32, uint64
+		// - float32, float64
+		// - string
+		// - []byte
+		// - map[string]interface{}
+		// - []interface{}
+		Value any
+	}
+)
+
+func (ParameterizeArgs) isParameterizeParameters()  {}
+func (ParameterizeValue) isParameterizeParameters() {}
```

This is the new interface exposed in `plugin`. The rest of the PR is
simply complying with the new interface.

While this change is technically breaking (since it was released in
v3.116.0), its a heavily experimental feature and the providers team (my
team) is the biggest consumer of `plugin`. Before this PR,
`Parameterize` was the *only* method that dirrectly exposed the gRPC
interface it was "abstracting".

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

## Checklist

- [X] I have run `make tidy` to update any new dependencies
- [X] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-30 03:17:49 +00:00
Fraser Waters ab7acdb602
Add Paramaterize to provider interface ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This is the bare bones changes required to update the provider interface
for parametrization. Nothing in the engine, sdks, cli, or codegen makes
use of these new methods yet.

But providers team can start building on top of this interface. Note
that this is subject to change, while this is _likely_ the right design
for parametrised providers there is a chance that we need to edit this
interface before GA release.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-15 16:22:39 +00:00
Germán Lena d7f24dfcfb
Refactor: move plugin kind to apitype ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR moves PluginKind to apitype to prevent circular dependencies
when adding apitype as a dependency of the workspace module.
It also re-exports PluginKind to keep backward compatibility

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-25 17:30:30 +00:00
Anton Tayanovskyy ee17f803aa
Schema loader made to respect PULUMI_DEBUG_PROVIDERS ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

With this change pulumi-yaml can pick up local provider and attach to it
from PULUMI_DEBUG_PROVIDERS for the purposes of schema resolution, which
enables using non-existent test-only providers. Before the change it
would fail hard trying to download it.


## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-03-04 21:54:05 +00:00
Fraser Waters 556cd9b6cd
Allow component providers to return outputs to the engine ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

This adds a new flag to provider Construct and Call requests to indicate
that the engine will accept output values in the response.

If a component provider sees that flag set they can return Output
PropertyValues and optionally skip filling in the dependency map in the
response.

The engine will use the combined information of the dependency map and
any dependencies found on any output values in the property to build the
final dependency map that is returned to programs.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-02-14 08:15:24 +00:00
Ian Wahbe 9a06694676
Move assets and archives to their own package ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR is motivated by https://github.com/pulumi/pulumi/pull/15145.

`resource.*` should be built on top of `property.Value`,[^1] which means
that `resource`
needs to be able to import `property.Value`, and so `property` cannot
import
`resource`. Since Assets and Archives are both types of properties, they
must be moved out
of `resource`.

[^1]: For example:
a1d686227c/sdk/go/common/resource/resource_state.go (L35-L36)

## Open Question

This PR moves them to their own sub-folders in `resource`. Should
`asset` and `archive`
live somewhere more high level, like `sdk/go/property/{asset,archive}`?

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-25 20:39:31 +00:00
Thomas Gummerer baecc85eaf
turn on the golangci-lint exhaustive linter ()
Turn on the golangci-lint exhaustive linter.  This is the first step
towards catching more missing cases during development rather than
in tests, or in production.

This might be best reviewed commit-by-commit, as the first commit turns
on the linter with the `default-signifies-exhaustive: true` option set,
which requires a lot less changes in the current codebase.

I think it's probably worth doing the second commit as well, as that
will get us the real benefits, even though we end up with a little bit
more churn. However it means all the `switch` statements are covered,
which isn't the case after the first commit, since we do have a lot of
`default` statements that just call `assert.Fail`.
 
Fixes  

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-17 16:50:41 +00:00
Fraser Waters 3be1b6289c
Remove deprecated Protobufs imports ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

github.com/golang/protobuf is marked deprecated and I was getting
increasingly triggered by the inconsistency of importing the `Empty`
type from "github.com/golang/protobuf/ptypes/empty" or
"google.golang.org/protobuf/types/known/emptypb" as "pbempty" or "empty"
or "emptypb". Similar for the struct type.

So this replaces all the Protobufs imports with ones from
"google.golang.org/protobuf", normalises the import name to always just
be the module name (emptypb), and adds the depguard linter to ensure we
don't use the deprecated package anymore.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-17 09:35:20 +00:00
Fraser Waters 16d9f4c167
Enable perfsprint linter ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945

This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.

Most of these edits were made by running `perfsprint --fix`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-12 12:19:42 +00:00
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Fraser Waters 478c29908f
Add a strict opinionated promise library ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by having another case in
https://github.com/pulumi/pulumi/pull/14548 that looked like "this is
just a promise".

We already had a mini-promise library for provider config in
sdk/go/common/resource/plugin/provider_plugin.go. Plus a couple of
places where we did a poor mans Go promise by having a WaitGroup plus a
variable to set (see the two changes in /pkg).

This adds a little promise library to safely cover all three of these
cases plus the case I'll be adding in . It is _minimal_ adding
just the features of promises needed for these initial cases. We can add
to it as needed.

I suspect we've got a number of places in test code that could probably
use this as well, but I haven't gone through that. I also suspect that
having this type available will result in more places in the future
being simpler because "its just a promise" is a fairly common scenario
in async systems. In fact Output's internally _are just a promise_ so we
could probably rewrite their internals using this, add a `Then()` like
method for apply and all the async complexity gets handled in the
promise layer while the output layer just cares about
unknowns/secrets/etc.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change - This is in
sdk/go/common and I'm considering it an non-public api for now.
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-15 14:53:12 +00:00
Fraser Waters 1776c30eb8
Ignore spurious error from kubernetes DiffConfig ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14529.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-11 09:59:39 +00:00
Fraser Waters 6277df0d8f
Return errors from DiffConfig ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14342.

I suspect this has just been a long standing typo. I can't see any
reason to ignore this error.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-30 09:28:17 +00:00
Fraser Waters ca12edfb4e
Send old inputs to Delete ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14115.

This was missed as part of https://github.com/pulumi/pulumi/pull/13139.

Adds a new configure flag (sends_old_inputs_to_delete) which the engine
will now always set to true. If that's set providers can rely on the old
inputs being sent to delete, otherwise they'll get nil.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-13 14:12:26 +00:00
Joe Duffy 96a9a77167
Policy remediations feature ()
This PR implements the new policy transforms feature, which allows
policy packs to not only issue warnings and errors in response to policy
violations, but actually fix them by rewriting resource property state.
This can be used, for instance, to auto-tag resources, remove Internet
access on the fly, or apply encryption to storage, among other use
cases.
2023-10-09 18:31:17 +00:00
Zaid Ajaj 142a25216b
[cli/engine] Restore elided asset contents from returned inputs from Read operations ()
# Description

When we send inputs to `Read` operations due to `pulumi refresh` we skip
sending asset contents. However, `Read` implementation might return the
asset-valued input properties unchanged (without their asset contents)
and the engine will simply write out the empty assets to the state
causing an invalid shape of asset in the inputs of resources.

This PR implements a function `restoreElidedAssetContents` and uses it
in the `Read` implementation on the new inputs and new state returned
from the provider plugins.

Testing the CLI locally using this fix, it does correctly write out the
asset contents to the state _after_ a `pulumi refresh`. ~However, when
running `preview` after `refresh`, the nodejs SDK still fails with the
same error in  because `RegisterResource` still returns the inputs
that contain empty/invalid assets~

**EDIT**: Fixed the issue with the inputs sent to the nodejs SDK are
missing asset contents (thanks @Frassle 🙏)

Fixes  

### TO-DO
- [x] Figure out why `RegisterResource` is sending empty assets for
during `preview` after `refresh`

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-06 16:47:01 +00:00
Fraser Waters 183b47bf06
More efficent mapping lookup ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Inspired by a comment from Zaid. This allows providers to return what
providers they have mapping information for without having to marshal
all their mapping data to the engine at the same time, this could save
transmitting a lot of data that the engine might not ever need (for
example if it's not converting code for that specific provider). It also
allows provider to support mulitple mappings.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-09-21 11:45:07 +00:00
Fraser Waters a691975202 Warn about ambient plugins loaded from $PATH
By default Pulumi will load ambient plugins from $PATH before looking in
the plugins directory or at bundled plugins.

While this is very useful for development it often causes confusion when
people have forgotten that they have plugins left on $PATH.

This makes the use of these $PATH plugins a diagnostic warning to try
and make that failure mode a little less silent.

Normal users shouldn't ever have plugins on $PATH and so won't see this
new warning.

Re-instates https://github.com/pulumi/pulumi/pull/13607 with a fix for
symlinks included.
2023-08-08 13:11:34 +01:00
Kyle Dixler 86ebe1bbd3 Revert "Warn about ambient plugins loaded from $PATH" 2023-08-04 16:54:16 -07:00
Fraser Waters a5b1590499 Warn about ambient plugins loaded from $PATH
By default Pulumi will load ambient plugins from $PATH before looking in
the plugins directory or at bundled plugins.

While this is very useful for development it often causes confusion when
people have forgotten that they have plugins left on $PATH.

This makes the use of these $PATH plugins a diagnostic warning to try
and make that failure mode a little less silent.

Normal users shouldn't ever have plugins on $PATH and so won't see this
new warning.
2023-07-27 17:59:44 +01:00
Fraser Waters a5797c1f4d Extend unknown providers test to cover Construct 2023-07-25 17:13:28 +01:00
Fraser Waters a9b84d3676 Assert that we don't call Delete with unknowns 2023-07-25 08:50:06 +01:00
Fraser Waters 571fadae3f Use slice.Prealloc instead of make([]T, 0, ...)
Fixes https://github.com/pulumi/pulumi/issues/12738

https://github.com/pulumi/pulumi/pull/11834 turned on the prealloc
linter and changed a load of slice uses from just `var x T[]` to `x :=
make([]T, 0, preallocSize)`. This was good for performance but it turns
out there are a number of places in the codebase that treat a `nil`
slice as semnatically different to an empty slice.

Trying to test that, or even reason that through for every callsite is
untractable, so this PR replaces all expressions of the form `make([]T,
0, size)` with a call to `slice.Prealloc[T](size)`. When size is 0 that
returns a nil array, rather than an empty array.
2023-06-29 11:27:50 +01:00
Fraser Waters 841c99a9dd Send old inputs to diff and update
Fixes https://github.com/pulumi/pulumi/issues/5973

This adds two new properties to the RPC interfaces with providers.
Firstly we pass a new bool flag to Configure to tell the provider if it
will get sent old inputs as part of Diff.
Secondly we now pass the old inputs and the old outputs to Diff and
Update as opposed to just the old outputs.

The flag passed to Configure isn't strictly needed. Providers should be
able to tell that the old inputs aren't being sent to Diff and Updae by
virtue of them being `null/nil/None` rather than an empty map. For
DiffConfig this is the only way to detect this case because it's called
before Configure. But its still useful to send this explictly to
Configure to allow providers to return a configure error that they need
to be ran against a newer engine version.

There is one fairly major failure case that can happen with this feature
and providers who update to make use of this feature should ensure they
call this out in their changelog.
That is if a provider has been saving old inputs in outputs as a way to
get access to them in diff, and then updates to use these new protocol
properties instead and stops saving the inputs in outputs _and then_ the
user downgrades back to an old provider version things will probably get
_very_ confused.
2023-06-21 21:04:50 +01:00
Fraser Waters 61219fe775 Add debug logs to Mapping calls
Makes it easier to trace conversion mappings via debug logs.
2023-06-05 10:05:16 +01:00
Fraser Waters e1663b7809 Log the version from GetPluginInfo 2023-06-02 22:07:04 +01:00
Abhinav Gupta ccb7a9598e
sdk/plugin: Add more resource options to ConstructOptions
Adds a bunch of new resource options to plugin.ConstructOptions
and propagates them to the ConstructRequest sent to an MLC provider.
2023-04-21 10:57:21 -07:00
Fraser Waters b5952e4bfd Pass PULUMI_CONFIG through to provider plugins 2023-04-05 10:17:18 +01:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Abhinav Gupta cc32691bc1
chore(all): strings.Replace(..., -1) => strings.Replace(...)
Replaces all instances of `strings.Replace(s, old, new, -1)`
with the equivalent `strings.ReplaceAll(s, old, new)`.
This function has been available since Go 1.12.
2023-03-01 13:22:33 -08:00
Abhinav Gupta 7bddec255d
sdk/go: Prefer contract.Assertf over Assert
Migrates all uses of contract.{Assert, AssertNoError, Require} in sdk/
to the `*f` variants that are required to provide more error context.

Step towards deprecating non-f variants entirely.

For context, `contract.Require` is similar to `contract.Assert`,
except it has a required parameter name as an argument:

    func Require(cond bool, param string)
    func Requiref(cond bool, param string, msg string, args ...any)

It includes the parameter name in the error message by default,
so the `msg` and `args` should only describe the constraint
without naming the parameter.

Refs 
2023-02-15 10:22:43 -08:00
Abhinav Gupta 5db1fcabeb
sdk/plugin/provider: Prevent use-before-configure races
This is a follow-up to  with a more permanent fix.

The cause of the original data race was that
some of the invocations accessed the configuration
before the Configure RPC succeeded.

Notably, there was a case where we had:

    doStuff(cfg)
    waitForConfiguration()
    doMoreStuff(cfg)

The issue was fixed by simple re-ordering.

    waitForConfiguration()
    doStuff(cfg)
    doMoreStuff(cfg)

However, it remains very easy to read the configuration
before it's actually configured.
It's only a matter of time before the same bug arises again.

To fix this, we pull out the configuration into its own struct,
and make it accessible only by calling a specific function.

This separation is enforced by a promise-like object
with a consumer (Await) and a producer (Fulfill) end.
Any method that needs the configuration must wait for it
with Await first.

NOTE:
Switching to this new method was straightforward
for nearly all methods because they run after Configure.
However, in local experiments, CheckConfig ran before Configure;
DiffConfig looks like it might too.
Both these methods did not have any "wait for configure" blockers
like the other methods,
but they **still used the incorrect configuration**.
I've filled in default values (false) for the configuration parameters
that they consumed before the provider was configured
because that's what they were using anyway.

Refs , 
2023-01-27 08:56:42 -08:00
Abhinav Gupta 99a3f18ee4
sdk/go/plugin: Fix data race between Configure and Delete
The data race results from Delete accessing configuration
before Configure has been called.

The immediate fix for this is to fix the call ordering.

A longer term fix is to make it impossible
to access configuration before Configure is called.

Resolves 
2023-01-24 16:55:36 -08:00
bors[bot] 8915911825
Merge
11819: Cleanup diag error for protected resources r=dixler a=iwahbe

Change the error message for deleting protected resources so it sounds less like we tried and failed to delete the resource, and more like we chose not to. 

Fixes 


11834: Preallocate slices with a known capacity. r=dixler a=RobbieMcKinstry

Enable the prealloc linter, which identifies slices with a known capacity, but are not preallocated, which results in unnecessary allocations and memcpys.

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] N/A: enabling a linter. I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] **N/A: not a user-facing change.** I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11842: Changelog and go.mod updates for v3.51.1 r=dixler a=justinvp



11845: Move SecretsProvider to pkg/secrets r=dixler a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Trying to push DefaultSecretsProvider up past the backends resulted in a module loop between pkg/resource/stack and pkg/resource/deploy. This places SecretProvider in the secrets module to avoid that.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works - N/A Just moving interface declaration 
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - N/A Internal code changes
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version - No
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Ian Wahbe <ian@wahbe.com>
Co-authored-by: Robbie McKinstry <robbie@pulumi.com>
Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
Co-authored-by: Robbie McKinstry <thesnowmancometh@gmail.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-01-12 20:11:42 +00:00
Robbie McKinstry f58a2942fe
Fix instance of missing length in preallocation of slice. 2023-01-12 08:57:05 -08:00
Robbie McKinstry 4959522a53
Repair tests expecting nil slices 2023-01-11 21:53:04 -08:00
Robbie McKinstry 1f78baae71
Preallocate slices with a known capacity.
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
2023-01-11 12:52:51 -08:00
Abhinav Gupta 69762acedf
grpc: WithInsecure is deprecated
grpc.WithInsecure is deprecated [1].
Instead, we're expected to use:

    grpc.WithTransportCredentials(insecure.NewCredentials())

This updates all instances across pkg/, sdk/ and tests/.

  [1]: https://pkg.go.dev/google.golang.org/grpc#WithInsecure

In service to 
2023-01-11 12:15:40 -08:00
Fraser Waters 1839e15e4a Phase 3 of the convert mapper 2022-12-01 23:43:43 +00:00
Fraser Waters 14d8b59ea2 First pass of adding the convert mapper 2022-11-29 09:46:44 +00:00
Fraser Waters 9e5f1cc618 Engine and Golang support for shimless providers
This allows the pulumi-language-go plugin to start up providers directly
from .go source files.

The other language providers will be extended to support this as well in
time.
2022-11-14 11:25:41 +00:00
Anton Tayanovskyy 9276574c80 Introduce PULUMI_DEBUG_GRPC support 2022-11-08 14:01:15 -05:00
杨成锴 7b09ad142a chore: Update doc comments, coding style, fix lint
* about error string: error string should not be capitalized or end with punctuation mark
* apply suggestions from code review
* lint: fix format error and simplify the code

Co-authored-by: Fraser Waters <frassle@gmail.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-10-13 13:50:49 -07:00
Kyle Pitzen 95e2505de5 feat(engine): Adds structured alias support to the engine
feat(engine): Adds alias lineage calculation to the engine
2022-10-11 17:56:32 -04:00