Commit Graph

58 Commits

Author SHA1 Message Date
Fraser Waters 8588aa365c
Lift context parameter to SerializeDeployment/Resource/Operations/Properties ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

SerializePropertyValue needed a `context.Context` object to pass to the
`config.Encrypter`. It was using `context.TODO()`, this change instead
accepts a context on the parameters and lifts that up to
SerializeProperties, SerializeResource, SerializeOperation, and
SerializeDeployment.

There were a few call sites for those methods that already had a context
on hand, and they now pass that context. The other calls sites now use
`context.TODO()`, we should continue to iterate in this area to ensure
everywhere that needs a context has one passed in.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-15 07:45:46 +00:00
Thomas Gummerer 43322f276a
fix panic on deserializing deployment ()
When a serialized deployment doesn't include a secrets provider
configuration, but does include ciphertexts, currently we end up with a
panic. Error out earlier if this is the case to avoid the panic.

This fixes the panic seen in
https://github.com/pulumi/pulumi/issues/15547 and
https://github.com/pulumi/pulumi/issues/14761, but it doesn't quite
explain why this is happening in the first place. I asked for some more
info from the users in these issues for that.

Putting this up as PR anyway in case anyone has any idea of why this
could be happening in the first place. I've tried spelunking through the
code, but nothing obvious stood out. It is possible that these were
still v2 snapshots that included no secrets manager, but I would be
somewhat surprised if those still existed in the wild.

Another potential solution here would be to try to pass the secret
manager from the config in to the DeserializeDeployment function, so we
could use that in these cases. It might not always be correct though, so
I'm not sure it's the right thing to do.

Thoughts?

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-03-07 09:31:57 +00:00
Fraser Waters 4c1471ec06
Don't encrypt secrets to get ciphertext that isn't actually used ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

While investigating into https://github.com/pulumi/pulumi/issues/15293 I
noticed that `SerializePropertyValue` was always calling into
`Encrypter.EncryptValue` (or `cachingCrypter.encryptSecret`) even when
`showSecrets` was true and the cipher text wasn't being returned.

This is a simple move of that logic into the `!showSecrets` branch so it
gets skipped entirely when the cipher text isn't needed.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-30 09:51:51 +00:00
Ian Wahbe 9a06694676
Move assets and archives to their own package ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR is motivated by https://github.com/pulumi/pulumi/pull/15145.

`resource.*` should be built on top of `property.Value`,[^1] which means
that `resource`
needs to be able to import `property.Value`, and so `property` cannot
import
`resource`. Since Assets and Archives are both types of properties, they
must be moved out
of `resource`.

[^1]: For example:
a1d686227c/sdk/go/common/resource/resource_state.go (L35-L36)

## Open Question

This PR moves them to their own sub-folders in `resource`. Should
`asset` and `archive`
live somewhere more high level, like `sdk/go/property/{asset,archive}`?

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-25 20:39:31 +00:00
Fraser Waters 01dd48ed4a
Add Base64SecretsProvider for testing ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
To support plugin based secret providers like (e.g. AGE
https://github.com/pulumi/pulumi/issues/11493) the default secrets
provider will need to become more complicated, requiring access to a
plugin host at least.

As such for tests it will be simpler to have a basic secrets provider
that only supports base64 and can be created without any setup.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-05 08:32:56 +00:00
Fraser Waters 3560333ae6
Clean up uses of .Error() ()
Combination of a few cleanups.

1. Don't call .Error() on errors that are being passed to "%s" format
functions. Format will call `Error()` itself.
2. Don't call assert.Error then assert.Equal/Contains, just use
assert.ErrorEqual/ErrorContains instead.
3. Use "%w" if appropriate, instead of "%v"/"%s".
2023-12-20 15:54:06 +00:00
Fraser Waters 516979770f
Allow anything in resource names ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/13968.
Fixes https://github.com/pulumi/pulumi/issues/8949.

This requires changing the parsing of URN's slightly, it is _very_
likely that
providers will need to update to handle URNs like this correctly.

This changes resource names to be `string` not `QName`. We never
validated this before and it turns out that users have put all manner of
text for resource names so we just updating the system to correctly
reflect that.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-20 08:59:00 +00:00
Pat Gavlin 948bb36e7e [engine] Add support for source positions
These changes add support for passing source position information in
gRPC metadata and recording the source position that corresponds to a
resource registration in the statefile.

Enabling source position information in the resource model can provide
substantial benefits, including but not limited to:

- Better errors from the Pulumi CLI
- Go-to-defintion for resources in state
- Editor integration for errors, etc. from `pulumi preview`

Source positions are (file, line) or (file, line, column) tuples
represented as URIs. The line and column are stored in the fragment
portion of the URI as "line(,column)?". The scheme of the URI and the
form of its path component depends on the context in which it is
generated or used:

- During an active update, the URI's scheme is `file` and paths are
  absolute filesystem paths. This allows consumers to easily access
  arbitrary files that are available on the host.
- In a statefile, the URI's scheme is `project` and paths are relative
  to the project root. This allows consumers to resolve source positions
  relative to the project file in different contexts irrespective of the
  location of the project itself (e.g. given a project-relative path and
  the URL of the project's root on GitHub, one can build a GitHub URL for
  the source position).

During an update, source position information may be attached to gRPC
calls as "source-position" metadata. This allows arbitrary calls to be
associated with source positions without changes to their protobuf
payloads. Modifying the protobuf payloads is also a viable approach, but
is somewhat more invasive than attaching metadata, and requires changes
to every call signature.

Source positions should reflect the position in user code that initiated
a resource model operation (e.g. the source position passed with
`RegisterResource` for `pet` in the example above should be the source
position in `index.ts`, _not_ the source position in the Pulumi SDK). In
general, the Pulumi SDK should be able to infer the source position of
the resource registration, as the relationship between a resource
registration and its corresponding user code should be static per SDK.

Source positions in state files will be stored as a new `registeredAt`
property on each resource. This property is optional.
2023-07-10 14:35:40 -07:00
Bryce Lampe 7e0dd3f3c2
This commit adds the `Created` and `Modified` timestamps to pulumi state that are optional.
`Created`: Created tracks when the remote resource was first added to state by pulumi. Checkpoints prior to early 2023 do not include this. (Create, Import)
`Modified`: Modified tracks when the resource state was last altered. Checkpoints prior to early 2023 do not include this. (Create, Import, Read, Refresh, Update)

When serialized they will follow RFC3339 with nanoseconds captured by a test case.
https://pkg.go.dev/time#RFC3339

Note: Older versions of pulumi may strip these fields when modifying the state.

For future expansion, when we inevitably need to track other timestamps, we'll add a new "operationTimestamps" field (or something similarly named that clarified these are timestamps of the actual Pulumi operations).

	operationTimestamps: {
		created: ...,
		updated: ...,
		imported: ...,
	}

Fixes https://github.com/pulumi/pulumi/issues/12022
2023-03-27 09:28:26 -07:00
Abhinav Gupta 5e346dfac9
deps: Upgrade to pgregory.net/rapid v0.5
The 0.5 release of rapid exposes a generics-based API
instead of `interface{}` and casting everywhere.
This makes for much cleaner usage.

There are a handful of cases where strongly typed generators,
e.g. `Generator[bool]`, need to be turned into `interface{}` (`any`),
which is doable with `AsAny()`.

API changes:
The only non-test changes to the SDK package are in
go/common/resource/testing, which contains testing utiltiies.
Functions that previously returned the old, untyped `Generator`
now return a strongly typed `Generator[T]`.
2023-02-16 12:04:07 -08:00
Abhinav Gupta 0bff0b8716 sdk/go: Remove 'nolint' directives from package docs
Go treats comments that match the following regex as directives.

    //[a-z0-9]+:[a-z0-9]

Comments that are directives don't show in an entity's documentation.
5a550b6951 (diff-f56160fd9fcea272966a8a1d692ad9f49206fdd8dbcbfe384865a98cd9bc2749R165)

Our code has `//nolint` directives that now show in the API Reference.
This is because these directives are in one of the following forms,
which don't get this special treatment.

    // nolint:foo
    //nolint: foo

This change fixes all such directives found by the regex:
`// nolint|//nolint: `.
See bottom of commit for command used for the fix.

Verification:
Here's the output of `go doc` on some entities
before and after this change.

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

nolint: lll, interfacer

nolint: lll, interfacer

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
```

After
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi | head -n8
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

const EnvOrganization = "PULUMI_ORGANIZATION" ...
var ErrPlugins = errors.New("pulumi: plugins requested")
func BoolRef(v bool) *bool
func Float64Ref(v float64) *float64
func IntRef(v int) *int
func IsSecret(o Output) bool
```

Before
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine. nolint: revive
```

After:
```
% go doc github.com/pulumi/pulumi/sdk/v3/go/pulumi URN_
package pulumi // import "github.com/pulumi/pulumi/sdk/v3/go/pulumi"

func URN_(o string) ResourceOption
    URN_ is an optional URN of a previously-registered resource of this type to
    read from the engine.
```

Note that golangci-lint offers a 'nolintlint'  linter
that finds such miuses of nolint,
but it also finds other issues so I've deferred that to a follow up PR.

Resolves 

Related: https://github.com/golangci/golangci-lint/issues/892

[git-generate]
FILES=$(mktemp)
rg -l '// nolint|//nolint: ' |
  tee "$FILES" |
  xargs perl -p -i -e '
    s|// nolint|//nolint|g;
    s|//nolint: |//nolint:|g;
  '
rg '.go$' < "$FILES" | xargs gofmt -w -s
2023-01-06 09:06:47 -08:00
Sam Eiderman 4bbe365f15 Add DeletedWith resource option
In many cases there is no need to delete resources if the container
resource is going to be deleted as well.

A few examples:
 * Database object (roles, tables) when database is being deleted
 * Cloud IAM bindings when user itself is being deleted

This helps with:
 * Speeding the deletion process
 * Removing unnecessary calls to providers
 * Avoiding failed deletions when the pulumi user running the
   plan has access to the container resource but not the contained
   ones

To avoid deleting contained resources, set the `DeletedWith` resource
option to the container resource.

TODO:
 Should we support DeletedWith with PendingDeletes?
 Special case might be when the contained resource is marked as pending
 deletion but we now want to delete the container resource, so
 ultimately there is no need to delete the contained anymore
2022-10-31 12:03:18 +02:00
Fraser Waters 9dfbee3b2d
Remove sequence numbers ()
* Remove sequenceNumber from protobufs

* Regenerate protobufs

* Remove setting and reading of sequence number in Check

* Remove sequence numbers from state

* Replace sequenceNumber with randomSeed in Check

* Fix tests

* Add to CHANGELOG
2022-07-25 12:08:03 +01:00
Anton Tayanovskyy 96a3783fb2
Fix orphan bulk decrypt traces ()
* Propagate context through crypter to correctly parent service crypter tracing spans

* Remove one of the context.TODO that is needed

* Fix new_test.go

* Fix pulumi_test.go

* Fix resources_test.go

* Fix pkg/resource/stack tests

* Add CHANGELOG_PENDING

* Fix value_test.go

* Fix stack_test.go

* Feedback
2022-07-18 09:36:31 -04:00
Fraser Waters 859052d6d9
Revert "Strip Aliases from state ()" ()
This reverts commit 17068e9b49.

Turns out NormalizeURNReferences needs this in the state to fix up URNs while the deployment is running. It feels like we should be able to either thread this information through to the snapshot manager another way but it's not obvious how. It's also tricky to test because snapshot code differs massively in unit tests compared to proper runs.
2022-03-24 20:08:18 +01:00
Fraser Waters 17068e9b49
Strip Aliases from state ()
* Strip Aliases from state

* chore: note fix, reason for change in changelog

* remove redundant test

see: ed2923653c/pkg/engine/lifeycletest/step_generator_test.go (L16)

Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2022-03-23 17:55:06 -07:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Fraser Waters 5d91f8f322
Add RetainOnDelete resource option ()
* Plumb in basics of retainOnDelete

* Add test

* Make test pass

* Add to changelog

* Add to API list

* lint

* Add semicolon

* Fix Infof call

* Fix method call

* new delete mode work

* cleanup

* protectTest

* Fix up test

* Fix replace

* Fix up test

* Warn on drop

* lint

* Change to just a bool flag

* Regenerate proto

* Rework to just a bool flag with no error

* Remove old comment

* Fix C# typo

* rm extra space

* Add missing semicolon

* Reformat python

* False typo

* Fix typo in js function name

* Reword docs

* lint

* Read doesn't need retainOnDelete
2022-02-16 22:11:12 +00:00
Paul Stack e3720b3a93
Using a decryptAll functionality when deserializing a deployment () 2022-01-24 22:33:40 +02:00
Fraser Waters ec3ef44841
Make resource autonames determinstic ()
* Start adding SequenceNumber

* Start adding sequence number to state

* New generate functions

* notes

* Don't increment if unknown

* Deterministic name test

* Check replace

* typo

* lint

* Increment on targetted replace

* Some comments and external fixes

* Add test for resetting sequence number after replace

* Reset sequence numbers after replace

* assert check we never pass -1 to check

* Add to dynamic providers

* lint

* Add to changelog
2022-01-20 11:18:54 +00:00
Pat Gavlin 134d7cb818
[apitype] Add a JSON schema for deployments. ()
This schema can be used to validate the contents of a Pulumi deployment.
If a deployment validates against this schema, it should be considered
syntactically valid, though it may contain certain classes of semantic
errors (e.g. references to unknown resources in dependency lists,
dependency cycles, etc.).

This schema is not yet used for validation in practice and may not be
entirely accurate.

These changes also add this schema (and the schemas on which it depends)
to the developer documentation. jsonschema2md.go has been updated to
support multi-file schemas.
2021-09-21 21:37:06 -07:00
Pat Gavlin cbdecf2cd5
[testing] Add rapid generators for PropertyValues. ()
And use those generators to test property value serialization and
deserialization paths.
2021-09-21 15:02:10 -07:00
Luke Hoban 8587f5410e
Error instead of assert on invalid resource in state file ()
* Error instead of assert on invalid resource in state file

Fixes 

* Add CHANGELOG
2021-05-17 09:47:28 +01:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Levi Blackstone f6ec691544
Fix malformed resource value bug ()
* Fix resource-ref-as-ID marshaling. ()

This reapplies 2f0dba23ab.

* Fix malformed resource value bug

PR  introduced a bug by marshaling resource
ids as PropertyValues, but not handling that case on
the unmarshaling side. The previous code assumed
that the id was a simple string value. This bug prevents
any stack update operations (preview, update, destroy,
refresh). Since this change was already
released, we must now handle both cases in the
unmarshaling code.

* Add resource ref unit tests for the Go SDK. ()

This reapplies 3d505912b8.

Co-authored-by: Pat Gavlin <pat@pulumi.com>
2021-01-21 16:40:27 -07:00
Paul Stack 48f906e026
Allow pulumi stack export to decrypt secrets () 2020-05-11 19:16:30 +01:00
Justin Van Patten 7f27618e2d
Avoid replace on second update with import applied ()
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.

It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.

Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.

To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
2020-04-15 18:52:40 -07:00
CyrusNajmabadi 66bd3f4aa8
Breaking changes due to Feature 2.0 work
* Make `async:true` the default for `invoke` calls ()

* Switch away from native grpc impl. ()

* Remove usage of the 'deasync' library from @pulumi/pulumi. ()

* Only retry as long as we get unavailable back.  Anything else continues. ()

* Handle all errors for now. ()


* Do not assume --yes was present when using pulumi in non-interactive mode ()

* Upgrade all paths for sdk and pkg to v2

* Backport C# invoke classes and other recent gen changes ()

Adjust C# generation

* Replace IDeployment with a sealed class ()

Replace IDeployment with a sealed class

* .NET: default to args subtype rather than Args.Empty ()

* Adding system namespace for Dotnet code gen

This is required for using Obsolute attributes for deprecations

```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```

* Fix the nullability of config type properties in C# codegen ()
2020-04-14 09:30:25 +01:00
evanboyle f754b486b8 move pkg/resource/config -> sdk/go/common/resource/config 2020-03-18 15:03:37 -07:00
evanboyle 67cb405c93 move pkg/apitype -> sdk/common/apitype 2020-03-18 15:00:30 -07:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00
evanboyle fba783caf9 move pkg/resource -> sdk/go/common/resource, but leave nested resource packages 2020-03-18 13:36:19 -07:00
Pat Gavlin 0e9c4ed2db
Serialize null property values. ()
Eliding these values prevents us from properly round-tripping resource
states that include null property values.

This is part of the fix for
https://github.com/pulumi/pulumi-azure/issues/383.
2019-11-22 11:03:02 -08:00
Chris Smith 2ff0e6dee3
Persist computed values () 2019-11-21 14:58:30 -08:00
Chris Smith 4e59263a9c
Add tests for serializing PropertyMaps ()
* WIP - Add tests for serializing PropertyMaps

* Apply suggestions from code review

Co-Authored-By: Pat Gavlin <pat@pulumi.com>

* Cleanup tests
2019-11-19 21:10:51 -08:00
Chris Smith 17ee050abe
Refactor the way secrets managers are provided () 2019-08-01 10:33:52 -07:00
Paul Stack 02ffff8840
Addition of Custom Timeouts ()
* Plumbing the custom timeouts from the engine to the providers

* Plumbing the CustomTimeouts through to the engine and adding test to show this

* Change the provider proto to include individual timeouts

* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface

* Change how the CustomTimeouts are sent across RPC

These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest

```
req=&pulumirpc.RegisterResourceRequest{
           Type:                    "aws:s3/bucket:Bucket",
           Name:                    "my-bucket",
           Parent:                  "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
           Custom:                  true,
           Object:                  &structpb.Struct{},
           Protect:                 false,
           Dependencies:            nil,
           Provider:                "",
           PropertyDependencies:    {},
           DeleteBeforeReplace:     false,
           Version:                 "",
           IgnoreChanges:           nil,
           AcceptSecrets:           true,
           AdditionalSecretOutputs: nil,
           Aliases:                 nil,
           CustomTimeouts:          &pulumirpc.RegisterResourceRequest_CustomTimeouts{
               Create:               300,
               Update:               400,
               Delete:               500,
               XXX_NoUnkeyedLiteral: struct {}{},
               XXX_unrecognized:     nil,
               XXX_sizecache:        0,
           },
           XXX_NoUnkeyedLiteral: struct {}{},
           XXX_unrecognized:     nil,
           XXX_sizecache:        0,
       }
```

* Changing the design to use strings

* CHANGELOG entry to include the CustomTimeouts work

* Changing custom timeouts to be passed around the engine as converted value

We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-16 00:26:28 +03:00
Luke Hoban 15e924b5cf
Support aliases for renaming, re-typing, or re-parenting resources ()
Adds a new resource option `aliases` which can be used to rename a resource.  When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.

There are two key places this change is implemented. 

The first is the step generator in the engine.  When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource.  That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order).  This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.

The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs.  This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases.  It is similar to how many other resource options are "inherited" implicitly from the parent.

Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
2019-05-31 23:01:01 -07:00
Matt Ellis 480a2f6c9e Augment secret outputs based on per request options 2019-05-10 17:07:52 -07:00
Matt Ellis cc74ef8471 Encrypt secret values in deployments
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.

The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).

The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.

A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
2019-05-10 17:07:52 -07:00
Pat Gavlin 35c60d61eb
Follow up on ()
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
  layers and language SDKs
2019-01-28 17:38:16 -08:00
Pat Gavlin 1ecdc83a33 Implement more precise delete-before-replace semantics. ()
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.

We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.

This is perhaps clearer when described by example. Consider the
following dependency graph:

  A
__|__
B   C
|  _|_
D  E F

In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.

In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
2019-01-28 09:46:30 -08:00
Pat Gavlin a222705143
Implement first-class providers. ()
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.

### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.

These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
  or unknown properties. This is necessary because existing plugins
  only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
  values are known or "must replace" if any configuration value is
  unknown. The justification for this behavior is given
  [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
  configures the provider plugin if all config values are known. If any
  config value is unknown, the underlying plugin is not configured and
  the provider may only perform `Check`, `Read`, and `Invoke`, all of
  which return empty results. We justify this behavior becuase it is
  only possible during a preview and provides the best experience we
  can manage with the existing gRPC interface.

### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.

All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.

Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.

### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
  provider plugins. It must be possible to add providers from a stack's
  checkpoint to this map and to register new/updated providers during
  the execution of a plan in response to CRUD operations on provider
  resources.
- In order to support updating existing stacks using existing Pulumi
  programs that may not explicitly instantiate providers, the engine
  must be able to manage the "default" providers for each package
  referenced by a checkpoint or Pulumi program. The configuration for
  a "default" provider is taken from the stack's configuration data.

The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").

The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.

During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.

While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.

### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
  to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
  a particular provider instance to manage a `CustomResource`'s CRUD
  operations.
- A new type, `InvokeOptions`, can be used to specify options that
  control the behavior of a call to `pulumi.runtime.invoke`. This type
  includes a `provider` field that is analogous to
  `ResourceOptions.provider`.
2018-08-06 17:50:29 -07:00
Sean Gillespie 48aa5e73f8
Save resources obtained from ".get" in the snapshot ()
* Protobuf changes to record dependencies for read resources

* Add a number of tests for read resources, especially around replacement

* Place read resources in the snapshot with "external" bit set

Fixes . This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.

* Fix an omission of OpReadReplace from the step list

* Rebase against master

* Transition to use V2 Resources by default

* Add a semantic "relinquish" operation to the engine

If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.

* Typo fix

* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
2018-08-03 14:06:00 -07:00
Alex Clemmer f037c7d143 Checkpoint resource initialization errors
When a resource fails to initialize (i.e., it is successfully created,
but fails to transition to a fully-initialized state), and a user
subsequently runs `pulumi update` without changing that resource, our
CLI will fail to warn the user that this resource is not initialized.

This commit begins the process of allowing our CLI to report this by
storing a list of initialization errors in the checkpoint.
2018-07-20 17:59:06 -07:00
Sean Gillespie 924c49d7e0
Fail fast when attempting to load a too-new or too-old deployment ()
* Error when loading a deployment that is not a version that the CLI understands

* Add a test for 'pulumi stack import' on a badly-versioned deployment

* Move current deployment version to 'apitype'

* Rebase against master

* CR: emit CLI-friendly error message at the two points outside of the engine calling 'DeserializeDeployment'
2018-05-25 13:29:59 -07:00
joeduffy 5967259795 Add license headers 2018-05-22 15:02:47 -07:00
Sean Gillespie 55711e4ca3
Revert "Lift snapshot management out of the engine and serialize writes to snapshot ()" ()
This reverts commit 2c479c172d.
2018-04-16 23:04:56 -07:00
Sean Gillespie 2c479c172d
Lift snapshot management out of the engine and serialize writes to snapshot ()
* Lift snapshot management out of the engine

This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.

As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.

Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.

Remove dead code, Steps no longer need a reference to the plan iterator that created them

Fixing various issues that arise when bringing up pulumi-aws

Line length broke the build

Code review: remove dead field, fix yaml name error

Rebase against master, provide implementation of StackPersister for cloud backend

Code review feedback: comments on MutationStatus, style in snapshot.go

Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager

Code review feedback: use a channel for synchronization

Add a comment and a new test

* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots

* Add a lot of tests - covers all of the non-error paths of BeginMutation and End

* Fix a test resource provider

* Add a few tests, fix a few issues

* Rebase against master, fixed merge
2018-04-12 09:55:34 -07:00
Pat Gavlin a23b10a9bf
Update the copyright end date to 2018. ()
Just what it says on the tin.
2018-03-21 12:43:21 -07:00