From SDKs, we call invokes in one of two ways:
* In a "non-output" context (e.g. `getX`), which has a result dependent
on the language (e.g. a `Promise` in NodeJS) that _does not_ track
dependencies.
* In an "output" context (e.g. `getXOutput`), which has an `Output` type
and does track dependencies.
In the non-output case, `dependsOn` really doesn't make sense, since
this style of invoke is inherently ignoring dependency tracking/outputs.
This commit thus reverts 492c57c7dd so
that we can rethink the design before people's programs are subtly
broken in this case.
This commit adds support for passing `dependsOn` to invokes (whether
streamed or not) in the NodeJS SDK. This allows programs to ensure that
certain invokes are executed after things they depend on, even if that
dependency is not implicitly captured with an input-output relationship.
Part of #14243
As @denbezrukov notes in #13885, Rome (https://github.com/rome/tools,
the JavaScript toolchain we have been using to format and lint code in
the NodeJS SDK) has been deprecated. Biome (https://biomejs.dev/) has
sprung up in its place as a community fork and appears to be the best
bet for migration going forward. This commit introduces Biome, ports the
bits of configuration that need changing and updates formatting
accordingly.
Closes#13885
Co-authored-by: Denis Bezrukov <6227442+denbezrukov@users.noreply.github.com>
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
Fixes https://github.com/pulumi/pulumi/issues/15733
Historically these packages were direct dependencies of
`@pulumi/pulumi`. To decouple the node SDK from the precise version of
TypeScript, the packages are now declared as optional peer pependencies
of `@pulumi/pulumi` and customers can pick the versions they want.
The reason we mark the peer dependencies as *optional* is to prevent
package managers from automatically installing them. This avoids the
situation where the package manger would install a more recent version
of TypeScript without the user explictly opting in. Newer versions have
stricter type checks, and can thus stop existing programs from running
successfully.
When the peer dependencies are not present, we load the vendored
versions of the modules.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
Builds on top of https://github.com/pulumi/pulumi/pull/15753
Fixes https://github.com/pulumi/pulumi/issues/15735
There are a couple breaking changes in the typescript API that we use in
`sdk/nodejs/runtime/closure/rewriteSuper.ts`. This PR adds a shim that
is used to bridge the differences and versions the snapshots where
needed.
This does not make typescript a peer dependency yet. Instead the tests
force a specific version to be used via [yarn
resolutions](https://classic.yarnpkg.com/lang/en/docs/selective-version-resolutions/).
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
The tests using `mockpackage` really should be integration tests. We
worked around this by including `mockpackage` in our dependencies, but
this has required some hackery in the build step to remove it again
there.
Now that we have the closure tests setup as integration tests, we can
add the tests using mockpackage there.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
In preparation of https://github.com/pulumi/pulumi/issues/15735 we make
the closure tests proper integration tests so that we can run them with
different typescript versions. Move each test to its own folder instead
of one large file.
This PR only changes tests, and does not touch any of the function
serialisation code.
Some snapshots had to be updated for indentation changes.
Hidden after all the test cases is the test script
[sdk/nodejs/tests/runtime/testdata/closure-tests/test.ts](dfcc953c08/sdk/nodejs/tests/runtime/testdata/closure-tests/test.ts)
Verified that tests run in CI
https://github.com/pulumi/pulumi/actions/runs/8389170587/job/22975068167?pr=15753
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
When using `tsc` to precompile typescript in a monorepo, we need to work
relative to the location of `package.json`, not where the pulumi program
lives (which is usually nested further down).
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
read-package-tree is deprecated. Additionally it has a dependency that
is flagged by security scanners.
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes https://github.com/pulumi/pulumi/issues/9129
Ref https://github.com/pulumi/pulumi/issues/12688
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
Make `pulumi install` detect npm and yarn workspaces setups and
successfully install dependencies.
Fixes#15435
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
NodeJS equivalent of the StackReference.GetOutputDetails
method and accompanying type added to the Go SDK in #12034.
This will allow users of the TypeScript and JavaScript SDKs
to fetch outputs from stack references directly--without going through
an Output type.
Couple notes about testing:
- `MockMonitor.readResource` kept exploding because the getCustom method
was missing on the provided object.
I didn't find any examples in the Node SDK
of using mocks to test StackReferences,
so I'm guessing this was an unexercised code path.
I've fixed that.
- It seems that the JavaScript SDK promotes an entire map to secret
if an item inside it is a secret.
So I had to isolate the two test cases into separate outputs
to get the plain text case to be written as a "value".
If there's a more correct way of setting up that mock,
I'm happy to merge the outputs back into a single map
for a more representative test case.
Refs #10839, #5035
previously, when both index.ts and index.js were present
and no main key was provided in Pulumi.yaml, Pulumi would quietly
resolve the entrypoint as index.js. This messages that decision in a warning
* Make tsconfig.json valid JSON
* Include invoke.ts in tsconfig.files
.. and consequently, the type therein `InvokeOptions`, in generated
docs. All functions in Pulumi SDKs take an InvokeOptions-typed argument,
and all the generated documentation for those functions has a broken
link because it's not part of the NodeJS SDK documentation generated for
pulumi/pulumi.
The version of TypeDoc used is quite old, and I suspect it consults
`tsconfig.json` to find the source files (evidence: someone complaining
that it does not do this well enough:
https://github.com/TypeStrong/typedoc/issues/1515). I have
conservatively just added the file in question to `files`, since:
- I'm not sure if .files should list _all_ the files
- Upgrading TypeDoc would surely have many unintended consequences
* Added support for complying with nodejs package.json exports field if provided.
Co-authored-by: Kyle Dixler <kyle@pulumi.com>
Co-authored-by: Ian Wahbe <ian@wahbe.com>
These changes add support for gathering code coverage data during tests.
For tests that do not involve the Pulumi CLI, this is straightforward: all of
the ecosystems we target already support gathering coverage data, and we follow
the rules accordingly. Support for each language is broken out into its own
commit.
For tests that do involve the Pulumi CLI, the picture is a bit more complicated.
Go does not make it trivial to perform a coverage-instrumented build (go build
does not have a -cover flag, for example). In lieu of official support, we abuse
go test -c and TestMain to produce a build of the CLI that supports collecting
and reporting coverage data.
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
These changes add support for mocking the resource monitor to the NodeJS
and Python SDKs. The proposed mock interface is a simplified version of
the standard resource monitor that allows an end-user to replace the
usual implementations of ReadResource/RegisterResource and Invoke with
their own. This can be used in unit tests to allow for precise control
of resource outputs and invoke results.
A user who calls `StreamInvoke` probably expects the `AsyncIterable`
that is returned to gracefully terminate. This is currently not the
case.
Where does something like this go wrong? A better question might be
where any of this went right, because several days later, after
wandering into civilization from the great Wilderness of Bugs, I must
confess that I've forgotten if any of it had.
`AsyncIterable` is a pull-based API. `for await (...)` will continuously
call `next` ("pull") on the underlying `AsyncIterator` until the
iterable is exhausted. But, gRPC's streaming-return API is _push_ based.
That is to say, when a streaming RPC is called, data is provided by
callback on the stream object, like:
call.on("data", (thing: any) => {... do thing ...});
Our goal in `StreamInvoke` is to convert the push-based gRPC routines
into the pull-based `AsyncIterable` retrun type. You may remember your
CS theory this is one of those annoying "fundamental mismatches" in
abstraction. So we're off to a good start.
Until this point, we've depended on a library,
`callback-to-async-iterator` to handle the details of being this bridge.
Our trusting nature and innocent charm has mislead us. This library is
not worthy of our trust. Instead of doing what we'd like it to do, it
returns (in our case) an `AsyncIterable` that will never complete.
Yes,, this `AsyncIterable` will patiently wait for eternity, which
honestly is kind of poetic when you sit down in a nice bath and think
about that fun time you considered eating your computer instead of
finishing this idiotic bug.
Indeed, this is the sort of bug that you wonder where it even comes
from. Our query libraries? Why aren't these `finally` blocks executing?
Is our language host terminating early? Is gRPC angry at me, and just
passive-aggrssively not servicing some of my requests? Oh god I've been
up for 48 hours, why is that wallpaper starting to move? And by the way,
a fun interlude to take in an otherwise very productive week is to try
to understand the gRPC streaming node client, which is code-gen'd, but
which also takes the liberty of generating itself at runtime, so that
gRPC is code-gen'ing a code-gen routine, which makes the whole thing
un-introspectable, un-debuggable, and un-knowable. That's fine, I didn't
need to understand any of this anyway, thanks friends.
But we've come out the other side knowing that the weak link in this
very sorry chain of incredibly weak links, is this dependency.
This commit removes this dependency for a better monster: the one we
know.
It is at this time that I'd like to announce that I am quitting my job
at Pulumi. I thank you all for the good times, but mostly, for taking
this code over for me.
This command will cause `pulumi policy publish` to behave in much the
same way `pulumi up` does -- if the policy program is in TypeScript, we
will use ts-node to attempt to compile in-process before executing, and
fall back to plain-old node.
We accomplish this by moving `cmd/run/run.ts` into a generic helper
package, `runtime/run.ts`, which slightly generalizes the use cases
supported (notably, allowing us to exec some program outside of the
context of a Pulumi stack).
This new package is then called by both `cmd/run/index.ts` and
`cmd/run-policy-pack/index.ts`.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
* Enable unit testing for Pulumi programs
This change enables rudimentary unit testing of your Pulumi programs, by introducing a `PULUMI_TEST_MODE` envvar that, when set, allows programs to run without a CLI. That includes
* Just being able to import your Pulumi modules, and test ordinary functions -- which otherwise would have often accidentally triggered the "Not Running in a CLI" error message
* Being able to verify a subset of resource properties and shapes, with the caveat that outputs are not included, due to the fact that this is a perpetual "dry run" without any engine operations occurring
In principle, this also means you can attach a debugger and step through your code.
* Finish the unit testing features
This change
1) Incorporates CR feedback, namely requiring that test mode be
explicitly enabled for any of this to work.
2) Implements Python support for the same capabilities.
3) Includes tests for both JavaScript and Python SDKs.
* Add a note on unit testing to the CHANGELOG
* Use Node 8 friendly assert API
* Embellish the CHANGELOG entry a bit