Commit Graph

26 Commits

Author SHA1 Message Date
Julien 371f7b1bbb
Run Environment.DeleteIfNotFailed after tests complete ()
Ensure we always run `Environment.DeleteIfNotFailed` in our Environment
based tests.

Also standardize on this instead of manually checking with `if t.Failed`
and calling `Environment.DeleteEnvironment()`.
2024-07-23 10:37:01 +00:00
Julien d1a401fbcd
Don't use indentation in converted pyproject.toml files ()
When converting from a requirements.txt to a pyproject.toml, we were
previously indenting each logical level in the file. Disable the
indentation to match the style of pyproject.toml files as generated
directly by Poetry.

Fixes https://github.com/pulumi/pulumi/issues/16657
2024-07-17 07:28:56 +00:00
Julien 99a401a241
Use default values for language specific prompts when using --yes ()
When passing `yes` in interactive mode (or non-interactive mode),
language specific options should use their deafult values without
prompting the user.
2024-07-08 11:00:33 +00:00
Julien P 531036ac07
Provide installation instructions for the python typechecker ()
When the typechecker option is set, but the selected typechecker is not
installed in the project's virtual environment, the pulumi operation
will fail with an unhelpful message like `mypy failed: exit status 1`.

To make this more actionable, we check if the typechecker is installed,
and if not, we provide installation instructions to remediate the issue.
2024-07-01 15:17:03 +00:00
Julien P 98b90f1902
Add packagemanager prompt to pulumi new for nodejs ()
https://github.com/pulumi/pulumi/pull/16346 introduced the capability to
query the language runtime for additional prompts. We use this to let
the user pick a package manager among npm, yarn and pnpm during `pulumi
new` when using the nodejs runtime.

When there is no explicitly configured package manager, we re-use the
previous behaviour for determining the package manager (check
`PULUMI_PREFER_YARN` env variable, look for lock files).

Defaults to `npm` when running `new` in non-interactive mode.
2024-06-21 11:35:06 +00:00
Julien P 6f42fba445
Instruct the user to use 'pulumi install' when using --generate-only ()
When running with `--generate-only`, the user can run `pulumi install`
to install all dependencies. This takes care of creating virtual
environments or node_modules, respecting the options specified in
Pulumi.yaml. This also removes language specific code from the CLI.

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-06-20 10:27:41 +00:00
Julien P a59b694515
Query language runtime for options during “pulumi new” ()
# Description

Fixes https://github.com/pulumi/pulumi/issues/16309

During `pulumi new` we query the language runtime using the new
`RuntimeOptionsPrompts` RPC call to get additional prompts to ask the
user.

<img width="900" alt="Screenshot 2024-06-07 at 14 28 58"
src="https://github.com/pulumi/pulumi/assets/387068/e68ef702-978b-47f7-9d4b-afdf10409ed8">

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!-- av pr metadata
This information is embedded by the av CLI when creating PRs to track
the status of stacks when using Aviator. Please do not delete or edit
this section of the PR.
```
{"parent":"master","parentHead":"","trunk":"master"}
```
-->

---------

Co-authored-by: Will Jones <will@sacharissa.co.uk>
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-06-17 17:10:55 +00:00
Julien P 6cee283a42
Support Poetry as Python toolchain ()
# Description

Implement `Toolchain` interface for Poetry.

This does not yet allow the user to select the toolchain during `pulumi
new`, but can be used with a template that has explicit poetry support
https://github.com/pulumi/templates/pull/795/files

```
pulumi new https://github.com/pulumi/templates/tree/julienp/poetry-template/aws-python-poetry 
```
Stacked on top of https://github.com/pulumi/pulumi/pull/16311

Fixes https://github.com/pulumi/pulumi/issues/16314
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-06 14:16:24 +00:00
Julien P 578e0937a9
[Python] Move existing dependency installation and python command invocation to a Toolchain interface ()
# Description

This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.

Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-03 13:52:27 +00:00
Thomas Gummerer ae8134f5ad
upgrade to latest version of golangci-lint ()
The version we currently have doesn't support Go 1.22 properly, so it
throws a bunch of warnings locally when trying to run it with the latest
Go version installed. Just running the latest version locally also
doesn't quite work, since it throws a bunch of errors from the
perfsprint linter, which seems to have gotten stricter.

Upgrade to the latest version of golangci-lint, and fix all the errors
we're getting from it. Mostly done via `perfsprint -fix`, with some
manual changes that `perfsprint -fix` wouldn't touch.
2024-04-19 06:20:33 +00:00
Fraser Waters a7e0551153
Reenable TestPythonTranslation and delete stack reference tests ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14765.

TestPythonTranslation wasn't flaky in of itself, it was just the
standard venv logic being iffy. No reason to disable this one test.

The stack reference tests haven't ran for ages, taking some advice from
@lunaris and just deleting them all, I've raised
https://github.com/pulumi/pulumi/issues/15932 to rewrite them as
conformance tests.
2024-04-15 12:36:03 +00:00
Fraser Waters 19f67eb163
Add 'typechecker' option to python runtime ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/15724.

This adds a new runtime option to the python language host `typeChecker`
which can be set to either "mypy" or "pyright". If it's set the language
host will run the mypy/pyright module before running the program.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-28 10:41:22 +00:00
Fraser Waters a08b41832a
Make TestConstructComponentConfigureProviderPython less flaky ()
Update TLS library to a more recent version that doesn't have the old
"install the plugin as part of setup.py" logic.

Then enable the engine to do the plugin download for this test by
disabling `PULUMI_DISABLE_AUTOMATIC_PLUGIN_ACQUISITION`. Should make the
test less flaky as the engine is managing the download and can retry,
but at least makes the error much more clear if it does fail.
2024-03-22 15:32:59 +00:00
Thomas Gummerer 697c8216ca
acquire plugin in python acceptance test ()
This test was disabled on MacOS due to what looks like a failure to
acquire a plugin. This probably worked accidentally (and is working
accidentally on other platforms) because presumably the plugin gets
acquired by another test and is thus available. We should however not
rely on that, end enable plugin acquisition where necessary. With that
we also fix the test on MacOS and can unskip it.

This addresses part of https://github.com/pulumi/pulumi/issues/15240.
Note that I don't think it fixes the flakyness, looking back through the
test logs I think I was conflating two separate things here.
2024-03-01 18:01:20 +00:00
Justin Van Patten 0211ab4a9f
Prepare for v3.106.0 release ()
### Features

- [components] Component providers can return output values in their
response to the engine.

- [programgen/{dotnet,go,nodejs,python}] Emit missing trivia for
resources and local variables
  [](https://github.com/pulumi/pulumi/pull/15419)


### Bug Fixes

- [engine] Engine no longer assumes program dependencies are specified
as semantic versions.
  [](https://github.com/pulumi/pulumi/pull/15409)

- [components/go] Component providers now correctly return output values
to the engine.
  [](https://github.com/pulumi/pulumi/pull/15405)


### Miscellaneous

- [protobuf] Split CallRequest into ResourceCallRequest for the
ResourceMonitor service.
  [](https://github.com/pulumi/pulumi/pull/15404)

- [backend/diy] Add a warning that non-project mode will be deprecated
at some point.
  [](https://github.com/pulumi/pulumi/pull/15411)
2024-02-15 06:06:26 +00:00
Justin Van Patten d446736e59
Unskip tests ()
- Unskip Python tests that needed to be skipped until v3.103.0 of the
`pulumi` PyPi package was released with support for Python 3.12.
- Unskip Node.js test that needed to be skipped until v3.103.1 of the
`@pulumi/pulumi` NPM package was published with a fix for the semver
type issue.

Fixes 
2024-01-25 19:24:22 +00:00
Justin Van Patten 7309681b5b
Support Python 3.12 ()
Python 3.12 requires `grpcio` 1.59.0 or higher. Unfortunately, there is
a regression in `grpcio` 1.58.0 through the latest version (currently
1.60.0) which causes any error returned from a Python gRPC server to be
written to stderr, including UNIMPLEMENTED errors. This primarily
affects Python dynamic providers, which don't have implementations for
`CheckConfig` and `DiffConfig`, resulting in a traceback error being
emitted to stderr when the engine calls these, which is visible to
users. This `grpcio` regression has been fixed upstream, but the fix has
not been released yet. We've been waiting for a 1.60.1 patch release.

This has not been great for our Python users who are using Python 3.12.
It's particularly bad for new Pulumi users who are using Python 3.12 and
are trying to get started with Pulumi. For these users, when trying to
install the `pulumi` PyPi package (i.e. via `pulumi new python`) the
installation fails with an error because it is pinned to depending on an
older version of `grpcio` which doesn't work on Python 3.12.

This commit works around the problem by providing default
implementations of `CheckConfig` and `DiffConfig` for python dynamic
providers and the component provider API, so that no error is emitted to
stderr when the engine calls these methods. The default implementations
for these are the same behavior that the engine would use if these
methods had returned UNIMPLEMENTED. I believe these are the only two
methods affected by this. Other methods like `Invoke`, `Call`,
`StreamInvoke`, `Construct`, `Attach`, `GetMapping`, and `GetMappings`,
continue to return UNIMPLEMENTED for dynamic providers, which I think is
OK; I don't believe these will be called by the engine under normal
circumstances.

Out of an abundance of caution, the `pulumi` package continues to depend
on the pinned version of `grpcio` when installing on versions of Python
less than 3.12. On Python 3.12 or greater, we now depend on `grpcio`
`~=1.60.0`. 1.60.0 doesn't have the fix for the regression, but the
workaround should allow things to work on Python 3.12 as before.

Once 1.60.1 is released, we can look into updating the `grpcio`
dependency to `~=1.60.1` for all versions of Python, and possibly revert
the workarounds, if we want.

Note:  added a test for dynamic providers to ensure nothing is
written to stderr. The test would fail if the workaround in this PR did
not work as intended:
https://github.com/pulumi/pulumi/pull/14474/files#diff-d92ccd283e08eadab2597825103e45cdaa96fea93324bc4d4d3b1d2b83c51b76

This PR depends on several other smaller PRs:
- https://github.com/pulumi/pulumi/pull/15220
- https://github.com/pulumi/pulumi/pull/15221
- https://github.com/pulumi/pulumi/pull/15222
- https://github.com/pulumi/pulumi/pull/15223
- https://github.com/pulumi/pulumi/pull/15224
- https://github.com/pulumi/pulumi/pull/15225
- https://github.com/pulumi/pulumi/pull/15226

Fixes 
2024-01-24 22:24:34 +00:00
Thomas Kappler 967f92b566
Don't swallow file not found in fsutil.FileCopy ()
# Description

`FileCopy` silently swallows file not found errors which leads to hard
to debug issues. For instance, as described in , tests with edit
steps unexpectedly run twice against the first step.

Fixes 
Fixes  

Merging this will break plenty of existing tests if they're not fixed
first. [This
query](https://github.com/search?q=org%3Apulumi+path%3A_test.go+%2F%5E%5Ct*Dir%3A+*%22%2F&type=code)
is a rough indication.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-12-06 13:29:40 +00:00
Fraser Waters d4c913aa34
Fix remaining lint issues ()
All paralleltest issues, also turns on the linter for these files now
they lint ok.
2023-12-02 17:16:09 +00:00
Fraser Waters 075a03ae9e
Actually save root on plugin.Context ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Pretty sure this has been broken since forever.

Pretty sure it's only _mostly_ worked because we nearly always run
pulumi from the directory with the Pulumi.yaml file and there's a bit of
code in `buildArgsForNewPlugin` that calls `filepath.Abs(root)` before
passing it to the language plugins. `Abs("")` will just resolve to the
working directory, which as above is nearly always the one with the
Pulumi.yaml file in it.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-29 14:30:52 +00:00
Fraser Waters 762fcee66d
Fix some more lint issues ()
Again same as https://github.com/pulumi/pulumi/pull/14644, just fixing
some issues that make lint sees once you enable build tags.
2023-11-28 16:43:48 +00:00
Anton Tayanovskyy ba039c20f8
Support returning plain values from methods ()
Support returning plain values from methods.

Implements Node, Python and Go support.

Remaining:

- [x] test receiving unknowns
- [x] acceptance tests written and passing locally for Node, Python, Go
clients against a Go server
- [x] acceptance tests passing in CI
- [x] tickets filed for remaining languages
   - [x] https://github.com/pulumi/pulumi-yaml/issues/499
   - [x] https://github.com/pulumi/pulumi-java/issues/1193
   - [x] https://github.com/pulumi/pulumi-dotnet/issues/170 

Known limitations:

- this is technically a breaking change in case there is code out there
that already uses methods that return Plain: true

- struct-wrapping limitation: the provider for the component resource
needs to still wrap the plain-returning Method response with a 1-arg
struct; by convention the field is named "res", and this is how it
travels through the plumbing

- resources cannot return plain values yet

- the provider for the component resource cannot have unknown
configuration, if it does, the methods will not be called
- Per Luke https://github.com/pulumi/pulumi/issues/11520 this might not
be supported/realizable yet

<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/12709

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-11-18 06:02:06 +00:00
Justin Van Patten 3d78b037f4
Fix traceback diagnostic from being printed when using Python dynamic providers ()
 upgraded grpcio to 1.59.2, to support Python 3.12. Unfortunately,
grpcio >=1.58.0 has a problem where calls to unimplemented gRPC methods
cause a traceback to be emitted to the server's stderr, which affects
Python dynamic providers, which don't implement `DiffConfig`. The result
is a traceback diagnostic shown for the Pulumi program using the dynamic
provider.

This PR fixes this by reverting the change, downgrading grpcio back to
1.56.2.

The existing Python dynamic provider diagnostic test has been extended
to check to make sure there are no unexpected diagnostics, which fails
before the revert, and passes after.

An upstream issue has been opened:
https://github.com/grpc/grpc/issues/34853

The issue to support Python 3.12 has been re-opened:  (currently
blocked on the upstream issue)

Fixes 
2023-11-01 22:19:37 +00:00
Fraser Waters 4baf4a1468 Always mark dynamic provider code as secret 2023-07-06 13:50:59 +01:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Robbie McKinstry a96a69db28
Rename "Smoke" test to "Acceptance" tests
After internal discussion, we determined "smoke" is a misleading
adjective for this category of tests. What we called "smoke tests"
are short integration tests for basic cross-platform functionality.
As a result, these are better named "acceptance" tests, since smoke
tests are intended to be a low water mark at the unit level to sniff
out bigger issues with the build as a whole.
2023-01-30 15:38:37 -05:00