# Description
This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.
Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
The Python language host determines which Pulumi plugins are required by
the program by calling out to `python -m pip list -v -format json` to
get the list of installed packages. If a package contains a
`pulumi-plugin.json` file that indicates it is for a resource plugin,
then that package is considered a Pulumi package that has an associated
plugin.
As a fallback for older packages that do not have a `pulumi-plugin.json`
file, if the package name is prefixed with `pulumi-`, it is also
considered to have an associated Pulumi plugin.
However, this fallback no longer works when using recent versions of
setuptools because of a behavior change in v69.0.3 and later of
setuptools. Previously, underscores were replaced by dashes. So `python
-m pip list --format json` would return the name of the package as:
```
{"name": "pulumi-cloudinit", "version": "1.3.0"}
```
But now with setuptools v69.0.3 and later, it returns the name as
pulumi_cloudinit (with an underscore rather than dash):
```
{"name": "pulumi_cloudinit", "version": "1.3.0"}
```
The underscore in the name is now preserved and the name is no longer
prefixed with pulumi-, so the langhost no longer considers it to have an
associated Pulumi plugin.
This commit fixes the fallback to consider packages prefixed by either
`pulumi_` or `pulumi-` to be Pulumi packages.
Fixes#15536
In preparation for supporting Python 3.12...
The test is installing `pulumi-random` which currently depends on
`pulumi`, which will fail on Python 3.12 because the currently released
version of `pulumi` is pinned to a version of `grpcio` that doesn't work
on Python 3.12.
Instead of depending on the public `pulumi` package, install the locally
built Python SDK. Then install `pulumi-random`. That way, it'll be using
a version of `grpcio` that works with Python 3.12.
When creating a PyPi package, Python setuptools prior to v69 would
record the name of a package in egg_info by replacing underscores with
dashes. So after installing `pulumi-random==4.15.0`, `pip list` would
report the package name as `pulumi-random`.
However, setuptools >= 69 now retains the underscore in egg_info
(https://github.com/pypa/setuptools/pull/4159). So after installing the
latest version of `pulumi-random==4.15.1`, `pip list` now reports the
package name as `pulumi_random` instead of `pulumi-random`.
We use pip to list installed packages so we can determine which ones are
Pulumi packages for `GetRequiredPlugins`.
It turns out that this change is largely benign for us. We don't really
care if the reported name is `pulumi-random` or `pulumi_random`, because
we will replace dashes with underscores in the returned name anyway,
because the location the package is installed on disk is going to have
the underscore, and it's in this location where we look for
`pulumi-plugin.json`.
8bcef51fb8/sdk/python/cmd/pulumi-language-python/main.go (L393-L398)
All that's required is updating the test's expected value, which is now
`pulumi_random` as of 4.15.1 of that package.
Fixes#15192
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
This is a pass over all of /sdk to replace asserts that just checked we
had an error with asserts for what the error value is.
Just checking for an error is a weak test that can result in error paths
being broken and tests not detecting it.
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
* about error string: error string should not be capitalized or end with punctuation mark
* apply suggestions from code review
* lint: fix format error and simplify the code
Co-authored-by: Fraser Waters <frassle@gmail.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
* Respond to SIGINT
With the current state of the PR, crash on SIGINT. This is progress.
* Don't crash responding to SIGINT
* Close on cancel instead of terminate
* CL
* Fix lint
* Add ctx for node
* Be consistent for test context
* Add coverage folder
* Adjust GHA to run Test target on Windows
* Extend error message on packages not found
* Try with path normalizer
* Edit Makefiles
* Skip SDK codegen tests on Windows
* Skip program gen tests on Windows
* Skip tests that started running on Windows and failing
* Fix non-compiling test
* Skip failing Windows HCL2 tests
* Fix lint
* Merged constants
* Skip one more test failing on Windows
* Disable cov-enabled builds on Windows
* Fix TestDeterminePulumiPackages on Windows
* Fix TestInstallCleansOldFiles test on Windows
* Fix TestCreatingProjectWithPulumiBackendURL on Windows
* Weaken TestAbbreviateFilePath to pass on Windows
* Fix TestDepRootCalc on Windows
* Fix LocalUrl() to be sensible on Windows
* Fix Go lint issue
* Windows fix for TestComments
* Cross-ref skip tracking issue
* Cross-ref issue to fix asset_test skips
* More cross-ref issues
* Use choco not chocolatey
* Use yarn run to ensure the right tools are selected
* Revert python3->python change in common.mk
* In CI context, use python not python3
* More randomness in temp folder names to aoid Windows collisions
* Go lint
Current/older versions of the Python language host cannot read `pulumiplugin.json` without returning an error. This hasn't been a problem because none of our packages have included the file. However, we're going to start including this file by default in packages. To avoid this error on older CLIs, we'll use a different file name. And since we're introducing the use of this file in the .NET and Go language hosts, we'll use the new name for these languages as well for consistency.
* Check for pulumiplugin.json on package lookup
* Update CHANGELOG_PENDING.md
* Add tests
Tests caught a bug in the implementation of `LoadPulumiPluginJSON`. +1
for tests.
* Fix test for CI
* Check error (fix lint)
* Avoid repeatedly invoking `pip show` in Python lang host
The Python language host invokes `pip show` for each Pulumi package in a
project at startup. But `pip show` is quite slow in large projects: it
takes 2+ seconds per invocation in a project at @MaterializeInc.
`pip show` is invoked to compute the installed location of each plugin
package. But it turns out `pip list` takes a `-v` flag that can supply
this information in one shot, avoiding the need to ever invoke `pip
show`.
This patch shaves about 20s off our boot time for `pulumi up`.
(There's probably a separate bug in Pip that causes `pip show` to be so
slow, because `pip list` is an order of magnitude faster and does a lot
more work, but I didn't bother tracking that down.)
* Test and fix issue with parsing non-JSON trailer returned by pip
* Fix issues found by Go lint
* CHANGELOG entry
Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.