Commit Graph

98 Commits

Author SHA1 Message Date
Julien 86a16d92e8
Switch to ruff for linting and formatting ()
Replace pylint and black with Ruff. Ruff is a lot faster and combines
both linting and formatting. It also has [very good editor
integration](https://docs.astral.sh/ruff/editors/setup/) via
https://github.com/astral-sh/ruff-lsp.

The formatting is mostly compatible with black, and only introduces some
minor changes.

A lot of the removed `pylint: XXX` comments where noops when linting
with pylint, most likely to due to updates to pylint catching more false
positives.

---------

Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-12-02 12:41:19 +00:00
Thomas Gummerer d03528dc6e
add lint_fix Makefile targets ()
Some of the linters we're using have an option to fix some of the lint
errors they are finding. This obviously only works for the simple cases,
and nothing too complex, but can still be useful.

Since there are a bunch of different tools we're using, it's not always
easy to remember which option needs to be passed to which tool. Add
`lint_fix` Makefile targets that pass these options to our linters where
available, to help fix these issues more easily.
2024-11-15 09:58:20 +00:00
Aaron Friel 2c900ddadb
Fix Pyright type checking of "StackReference#outputs" ()
Updates the type for the `outputs` property of the `StackReference`
class to be a `dict[str, Any]` instead of a `dict` (with unknown keys
and values). This change fixes the type error in the referenced issue.

Fixes 

Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-08-14 05:14:23 +00:00
Fraser Waters 48dbd6c596
Use black to format lib/test ()
Test code should be formatted and linted the same as library code. This
is the first step of that, simply including ./lib/test to the folder
that the black formatter runs on.
2024-04-23 08:29:58 +00:00
Julien P c26874d411
[auto/python] Add new API to install the Pulumi CLI ()
# Description

Provide a way for the Automation API to install the Pulumi CLI so that
Automation API can be used in a more standalone manner.

https://github.com/pulumi/pulumi/issues/14987

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-01-26 14:30:19 +00:00
Justin Van Patten 3a1f137203
Switch to API Token for PyPi uploads ()
Username/Password authentication is no longer supported. This PR
migrates to using API Tokens per https://pypi.org/help/#apitoken. We use
`__token__` for the username and the org-wide `PYPI_API_TOKEN` secret
for the password.

Part of https://github.com/pulumi/pulumi/issues/15043

Reference:
- https://github.com/pulumi/ci-mgmt/issues/751
- https://github.com/pulumi/ci-mgmt/pull/767
2024-01-04 14:44:52 +00:00
Abhinav Gupta 0326af4461
make(sdk/py): Fix install, dist, test_go, brew ()
Some `make` targets broke 
when we turned pulumi-language-python into
an independent Go module.
It also has a different import path than it did before.

This updates the Makefile to match the new path
and uses `-C` to cd into that directory for relevant Go commands.
2023-08-31 20:57:59 +00:00
Fraser Waters e5983715bf Fix makefiles 2023-05-26 11:00:45 +01:00
Justin Van Patten c0ec513dcb [sdk/python] Allow editable installs without build step
This commit modifies `setup.py` to use a Python variable as the source of the package version instead of a placeholder string and adds a fallback for the README definition. This makes the package installable via the `-e` flag directly from its source directory (`sdk/python/lib`), rather than having a build step and having to `-e` install the built directory (`sdk/python/env/src`).
2022-09-30 17:21:44 -07:00
Aaron Friel 6ef7ef64d6 ci: Enable testing of language version sets 2022-09-21 09:48:38 -07:00
Aaron Friel 746c5f6e3c ci: Enable robust retries on tests 2022-09-14 12:19:09 -07:00
Aaron Friel 272b987791 ci: ensure trunk is always green 2022-09-13 13:38:14 -07:00
Fraser Waters 22fdbfea4f
Require python 3.7 ()
* Require python 3.7

As per our newly documented supported versions (https://www.pulumi.com/docs/intro/languages/python/)

* Add to CHANGELOG
2022-09-01 12:21:49 +01:00
Anton Tayanovskyy b06b860c2a
Remove PROJECT_ROOT Make var () 2022-06-24 10:04:40 -04:00
Anton Tayanovskyy 0e64874da7
Avoid realpath Makefile func that seems flaky on Win ()
* Avoid realpath Makefile func that seems flaky on Win

* Correct relpath for top-level Make
2022-06-23 15:32:18 -04:00
Ian Wahbe 9bad651337
Cleanup make ensure ()
* Don't format generated files

* Allow ensure to be stateful

* Cleanup comments and echo flags
2022-05-16 16:47:04 -07:00
Ian Wahbe 5c30ac0c9a
Don't format generated files () 2022-05-16 11:47:17 -07:00
Fraser Waters c3e084d028
Makefiles are very whitespace sensitive () 2022-03-26 09:00:21 +00:00
Matthieu Coudron d4b9d61d70
allow to override makefile variables ()
* allow to override makefile variables

When packaging pulumi-python for www.nixos.org, there is no /bin/bash
available so allow for an alternative path to the shell.

Same for versions, PYPI version can be set via the environment variable
PULUMI_VERSION.

* Update sdk/nodejs/Makefile

Co-authored-by: Fraser Waters <frassle@gmail.com>

* Update sdk/go/Makefile

Co-authored-by: Fraser Waters <frassle@gmail.com>

Co-authored-by: Fraser Waters <frassle@gmail.com>
2022-03-25 21:56:11 +00:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Aaron Friel 7b7ec02521 ci: divide and conquer integration tests by sdk and package group 2022-03-04 18:08:23 -08:00
Ian Wahbe 06ba63bb57
Use black code formatter for the python sdk ()
* Add black as a lint + format target for python

* Update lib/pulumi formatting
2022-02-04 01:16:16 +01:00
Anton Tayanovskyy b65a1b89a6
Revert "Lower test parallelism on Windows ()" ()
This reverts commit 3b20dc37be.
2022-01-21 17:55:38 -05:00
Aaron Friel d23100839c
Merge pull request from pulumi/python3-fixes
[sdk/python] Fix makefile target ordering, +x .py
2022-01-19 16:03:05 -08:00
Anton Tayanovskyy 3b20dc37be
Lower test parallelism on Windows () 2022-01-19 17:58:12 -05:00
Aaron Friel 7299f016fc fix: test harness sourcing venv on Windows
On Windows, sourcing the activation script caused incorrect behavior and mangled the path, causing tests to fail.
2022-01-19 14:38:21 -08:00
Aaron Friel b61c846943 revert to prior method of running pip 2022-01-19 14:38:21 -08:00
Aaron Friel a9b555df1d [sdk/python] Fix makefile target ordering, +x .py
* `pip` & the `wheel` package are dependencies before lint/build.
* Fix a .py file committed without the execute bit

Used these commands with `watchexec` to make a variety of changes and
then backed them out to find the minimum viable patch.

`watchexec "git clean -xfd; make lint"`
`watchexec "git clean -xfd; make build_package"`

And finally:

`watchexec "git clean -xfd; make test_all"`

Initially I thought the issue was that my distro only has
`/usr/bin/python3` as I was getting errors on running `python`, but it
looks like the issue was primarily activating the environment and making
sure dependencies were installed correctly.

There was an idempotency issue as well, where the commands would fail on
first run, but on second the `bdist_wheel` package would be set up and
they would succeed.
2022-01-19 14:38:21 -08:00
Anton Tayanovskyy 76ba788985
Toward replacing MSBuild with make+bash on Windows ()
* Add coverage folder

* Adjust GHA to run Test target on Windows

* Extend error message on packages not found

* Try with path normalizer

* Edit Makefiles

* Skip SDK codegen tests on Windows

* Skip program gen tests on Windows

* Skip tests that started running on Windows and failing

* Fix non-compiling test

* Skip failing Windows HCL2 tests

* Fix lint

* Merged constants

* Skip one more test failing on Windows

* Disable cov-enabled builds on Windows

* Fix  TestDeterminePulumiPackages on Windows

* Fix TestInstallCleansOldFiles test on Windows

* Fix TestCreatingProjectWithPulumiBackendURL on Windows

* Weaken TestAbbreviateFilePath to pass on Windows

* Fix TestDepRootCalc on Windows

* Fix LocalUrl() to be sensible on Windows

* Fix Go lint issue

* Windows fix for TestComments

* Cross-ref skip tracking issue

* Cross-ref issue to fix asset_test skips

* More cross-ref issues

* Use choco not chocolatey

* Use yarn run to ensure the right tools are selected

* Revert python3->python change in common.mk

* In CI context, use python not python3

* More randomness in temp folder names to aoid Windows collisions

* Go lint
2022-01-07 22:27:14 -05:00
Ian Wahbe c95e49e5ab
Use $(PYTHON) for makefile ()
* Use $(PYTHON) for makefile

* Normalize linting
2021-12-08 13:39:39 -08:00
Justin Van Patten 096b667ee1
Replace the use of pipenv with Python's venv ()
We've had a few issues in the recent past related to pipenv oddities in
CI which lead us to temporarily globally install the Python SDK in CI.

This change removes the use of pipenv in favor of Python's built-in
venv and avoids globally installing the Python SDK.
2021-12-07 07:59:07 -08:00
Pat Gavlin d3e49ecb65
[cli, testing, github] Gather code coverage data in CI. ()
These changes add support for gathering code coverage data during tests.

For tests that do not involve the Pulumi CLI, this is straightforward: all of
the ecosystems we target already support gathering coverage data, and we follow
the rules accordingly. Support for each language is broken out into its own
commit.

For tests that do involve the Pulumi CLI, the picture is a bit more complicated.
Go does not make it trivial to perform a coverage-instrumented build (go build
does not have a -cover flag, for example). In lieu of official support, we abuse
go test -c and TestMain to produce a build of the CLI that supports collecting
and reporting coverage data.
2021-11-30 17:24:01 -08:00
Anton Tayanovskyy 99fdad0ed9
Fix python dependencies issues in GHA () 2021-11-06 00:55:17 -07:00
Pat Gavlin 2dcf3806bd
[automation-api] Exclude tests from test_fast. ()
The Automation API tests take long enough that thay don't really fit
into `test_fast`. These changes move those tests to `test_all`.
2021-09-16 17:33:33 -07:00
Anton Tayanovskyy ba70b3fdba
Attempt to avoid triple-building projects in the solution () 2021-07-28 20:31:11 -04:00
Anton Tayanovskyy 3aa97a4b7d
Fanout build experiment ()
* Experiment with gotestsum and test timings

* Fix to locating the helper script

* Fix the code for installing gotestsum

* Try alternative installation method

* Use go to compute test stats; Python fails parsing time values

* Try version without v

* Try with fixed gorelaser config

* Fix test time correlation

* Try a stable test stat sort finally

* Use more accurate test duration aggregation

* Include python and auto-api tests in the Go timing counts

* Bring back TESTPARALLELISM

* Fix test compilation

* Only top 100 slow tests

* Try to fracture build matrix to fan out tests

* Do not run Publish Test Results on unsuppored Mac

* Auto-create test-results-dir

* Fix new flaky test by polling for logs

* Try to move native tests to their own config

* Actually skip

* Do not fail on empty test-results folder

* Try again

* Try once more

* Integration test config is the crit path - make it smaller

* Squash underutilized test configurations

* Remove the test result summary box from PR - counts now incorrec

* Remove debugging step
2021-07-27 10:07:15 -04:00
Anton Tayanovskyy b77f32930c
Remote component py SDK ()
* Python support for authoring resource providers for multi-lang

* Support for passing prompt values to Python resource providers
2021-04-15 14:49:51 -04:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Paul Stack 1d3c9edb6c
Ensure that make brew works as expected rather than passing empty version ()
Fixes:#6565

As part of , the logic for determing the version of the build was
moved to be a dependency on pulumictl.

Unfortunately, the homebrew installs use the "make dist" command to
build + install Pulumi to the user maching and as that would have a
dependency on pulumictl and it not existing on the user machine, it
would pass an empty version to the ldflag

This then manifested to the user as:

```
▶ pulumi version
warning: A new version of Pulumi is available. To upgrade from version '0.0.0' to '2.22.0', run
   $ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
```

We are able to mitigate this behaviour by bringing back the get-version
script and using that script as part of the make brew installation

We can see that the versions are the same between the 2 different
installation techniques

```
make dist <------- uses pulumict
DIST:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=2.24.0-alpha.1616029310+787eb70a" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
DIST:
BUILD:
```

```
make brew <----- uses the legacy script
▶ make brew
BREW:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=v2.24.0-alpha.1616029310+g787eb70a2" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
BREW:
```

A full post mortem will be carried out to ensure we mitigate these
types of errors going forward and that we are able to better test
these types of situations
2021-03-18 02:07:02 +00:00
Paul Stack c48ba37fcf
Migrate the version calculation to use pulumictl () 2021-03-10 19:03:29 +00:00
Justin Van Patten 5ab051cd97
Implement GetRequiredPlugins for Python ()
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.

Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
2020-12-03 19:22:16 -08:00
Vivek Lakshmanan 55e86bf078
Update sdk/python/Makefile
Co-authored-by: Komal <komal@pulumi.com>
2020-11-18 09:21:41 -08:00
Vivek Lakshmanan 0d6f9fdcbf Switch to pytest since it captures stdout/stderr and only prints on failure 2020-11-17 23:09:18 -08:00
Vivek Lakshmanan 98bc43d6d6 Disable verbose logging for python tests 2020-11-17 23:09:18 -08:00
Komal f1f6a126bc
Support python 3.9 ()
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Co-authored-by: stack72 <public@paulstack.co.uk>
2020-11-03 16:28:45 -08:00
Lee Briggs 95bc138b41
add master branch workflow () 2020-09-21 16:20:05 -07:00
Paul Stack 45e1917a30
Upgrade to Go 1.14.x () 2020-05-28 12:01:33 +01:00
Justin Van Patten af3d4b890d
Allow `pulumi.export` calls from unit tests ()
The previous attempt to allow this didn't actually allow it, so this is
take two. As part of the previous attempt, I thought after tweaking the
test I had observed the test failing, and then succeeding after making
the product changes, but I must have been mistaken.

It turns out that our existing mocks tests weren't running at all
because of a missing `__init__.py` file. Once the missing `__init__.py`
is added, the tests run, but other tests ("test mode" tests) fail
because the code that creates the mocks and resources will run during
test discovery, and setting the mocks modifies global state.

To address the test issue, I've moved the mocks tests into their own
`test_with_mocks` package that can be run separately from other tests.

And addressed the original issue, by creating a root Stack resource if
one isn't already present when the mocks are set.
2020-05-20 09:54:40 -07:00
Lee Briggs 8c314ec39a
Add brew make targets to sdk makefiles 2020-05-13 20:42:01 -07:00
Pat Gavlin 9b1b9cca24
Makefile: properly encode dependencies ()
This allows for the use of `make -j`, which speeds up a full build
dramatically at the cost of rather incomprehensible logs.
2020-05-04 14:26:52 -07:00