Commit Graph

30 Commits

Author SHA1 Message Date
Justin Van Patten e13780c0bd
[sdk/python] Upgrade to grpcio 1.60.1 ()
grpcio 1.60.1 includes a fix for the regression that results in a
traceback being written to stderr for unimplemented gRPC methods.
https://github.com/grpc/grpc/releases/tag/v1.60.1

This change upgrades to 1.60.1 and reverts the workarounds we put in
place in https://github.com/pulumi/pulumi/pull/15190.

Note that we have an integration test for Python dynamic providers that
ensures nothing unexpected is written to stderr:
16a1fb31c2/tests/integration/integration_python_acceptance_test.go (L111-L117)

Fixes 
2024-02-05 19:49:08 +00:00
Fraser Waters b9c92dfb50
Pin pytest to v7 () 2024-01-28 13:05:32 +00:00
Justin Van Patten 7309681b5b
Support Python 3.12 ()
Python 3.12 requires `grpcio` 1.59.0 or higher. Unfortunately, there is
a regression in `grpcio` 1.58.0 through the latest version (currently
1.60.0) which causes any error returned from a Python gRPC server to be
written to stderr, including UNIMPLEMENTED errors. This primarily
affects Python dynamic providers, which don't have implementations for
`CheckConfig` and `DiffConfig`, resulting in a traceback error being
emitted to stderr when the engine calls these, which is visible to
users. This `grpcio` regression has been fixed upstream, but the fix has
not been released yet. We've been waiting for a 1.60.1 patch release.

This has not been great for our Python users who are using Python 3.12.
It's particularly bad for new Pulumi users who are using Python 3.12 and
are trying to get started with Pulumi. For these users, when trying to
install the `pulumi` PyPi package (i.e. via `pulumi new python`) the
installation fails with an error because it is pinned to depending on an
older version of `grpcio` which doesn't work on Python 3.12.

This commit works around the problem by providing default
implementations of `CheckConfig` and `DiffConfig` for python dynamic
providers and the component provider API, so that no error is emitted to
stderr when the engine calls these methods. The default implementations
for these are the same behavior that the engine would use if these
methods had returned UNIMPLEMENTED. I believe these are the only two
methods affected by this. Other methods like `Invoke`, `Call`,
`StreamInvoke`, `Construct`, `Attach`, `GetMapping`, and `GetMappings`,
continue to return UNIMPLEMENTED for dynamic providers, which I think is
OK; I don't believe these will be called by the engine under normal
circumstances.

Out of an abundance of caution, the `pulumi` package continues to depend
on the pinned version of `grpcio` when installing on versions of Python
less than 3.12. On Python 3.12 or greater, we now depend on `grpcio`
`~=1.60.0`. 1.60.0 doesn't have the fix for the regression, but the
workaround should allow things to work on Python 3.12 as before.

Once 1.60.1 is released, we can look into updating the `grpcio`
dependency to `~=1.60.1` for all versions of Python, and possibly revert
the workarounds, if we want.

Note:  added a test for dynamic providers to ensure nothing is
written to stderr. The test would fail if the workaround in this PR did
not work as intended:
https://github.com/pulumi/pulumi/pull/14474/files#diff-d92ccd283e08eadab2597825103e45cdaa96fea93324bc4d4d3b1d2b83c51b76

This PR depends on several other smaller PRs:
- https://github.com/pulumi/pulumi/pull/15220
- https://github.com/pulumi/pulumi/pull/15221
- https://github.com/pulumi/pulumi/pull/15222
- https://github.com/pulumi/pulumi/pull/15223
- https://github.com/pulumi/pulumi/pull/15224
- https://github.com/pulumi/pulumi/pull/15225
- https://github.com/pulumi/pulumi/pull/15226

Fixes 
2024-01-24 22:24:34 +00:00
Justin Van Patten eb3633d34f
Upgrade pylint to 3.0.3 ()
In preparation for supporting Python 3.12...

We need to be running a more recent version of pylint that runs on
Python 3.12 (otherwise it errors). This required disabling some new
lints that were resulting in new lint errors and fixing how
`overgeneral-exceptions` is specified.
2024-01-23 23:48:02 +00:00
Justin Van Patten 3d78b037f4
Fix traceback diagnostic from being printed when using Python dynamic providers ()
 upgraded grpcio to 1.59.2, to support Python 3.12. Unfortunately,
grpcio >=1.58.0 has a problem where calls to unimplemented gRPC methods
cause a traceback to be emitted to the server's stderr, which affects
Python dynamic providers, which don't implement `DiffConfig`. The result
is a traceback diagnostic shown for the Pulumi program using the dynamic
provider.

This PR fixes this by reverting the change, downgrading grpcio back to
1.56.2.

The existing Python dynamic provider diagnostic test has been extended
to check to make sure there are no unexpected diagnostics, which fails
before the revert, and passes after.

An upstream issue has been opened:
https://github.com/grpc/grpc/issues/34853

The issue to support Python 3.12 has been re-opened:  (currently
blocked on the upstream issue)

Fixes 
2023-11-01 22:19:37 +00:00
Alexander Hungenberg 85b41ecef3
updating grpcio version ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

The PR updates the `grpcio` dependency for the Python SDK.

I also made a change here to [no longer fix the grpcio version to a
specific
one](https://iscinumpy.dev/post/bound-version-constraints/#tldr) - which
makes it very hard to use pulumi in other projects with other
dependencies.

Fixes 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->

---------

Co-authored-by: Fraser Waters <frassle@gmail.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-10-22 18:30:21 +00:00
snyk-bot 8f8f30eeb8
fix: sdk/python/requirements.txt to reduce vulnerabilities
The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-GRPCIO-5834443
2023-08-10 18:51:34 +00:00
Abhinav Gupta 1447ee55c3
deps(py): upgrade to latest grpcio
Upgrades to the latest release of grpcio for the Python SDK.

Supersedes , 
Fixes CVE-2023-32731
2023-07-06 09:04:59 -07:00
Fraser Waters 7ef6f61b11 Support WhoAmI in automation api for old CLI versions 2023-03-22 13:30:08 +00:00
Ringo De Smet 91a3e4df6a
Python grpc 1.51.3 contains universal2 binaries (x86_64 & arm64)
Signed-off-by: Ringo De Smet <ringo@de-smet.name>
2023-03-01 15:30:45 +01:00
Kyle Dixler 30c0f7e1e8
pinned mypy to 0.991 to make lints pass for release 3.54.0 2023-02-06 12:36:03 -08:00
Kyle Pitzen 0752b7921a Bumps grpcio to version 1.50
Currently, pinning to a version of grpcio <1.50 can cause build errors
on newer versions of python+pip - the existing setup.py install method
of building libararies is being deprecated and wheels are not being
backported for older versions of grpcio.  Since this change is a no-op
as far as our python proto clients are concerned, it feels safe to bump
to a version that will be supported going forward
2022-12-21 08:45:21 -05:00
Fraser Waters c10ff7efad Fix protobuf checksum code and regenerate
Updates the protobuf generation to use mypy_protobuf to get types.
2022-11-04 09:57:05 +00:00
Kyle Dixler 613e4d5d88
pinned pylint==2.14.5 () 2022-08-26 17:25:05 -04:00
Robbie McKinstry 237da1637b
Update PyYAML to v6.0 ()
* Update PyYAML to v6.0

* Update CHANGELOG
2022-08-18 10:39:36 -04:00
Kyle Dixler 0ba7aaf8c7
pinned grpcio to 1.47 to fix hanging tests ()
* pinned grpcio==1.47
2022-07-29 17:10:02 -07:00
Fraser Waters cd85d614cb
Update python protobuf ()
* Update python protobuf

* changelog entry

Co-authored-by: Zaid Ajaj <zaid.naom@gmail.com>
2022-07-12 21:24:51 +01:00
Fraser Waters 6478b6b347
Update python requirments to pin to major version () 2022-05-27 12:54:40 +01:00
Fraser Waters e2c73e57b7
Pin python protobuf to <4 ()
* Pin python protobuf to <4

Fixes https://github.com/pulumi/pulumi/issues/9694

* Add to CHANGELOG
2022-05-26 11:29:52 +01:00
Ian Wahbe c704bb0d22
Unpin python grpcio version () 2022-04-19 22:48:02 +02:00
Fraser Waters 4145b1bdac
Unfix protobuf, teach pylint about generated members () 2022-04-04 13:28:04 +01:00
Fraser Waters 2151ab6e16
Fix python protobuf to < 3.20.0 () 2022-04-02 08:05:11 +01:00
Ian Wahbe a784cb58b3
Restrict grpcio version to below 1.44.0 ()
* Restrict grpcio version to below 1.44.0

* Update CHANGELOG_PENDING.md
2022-03-07 18:43:04 -08:00
Ian Wahbe 06ba63bb57
Use black code formatter for the python sdk ()
* Add black as a lint + format target for python

* Update lib/pulumi formatting
2022-02-04 01:16:16 +01:00
Justin Van Patten 096b667ee1
Replace the use of pipenv with Python's venv ()
We've had a few issues in the recent past related to pipenv oddities in
CI which lead us to temporarily globally install the Python SDK in CI.

This change removes the use of pipenv in favor of Python's built-in
venv and avoids globally installing the Python SDK.
2021-12-07 07:59:07 -08:00
Sean Gillespie 89b052fc6d
Use Pipenv to manage Python environments ()
* Use Pipenv to manage Python environments

* Rename PIPENV_ARGS to PIPENV_PYTHON_VERSION to avoid confusion. Also remove two unused variables
2018-06-21 18:01:23 -07:00
joeduffy 440ffb27e5 Prefer Python2 tools; fail-fast when wrong
This change includes a few things:

1) Prefer python2 and pip2 when on the PATH, over the undecorated
   names python and pip.  This is the standard convention for package
   managers like Pip, etc., to support Python2 and Python3 side-by-side.

2) Fail-fast if neither can be found on the PATH.

3) Check the reported version number for python, pip, and pylint, and
   fail-fast if it doesn't report back 2.7, just to safeguard against
   undecorated binaries with unsupported versions.

Also, we had not listed wheel as a dependency in the requirements.txt
file.  This needs to be there to support building bdist_wheels.  Fixed.
2018-03-06 17:50:42 -08:00
joeduffy 4f591bed76 Use a newer pylint
This ensures the new exclusion syntax is accepted, which is required
to ignore the Protobuf-generated files which have numerous lint warnings.
2018-02-24 09:52:30 -08:00
joeduffy a045e2fb1e Implement more of the Python runtime
This change includes a lot more functionality.  Enough to actually
run the webserver-py example through previews, updates, and destroys!

* Actually wire up the gRPC connections to the engine/monitor.

* Move the Node.js and Python generated Protobuf/gRPC files underneath
  the actual SDK directories to simplify this generally.  No more
  copying during `make` and, in fact, this was required to give a smoother
  experience with good packages/modules for the Python's SDK development.

* Build the Python egg during `make build`.

* Add support for program stacks.  Just like with the Node.js runtime,
  we will auto-parent any resources without explicit parents to a single
  top-level resource component.

* Add support for component resource output properties.

* Add get_project() and get_stack() functions for retrieving the current
  project and stack names.

* Properly use UNKNOWN sentinels.

* Add a set_outputs() function on Resource.  This is defined by the
  code-generator and allows custom logic for output property setting.
  This is cleaner than the way we do this in Node.js, and gives us a
  way to ensure that output properties are "real" properties, complete
  with member documentation.  This also gives us a hook to perform
  name demangling, which the code-generator typically controls anyway.

* Add package dependencies to setuptools.py and requirements.txt.
2018-02-24 08:58:34 -08:00
joeduffy 74563afdc8 Get the empty Python program working
This change gets enough of the Python SDK up and running that the
empty Python program will work.  Mostly just scaffolding, but the
basic structure is now in place.  The primary remaining work is to
wire up resource creation to the gRPC interfaces.

In summary:

* The basic structure is as follows:

    - Everything goes into sdk/python/.

    - sdk/python/cmd/pulumi-langhost-python is a Go language host
      that simply knows how to spawn Python processes to run out
      entrypoint in response to requests by the engine.

    - sdk/python/cmd/pulumi-langhost-python-exec is a little Python
      shim that is invoked by the language host to run Python programs,
      and is responsible for setting up the minimal goo before we can
      do so (RPC connections and the like).

    - sdk/python/lib/ contains a Python Pip package suitable for PyPi.

    - In there, we have two packages: the root pulumi package that
      contains all of the basic Pulumi programming model abstractions,
      and pulumi.runtime, which contains the implementation of
      resource registration, RPC interfacing with the engine, and so on.

* Add logic in our test framework to conditionalize on the language
  type and react accordingly.  This will allow us to skip Yarn for
  Python projects and eventually run Pip if there's a requirements.txt.

* Created the basic project structure, including all of the usual
  Make targets for installing into the proper places.

* Building also runs Pylint and we are clean.

There are a few other minor things in here:

* Add an "empty" test for both Node.js and Python.  These pass.

* Fix an existing bug in plugin shutdown logic.  At some point, we
  started waiting for stderr/stdout to flush before shutting down
  the plugin; but if certain failures happen "early" during the
  plugin launch process, these channels will never get initialized
  and so waiting for them deadlocks.

* Recently we seem to have added logic to delete test temp
  directories  if a failure happened during initialization of said
  temp directories.  This is unfortunate, because you often need to
  look at the temp directory to see what failed.  We already clean
  them up elsewhere after the full test completes successfully, so
  I don't think we need to be doing this, and I've removed it.

Still many loose ends (config, resources, etc), but it's a start!
2018-02-23 19:33:02 -08:00