Commit Graph

52 Commits

Author SHA1 Message Date
Julien 487b4a8494
Install missing python versions using pyenv during installation ()
Adds the flag `--use-language-version-tools` to `pulumi install`. When
passed, and `pyenv` is installed, and a `.python-version` file is in the
project directory or any of its parent directories, Pulumi will install
the requested python version if it is not already installed.

`LanguageRuntime.InstallDependencies` now takes a struct
`InstallDependenciesRequest` as argument to make it easier to extend
this call with options.

See https://github.com/pulumi/pulumi-docker-containers/pull/232

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-08-19 15:55:54 +00:00
Daniel Bradley c6a01328d2
Set range in python codegen default SDK version ()
If the schema doesn't set any specific dependencies, we should be
defaulting to requiring pulumi within the current major version rather
than completely unconstrained.

Discussion:
https://github.com/pulumi/pulumi-kafka/pull/410/files#r1594324390
2024-06-24 13:57:57 +00:00
Julien P 05574aaaa4
Generate TypedDicts for python inputs ()
Epic: Improved Typing https://github.com/pulumi/pulumi/issues/12689

This PR adds the flag `Languages.Python.InputTypes` to the schema, which
can take the values `classes` or `classes-and-dicts`. In the first
iteration of the feature, this defaults to `classes`, which leaves code
generation as is and does not change input types.. If the flag is set to
`classes-and-dicts`, `TypedDict` based types are generated next to the
current `<Type>Args` classes. In the future this could be extended to
support `dicts` to generate only `TypedDict` types.

The generated types are conditional on the used type checker to work
around perf issues in MyPy and PyCharm, see
https://github.com/pulumi/pulumi/issues/12689#issuecomment-2117240276

```python
if not MYPY:
    class DeploymentArgsDict(TypedDict):
        api_version: NotRequired[Input[str]]
        kind: NotRequired[Input[str]]
        metadata: NotRequired[Input['ObjectMetaArgsDict']]
        ...
elif False:
    DeploymentArgsDict: TypeAlias = Mapping[str, Any]
```

Removing the workaround is tracked in
https://github.com/pulumi/pulumi/issues/16408

---------

Co-authored-by: Anthony King <anthony@datapane.com>
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
2024-06-18 15:24:42 +00:00
Julien P 578e0937a9
[Python] Move existing dependency installation and python command invocation to a Toolchain interface ()
# Description

This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.

Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-03 13:52:27 +00:00
Justin Van Patten cc631241b3
[sdkgen/python] Require Python >=3.8 ()
Python 3.7 is unsupported and has been end-of-life since 6/27/2023. See
https://devguide.python.org/versions/

The core `pulumi` Python SDK package is being updated to indicate that
Python 3.8 or greater is required (see ).

This change updates the default minimum required Python version for
generated provider SDKs to Python 3.8 or greater as well.

Fixes 
2024-02-04 19:07:53 +00:00
Justin Van Patten 87b611b1f4
[sdkgen/python] Use `importlib.metadata` instead of `pkg_resources` ()
This addresses two issues:

1. `pkg_resource` is deprecated in favor of `importlib.resources` and
`importlib.metadata`
(https://setuptools.pypa.io/en/latest/pkg_resources.html)

2. Generated provider SDKs don't indicate that they have a dependency on
`setuptools` (which includes `pkg_resources`), which can cause problems
when installing the package in environments that don't have `setuptools`
installed. That's not often common in Pulumi projects, as the virtual
environment created by the CLI will include `setuptools`, however, if
creating the virtual environment manually with `python -m venv`,
`setuptools` is no longer included in the created virtual environment as
of Python 3.12.

Fixes 

Based on , thanks @edgarrmondragon!

---------

Co-authored-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
2024-01-27 02:13:37 +00:00
Fraser Waters 16d9f4c167
Enable perfsprint linter ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945

This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.

Most of these edits were made by running `perfsprint --fix`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-12 12:19:42 +00:00
Robbie McKinstry 8b98abdf2d
Regen test docs 2023-05-28 14:08:57 -04:00
Robbie McKinstry 625a4f55d4
Set a minimum version of Python supported.
This commit sets the minimum version of Python supported by
the package for pyprojec.toml builds.
2023-05-28 14:08:56 -04:00
Abhinav Gupta ec55d4c6f1
chore(all): Log to iotest.LogWriter from tests
There are very few direct uses of os.Stderr and os.Stdout in our tests.
Switch these to use iotest.LogWriter where appropriate,
so they log messages to the appropriate testing.TB.
In case of failure, these messages will be properly associated to the
test/subtest that they were for
instead of being printed all to stdout/stderr.

Uses of os.Stdout/Stderr that were omitted:

- Example tests: These don't have a testing.TB to log to.
- pulumi/main_test: Replaces the global os.Stdout/Stderr in TestMain.
  LogWriter isn't relevant there.
2023-03-07 11:27:47 -08:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Ian Wahbe 60481daba8
Fix nodejs external enums ()
* Add a test

* Fix empty submodules for python

* Handle enums without other supporting types

* Run global tests

* Support external enums

* CL
2022-07-29 17:07:27 +02:00
Mikhail Shilkov 48a5677d1d
Simplify and harden escaping in Python doc strings SDK gen () 2022-07-07 09:21:30 +02:00
Ian Wahbe c74ff6f891
[codegen/go] Allow nested collections ()
* Add test & fix header bug

* Get working

* Fix python test

* CL
2022-06-21 11:04:13 -07:00
Aaron Friel aae9c20ffc
[codegen/python] Implement deep merge on resource and invoke options, matching other SDKs ()
* [codegen/python] Implement deep merge on resource and invoke options, matching other SDKs

* pr feedback
2022-06-14 16:52:24 -07:00
Ian Wahbe f3e430a1da
Respond to SIGINT during plugin install ()
* Respond to SIGINT

With the current state of the PR, crash on SIGINT. This is progress.

* Don't crash responding to SIGINT

* Close on cancel instead of terminate

* CL

* Fix lint

* Add ctx for node

* Be consistent for test context
2022-06-09 14:57:56 -07:00
Ian Wahbe 741310d4d7
[python,nodejs] Speed up codegen tests ()
* Don't run pyTestCheck unless there were mixins

On my machine this changes the python test time from ~340s to ~55s

* TS: assume subsidiary packages are correct

'--skipLibCheck true --skipDefaultLibCheck true' cuts ~50s off of the
total time needed in github.com/pulumi/pulumi/pkg/v3/codegen/nodejs.
2022-06-02 14:56:56 -07:00
Ian Wahbe 183fa89163
Add an EnumType to the PCL model ()
* Add an EnumType to the PCL model

Languages implementation:
- [X] C#
- [X] Python
- [X] TypeScript
- [X] Go

* Fix assert

* Take namespace settings into account

* Implement enums for Go

* Update tests and add documentation.

* Add a test + handle output&unsafe

* Get outputs compiling for Go

* Line up types for Go

* Add nodejs

* Add Python

* Change expression form

* Fix nil check (`==` -> `!=`)

* Standardize nodejs & python checks

This should fix the Node OOM error.

* Rename enum-py to typed-enum-pp

* Don't reference schema types in `hcl2/model`

* Fix nits
2022-04-18 11:03:42 +02:00
Aaron Friel ed2923653c ci: radical idea - what if slow tests & no stdout makes GH consider runner dead? 2022-03-06 14:52:13 -08:00
Ian Wahbe a1e18dae4d
export codegen tests for internal use ()
* Export Codegen test modules

* Document pkg stability guarantee

* Expose programgen

* Recommendation: Improve wording

* Move `internal` to `testing`

* Re-rout references to codegen/internal

* Fix some other "internal" references
2022-02-07 12:10:04 +01:00
Ian Wahbe 0e859d72cc
Cleanup sdk codegen tests ()
* Pass missing tests for python

* Don't run missing nodejs tests

* Fix error message

* Re-enable "hyphen-url" (8370)

It seems like this was fixed in another commit. It passes now.

* Fix comment
2021-11-30 11:09:36 -08:00
Ian Wahbe 9f6589ed8c
iwahbe/7802/compile program generator test output ()
* Move program tests into folders

* update package schema

* Enabled tests pass

* Fix lints and begin to update test cases

* Re-enable tests

* Update aws version to v4

* Refactor language specific parts

* Hook up dotnet and nodejs

* Update tests from master

* SSOT for schema/version

* Name blocking errors. Leave tests in valid state

* Give each language its own folder

* Remove SkipCompile for azure-sa (bug was fixed)

* Fix nits + changes asserts to require

* Remove unused import

* One last assert => require
2021-09-29 11:33:57 -07:00
Pat Gavlin 0a45f8d3ab
[codegen/go] Rewrite cyclic types. ()
When computing the type name for a field of an object type, we must
ensure that we do not generate invalid recursive struct types. A struct
type T contains invalid recursion if the closure of its fields and its
struct-typed fields' fields includes a field of type T. A few examples:

Directly invalid:

    type T struct { Invalid T }

Indirectly invalid:

    type T struct { Invalid S }

    type S struct { Invalid T }

In order to avoid generating invalid struct types, we replace all
references to types involved in a cyclical definition with *T. The
examples above therefore become:

(1) type T struct { Valid *T }

(2) type T struct { Valid *S }

    type S struct { Valid *T }

We do this using a rewriter that turns all fields involved in reference
cycles into optional fields.

These changes also include an enhancement to the SDK codegen test
driver in the interest of making iterating and debugging more convenient:  if the -sdk.no-checks flag is passed, the driver will not run post-generation checks.
2021-09-28 07:33:14 -07:00
Anton Tayanovskyy 49ccd9ac97
Simplify output-funcs codegen test ()
* Consolidate output-funcs into a single normal schema.json

* Accept nodejs codegen output

* Accept dotnet output-funcs output; does not compile yet

* Accept docs output-funcs output

* Permit parallel test runs

* Accept nodejs codegen

* Fix and speed up Python codegen tests

* Dedup dash-named-schema

* Make dotnet tests pass

* Satisfy go lint
2021-09-23 13:42:20 -04:00
Anton Tayanovskyy 49298fb433
Codegen testing upgrades ()
* Multi-pass, in-place checks for SDK codegen tests; toward working Python checks

* Remove temp debug output

* Upgrade Node

* Update dotnet; need to follow up on version.txt quirks

* WIP

* Sounds like we can use non-github package names to ensure things are local

* Fix simple-enum-schema

* Fix dash-named-schema

* Fix nested-module

* Start building a test-running pass

* Infer skipping tests from skipping compiles

* Move tree schma tests to a proper place

* Address lint issues on Go code

* Build against local Go SDK

* Update pkg/codegen/internal/test/sdk_driver.go

Co-authored-by: Ian Wahbe <ian@wahbe.com>

* Make go tests work by copying them into the tree from go-extras

* Fix lint

* Fix bad merge

* Manifest-based file discovery

* Remove version-related TODO from dotnet codegen

* Add doc comment

* Do not overwrite go.mod if found from mixins

* Accept python codegen change

* Accept node codegen

* Ignore lint issue

* Accept docs changes

Co-authored-by: Ian Wahbe <ian@wahbe.com>
2021-09-22 13:55:20 -04:00
Ian Wahbe 67303e1b99
Run type checker against all languages ()
We run the best static check we can on generated code, ensuring that it is valid. 

* Run type checker against all languages (not docs)

* Fix package location, add some deps for schemas

* More tests passing

* These tests finally work

* Make linter happy

* Fix tests for merge from master

* Opt out of input-collision(nodejs) test

* Get more visibility into testing nodejs

* Fix type assumption

* Specify ts-node version

* Retrofit typescript dependencies for node14

* Give each go instance it's own module

* Attempt to diagnose remote go mod init failure

* Provide root for go mod init

* Make linter happy
2021-09-15 09:49:36 -07:00
Anton Tayanovskyy 4380a63ad9
Implement 5758 {fn}_output codgen for Python ()
* Implement 5758 {fn}_output codgen for Python

* Fix lint issues

* Accept codegen changes in expected examples

* Test and fix positional arg handling

* Add parameter descriptions to illustrate docstring codegen

* Generate docstrings

* Accept doc changes in expected go codegen file

* Use platform-agnostic API to drive Python tests
2021-08-30 16:52:58 -04:00
Pat Gavlin d07b325138
[codegen] Add type name generation tests. ()
The inputs and expected outputs for the tests are encoded using a
schema. Each property present in the schema forms a testcase; the
expected outputs for each language are stored in each property's
`Language` field with the language name "test". Expected outputs can be
regenerated using `PULUMI_ACCEPT`.
2021-07-09 10:23:10 -07:00
Pat Gavlin 46400d502b
[codegen] Unify SDK codegen testing ()
Rather than duplicating the list of tests and codegen driver across each
SDK, move its definition into `pkg/codegen/internal/test`. This has a
few notable benefits:

- All SDK code generators will be tested against each test. Though some
  tests may exercise a particular code generator more than others, the
  extra coverage will be generally beneficial.
- Adding a new test is simpler, as only a single file needs to be
  changed.
- All SDKs now honor the `PULUMI_ACCEPT` environment variable for
  updating baselines.
- Codegen tests now validate all generated files instead of only a
  particular subset.
2021-07-06 15:40:53 -07:00
James Nugent d525a8b331
Add complete third-party Python module and fix setup.py indentation ()
Co-authored-by: Komal <komalsali@gmail.com>
2021-06-24 16:01:49 -05:00
Pat Gavlin 7b1d6ec1ac
Reify `Input` and `Optional` types in the schema type system. ()
These changes support arbitrary combinations of input + plain types
within a schema. Handling plain types at the property level was not
sufficient to support such combinations. Reifying these types
required updating quite a bit of code. This is likely to have caused
some temporary complications, but should eventually lead to
substantial simplification in the SDK and program code generators.

With the new design, input and optional types are explicit in the schema
type system. Optionals will only appear at the outermost level of a type
(i.e. Input<Optional<>>, Array<Optional<>>, etc. will not occur). In
addition to explicit input types, each object type now has a "plain"
shape and an "input" shape. The former uses only plain types; the latter
uses input shapes wherever a plain type is not specified. Plain types
are indicated in the schema by setting the "plain" property of a type spec
to true.
2021-06-24 09:17:55 -07:00
Komal 0f97a9bdfa
Fix python module nesting () 2021-06-22 15:43:21 -07:00
Justin Van Patten 4e9e017cd2
[codegen/python] Rename conflicting ResourceArgs classes ()
Python resource constructor overloads were recently added that accept a
`<Resource>Args` class for input properties, as an alternative to the
other constructor overload that accepts keyword arguments. The name of
the new args class is the name of the resource concatenated with an
`Args` suffix.

Some providers (e.g. Kubernetes, Azure Native, and Google Native) have
input types with the same name as resources in the same module, which
results in two different `<Resource>Args` classes in the same module.

When you try to use the new args class with the constructor, e.g.:

```python
pulumi_kubernetes.storage.v1.StorageClass(
            resource_name='string',
            args=pulumi_kubernetes.storage.v1.StorageClassArgs(...),
            opts=pulumi.ResourceOptions(...),
)
```

You run into an error, because
`pulumi_kubernetes.storage.v1.StorageClassArgs` is actually referring to
the existing input type rather than the intended `StorageClassArgs`
class for the constructor arguments.

Having the duplicate classes hasn't broken existing usage of the input
type because we "export" all the input types for a module _after_ all
the resources and resource args classes are exported, so the input type
just ends up "overwriting" the duplicate resource args class.

Other languages don't have this problem because the input type is either
in it's own module/namespace (e.g. Node.js and .NET) or a different name
is used for the input type (Go). But with Python, the input types and
resources are all available in the same module.

To address this for Python, when there is an input type in the same
module with the same name as the resource, the args class for the
resource will be emitted as `<Resource>InitArgs` instead of
`<Resource>Args`.
2021-06-10 10:41:49 -07:00
James Nugent 73c8cc3ec5
Permit override of Python package naming ()
This commit adds a new language option for Python generation to specify
the package name instead of using `pulumi_x` where x is the name defined
in the schema.

A new test is added, and this has also been shown to produce no diff
when run against `pulumi-eks`.
2021-05-06 11:18:14 -07:00
Anton Tayanovskyy 6f1ea08993
Import subpackages lazily ()
* Import subpackages lazily

* Tighten up lazy_import impl

* Eagerly register resources, but lazily load their impl

* Add CHANGELOG entry

* Satisfy lint

* Restore mypy behavior so the change is not breaking

* Fix golden tests
2021-04-29 16:08:22 -04:00
pulumi-bot 73a66f48ea [breaking] Changing the version of go.mod in sdk / pkg to be v3 2021-04-14 19:32:18 +01:00
Justin Van Patten 572c74826c
Add schema & codegen support for plain properties ()
This change adds schema and codegen support for plain properties which
are emitted typed as the plain type rather than wrapped as an `Input`.
Plain properties require a prompt value and do not accept a value that
is `Output`.
2021-03-10 07:08:08 -08:00
Vivek Lakshmanan 8d022e39e8 Fix tests 2021-02-25 22:26:54 -08:00
Levi Blackstone 2dfec71a5c
[codegen/go] Register resource modules/packages () 2021-01-19 16:59:51 -07:00
Komal 9b33dd84d5
[codegen/*] - Fix enum names that start with an underscore. () 2020-12-16 09:22:44 -08:00
Komal cae129c7fa
[codegen/*] - Align makeValidIdentifier between languages () 2020-12-15 12:41:27 -08:00
Komal 600e296ee2
[codegen/python] - Enums () 2020-11-24 23:43:32 -06:00
Justin Van Patten a9d213107b
[codegen/(dotnet|python)]: Emit refs to provider resources () 2020-11-23 12:28:00 -08:00
Levi Blackstone 3586ab5d85
[codegen/python] Handle external schema refs () 2020-11-19 13:56:28 -07:00
Komal af9f636eef
Split schema tests by language () 2020-10-10 13:47:23 -07:00
Levi Blackstone 38172999e7
[codegen] Extract common test logic () 2020-09-30 14:41:52 -06:00
Levi Blackstone 64602798b1
[codegen/python] Don't generate get methods for ComponentResources () 2020-09-30 10:16:25 -06:00
Komal 59b906eb76
Update nodejs gen-tests () 2020-09-29 16:00:03 -07:00
Levi Blackstone 9363f606b6
[codegen/schema] Add IsComponent to resource schema ()
Add support for ComponentResources to the schema by
adding a new IsComponent field to the Resource spec.
2020-09-23 18:23:46 -06:00
Levi Blackstone 81580289a7
[codegen/python] Handle ResourceType () 2020-09-23 13:39:25 -06:00