Adds the flag `--use-language-version-tools` to `pulumi install`. When
passed, and `pyenv` is installed, and a `.python-version` file is in the
project directory or any of its parent directories, Pulumi will install
the requested python version if it is not already installed.
`LanguageRuntime.InstallDependencies` now takes a struct
`InstallDependenciesRequest` as argument to make it easier to extend
this call with options.
See https://github.com/pulumi/pulumi-docker-containers/pull/232
---------
Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
Epic: Improved Typing https://github.com/pulumi/pulumi/issues/12689
This PR adds the flag `Languages.Python.InputTypes` to the schema, which
can take the values `classes` or `classes-and-dicts`. In the first
iteration of the feature, this defaults to `classes`, which leaves code
generation as is and does not change input types.. If the flag is set to
`classes-and-dicts`, `TypedDict` based types are generated next to the
current `<Type>Args` classes. In the future this could be extended to
support `dicts` to generate only `TypedDict` types.
The generated types are conditional on the used type checker to work
around perf issues in MyPy and PyCharm, see
https://github.com/pulumi/pulumi/issues/12689#issuecomment-2117240276
```python
if not MYPY:
class DeploymentArgsDict(TypedDict):
api_version: NotRequired[Input[str]]
kind: NotRequired[Input[str]]
metadata: NotRequired[Input['ObjectMetaArgsDict']]
...
elif False:
DeploymentArgsDict: TypeAlias = Mapping[str, Any]
```
Removing the workaround is tracked in
https://github.com/pulumi/pulumi/issues/16408
---------
Co-authored-by: Anthony King <anthony@datapane.com>
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
# Description
This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.
Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
Python 3.7 is unsupported and has been end-of-life since 6/27/2023. See
https://devguide.python.org/versions/
The core `pulumi` Python SDK package is being updated to indicate that
Python 3.8 or greater is required (see #15365).
This change updates the default minimum required Python version for
generated provider SDKs to Python 3.8 or greater as well.
Fixes#15367
This addresses two issues:
1. `pkg_resource` is deprecated in favor of `importlib.resources` and
`importlib.metadata`
(https://setuptools.pypa.io/en/latest/pkg_resources.html)
2. Generated provider SDKs don't indicate that they have a dependency on
`setuptools` (which includes `pkg_resources`), which can cause problems
when installing the package in environments that don't have `setuptools`
installed. That's not often common in Pulumi projects, as the virtual
environment created by the CLI will include `setuptools`, however, if
creating the virtual environment manually with `python -m venv`,
`setuptools` is no longer included in the created virtual environment as
of Python 3.12.
Fixes#12414
Based on #14002, thanks @edgarrmondragon!
---------
Co-authored-by: Edgar Ramírez Mondragón <edgarrm358@gmail.com>
Co-authored-by: Fraser Waters <fraser@pulumi.com>
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945
This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.
Most of these edits were made by running `perfsprint --fix`.
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
There are very few direct uses of os.Stderr and os.Stdout in our tests.
Switch these to use iotest.LogWriter where appropriate,
so they log messages to the appropriate testing.TB.
In case of failure, these messages will be properly associated to the
test/subtest that they were for
instead of being printed all to stdout/stderr.
Uses of os.Stdout/Stderr that were omitted:
- Example tests: These don't have a testing.TB to log to.
- pulumi/main_test: Replaces the global os.Stdout/Stderr in TestMain.
LogWriter isn't relevant there.
Per team discussion, switching to gofumpt.
[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.
[1]: https://github.com/mvdan/gofumpt
See the full list of [Added rules][2], but it includes:
- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
`) {` on a separate line.
[2]: https://github.com/mvdan/gofumpt#Added-rules
gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.
gofumpt support is built into the tooling we use for development
so this won't change development workflows.
- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
(see [installation instrutions][3])
[3]: https://github.com/mvdan/gofumpt#installation
This change was generated by running:
```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```
The following files were manually tweaked afterwards:
- pkg/cmd/pulumi/stack_change_secrets_provider.go:
one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
`var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
long line because of error message
- pkg/backend/snapshot_test.go:
long line trying to assign three variables in the same assignment
I have included mention of gofumpt in the CONTRIBUTING.md.
* Respond to SIGINT
With the current state of the PR, crash on SIGINT. This is progress.
* Don't crash responding to SIGINT
* Close on cancel instead of terminate
* CL
* Fix lint
* Add ctx for node
* Be consistent for test context
* Don't run pyTestCheck unless there were mixins
On my machine this changes the python test time from ~340s to ~55s
* TS: assume subsidiary packages are correct
'--skipLibCheck true --skipDefaultLibCheck true' cuts ~50s off of the
total time needed in github.com/pulumi/pulumi/pkg/v3/codegen/nodejs.
* Add an EnumType to the PCL model
Languages implementation:
- [X] C#
- [X] Python
- [X] TypeScript
- [X] Go
* Fix assert
* Take namespace settings into account
* Implement enums for Go
* Update tests and add documentation.
* Add a test + handle output&unsafe
* Get outputs compiling for Go
* Line up types for Go
* Add nodejs
* Add Python
* Change expression form
* Fix nil check (`==` -> `!=`)
* Standardize nodejs & python checks
This should fix the Node OOM error.
* Rename enum-py to typed-enum-pp
* Don't reference schema types in `hcl2/model`
* Fix nits
* Pass missing tests for python
* Don't run missing nodejs tests
* Fix error message
* Re-enable "hyphen-url" (8370)
It seems like this was fixed in another commit. It passes now.
* Fix comment
* Move program tests into folders
* update package schema
* Enabled tests pass
* Fix lints and begin to update test cases
* Re-enable tests
* Update aws version to v4
* Refactor language specific parts
* Hook up dotnet and nodejs
* Update tests from master
* SSOT for schema/version
* Name blocking errors. Leave tests in valid state
* Give each language its own folder
* Remove SkipCompile for azure-sa (bug was fixed)
* Fix nits + changes asserts to require
* Remove unused import
* One last assert => require
When computing the type name for a field of an object type, we must
ensure that we do not generate invalid recursive struct types. A struct
type T contains invalid recursion if the closure of its fields and its
struct-typed fields' fields includes a field of type T. A few examples:
Directly invalid:
type T struct { Invalid T }
Indirectly invalid:
type T struct { Invalid S }
type S struct { Invalid T }
In order to avoid generating invalid struct types, we replace all
references to types involved in a cyclical definition with *T. The
examples above therefore become:
(1) type T struct { Valid *T }
(2) type T struct { Valid *S }
type S struct { Valid *T }
We do this using a rewriter that turns all fields involved in reference
cycles into optional fields.
These changes also include an enhancement to the SDK codegen test
driver in the interest of making iterating and debugging more convenient: if the -sdk.no-checks flag is passed, the driver will not run post-generation checks.
* Multi-pass, in-place checks for SDK codegen tests; toward working Python checks
* Remove temp debug output
* Upgrade Node
* Update dotnet; need to follow up on version.txt quirks
* WIP
* Sounds like we can use non-github package names to ensure things are local
* Fix simple-enum-schema
* Fix dash-named-schema
* Fix nested-module
* Start building a test-running pass
* Infer skipping tests from skipping compiles
* Move tree schma tests to a proper place
* Address lint issues on Go code
* Build against local Go SDK
* Update pkg/codegen/internal/test/sdk_driver.go
Co-authored-by: Ian Wahbe <ian@wahbe.com>
* Make go tests work by copying them into the tree from go-extras
* Fix lint
* Fix bad merge
* Manifest-based file discovery
* Remove version-related TODO from dotnet codegen
* Add doc comment
* Do not overwrite go.mod if found from mixins
* Accept python codegen change
* Accept node codegen
* Ignore lint issue
* Accept docs changes
Co-authored-by: Ian Wahbe <ian@wahbe.com>
We run the best static check we can on generated code, ensuring that it is valid.
* Run type checker against all languages (not docs)
* Fix package location, add some deps for schemas
* More tests passing
* These tests finally work
* Make linter happy
* Fix tests for merge from master
* Opt out of input-collision(nodejs) test
* Get more visibility into testing nodejs
* Fix type assumption
* Specify ts-node version
* Retrofit typescript dependencies for node14
* Give each go instance it's own module
* Attempt to diagnose remote go mod init failure
* Provide root for go mod init
* Make linter happy
The inputs and expected outputs for the tests are encoded using a
schema. Each property present in the schema forms a testcase; the
expected outputs for each language are stored in each property's
`Language` field with the language name "test". Expected outputs can be
regenerated using `PULUMI_ACCEPT`.
Rather than duplicating the list of tests and codegen driver across each
SDK, move its definition into `pkg/codegen/internal/test`. This has a
few notable benefits:
- All SDK code generators will be tested against each test. Though some
tests may exercise a particular code generator more than others, the
extra coverage will be generally beneficial.
- Adding a new test is simpler, as only a single file needs to be
changed.
- All SDKs now honor the `PULUMI_ACCEPT` environment variable for
updating baselines.
- Codegen tests now validate all generated files instead of only a
particular subset.
These changes support arbitrary combinations of input + plain types
within a schema. Handling plain types at the property level was not
sufficient to support such combinations. Reifying these types
required updating quite a bit of code. This is likely to have caused
some temporary complications, but should eventually lead to
substantial simplification in the SDK and program code generators.
With the new design, input and optional types are explicit in the schema
type system. Optionals will only appear at the outermost level of a type
(i.e. Input<Optional<>>, Array<Optional<>>, etc. will not occur). In
addition to explicit input types, each object type now has a "plain"
shape and an "input" shape. The former uses only plain types; the latter
uses input shapes wherever a plain type is not specified. Plain types
are indicated in the schema by setting the "plain" property of a type spec
to true.
Python resource constructor overloads were recently added that accept a
`<Resource>Args` class for input properties, as an alternative to the
other constructor overload that accepts keyword arguments. The name of
the new args class is the name of the resource concatenated with an
`Args` suffix.
Some providers (e.g. Kubernetes, Azure Native, and Google Native) have
input types with the same name as resources in the same module, which
results in two different `<Resource>Args` classes in the same module.
When you try to use the new args class with the constructor, e.g.:
```python
pulumi_kubernetes.storage.v1.StorageClass(
resource_name='string',
args=pulumi_kubernetes.storage.v1.StorageClassArgs(...),
opts=pulumi.ResourceOptions(...),
)
```
You run into an error, because
`pulumi_kubernetes.storage.v1.StorageClassArgs` is actually referring to
the existing input type rather than the intended `StorageClassArgs`
class for the constructor arguments.
Having the duplicate classes hasn't broken existing usage of the input
type because we "export" all the input types for a module _after_ all
the resources and resource args classes are exported, so the input type
just ends up "overwriting" the duplicate resource args class.
Other languages don't have this problem because the input type is either
in it's own module/namespace (e.g. Node.js and .NET) or a different name
is used for the input type (Go). But with Python, the input types and
resources are all available in the same module.
To address this for Python, when there is an input type in the same
module with the same name as the resource, the args class for the
resource will be emitted as `<Resource>InitArgs` instead of
`<Resource>Args`.
This commit adds a new language option for Python generation to specify
the package name instead of using `pulumi_x` where x is the name defined
in the schema.
A new test is added, and this has also been shown to produce no diff
when run against `pulumi-eks`.
* Import subpackages lazily
* Tighten up lazy_import impl
* Eagerly register resources, but lazily load their impl
* Add CHANGELOG entry
* Satisfy lint
* Restore mypy behavior so the change is not breaking
* Fix golden tests
This change adds schema and codegen support for plain properties which
are emitted typed as the plain type rather than wrapped as an `Input`.
Plain properties require a prompt value and do not accept a value that
is `Output`.