Commit Graph

126 Commits

Author SHA1 Message Date
Thomas Gummerer c5889864c7
plugins: don't panic when the given path is not clean ()
When the user specifies a provider with a specific path in the
`Pulumi.yaml`, we later in the program assert that the path that was
passed in also matches with the path where we found the plugin. We do
this by using a string comparison, however that doesn't work if the path
the user passes is not clean, e.g. has a trailing slash, or has a double
slash, or some such.

Fix this by using `filepath.Clean` on the user supplied path, which
these things up.

I was also briefly wondering if this works properly if the user passes
in a path that is a symlink (it does), and wrote a test for that,
checking that behaviour.

Fixes https://github.com/pulumi/pulumi/issues/17130
2024-09-04 10:08:44 +00:00
Fraser Waters f8d05644e3
Fix GetPluginInfo with shimless project plugins ()
Before this fix GetPluginInfo would error because stat on the expected
`pulumi-resource-exe` file would fail (because it didn't exist). This
fixes it to fallback to looking at the folder instead.
2024-08-30 15:51:15 +00:00
Julien 487b4a8494
Install missing python versions using pyenv during installation ()
Adds the flag `--use-language-version-tools` to `pulumi install`. When
passed, and `pyenv` is installed, and a `.python-version` file is in the
project directory or any of its parent directories, Pulumi will install
the requested python version if it is not already installed.

`LanguageRuntime.InstallDependencies` now takes a struct
`InstallDependenciesRequest` as argument to make it easier to extend
this call with options.

See https://github.com/pulumi/pulumi-docker-containers/pull/232

---------

Co-authored-by: Thomas Gummerer <t.gummerer@gmail.com>
2024-08-19 15:55:54 +00:00
Mikhail Shilkov d4f1cf5c87
URL-based plugin source overrides via env var ()
### Motivation

Pulumi plugin binaries can be downloaded by the CLI from multiple
sources. By default, it's downloaded from Pulumi's GitHub releases or
get.pulumi.com, but plugins can also specify their binary sources via
the `PluginDownloadURL` schema option. They can point to custom GitHub,
Gitlab, or HTTP locations.

Enterprise customers ask for a way to isolate the CLI from downloads
from random locations and to configure the CLI to go to their internal
pre-approved artefact location instead. This way, Pulumi can run in
"air-gapped" environments (which still have access to Cloud APIs, of
course).

Related issues:
- https://github.com/pulumi/pulumi/issues/14459
- https://github.com/pulumi/pulumi/issues/16240

Currently, there is a basic mechanism to do so via the variable
`pluginDownloadURLOverrides`, but it has two major limitations:
- The variable value is set via a compile-time flag, so it requires a
custom build of the CLI
- The overrides are based on the plugin name, so the rules must be
defined without access to the original URL, which makes it hard to
provide universal rules and still distinguish between first-party,
public third-party, or private in-house plugins
- We ignore overrides for all plugins that have `PluginDownloadURL` set
- Overrides can set a plugin replacement redirect only to HTTP(s)
addresses

### Proposal

This PR makes two sets of changes:

1. It allows passing overrides via the
`PULUMI_PLUGIN_DOWNLOAD_URL_OVERRIDES` environment variable. The
compile-time flag is still supported, but the env var takes priority.

More configuration levers could be supported, but it not clear if we
have good ones until [Support .pulumirc file for global
config](https://github.com/pulumi/pulumi/issues/13484) is implemented. I
don't expect users to want to set this via their stack configs, but I'm
curious what others think. In any case, more sources can be added later.

2. The overrides now apply based on the original download URL, not just
on plugin names. Actually, it's the base URL of a download source that
is passed to the regexp matcher. Examples of possible options are:

- `github://api.github.com/pulumi/pulumi-xyz` for a first-party plugin
(note that we don't pass `get.pulumi.com`
- `github://api.github.com/pulumiverse/pulumi-grafana` for a community
plugin that sets `PluginDownloadURL`
- `gitlab://gitlab-host/proj-name` for a community plugin hosted on
Gitlab
    - `https://example.com/downloads/` for HTTP sources

So, the override
`^github://api.github.com/pulumi/pulumi-xyz=https://example.com/downloads/pulumi-xyz/`
will override the single provider URL from our GitHub releases to the
given HTTP location.

On top of that, regular expressions may contain name groups to capture
and use templated values. For example,
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/${org}/${repo}`
captures any GitHub plugin and redirects it to its corresponding HTTP
location. Group indices are also supported: the above override can also
be written as
`^github://api.github.com/(?P<org>[^/]+)/(?P<repo>[^/]+)=https://example.com/downloads/$1/$2`,
with `$0` meaning the full match.

The override URLs have the same semantics as `PluginDownloadURL`, so
they can point to GitHub, Gitlab, HTTP, or anything we introduce in the
future.

### Impact

Technically, this is a breaking change, because name-based overrides
will stop working. However, we are fairly certain that we have a single
customer using the existing compile-time approach, and they indicated
that they don't need the name-based overrides if they have URL-based
overrides. I reviewed this PR with them and made sure they can migrate
immediately after the change is released.

Backwards compatibility is slightly tricky, because we'd need to keep
name-based override _and_ not applying them to third-party plugins. But
we can do it if necessary.

Resolve 
2024-07-26 10:37:09 +00:00
Julien P 98b90f1902
Add packagemanager prompt to pulumi new for nodejs ()
https://github.com/pulumi/pulumi/pull/16346 introduced the capability to
query the language runtime for additional prompts. We use this to let
the user pick a package manager among npm, yarn and pnpm during `pulumi
new` when using the nodejs runtime.

When there is no explicitly configured package manager, we re-use the
previous behaviour for determining the package manager (check
`PULUMI_PREFER_YARN` env variable, look for lock files).

Defaults to `npm` when running `new` in non-interactive mode.
2024-06-21 11:35:06 +00:00
Zaid Ajaj 2766475bd1
[cli/plugin] Fix plugin install command when plugin type is tool ()
# Description

When trying to install a plugin of type `tool` from GitHub, the Pulumi
convention is to have these plugins available in repositories named
`pulumi-tool-<name>` so that we can install them via the CLI as follows:
```
pulumi plugin install tool <name>
```
However, today this fails because we don't prefix the repository name
correctly with `"pulumi-tool-"`. This PR fixes that. Tested against
[Zaid-Ajaj/pulumi-tool-importer](https://github.com/Zaid-Ajaj/pulumi-tool-importer)

Also removes hardcoded plugin download URL for known converter plugins
of mine. These are moved to the `pulumi` organisation and no longer
require a separate URL.

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-06-17 13:25:57 +00:00
Mikhail Shilkov 642cb5b5c7
Revert "Prefer pluginDownloadURLOverrides over PluginDownloadURL specified in the package" ()
Reverts 
Resolves https://github.com/pulumi/pulumi/issues/16316
2024-06-04 17:37:34 +00:00
Julien P 578e0937a9
[Python] Move existing dependency installation and python command invocation to a Toolchain interface ()
# Description

This PR refactors the existing Python dependency installation and
command running code to use the `Toolchain` interface. This will make it
possible to swap out the default Pip based toolchain for a Poetry based
toolchain.

Fixes https://github.com/pulumi/pulumi/issues/16285
Ref https://github.com/pulumi/pulumi/issues/15937

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`
2024-06-03 13:52:27 +00:00
Mikhail Shilkov 3e0aedeee2
Prefer pluginDownloadURLOverrides over PluginDownloadURL specified in the package ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

Overriding plugin download URLs with compilation flags was originally
added in . Its intent was allowing our customers to override
download locations for all plugins, so that only trusted pre-approved
plugins could be downloaded.

Since then, we've added `PluginDownloadURL` for a package, which is the
default URL for that package's binary if it's shipped outside our Pulumi
org. Currently, `PluginDownloadURL` takes precedence over
`pluginDownloadURLOverrides`, which means it's impossible to override
third-party package binary locations.

This PR changes plugin source resolution to flip the priority of those
two. If an override matches regex, its URL will take priority over the
default `PluginDownloadURL` specified in the package.

I have added tests to verify `pluginDownloadURLOverrides` with and
without `PluginDownloadURL`. The second one fails before my change.

Resolves 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-05-13 14:35:44 +00:00
Germán Lena d7f24dfcfb
Refactor: move plugin kind to apitype ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

This PR moves PluginKind to apitype to prevent circular dependencies
when adding apitype as a dependency of the workspace module.
It also re-exports PluginKind to keep backward compatibility

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-04-25 17:30:30 +00:00
​Andrzej Ressel 77e8d8ffee
Append .exe when installing local language plugins ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes 

## Checklist

- [ ] I have run `make tidy` to update any new dependencies
- [ ] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [X] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2024-03-17 22:26:22 +00:00
Fraser Waters 721d61115b
Add `plugin run` command ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Mostly for providers to experiment with, currently hidden behind
"PULUMI_DEV".
2024-02-05 08:35:48 +00:00
Thomas Gummerer e4c5d250fa
add a CR before when finishing the progress bar ()
We are using our terminal in raw mode, which means we're not getting CRs
automatically added. Add one after a progress bar has finished to
improve the output slightly.

The final output after this looks like: 

```
$ pulumi preview                     
Please choose a stack, or create a new one: dev
Previewing update (dev):
Downloading plugin: 18.39 MiB / 18.39 MiB [========================] 100.00% 15s

[resource plugin docker-3.6.1] installing
Downloading plugin: 20.19 MiB / 20.19 MiB [========================] 100.00% 15s

[resource plugin awsx-1.0.5] installing
Downloading plugin: 164.18 MiB / 164.18 MiB [======================] 100.00% 32s

[resource plugin aws-5.42.0] installing
[...]
```

Which seems slightly better, but not a huge improvement, and the
progress bar is unfortunately also still jumpy. I'll write down the rest
of my learnings here in https://github.com/pulumi/pulumi/issues/14250.

I'm not entirely sure it's even worth merging this, but I'm putting this
up as a PR for further discussion and to show where the investigation
led.
2024-01-23 09:27:26 +00:00
Fraser Waters 3560333ae6
Clean up uses of .Error() ()
Combination of a few cleanups.

1. Don't call .Error() on errors that are being passed to "%s" format
functions. Format will call `Error()` itself.
2. Don't call assert.Error then assert.Equal/Contains, just use
assert.ErrorEqual/ErrorContains instead.
3. Use "%w" if appropriate, instead of "%v"/"%s".
2023-12-20 15:54:06 +00:00
Fraser Waters 16d9f4c167
Enable perfsprint linter ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Prompted by a comment in another review:
https://github.com/pulumi/pulumi/pull/14654#discussion_r1419995945

This lints that we don't use `fmt.Errorf` when `errors.New` will
suffice, it also covers a load of other cases where `Sprintf` is
sub-optimal.

Most of these edits were made by running `perfsprint --fix`.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-12 12:19:42 +00:00
Fraser Waters 1f28042b2d
Prefer stable plugin release to pre-releases ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Fixes https://github.com/pulumi/pulumi/issues/14680.

Updated the plugin logic (which we still use when no explict version is
given) to prefer selecting a stable version over a pre-release version
when no explict requested version is given.


## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [ ] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-12-03 09:15:07 +00:00
Fraser Waters a6050592f1
Support {NAME} in plugin download url ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Allows `${NAME}` in the download url template string. 

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-10-30 09:29:12 +00:00
Fraser Waters 0ade454f11
Allow language plugins to return plugin checksums ()
<!--- 
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->

Also fix an issue where if a platform was missing a checksum it would
error with "invalid checksum, expected , actual 01234".

None of the language runtimes yet return anything for this, but it's a
simple plumbing to expose it for the future.

We'll _probably_ start adding checksums to the pulumi-plugin.json files,
and then GetRequiredPlugins can simply return that data.

## Checklist

- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
  - [x] I have formatted my code using `gofumpt`

<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
2023-08-25 15:26:25 +00:00
Fraser Waters a691975202 Warn about ambient plugins loaded from $PATH
By default Pulumi will load ambient plugins from $PATH before looking in
the plugins directory or at bundled plugins.

While this is very useful for development it often causes confusion when
people have forgotten that they have plugins left on $PATH.

This makes the use of these $PATH plugins a diagnostic warning to try
and make that failure mode a little less silent.

Normal users shouldn't ever have plugins on $PATH and so won't see this
new warning.

Re-instates https://github.com/pulumi/pulumi/pull/13607 with a fix for
symlinks included.
2023-08-08 13:11:34 +01:00
Robbie McKinstry c8d331ca37 Check if Plugin is bundled before installing
This commit adds a check to `pulumi plugin install` to confirm
that the plugin is not bundled with Pulumi before performing the download.
Without this check, the CLI will produce a 404 error because the language
plugins aren't distributable independently of the CLI executable.

If PULUMI_DEV is set we'll skip the error and try to download anyway.
2023-08-07 15:33:12 +01:00
Kyle Dixler 86ebe1bbd3 Revert "Warn about ambient plugins loaded from $PATH" 2023-08-04 16:54:16 -07:00
Fraser Waters a5b1590499 Warn about ambient plugins loaded from $PATH
By default Pulumi will load ambient plugins from $PATH before looking in
the plugins directory or at bundled plugins.

While this is very useful for development it often causes confusion when
people have forgotten that they have plugins left on $PATH.

This makes the use of these $PATH plugins a diagnostic warning to try
and make that failure mode a little less silent.

Normal users shouldn't ever have plugins on $PATH and so won't see this
new warning.
2023-07-27 17:59:44 +01:00
Fraser Waters 753a21daa2 Fix lookup of side-by-side binaries when PULUMI_IGNORE_AMBIENT_PLUGINS is set 2023-07-19 11:47:00 +01:00
Fraser Waters 1949f3a920 Fix http source URL interpolation
Turns out this wasn't tested and regressed in
51cc6461df
when we started URL parsing the input string and so ended up with a text
string with "%7B" instead of "{" to replace.

This fixes the interpolation and changes a couple of the http tests to
actually use this feature so it's tested.
2023-07-10 11:05:30 +01:00
Fraser Waters 591774afb1 Handle installing plugins without versions in the defaultHost
Maybe fixes https://github.com/pulumi/pulumi-terraform-bridge/issues/1209
Fixes https://github.com/pulumi/pulumi-terraform-bridge/issues/1200

When looking up schemas we would call into `defaultHost.InstallPlugin` to
install any missing plugins, and we would generally _not_ have version
information for that. Unfortunately `defaultHost.InstallPlugin` didn't
handle the case of version not being set, double unfortunately it also
didn't error.

This fixes that install path to call `GetLatestVersion` if version isn't
set.
2023-06-12 12:22:08 +01:00
Fraser Waters 768f329177 Don't lookup bundled plugins on path if IGNORE_AMBIENT_PLUGINS is set
Fixes https://github.com/pulumi/pulumi/issues/13087
2023-06-02 17:49:55 +01:00
Lee Briggs f42268be21 add rate limited url to http requests 2023-04-24 16:16:35 +01:00
Fraser Waters 2795e9b059 Log that we're falling back to get.pulumi.com 2023-04-12 15:01:54 +01:00
Abhinav Gupta f9458a3510
sdk/workspace/pluginDownloader: Re-use util/retry
The retry logic wants to make at most 5 attempts
(that's what the "limit" says)
but it actually makes 6 requests.

This fixes the off-by-one error
by switching to the re-usable util/retry package
instead of re-implementing retries here.

As with util/retry.Retryer,
pluginDownloader allows overriding time.After from tests,
so that we don't sleep in tests.
2023-03-24 18:11:40 -07:00
Abhinav Gupta 82e8aeba0b
sdk/workspace/pluginDownloader: Fix multiplied retries
The pluginDownloader retries downloads with specialized logic,
and it relies on getHTTPResponse to make the HTTP requests.
However, getHTTPResponse retries the requests *it* makes as well.

This changes getHTTPResponse to not perform any retries,
and makes a copy of it, getHTTPResponseWithRetry, that does.
The pluginDownloader uses getHTTPResponse,
and other prior uses of getHTTPResponse have switched to the new one.

This resolves the multiplied retries for pluginDownloader,
however there's still an off-by-one error.
2023-03-24 18:11:10 -07:00
Abhinav Gupta e5aafcd382
refactor(sdk/workspace/DownloadToFile): Pull state into a struct
All state for DownloadToFile was in a bunch of anonymous functions
that called each other.
The only way to inject new options was via new positional parameters.

This turns the core logic of DownloadToFile into
a pluginDownloader struct,
with retry and wrapper injected as fields rather than parameters.
It further breaks out all anonymous functions in DownloadToFile
into methods on that struct.
Their order has been retained to make this easy to review.

This change contains no logic changes.
2023-03-24 18:07:42 -07:00
Fraser Waters b0ab245078 Move automatic plugin-install to engine 2023-03-20 23:41:07 +00:00
Fraser Waters 6d5b255184 Hardcode repo for yaml converter 2023-03-13 18:30:01 +00:00
Fraser Waters 5013d427e6 Add converter plugin 2023-03-09 22:16:21 +00:00
Fraser Waters 67f26e5a51 Don't send empty token/Bearer headers 2023-03-09 09:43:04 +00:00
Abhinav Gupta e395deef6b
all: Assert => Assertf
Migrates all remaining usages of
`contract.Assert*` and `contract.Require*` to the f variants,
which require adding meaningful error messages.

There were a couple cases where a `testing.T` or `testing.B`
was already available.
For those, this uses t.FailNow or require.NoError.

Refs 
2023-03-03 14:37:43 -08:00
Abhinav Gupta 7aa5b77a0c
all: Reformat with gofumpt
Per team discussion, switching to gofumpt.

[gofumpt][1] is an alternative, stricter alternative to gofmt.
It addresses other stylistic concerns that gofmt doesn't yet cover.

  [1]: https://github.com/mvdan/gofumpt

See the full list of [Added rules][2], but it includes:

- Dropping empty lines around function bodies
- Dropping unnecessary variable grouping when there's only one variable
- Ensuring an empty line between multi-line functions
- simplification (`-s` in gofmt) is always enabled
- Ensuring multi-line function signatures end with
  `) {` on a separate line.

  [2]: https://github.com/mvdan/gofumpt#Added-rules

gofumpt is stricter, but there's no lock-in.
All gofumpt output is valid gofmt output,
so if we decide we don't like it, it's easy to switch back
without any code changes.

gofumpt support is built into the tooling we use for development
so this won't change development workflows.

- golangci-lint includes a gofumpt check (enabled in this PR)
- gopls, the LSP for Go, includes a gofumpt option
  (see [installation instrutions][3])

  [3]: https://github.com/mvdan/gofumpt#installation

This change was generated by running:

```bash
gofumpt -w $(rg --files -g '*.go' | rg -v testdata | rg -v compilation_error)
```

The following files were manually tweaked afterwards:

- pkg/cmd/pulumi/stack_change_secrets_provider.go:
  one of the lines overflowed and had comments in an inconvenient place
- pkg/cmd/pulumi/destroy.go:
  `var x T = y` where `T` wasn't necessary
- pkg/cmd/pulumi/policy_new.go:
  long line because of error message
- pkg/backend/snapshot_test.go:
  long line trying to assign three variables in the same assignment

I have included mention of gofumpt in the CONTRIBUTING.md.
2023-03-03 09:00:24 -08:00
Fraser Waters c4083d3821 Nicer github rate limit error 2023-03-02 21:33:38 +00:00
bors[bot] 51cc6461df
Merge
12190: Rename http source r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Just a little tidy up some things I noticed while working on the gitlab source.
1. Rename pluginURLSource to httpSource. A plugin URL is _anything_ that is given as a pluginDownloadURL. This type only handles the downloads from http/https locations.
2. Make GetSource error for unrecognized source schemes. If we add new schemes in the future (fairly likely) if someone tries to use one of the new schemes on an old CLI they'll get a fairly comprehensible error that the scheme wasn't recognized rather than some cryptic error from however the http library fails trying to parse the strange (to it) url.

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change - Not really user facing.
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
2023-02-17 09:20:38 +00:00
Fraser Waters 4ed7fb78bf Rename http source 2023-02-16 19:51:37 +00:00
bors[bot] 58af94753c
Merge
12180: sdk/go: Prefer contract.Assertf over Assert r=abhinav a=abhinav

Migrates all uses of contract.{Assert, AssertNoError, Require} in sdk/
to the `*f` variants that are required to provide more error context.

Step towards deprecating non-f variants entirely.

For context, `contract.Require` is similar to `contract.Assert`,
except it has a required parameter name as an argument:

    func Require(cond bool, param string)
    func Requiref(cond bool, param string, msg string, args ...any)

It includes the parameter name in the error message by default,
so the `msg` and `args` should only describe the constraint
without naming the parameter.

Refs 


Co-authored-by: Abhinav Gupta <abhinav@pulumi.com>
2023-02-15 20:11:58 +00:00
Abhinav Gupta 7bddec255d
sdk/go: Prefer contract.Assertf over Assert
Migrates all uses of contract.{Assert, AssertNoError, Require} in sdk/
to the `*f` variants that are required to provide more error context.

Step towards deprecating non-f variants entirely.

For context, `contract.Require` is similar to `contract.Assert`,
except it has a required parameter name as an argument:

    func Require(cond bool, param string)
    func Requiref(cond bool, param string, msg string, args ...any)

It includes the parameter name in the error message by default,
so the `msg` and `args` should only describe the constraint
without naming the parameter.

Refs 
2023-02-15 10:22:43 -08:00
Fraser Waters 99ebe52db6 Add gitlab source 2023-02-15 10:18:44 +00:00
Fraser Waters eb4565936f Replace pkg/errors in sdk/go/common/workspace 2023-01-19 20:22:54 +00:00
Robbie McKinstry 1f78baae71
Preallocate slices with a known capacity.
Enable the prealloc linter, which identifies slices
with a known capacity, but are not preallocated, which
results in unnecessary allocations and memcpys.
2023-01-11 12:52:51 -08:00
Abhinav Gupta 1158d4acee
all: Drop ioutil
Stop using io/ioutil across the entire repository.
The io/ioutil package was deprecated in Go 1.16 (2021-02)
with replacements provided in other packages.
Specifically:

    ioutil.Discard   => io.Discard
    ioutil.NopCloser => io.NopCloser
    ioutil.ReadAll   => io.ReadAll
    ioutil.ReadFile  => os.ReadFile
    ioutil.TempDir   => os.MkdirTemp
    ioutil.TempFile  => os.CreateTemp
    ioutil.WriteFile => os.WriteFile

This change switches all of these entities
across the repository.

Following this change,
the only references to ioutil are in schema files:

    % rg -l ioutil
    pkg/codegen/testing/test/testdata/aws-4.26.0.json
    pkg/codegen/testing/test/testdata/aws-4.36.0.json
    pkg/codegen/testing/test/testdata/aws-4.37.1.json
    pkg/codegen/testing/test/testdata/aws-5.4.0.json
    pkg/codegen/testing/test/testdata/aws-5.16.2.json

The bulk of this change was generated automatically
with manual touch ups afterwards.
2023-01-06 16:35:14 -08:00
bors[bot] 40e0ac20c9
Merge
11362: Library for environmental variable access r=iwahbe a=iwahbe

Add a library to encapsulate environmental variable declaration, lookup and parsing. 

### Usage
Variables strongly typed and declared in the module scope like so:
```go
// Automatically prefixed with PULUMI_ by default
var Experimental = env.Bool("EXPERIMENTAL", "Enable experimental options and commands")
```

Values are accessed by calling `.Value()` on the module level variable:
```go
if Experimental.Value() { // .Value() returns a bool here
  // do the new exiting thing
} else {
  // do the well tested thing
}
```

Environmental variables can be predicated on other boolean type variables:
```go
var NewThingParam = env.String("NEW_THING", "pass this to the new thing", 
    env.Needs(Expiremental))
```

Predicated variables will only show up as set if there predicate is `true`:

```go
if v := NewThingParam.Value(); v != "" {
  do_thing(v)
} else {
  // do the well tested thing
}
```
The above `if` check is equivalent to 
```go
if v = os.Getenv("PULUMI_NEW_THING"); v != "" && cmduilt.IsTruthy(os.Getenv("PULUMI_EXPERIMENTAL"))
```

This makes marking and unmarking a variable as experimental a 1 line change.

--- 

All declared variables can then be iterated on for documentation with `env.Variables()`. This is how we can use this lib to build documentation for our environmental variables.

### Assumptions

#### Immutability 
The library assumes that environmental variables are inherited from the environment. Values are captured during program startup, changes to the environment are not observed. We are not resilient to environmental variables changing out from under us, so this complies with current usage. This assumption can be changed without breaking the API.

#### Known values
The library assumes that the names of environmental variables are known at compile time. If the variable to be looked up is derived from user input, a different mechanism should be used.

### Adoption

Adopting this system over our pulumi/pkg codebase will take a while. My plan is to merge in this PR without moving over our existing env vars from `os.Getenv`. We can then gradually move over existing env vars piecemeal in subsequent PRs. Any new env vars added will use the new system.

Prerequisite for https://github.com/pulumi/pulumi/issues/10747.

### Other Options - Viper
Viper looks like an excellent config retrieval package, but doesn't support at-site documentation. It also has less type safety then I would prefer as a top level api. If it would be helpful, we could switch the underlying mechanism from `os.Lookup` to `viper.Get*` without changing the API this library exposes. I don't think viper is necessary right now, but adopting this library doesn't preclude viper. 

Co-authored-by: Ian Wahbe <ian@wahbe.com>
2022-12-15 12:15:34 +00:00
Ian Wahbe 84f50460e3 Add `pulumi env` commmand 2022-12-14 15:41:42 +01:00
Ian Wahbe 8ccb7fd4b8 Library for environmental variable access
Change function args to opts

Add tests

Support `env.Needs` api

Allow concrete environments
2022-12-14 15:41:42 +01:00
Christian Nunciato 2692efc42e
Add a space in plugin error message 2022-12-13 12:20:43 -08:00