pulumi/sdk/go/common/workspace/templates_zip.go

197 lines
4.8 KiB
Go
Raw Permalink Normal View History

Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
// Copyright 2016-2023, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package workspace
import (
"archive/zip"
"bytes"
"fmt"
"io"
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
"math"
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
"time"
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
)
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
const RetryCount = 6
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
// Sanitize archive file pathing from "G305: Zip Slip vulnerability"
func sanitizeArchivePath(d, t string) (v string, err error) {
v = filepath.Join(d, t)
if strings.HasPrefix(v, filepath.Clean(d)) {
return v, nil
}
return "", fmt.Errorf("%s: %s", "content filepath is tainted", t)
}
func isZIPTemplateURL(templateNamePathOrURL string) bool {
parsedURL, err := url.Parse(templateNamePathOrURL)
if err != nil {
return false
}
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
return parsedURL.Path != "" && strings.HasSuffix(parsedURL.Path, ".zip")
}
func retrieveZIPTemplates(templateURL string) (TemplateRepository, error) {
var err error
// Create a temp dir.
var temp string
if temp, err = os.MkdirTemp("", "pulumi-template-"); err != nil {
return TemplateRepository{}, err
}
Simplifies URL parsing for pulumi new <zip> (#14546) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Currently, we add a bit of "cleverness" to our URL parsing, by removing `.zip` suffixes when we've determined one exists (and started down the path of pulumi new-ing a zip ref). That requires upstream API providers to host their zipfiles at a location _without_ that suffix, which precludes any existing .zip files hosted at places like github etc which may have that extension. This PR removes that logic so we serve as a simple passthrough for whichever URL we're passed (extending the existing contract). It also depends on https://github.com/pulumi/pulumi.ai/pull/414 being merged and deployed. Fixes https://github.com/pulumi/pulumi.ai/issues/415 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [ ] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-13 17:08:52 +00:00
parsedURL, err := url.Parse(templateURL)
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
if err != nil {
return TemplateRepository{}, err
}
var fullPath string
if fullPath, err = RetrieveZIPTemplateFolder(parsedURL, temp); err != nil {
return TemplateRepository{}, fmt.Errorf("failed to retrieve zip archive: %w", err)
}
return TemplateRepository{
Root: temp,
SubDirectory: fullPath,
ShouldDelete: true,
}, nil
}
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
func backoff(retries int) time.Duration {
return time.Duration(math.Pow(2, float64(retries))) * (time.Second / 4)
}
func shouldRetry(err error, resp *http.Response) bool {
if err != nil {
return true
}
if resp.StatusCode == http.StatusBadGateway ||
resp.StatusCode == http.StatusServiceUnavailable ||
resp.StatusCode == http.StatusGatewayTimeout ||
resp.StatusCode == http.StatusNotFound {
return true
}
return false
}
func drainBody(resp *http.Response) {
if resp.Body != nil {
_, err := io.Copy(io.Discard, resp.Body)
if err != nil {
return
}
resp.Body.Close()
}
}
type retryableTransport struct {
transport http.RoundTripper
}
func (t *retryableTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// Clone the request body
var bodyBytes []byte
if req.Body != nil {
bodyBytes, _ = io.ReadAll(req.Body)
req.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
}
// Send the request
resp, err := t.transport.RoundTrip(req)
// Retry logic
retries := 0
for shouldRetry(err, resp) && retries < RetryCount {
// Wait for the specified backoff period
time.Sleep(backoff(retries))
// We're going to retry, consume any response to reuse the connection.
drainBody(resp)
// Clone the request body again
if req.Body != nil {
req.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
}
// Retry the request
resp, err = t.transport.RoundTrip(req)
retries++
}
// Return the response
return resp, err
}
func NewRetryableClient() *http.Client {
transport := &retryableTransport{
transport: &http.Transport{},
}
return &http.Client{
Transport: transport,
}
}
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
func RetrieveZIPTemplateFolder(templateURL *url.URL, tempDir string) (string, error) {
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
if templateURL.Scheme == "" {
return "", fmt.Errorf("invalid template URL: %s", templateURL.String())
}
client := NewRetryableClient()
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
packageRequest, err := http.NewRequest(http.MethodGet, templateURL.String(), bytes.NewReader([]byte{}))
if err != nil {
return "", err
}
packageRequest.Header.Set("Accept", "application/zip")
Adds Pulumi AI integrations with Pulumi New (#14685) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Now that we support .zip archive sources for Pulumi New, we have all of the API surface we need to provide a full Pulumi New experience using Pulumi AI. This PR introduces a few modes of interacting with Pulumi AI to generate Pulumi projects. - The default `pulumi new` experience now begins with a choice between `ai` and `template` modes - the `template` mode represents the current state of running `pulumi new`, while `ai` provides an interactive experience with Pulumi AI. - The user can iteratively ask questions to improve or change the resulting program - each time a prompt is completed, they are asked to `refine`, `no`, or `yes` their session - `refine` allows a follow-on prompt to be submitted. `no` ends the session without generating a pulumi project, and `yes` generates a Pulumi project from the most recent program returned by Pulumi AI. - Additionally, top-level flags, `--ai` and `--language` are provided to fill in default values for the AI mode. When a prompt is provided with a language, it is automatically submitted to Pulumi AI - if either is missing, the user is prompted for whichever value is missing. Fixes https://github.com/pulumi/pulumi.ai/issues/441 Fixes https://github.com/pulumi/pulumi.ai/issues/443 Fixes https://github.com/pulumi/pulumi.ai/issues/444 Depends on https://github.com/pulumi/pulumi.ai/pull/472 Depends on https://github.com/pulumi/pulumi.ai/pull/507 Depends on https://github.com/pulumi/pulumi.ai/pull/508 ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. --> --------- Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
2023-12-20 22:08:09 +00:00
packageResponse, err := client.Do(packageRequest)
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
if err != nil {
return "", err
}
packageResponseBody, err := io.ReadAll(packageResponse.Body)
if err != nil {
return "", err
}
archive, err := zip.NewReader(bytes.NewReader(packageResponseBody), int64(len(packageResponseBody)))
if err != nil {
return "", err
}
for _, file := range archive.File {
filePath, err := sanitizeArchivePath(tempDir, file.Name)
if err != nil {
return "", err
}
Adds nested directory support to pulumi new .zip (#14473) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> The previous ZIP implementation here only supported a flat .zip file structure - Pulumi Java projects often contain nested directories (and many templates will likely organize code in such a way that involves directories as well). This inspects each file during traversal of the ZIP archive, and creates any necessary directories it encounters along the way. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [x] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [x] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-11-01 21:59:29 +00:00
if file.FileHeader.FileInfo().IsDir() {
err = os.MkdirAll(filePath, 0o777)
if err != nil {
return "", err
}
} else {
fileReader, err := file.Open()
if err != nil {
return "", err
}
defer fileReader.Close()
destinationFile, err := os.Create(filePath)
if err != nil {
return "", err
}
defer destinationFile.Close()
_, err = io.Copy(destinationFile, fileReader) // #nosec G110
if err != nil {
return "", err
}
Adds remote .zip archive support to `pulumi new` (#14443) <!--- Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation. --> # Description <!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. --> Fixes https://github.com/pulumi/pulumi.ai/issues/363 This change introduces a new accepted suffix to `pulumi new` HTTP arguments - `.zip`. When a URL of the form `http[s]://<url>.zip` is provided to `pulumi new`, we send a GET request to `<url>` (without the .zip suffix) with the `accept` header set to `application/zip`. It is then up to whatever is hosted at that location to correctly return a binary zip archive in its response. We then parse that response in memory and write each file to a tempdir as we do with current templates, which can then be picked up by the standard `pulumi new` templating logic. ## Checklist - [x] I have run `make tidy` to update any new dependencies - [x] I have run `make lint` to verify my code passes the lint check - [ ] I have formatted my code using `gofumpt` <!--- Please provide details if the checkbox below is to be left unchecked. --> - [ ] I have added tests that prove my fix is effective or that my feature works <!--- User-facing changes require a CHANGELOG entry. --> - [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change <!-- If the change(s) in this PR is a modification of an existing call to the Pulumi Cloud, then the service should honor older versions of the CLI where this change would not exist. You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add it to the service. --> - [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Cloud API version <!-- @Pulumi employees: If yes, you must submit corresponding changes in the service repo. -->
2023-10-31 22:05:39 +00:00
}
}
return tempDir, nil
}