The existing overlays (e.g. Chart v3 in Kubernetes, or CallbackFunction
in AWS) are not available in every language Pulumi supports. This often
confuses users because the generated docs include all languages Pulumi
supports (e.g. see
https://github.com/pulumi/pulumi-kubernetes/issues/2181).
To solve that problem, this change adds a new optional parameter to the
schema that allows configuring the languages an overlay (resource or
function) supports.
To support this in docsgen the existing Language Chooser
(`LangChooserLanguages`) of resources is made configurable and extended
to functions.
Note: This doesn't support resource methods right now. They'll need
extra handling because and overlay resource method might not support all
of the languages its resource supports. I'll tackle this in a follow up
PR.
Here's a screenshot of how this will look like for the Helm v3 chart for
example:
<img width="1046" alt="Screenshot 2024-07-01 at 16 11 23"
src="https://github.com/pulumi/pulumi/assets/2453580/b1a1365a-6dee-4099-829a-2859639a4c8c">
The PR contains the following commits. I'd recommend to look at the
first three ones and then check the regenerated golden files in the last
one:
- **Add schema parameter to constrain supported languages for overlays**
- **Update developer docs and changelog**
- **Refactor LanguageChooser and always pass supported languages**
- **Regenerate testdata**
relates to #13231
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
Originally motivated by uncontrolled pseudo-recurcive type expansion in
AWS WafV2, we've limited the number of types that we should in the docs
to 200: https://github.com/pulumi/pulumi/pull/12070
Our large customer that publishes their own packages and docs came back
to us and said they have legitimate use cases with more than 200 types:
#15507
I've grabbed stats about our current packages and still found a few
offenders:
```
"aws:lex/v2modelsIntent:V2modelsIntent" 920
"aws:wafv2/ruleGroup:RuleGroup" 310
"aws:wafv2/webAcl:WebAcl" 523
"azure-native:datafactory:Dataset" 256
"azure-native:datafactory:LinkedService" 299
"azure-native:datafactory:Pipeline" 618
"azure-native:datamigration:ServiceTask" 291
"azure-native:datamigration:Task" 291
"aws-native:quicksight:Analysis" 589
"aws-native:quicksight:Dashboard" 606
"aws-native:quicksight:Template" 590
```
Therefore, I'm not entirely removing the limit in this PR, but
a) bumping the default to 1000
b) applying 200 to the known offenders only
I don't love it's hard coded, but I haven't found a place to add simple
configuration nob. Anyway, it's slightly less hard-coded than it used to
be.
Fixes#15507
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
- Existing docs gen tests cover that I haven't broken anything
- I re-generated the AWS docs and they had no changes
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
# Description
This PR is an initial implementation of emitting constructor syntax of
resources into the docs for typescript, python, go and csharp.
## Checklist
- [ ] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [ ] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
This pull request depends on corresponding changes in the
pulumi-terraform bridge.
The bridge will mark up each code section it finds (not just ones found
under an "Example Usage" header) with an HTML comment - currently this
will be `<!--Begin Code Chooser -->` and `<!--End Code Chooser -->` but
the name is subject to change.
In this PR, we are adding logic to detect if an incoming Description
block has an `{{% examples }}` shortcode markup: if yes, we will process
it as before. If no, we will switch to new functionality that relies on
documentation with code sections having no short codes, no "Examples
Section" logic, and will generate every code section it finds in an
incoming schema that is in the Description/Content section of a registry
page with a chooser and language choosables.
This work is part of https://github.com/pulumi/pulumi-aws/issues/2624.
Edit: unit tests are added
This PR is a prerequisite for
https://github.com/pulumi/pulumi-terraform-bridge/pull/1689. It aims to
support both legacy and new behavior.
<!---
Thanks so much for your contribution! If this is your first time
contributing, please ensure that you have read the
[CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md)
documentation.
-->
# Description
<!--- Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context. -->
Fixes # (issue)
## Checklist
- [x] I have run `make tidy` to update any new dependencies
- [x] I have run `make lint` to verify my code passes the lint check
- [x] I have formatted my code using `gofumpt`
<!--- Please provide details if the checkbox below is to be left
unchecked. -->
- [x] I have added tests that prove my fix is effective or that my
feature works
<!---
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the
`changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the
Pulumi Cloud,
then the service should honor older versions of the CLI where this
change would not exist.
You must then bump the API version in
/pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi
Cloud API version
<!-- @Pulumi employees: If yes, you must submit corresponding changes in
the service repo. -->
A change was made a while back to remove the use of shortcodes. As part
of that, the shortcode used to render markdown was replaced by a
function call that used goldmark to render markdown. The resulting HTML
from that wraps the text in paragraph tags.
This causes some problems for rendering the docs. For deprecations, the
deprecation message now shows up outside of the "Deprecated" box. This
happens because the rendered markdown was being wrapped in `<p></p>`
tags, leading to `<p class="resource-deprecated">Deprecated:
<p>message</p></p>`, which does not render correctly because paragraph
tags cannot be nested.
Also, in description lists, rendered markdown text was being wrapped in
paragraph tags (e.g. `<dd><p>text</p></dd>`), causing it to render
differently from text not wrapped in paragraph tags (e.g.
`<dd>text</dd>`).
This change addresses these issues.
- First, `<div class="resource-deprecated">` is used rather than `<p>`
to contain the deprecation information.
- Second, the `markdownify` function will now trim unnecessary paragraph
tags.
Fixes https://github.com/pulumi/pulumi-hugo/issues/2832
Fixes https://github.com/pulumi/registry/issues/3008
---
## Before
<img width="713" alt="Screen Shot 2023-08-24 at 5 21 44 PM"
src="https://github.com/pulumi/pulumi/assets/710598/1358cc74-dd8a-4bc0-bed8-603b1439b2aa">
## After
<img width="721" alt="Screen Shot 2023-08-24 at 5 21 58 PM"
src="https://github.com/pulumi/pulumi/assets/710598/ad501e68-1bac-486c-a723-7db2b3442c8a">
## Before
<img width="719" alt="Screen Shot 2023-08-24 at 5 22 28 PM"
src="https://github.com/pulumi/pulumi/assets/710598/6ca40b85-9008-429b-83c3-2f0d681752f3">
## After
<img width="714" alt="Screen Shot 2023-08-24 at 5 24 02 PM"
src="https://github.com/pulumi/pulumi/assets/710598/eec1dacf-8c24-470e-b1e3-da843b4aeaee">
There are a few resources with a *very* large number of nested types. We have seen this lead to generating docs pages that cannot be loaded in a browser.
Instead, only emit the first 200 nested types, and emit a note to users if we have done this.
In the AWS provider, this only triggers for wafv2.WebAcl and wafv2.RuleGroup.
Add line breaks and whitespace to avoid long horizontal scrolls for Python constructor/function arguments. Also, include the new ResourceArgs constructor overload.
Fixes: https://github.com/pulumi/docs/issues/4340
The deprecated message sits below the examples so it is not clear
to the user that the resource / datasource is deprecated
We're not going to generate language-specific API docs for the Azure NextGen provider, only resource docs. This change makes it so the resource docs do not emit any links to nonexistent API docs.
Resource doc changes for Python:
- Types are included in constructor/function args
- The property names for input/output types are now always snake_case, regardless of the generated mapping tables, to match the new input/output classes
- Some other minor tweaks to function/constructor signatures (e.g. removed the `__props__` arg, as it's not meant to be used directly; use `@staticmethod` for static `get` methods).
* Add a new template for examples section. Extract the examples section into a structured format for custom template processing.
* Update the description IFF we were able to extract examples sections from it.
* Update doc comments and add missing file header for the newly added file.
* Make the example description readable. Add a check for empty example sections.
* Add a chooser right below the Example Usage header. Remove javascript as a language.
* Allow an empty new-line between short-codes boundaries.
* Restore the API type links for C#.
* Also restore them in function.tmpl.
* Add package details to the Functions template as well. Add a global template function to detect if the APIDocLinks has links for a language. Don't generate C# API doc links for k8s.
* Add ids to headers so they can be linked to
* Make header ids more general
Rather than including the identifier in the name of the id, use a simpler more generic id.
* H3=>H2 for Modules/Resources/Functions/Package Details headings
This way, on individual resource/function pages, it doesn't make it look like "Package Details" is a sub heading under "Supporting Types" in the right-hand "ON THIS PAGE".
* Add auto-generated disclaimer for the resource.tmpl as well.
* Allow specifying the languages to show in the lang chooser. Hide the go language for k8s overlay resources.
* Add another exclusion for generating the C# constructor param for k8s overlay resources.
* Replace hard-coded exclusions for skipping input properties with checking for ConstValue attribute of input properties.
* Add a check for the Python code gen to prevent infinte recursion. Remove the custom logic for k8s for generating Python property case maps.
* Fix bug with C# links for k8s constructor resources.
* Apply a style on the deprecation message.
* Use the display name when rendering a property type that does not have a link.
* Fix the python type string generator for docs to inspect union types.
* Update cleanTypeString to account for nodejs package names.
* Add missing lookup names for packages.
* Add new templates for generating index files. Remove the now irrelevant code.
* Add comment for generating index files by deduping module names. Use the path format of a module name as the link.
* Add a trailing slash for module links in the index page.
* Move the categories rendered in the index file into their own template file. Handle kubernetes provider specific module name conversion while generating parent modules.
* Fix issue with the k8s provider resource being generated as a module rather than a resource on the package-level index page.
* Show the deprecation message of a resource at the top of the document.
* Move k8s specific things to its own file.
* Add title attribute to the list items. Show the last part of a module name, in case it contains path separators
* Lookup the package name from the Go language info object for k8s then use that to lookup the C# namespace.
* Move the logic for cleaning a property type string for display names to a separate function.
* Export the title function from go and dotnet code generators for use in the resource doc generator. Use the title function from the respective lang code gens to match the correct title case used there. Fix issue with the Kubernetes namespace not being stripped for C# property type strings.
* Skip including apiVersion and kind as input properties for Kubernetes until pulumi-kubernetes#1062 is merged.
* Generate the constructor params for Python along with other languages.
* Remove redundant py_function_param nested template. Declare a new type for defining property characteristics rather than using inlining formal params. Generate the Lookup functions for all languages similar to the constructor params with linking enabled.
* Fix bug with generating the input arg type name for Functions in Go.
* Add prefix for args param of a Go-based Resource Function.
* Input args for Go-based Functions use Lookup*Args and not Get*Args.
* Turns out that args for Go-based Functions use a different prefix based on whether the function is a package-level or module-level Function.
* Update the Python list and dictionary type names for the resource doc generator.
* Add a separate function for Python doc helper return a type string representing dictionaries that are simple maps and don't have a known nested element type.
* Also remove the Examples property from the resourceDocArgs type since the resource Comment will contain the multi-lang examples on their surrounded by a shortcode.
* Added a new template for Functions. Implement the genFunction method for generating the docs for Functions.
* Rename type resourceArgs to resourceDocArgs. Minor updates the resource template.
* Generate nested types for Functions.
* Unexport types that don't need to be exported. Create the doc language helper objects in an init function and reuse them rather than recreating them every time. Update genNestedTypes to work with schema functions or resources.
* Fixed bug in nested type generation for Functions. Fixed bug in generating input and output doc links for nested types.
* Update properties.tmpl to render property comment as-is. WIP splitting out properties to lang-specific tables.
* Generate the constructor dynamically from the resource per language.
* Add doc functions in each language generator package for getting doc links for types..and later other functions too.
* Render the constructor params in the Go code and inject into the template.
* Generate nodejs types using the nodejs lang generator.
* Add a templates bundler. Added a new Make target for autogenerating a static bundle for the resource docs generator.
* Generate type links for all languages based on their schema type. Render the property type with a link if the underlying elements have a supporting type. Fix word-breaks for Python type names.
* Various changes including the introduction of an interface type under the codegen package to help with generating some language-specific information for the resource docs generator.
* Add a function to explicitly generate links for input types of nested types. Fix the resource doc link generator for Go. Don't replace the module name from the nodejs language type.
* Fix bug with C# property type html encoding.
* Fix some template formatting. Pass the state inputs for Python to generate the lookup function for it.
* Do not generate the examples section if there are none.
* Generating the property types per language.
* Formatting. Rename function for readability.
* Add comments. Update README.
* Use relative URLs for doc links within the main site