mirrors
/
pulumi
mirror of https://github.com/pulumi/pulumi.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
pulumi/docs/architecture
History
Will Jones 1cd86ddee1
Clean up documentation to support the bridge (#17784) 
This commit makes a few changes to our developer documentation in order
to make way for pulumi/pulumi-terraform-bridge#2623, which adds Sphinx
developer documentation to the Terraform Bridge. Namely:

* We introduce `intersphinx`, which allows for cross-referencing between
multiple Sphinx documentation sites as though they were in one site. We
won't introduce such links until the bridge documentation is merged and
deployed, but in theory links such as `tfbridge:some-label` will resolve
correctly thereafter.
* We rename the root page from "Home", to "Pulumi developer
documentation", since both the `pulumi/pulumi` and bridge documentation
sites will have a root and it could be confusing if they were both
called "Home".
* While we're here, we copy a trick from the bridge PR that allows us to
just write Mermaid diagrams using `mermaid` instead of `{mermaid}`. This
means that GitHub will render them in our Markdown files, which is nice.
 		2024-11-15 14:41:07 +00:00
..
deployment-execution Clean up documentation to support the bridge (#17784) 2024-11-15 14:41:07 +00:00
providers Split up provider documentation (#17679) 2024-11-05 17:48:18 +00:00
testing Update integration.md (#17468) 2024-10-03 16:36:36 +00:00
types Document more of Pulumi's type system (#17680) 2024-11-06 09:47:24 +00:00
README.md Split up provider documentation (#17679) 2024-11-05 17:48:18 +00:00
languages.md Flesh out documentation on language hosts and MLCs (#17163) 2024-09-05 14:21:49 +00:00
plugins.md Document providers and plugins (#17159) 2024-09-05 09:25:16 +00:00

README.md

Architecture

We can approach Pulumi's architecture by breaking down how a user interacts with it. Pulumi (or "Pulumi IaC", "Pulumi Infrastructure as Code" when differentiating from other Pulumi products) allows users to manage their infrastructure using the programming languages and technologies they are already familiar with:

  • Users write programs in a language of their choice (e.g. TypeScript or Python).

  • Programs declare a number of resources (e.g. virtual machines, or storage buckets) that the user wants to manage in various providers (e.g. AWS, Azure or GCP).

  • Programs are written using the core Pulumi SDK (e.g. @pulumi/pulumi in TypeScript) and one or more provider SDKs (e.g. @pulumi/aws for AWS).

  • Users invoke the Pulumi command-line interface (CLI) to run their programs and invoke particular operations (e.g. preview, update, or refresh). An instance of a program is called a stack (e.g. development, or production). Stacks are grouped into projects.

  • The CLI creates an instance of the deployment engine to manage the lifecycle of the operation. The engine interacts with both the program and the set of relevant providers in order to achieve a desired state.

  • When the operation completes, the engine persists its view of the state to a configured storage backend (e.g. a local file, or a service like Pulumi Cloud).

In order to support the various languages and providers that Pulumi offers, these pieces are composed as a set of processes that communicate with each other using gRPC remote procedure calls. These calls are specified as Protocol Buffer interfaces (Protobufs). Excluding language-specific components (e.g. SDKs), the majority of these processes are implemented as libraries and executables written in Go:

  • Programs are executed by a language host process, which wraps the runtime for a given language in an interface that the deployment engine can interact with.

  • Provider processes (often just "providers") expose a standard interface for performing validation, diff, create, read, update, and delete operations on resources. Providers typically also support function and method calls on the provider and resource objects (known as invokes and calls respectively).

  • The deployment engine orchestrates language hosts and provider processes to achieve the desired state of the infrastructure. As well as calling procedures on these processes, the engine itself exposes procedures that can be called back into by the language hosts and providers. Perhaps the most important of these is resource registration, whereby a program indicates to the engine the desired state of a resource (which the engine will then interface with the relevant provider to achieve).

  • The Pulumi resource model is specified using schemata. Provider processes expose functions for returning the set of resources and functions that they expose in a schema. Language hosts expose functions for generating code (e.g. provider SDKs) from a supplied schema.

With this in mind, this architecture is reflected in this repository as follows:1

  • proto -- contains Protobuf definitions for the various services and messages that Pulumi uses to communicate between processes. Code generated from these is committed to the repository and used by various components.

  • pkg/codegen -- contains code generators for the various languages that Pulumi supports. As well as supporting language SDK generation as mentioned above ("SDKgen"), code generators also underpin the pulumi convert command, which can convert existing infrastructure-as-code (IaC) programs written in e.g. Terraform or CloudFormation to Pulumi programs ("Programgen").

  • sdk/{go,nodejs,python,...} -- contains the core Pulumi SDKs for the various languages that Pulumi supports. These SDKs provide the core functionality for writing Pulumi programs in a given language, exposing the Pulumi engine's gRPC interface (e.g. RegisterResource) in an idiomatic way for that language (e.g. new Resource(...) in TypeScript, or Resource(...) in Python).

    The language host for a given language (written in Go) is also typically found under that language's SDK directory, and named pulumi-language-<language> (e.g. pulumi-language-nodejs in the case of NodeJS/TypeScript).

  • pkg/{backend,engine,graph,operations,resource} -- contains the core components of the Pulumi engine. The various directories roughly segregate some of the responsibilities that fall to the engine, such as:

    • Persisting state to a configured backend;
    • Computing and respecting dependency graphs so that operations can be executed in the correct order;
    • Managing resource lifecycles and deployments.

    Of particular note is pkg/resource/deploy, which houses a number of the pieces required to enact a complete deployment.

  • pkg/{cmd,display} -- contains code pertinent to the Pulumi CLI. In particular, pulumi/cmd/pulumi is the entry point to the CLI and exposes the various pulumi preview, pulumi up, pulumi destroy, etc. commands that you may be familiar with, routing these commands to an instance of the engine that the CLI creates and manages.

  • With the exception of a number of test providers that live in the core pulumi/pulumi repository, provider implementations are typically housed in their own repositories (e.g. pulumi-aws for AWS). These repositories follow a similar pattern when it comes to several of the core components -- e.g. most providers will have an sdk/ directory exposing provider SDKs for each supported language.

The remainder of this section covers many of these components in more detail, as well as discussing how they are verified and tested.

:::{toctree} :maxdepth: 1 :titlesonly:

/docs/architecture/deployment-execution/README /docs/architecture/types/README /docs/architecture/languages /docs/architecture/providers/README /docs/architecture/plugins /pkg/codegen/README /docs/architecture/testing/README :::

  1. This is not intended to be an exhaustive list of directories in the repository, but rather a high-level overview of the most important ones that pertain to the pieces outlined above. Other concerns, such as infrastructure, documentation, testing, and so on, are covered in other parts of this documentation. ↩︎