Pulumi ESC: Retrieve Secrets from External Sources
Overview
With Pulumi ESC, you can safely store configuration values as plaintext or as secrets directly in the environment file. You also have the option to dynamically import secrets from external sources such as:
- AWS Secrets Manager
- Azure Key Vault
- Google Secret Manager
This enables you to centralize both new and existing secrets and configurations and use them in multiple places without duplicating values across providers or environments.
Import external secrets
If you have not done so already, make sure you have configured OIDC connectivity between Pulumi and a supported provider listed below. During the configuration, you will need to make sure that you add the permissions necessary to interact with secrets in your chosen provider.
To retrieve secret values from AWS Secrets Manager, you must first:
- ensure that your IAM role has the appropriate Secrets Manager permissions
- create a Secrets Manager secret
Once that is complete, you will need to update your environment definition to add the aws-secrets provider configuration. To do this, add the following configuration to your environment definition, making sure to:
- nest it under the
awssecond-level key - replace the value of
<your-aws-region-here>with your secret’s region - replace the value of
<your-secret-name-here>with the name of your secret
values:
aws: # your existing oidc configuration
login:
...
...
secrets: # add the aws-secrets provider config
fn::open::aws-secrets:
region: <your-aws-region-here>
login: ${aws.login}
get:
app-secret:
secretId: <your-secret-name-here>
Your updated environment file should look similar to the following:
# example environment definition
values:
aws:
login:
fn::open::aws-login:
oidc:
roleArn: arn:aws:iam::0123456789:role/esc-oidc
sessionName: pulumi-environments-session
duration: 1h
secrets:
fn::open::aws-secrets:
region: us-east-1
login: ${aws.login}
get:
app-secret:
secretId: my-app-secret
You can validate this configuration by opening the environment via the ESC console, clicking the Open button and then clicking the Show secrets slider.
Alternatively, you can validate the configuration by opening the environment via the ESC CLI. Run the esc env open <your-org>/<your-project-name>/<your-environment-name> command, making sure to replace the values of <your-org>, <your-project-name>, and <your-environment-name> with the names of your Pulumi organization, ESC project, and environment respectively.
$ esc env open pulumi/demos/aws-secrets-example
{
"aws": {
"login": {
"accessKeyId": "ASIA...",
"secretAccessKey": "Jdt...",
"sessionToken": "Fwo..."
},
"secrets": {
"app-secret": "pulumi-esc-get-started-secret"
}
}
}
If you need to retrieve multiple Secrets Manager secrets, you can do so as shown below:
values:
aws:
login:
...
...
secrets:
fn::open::aws-secrets:
region: us-east-1
login: ${aws.login}
get:
app-secret:
secretId: my-app-secret
api-key: # additional AWS Secrets Manager secret
secretId: my-api-key
To retrieve secret values from Azure Key Vault, you must first:
- create an Azure Key Vault
- add a Key Vault role assignment
- On your Key Vault’s service page, follow steps 3 - 8 in the Azure OIDC guide, making sure to select the “Key Vault Secrets Officer” role under the Job functions role tab
- create an Azure Key Vault secret
Once that is complete, you will need to update your environment definition to add the azure-secrets provider configuration. To do this, add the following configuration to your environment definition, making sure to:
- nest it under the
azuresecond-level key - replace the value of
<your-vault-name-here>with the name of your vault - replace the value of
<your-secret-name-here>with the name of your secret
values:
azure: # your existing oidc configuration
login:
...
...
secrets: # add the azure-secrets provider config
fn::open::azure-secrets:
login: ${azure.login}
vault: <your-vault-name-here>
get:
app-secret:
name: <your-secret-name-here>
Your updated environment file should look similar to the following:
# example environment definition
values:
azure:
login:
fn::open::azure-login:
clientId: 99b...
tenantId: 706...
subscriptionId: 028...
oidc: true
secrets:
fn::open::azure-secrets:
login: ${azure.login}
vault: pulumi-esc-vault
get:
app-secret:
name: my-app-secret
You can validate this configuration by opening the environment via the ESC console, clicking the Open button and then clicking the Show secrets slider.
Alternatively, you can validate the configuration by opening the environment via the ESC CLI. Run the esc env open <your-org>/<your-project>/<your-environment-name> command, making sure to replace the values of <your-org>, <your-project-name>, and <your-environment-name> with the names of your Pulumi organization, ESC project, and environment respectively.
$ esc env open pulumi/demos/azure-secrets-example
{
"azure": {
"login": {
"clientId": "99b...",
"oidc": {
"token": "eyJ..."
},
"subscriptionId": "028...",
"tenantId": "706..."
},
"secrets": {
"app-secret": "pulumi-esc-get-started-secret"
}
}
}
If you need to retrieve multiple Azure Key Vault secrets, you can do so as shown below:
values:
azure:
login:
...
...
secrets:
fn::open::azure-secrets:
login: ${azure.login}
vault: pulumi-esc-vault
get:
app-secret:
name: my-app-secret
api-key: # additional Azure Key Vault secret
secretId: my-api-key
To retrieve secret values from Google Secret Manager, you must first:
- ensure that your service account has the appropriate Secret Manager permissions
- create a Google secret
Once that is complete, you will need to update your environment definition to add the gcp-secrets provider configuration. To do this, add the following configuration to your environment definition, making sure to:
- nest it under the
gcpsecond-level key - replace the value of
<your-secret-name-here>with the name of your secret
values:
gcp: # your existing oidc configuration
login:
...
...
secrets: # add the gcp-secrets provider config
fn::open::gcp-secrets:
login: ${gcp.login}
access:
app-secret:
name: <your-secret-name-here>
Your updated environment file should look similar to the following:
# example environment definition
values:
gcp:
login:
fn::open::gcp-login:
project: 123456789
oidc:
workloadPoolId: pulumi-esc
providerId: pulumi-esc
serviceAccount: pulumi-esc@foo-bar-123456.iam.gserviceaccount.com
secrets:
fn::open::gcp-secrets:
login: ${gcp.login}
access:
app-secret:
name: my-app-secret
You can validate this configuration by opening the environment via the ESC console, clicking the Open button and then clicking the Show secrets slider.
Alternatively, you can validate the configuration by opening the environment via the ESC CLI. Run the esc env open <your-org>/<your-project-name>/<your-environment-name> command, making sure to replace the values of <your-org>, <your-project-name>, and <your-environment-name> with the names of your Pulumi organization, ESC project, and environment respectively.
$ esc env open pulumi/demos/gcp-secrets-example
{
"gcp": {
"login": {
"accessToken": "ya29....",
"expiry": "2024-01-25T12:52:15Z",
"project": 123456789,
"tokenType": "Bearer"
},
"secrets": {
"app-secret": "pulumi-esc-get-started-secret"
}
}
}
If you need to retrieve multiple Google Secret Manager secrets, you can do so as shown below:
values:
gcp:
login:
...
...
secrets:
fn::open::gcp-secrets:
login: ${gcp.login}
access:
api-key: # additional Google Secret Manager secret
name: api-key
app-secret:
name: app-secret
Thank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.
