Using AWS Secret Manager
Introduction
This technical document describes the approach of using AWS
Secrets Manager to securely manage secrets and credentials for application
running on Amazon EC2 instances. The document outlines the steps to configure
the necessary infrastructure components, including AWS Secrets Manager, AWS
Identity and Access Management (IAM), and Azure DevOps.
Solution defined in the document is generic and can be
applied to programs planning to implement AWS secret manager in their
respective projects leveraging Azure Devops
Approach
The approach described in this document consists of the
following steps:
·
Configure AWS Secrets Manager to store secrets
·
Create an IAM policy for the EC2 instance to
access Secrets Manager
·
Configure Azure DevOps to fetch secrets from
Secrets Manager
·
Pass secrets to the .NET Core application using
environment variables or application config
Detailed Solution
1. Configure AWS Secrets Manager to store
secrets
·
Create a new secret in AWS Secrets Manager for
each environment (dev, qa, prod) that the application uses
o
Non Prod
§
Dev_Secrets
§
QA_secrets
§
Qualif_Secrets
o
Prod Secrets
§
Prod_Secrets
·
Alternately we can also store all environment secrets
per service.
Note - If you have a small number of services and environments,
storing all environment secrets per service may simplify secret management and
reduce the risk of misconfigurations. However, if you have a large number of
services and environments or require more flexibility, storing secrets per
environment may be a better approach.
If you are expecting to develop
multiple micro services hence it’s better to go for storing secrets per
environment
·
Store all the secrets for each environment as a
key-value pairs in JSON format
2. Create an IAM policy for the EC2 instance
to access Secrets Manager
·
Create IAM Users for prod and non-prod environments
o
Generate AWS access key and secret access key
o
Access Type – Programmatic Access
·
Attach Policy SecretsManagerRead to Users created in step 2
·
If we push the secret to App.config then this is
not required
o Attach IAM roles for EC2 instances
for applications to access Secret manager at runtime
o
Attach the appropriate AWS managed policy that
grants permissions to access Secrets Manager
3. Configure Azure DevOps to fetch secrets
from Secrets Manager
·
Create a new service connection in Azure DevOps
for AWS, using the IAM credentials for accessing Secrets Manager
o
Navigate to the "Project Settings" in
Azure DevOps and click on "Service connections".
o
Click on "New service connection" and
select "AWS".
o
Follow the prompts to enter your AWS access key
and secret access key, as well as the region where your AWS resources are
located.
4. Create a new pipeline variable in Azure
DevOps for the secret name:
o
In the Azure DevOps pipeline, define variable
(e.g., "EnvironmentSecret") and set the value to the name of the
secret that you want to fetch from AWS Secrets Manager.
§
E.g: EnvironmentSecret = Dev_Secrets’ –
to Identify Build environment
§
Refer secret name created in Step 1
5.
Add an
Azure DevOps task to fetch the secret:
o
In Azure DevOps pipeline, add a task that
fetches the secret from AWS Secrets Manager.
o
aws secretsmanager get-secret-value --secret-id
$(EnvironmentSecret) --query SecretString --output text
o
aws secretsmanager
get-secret-value --secret-id EnvironmentSecret --query SecretString
6.
Pass the
secret value to your .NET Core application
There are multiple options to store the
secret . This can be evaluated and agreed upon
o
Use Environment Variable in EC2
§
Via Azure DevOps task and update environment
variable in EC2
§
MySecretName ={
SecretString }
(OR)
o
Use configuration files
§
Via Azure App Service Deploy task to deploy the
configuration file to the Azure App Service that hosts your .NET Core application
o
Store the Output secret value from the previous
step into a Key value pair
o
Create new Azure DevOps task called "Azure
App Service Settings"
(OR)
o
Store the secrets in Secure Files in Azure
Pipelines and push it into appconfig file
Go
to Pipelines > Library > Secure files
7. Aplication code
.NET Core application to retrieve the value
of "MySecretName":
string secretValue
= Environment.GetEnvironmentVariable("MySecretName");
References
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-app-service-settings-v1?view=azure-pipelines&viewFallbackFrom=azure-devops
https://aws.amazon.com/blogs/modernizing-with-aws/how-to-load-net-configuration-from-aws-secrets-manager/
