With so many options to manage your secrets in the cloud, the decision on how to handle secrets can be cumbersome for organizations of many sizes. In this blog post, we will examine three of the most popular solutions for Amazon Web Services (AWS) and compare their features, security concerns, and cost associated with each one.
Note: This blog post is based on a whitepaper authored earlier this year. Please feel free to take the time to read it as well.
Individuals who are reading this post most likely already know what a secrets manager is and why one is considered necessary for a robust cloud security strategy. For those of you new to the concept, here’s a short overview:
When developing applications and provisioning infrastructure secrets are inherently created. A secret is potentially sensitive information which can typically grant access to additional data. Using a secrets manager to store configuration in the environment and not hardcoded in the source code is important in the development of a twelve-factor application, and a secrets manager helps to facilitate this while protecting your secrets and providing additional benefits.
Now that we have (briefly) went over what a secrets manager is and why it is necessary to keep a cloud environment safe, it’s time to introduce some solutions. Solutions come in two different categories: native and third party. Native AWS solutions include AWS Systems Manager Parameter Store and AWS Secrets Manager. There is a plethora of third party solutions that work in a variety of environments, the most widely used one being HashiCorp Vault, therefore we will be examining it in this post. Other third party solutions may be visited in future posts. Comment below if you have any favorites not mentioned.
Native AWS solutions integrate well with services such as AWS CloudWatch and AWS Identity and Access Management (IAM), whereas third party solutions typically require a little more work to get running such as setting up instances or containers that run the software. Third party solutions are usually not tied to any specific cloud and can be very useful for those who want a unified secrets solution across on-premise or other clouds such as Google Cloud Platform and Microsoft Azure.
Now it is time to meet our secrets managers and compare the features that each one of them have. A secrets manager does not need a lot of bells and whistles, but there are many benefits to having a feature-rich toolset included with the solution. This isn’t an exhaustive list of features for each secrets manager, but does provide an outline of the most common features.
Most secrets come in the form of key value pairs, some examples being: AWS Access Key/AWS Secret Key and environment variable name/value. Being able to store key-value pairs is a fundamental feature to being a secrets manager and is present in all three solutions.
AWS Systems Manager Parameter Store standard parameters holds secrets up to 4 KB in size, whereas advanced parameters hold secrets up to 8 KB in size. For AWS Secrets Manager, key names for secrets cannot exceed 256 characters and values cannot exceed 10240 bytes (about 10 KB).
HashiCorp Vault does not enforce a size limit for key-value pairs, but it can vary depending on the storage backend. It should be noted that Vault’s HTTP API has a default limit of 32MB to help mitigate denial of service (DoS) attacks due to arbitrarily large requests, but this can be adjusted by changing the configuration file.
Being able to rotate secrets is essential in maintaining proper security hygiene for applications and services. By rotating secrets you are ensuring that any secrets that exist in code or infrastructure will not be valid if compromised, and in addition you are ensuring that your application is doing proper fetching from the secrets manager, thus encouraging a tight integration with the secrets manager and those applications that adopt it. In other words, regularly rotating secrets not only keeps your application secure, but can encourage the adoption of a secrets manager throughout your project and/or company. For those who are compliance conscious, NIST 800–57 prescribes the rotation of secrets and having a proper rotation mechanism can help significantly with achieving that or similar types of goals.
At the time of writing, AWS Systems Manager Parameter Store does not currently support secrets rotation. A solution such as chamber or the AWS SDK can be leveraged in conjunction with additional scripting to manually rotate secrets, and can possibly be automated by similar means.
AWS Secrets Manager supports the automatic rotation of RDS, Redshift, and DocumentDB secrets via the deployment of provided Lambda rotation functions. Custom Lambda rotation functions can be written to rotate other types of database credentials or secrets. A nice benefit of having a Lambda rotation function is that you can test the new version of the secret inside of the rotation function to ensure the application works correctly after the secret has been rotated.
HashiCorp Vault has a feature called dynamic secrets, which allows for Just-in-time (JIT) provisioning of secrets — basically this means that dynamic secrets are generated when they are accessed and don’t exist until they are read. This means there is a very low risk of someone stealing them or another client using the same secrets. As of the time of this writing, Vault supports dynamic secrets with a variety of backends including databases, messaging queues, cloud providers, Secure Shell (SSH), and Public Key Infrastructure (PKI). Other secrets types can be extended with plugins.
Password Generation is not an essential feature of a secrets manager, but can be useful for automation purposes and serves to be a useful tool to have in the cloud security toolbox.
Being a simple key-value store, it is not surprising that AWS Systems Manager Parameter Store doesn’t have password generation capabilities. On the other hand, AWS Secrets Manager has password generation capabilities via the *get-random-password* AWS CLI call. Secrets Manager has many options for this including generating both complex and simple passwords with the use of CLI flags. HashiCorp Vault has the vault-secrets-gen plugin to generate high entropy passwords and passphrases with a variety of options to control the secret’s length and complexity.
Infrastructure as Code (IaC) is important for cloud environments, and we at ScaleSec are indeed fans of it. The two most prominent declarative IaC solutions for AWS are HashiCorp Terraform and AWS CloudFormation. Using a secrets manager in your IaC ensures that secrets are not hardcoded into source code and the most recent version of a particular secret can be fetched on demand when code is run.
Both AWS Systems Manager Parameter Store and AWS Secrets Manager have first class support for CloudFormation. For example, when creating resources through CloudFormation (such as an RDS database), secrets (such as the master username and password) can be randomly generated and stored in AWS Secrets Manager. Note: The ability to randomly generate strings is only available in AWS Secrets Manager and not Systems Manager Parameter Store.
Terraform supports all three solutions and is able to both create and read secrets. There is a provider for Vault and a resource/data source for both AWS Secrets Manager and Systems Manager Parameter Store. Both native AWS solutions have KMS support via Terraform and the rotation mechanism for AWS Secrets Manager can also be invoked from Terraform.
A paramount concern for any secrets manager is how effective it will be in keeping secrets safe. Security for secrets managers must be examined in terms of confidentiality, integrity, and availability of the secrets that they are managing. We will examine the following topics: what access control mechanisms are in place to manage both the modification and the retrieval of secrets, how secrets are protected at rest, and how secrets are protected in transit. This is by no means an exhaustive list of security concerns that should be considered when selecting a secrets manager for use in the cloud, but highlights the most important points that should be taken into account before selecting a secrets manager.
In order to access secrets from AWS Secrets Manager or AWS Systems Manager Parameter Store, one must have valid AWS credentials with the proper permissions to read the desired secret. Although Parameter Store standard parameters don’t have much access control mechanisms past this, both Parameter Store advanced parameters and AWS Secrets Manager secrets have robust access control mechanisms via AWS Identity and Access Management (IAM) policies. IAM policies are deny by default, meaning that unless actions are overridden by an explicit allow, then access to or management of a secret governed by the policy are inherently protected. AWS IAM can also be utilized to share secrets between AWS accounts via a resource-based policy, which allows for attachment of a permissions policy for the secret.
A simple AWS IAM policy example that displays some capabilities for AWS Secrets Manager is given below to display access control capabilities provided by IAM. Note that the . syntax is used to express that the secret should start with secret-2 and be followed by exactly 6 characters:
| "Version" : " 2012-10-17 " , |
| "Statement" : [ |
| "Effect" : " Allow " , |
| "Action" : [ |
| " secretsmanager:DescribeSecret " , |
| " secretsmanager:List* " |
| ], |
| "Resource" : " * " |
| >, |
| "Effect" : " Allow " , |
| "Action" : " secretsmanager:* " , |
| "Resource" : [ " arn:aws:secretsmanager:::secret:secret-a1b2c3 " , " arn:aws:secretsmanager:::secret:secret2-. " |
| ] |
| > |
| ] |
| > |
HashiCorp Vault manages access control through path-based policies. Through the use of policies, Role-Based Access Control can be achieved by specifying access privileges via path-based matching. Vault policies can be written in JSON or HashiCorp Configuration Language (HCL) for easier human readability. Like IAM policies, Vault policies have an implicit deny as well, and paths in Vault must be explicitly allowed along with the corresponding capabilities.
An example of a Vault policy written in HCL is given below. Note the glob (*) syntax for allowing access to all paths under the specified prefix (in this case secret/), a path for specific access (such as secret/super-secret), and conditional access based on allowed parameters (shown in the secret/restricted block):
| path " secret/* " |
| capabilities = [ " create " , " read " , " update " , " delete " , " list " ] |
| > |
| path " secret/super-secret " |
| capabilities = [ " deny " ] |
| > |
| path " secret/restricted " |
| capabilities = [ " create " ] |
| allowed_parameters = |
| " foo " = [] |
| " bar " = [ " zip " , " zap " ] |
| > |
| > |
AWS Secrets Manager store uses AWS KMS to encrypt parameter values. A custom AWS KMS key can be used to encrypt your secrets instead of the default Secrets Manager customer managed key (CMK) for your account. Using a custom KMS key is generally better since it can unlock capabilities such as utilizing encryption at rest to help destroy your data, also known as cryptoshredding. When dealing with sensitive information such as secrets, it is important to verify that the data has been destroyed once a key has been rotated on hardware that is not your own (eg. the cloud) — especially since both of these solutions are AWS managed services and thus visibility into the data destruction process is limited. Using a CMK also allows the use of CloudHSM to easily generate and use your own encryption keys.
The storage backends used by Vault are designed with a zero trust mindset. Vault uses a security barrier for all requests made to the backend. The security barrier automatically encrypts all data leaving Vault using a 256-bit Advanced Encryption Standard (AES) cipher in the Galois Counter Mode (GCM) with 96-bit nonces. The nonce is randomly generated for every encrypted object. When data is read from the security barrier the GCM authentication tag is verified during the decryption process to detect integrity breaches.
By design, Vault needs an encryption key in order to decrypt data. This encryption key is stored with the data, but is encrypted with another encryption key known as the master key. The master key is actually not stored anywhere. Instead of distributing this single master key to an operator, Vault uses an algorithm known as Shamir’s Secret Sharing to split the key into shards. A certain threshold of shards is required to reconstruct the master key. This protects against insider threat because in the case of a detected intrusion, the Vault data can be sealed in order to quickly minimize damages. The master key must then be reconstructed using the distributed shards to reach a quorum and unseal the vault and so that applications and services can access secrets again. Cloud technologies such as AWS KMS, Google Cloud KMS, and Azure Key Vault can also be used to auto-unseal the Vault. HashiCorp Vault Enterprise can leverage a HSM for master key wrapping, automatic unsealing, and seal wrapping.
To protect secrets as they are in transit, it is best practice to ensure secrets are not transmitted over the Internet whenever possible. Fortunately, both AWS Systems Manager Parameter Store and AWS Secrets Manager support the use of VPC (Virtual Private Cloud) endpoints. The use of VPC endpoints are highly encouraged to prevent any breaches in confidentiality that comes from transmitting information publicly and helps to ensure the compliance goals of an organization. In addition, you get performance enhancements and VPC endpoints have their own IAM policies that can be used for an additional layer of protection.
Since HashiCorp Vault is run on hardware provisioned by the consumer of it, the networking situation can differ between installations. Transport Layer Security (TLS) can be configured on Vault via a listener to enable encryption in transit and thus protect data that travels over a network. Vault can be configured to be highly available and have end-to-end TLS over a network with relative ease.
Cost must also be considered since the pricing models differ between native and third party offerings. Both of the AWS native solutions have a per secret cost and cost per API call, but they have no operational cost due to the fact that AWS maintains them. On the other hand, the open source version of HashiCorp Vault has no per secret or access cost, but is self-hosted and has a cost for enterprise features.
AWS Secrets Manager is $0.40 per secret per month, for secrets that are stored in less than a month the price is prorated. There is an additional charge of $0.05 per 10,000 API calls. As an example, if 1000 secrets are stored using AWS Secrets Manager, with 400,000 API calls there is: a monthly charge of $400 per month and API calls will cost an additional $2.
AWS Systems Manager Parameter Store has different pricing models for standard and advanced parameters. Standard parameters are available at no additional charge. Advanced parameters are charged $0.05 per secret per month. Charges for parameters stored in less than a month are prorated. There is an additional charge of $0.05 per 10,000 API calls. For example, if 1000 advanced parameter secrets are stored with 400,000 API calls using AWS Systems Manager Parameter Store, there is: a monthly charge of $500 per month and API calls will cost an additional $2.
HashiCorp Vault comes in both open-source and two enterprise versions: Enterprise Platform and Enterprise Modules. Both enterprise solutions come at an additional cost, which can be given by contacting HashiCorp. Vault is also a self-hosted solution for both open source and enterprise, therefore both provisioned infrastructure and operational upkeep must be considered when calculating cost. There is no additional cost for number of stored secrets or secrets retrieval.
Using Vault’s reference architecture: three m5.large instances with 25 GB gp2 drives attached to achieve n-2 redundancy (where the loss of 2 objects within the failure domain can be tolerated), we can provide an estimated cost for Vault. The calculation below was performed using the AWS pricing calculator: 3 instances x $0.096 x 730 hours in month = $210.24 (monthly, on-demand instances)
For storing less than 10,000 secrets and no secrets greater than 4 KB in size, AWS Systems Manager Parameter Store standard parameters is free and can be useful for proof of concepts or non-production environments. If you are looking for a simple and native secrets manager that is production-ready, please consider AWS Systems Manager Parameter Store advanced parameters instead.
The best native secrets manager for AWS is AWS Secrets Manager. The security features along with secrets rotation and password generation capabilities gives this secrets manager the edge when it comes to AWS’s offerings.
The most feature rich and the solution of choice is HashiCorp Vault. It excels in most categories and has great community support. Arguably, it is much more than a secrets manager — it has been described as the “Swiss army knife” of cloud security because of its additional features such as brokering SSH access, being a Certificate Authority to generate X.509 certificates on demand, and performing encryption as a service. This is also the best option if you are considering a multi-cloud strategy in the future and would like to keep secrets in a centralized location.
