Aws where to store private key?

AWS Secrets Manager provides full lifecycle management for secrets within your environment. In this post, Maitreya and I will show you how to use Secrets Manager to store, deliver, and rotate SSH keypairs used for communication within compute clusters. Rotation of these keypairs is a security best practice, and sometimes a regulatory requirement. Traditionally, these keypairs have been associated with a number of tough challenges. For example, synchronizing key rotation across all compute nodes, enable detailed logging and auditing, and manage access to users in order to modify secrets.

However, rotating the keypair on all compute clusters’ nodes must be done in a tightly coordinated fashion, and failures generally result in availability risks. Moreover, the keypairs themselves are highly sensitive security credentials which must be carefully controlled with fine-grain access controls, detailed monitoring, and audit logging. These are precisely the types of tough challenges that AWS Secrets Manger solves for you.

In this post, we’ll show you how to secure, rotate, and use SSH keypairs for inter-cluster communication. You’ll use an AWS CloudFormation template to launch a cluster and configure Secrets Manager. Then we’ll show you how to use Secrets Manager to deliver the keypair to the cluster and use it for management operations, such as securely copying a file between nodes. Finally, we’ll use Secrets Manager to seamlessly rotate the keypair used by the cluster without any changes or outages. In this post, we’ve highlighted compute clusters, but you can use Secrets Manager to apply this solution directly to any SSH based use-case.

The following architecture diagram presents an overview of the solution:

The sample architecture created by CloudFormation includes one master node, three worker nodes, AWS Secret Manager—which utilizes a rotation AWS Lambda function—and AWS Systems Manager. Setting up the cluster is out of scope for this post; in our walkthrough, we’ll focus on the keypair rotation architecture.

Secrets Manager uses staging labels to identify different versions of a secret during rotation. A staging label is a text string. For example, by default, AWSCURRENT is attached to the current version of the secret, while AWSPENDING will be attached to new versions of the secret before they have been verified and deployed to corresponding resources.

As shown in the diagram:

Today, this solution deployed in the N. Virginia Region will cost $0.0914 an hour for the four t2.micro EC2 instances and NAT Gateway that comprise the sample cluster. Secrets manager has a 30-day trial period, after which one secret will cost $0.40 per month and $0.05 per 10,000 API calls. There is no additional charge for AWS Systems Manager.

In this section, you’ll deploy a test stack that demonstrates the entire solution. After deployment, you’ll log in to the master node and securely copy a file to one of the worker nodes. Finally, you’ll use Secrets Manager to rotate and deploy a new SSH keypair. The CloudFormation templates and secret rotation code are available in the AWS GitHub repository.

Set up the sample deployment by selecting the AWS CloudFormation Launch Stack button bellow; by default, the stack will be deployed in the us-east-1 (N. Virginia) Region.

The template creates an Amazon Amazon Virtual Private Cloud (VPC), private and public subnets, EC2 instances (master node and mock cluster), and the IAM role and policies used for the EC2 instances.

Next, create and configure a new secret from the Secrets Manager console to store the cluster communication SSH keypair.

The CloudFormation template did not deploy a secret, so follow these steps to create a secret from the console and rotation function configuration. To create a new secret:

With the secret configuration completed and the instances up and running, you’re now going to securely copy a file from the master node to one of the worker nodes, using the SSH key stored in Secrets Manager to test the solution.

Figure 6 shows ssh login to master node, and the copy_file.py command to worker node.

During execution, the python script will use the Secrets Manager get_secret_value API to retrieve the secret, which includes the private key. It will then use this key to establish a secure SSH connection with the worker nodes, without saving the private key on any master node storage.

You can review the copy_file.py on the master node or on GitHub. In the get_private_key() function, you can read the secret value, which includes the private key:

In the copy_file function, create a secured SSH tunnel to copy a file using the private key from memory, using Paramiko, a Python implementation of SSHv2.

To demonstrate the rotation of the SSH keypair, you’ll now manually invoke the rotation function:

The file has now been transferred successfully using a new key pair, with no updates required.

You can monitor and audit all APIs used to create and rotate your keys in Secrets Manager via AWS CloudTrail. To view CloudTrail events, follow these steps:

Additionally, Secrets Manager can work with CloudWatch Events to trigger alerts when administrator-specified operations occur in an organization (for example, to notify you of a secret deletion attempt).

To delete the entire CloudFormation stack:

In this post, we demonstrate how you can use AWS Secrets Manager to store, rotate, and deliver SSH keypairs in order to secure communication within a compute cluster. Keys are securely encrypted and stored in AWS Secret Manager, which will also rotate the keys and install public keys on all nodes for you. By using this method, you won’t have to manually deploy SSH Keys on the various EC2 instances or manually rotate them. APIs associated with secrets management and rotation are logged in CloudTrail for auditing and monitoring. This key rotation solution is serverless. It does not require any servers to maintain and can scale rapidly.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the AWS Secrets Manager forum.

Want more AWS Security news? Follow us on Twitter.

And accessing them from a Node.js app…

If your application needs to SSH to an EC2 instance and run some shell commands, you will need to use the SSH Private Key to access the instance.

The question is where do you store the SSH Key? Would you store the SSH key alongside your code and commit it to your repository? With a public repository, you would be exposing your key to the world. But even if your repository is private you would be allowing the key to be retrieved by anyone that has access to the repository. The secure and recommended way to store the key is to use the AWS Secrets Manager where the SSH key would be stored encrypted and would only ever be retrieved and used by your application during run-time, and never written to disk.

Here are the steps to accomplish this:

#1 Create the Secret

In the AWS Management Console, navigate to the AWS Secrets Manager Service and click on Store a new secret.

Select ‘Other type of secrets’. Select the Plaintext tab and paste the SSH Key from your .pem file into the text area. (Make sure you clear the {“”:””} json format first; the JSON parser returns a token error if the key is stored in json format and parsed out in the code).

Leave the DefaultEncryptionKey to encrypt your SSH Key secret.

Give a name to your secret, add tags, leave the rotation disabled, and store the secret.

#2 Create the EC2 Instance role with the Secrets Manager Access Policy

You can create a custom policy instead, with access to the specific Secret instead of the generic role provided by AWS.

Click on Create.

#3 Attach the Role to the EC2 Instance host from where you will be running your applicationNavigate to the EC2 service, select your EC2 Instance, Actions, Instance Settings, Modify IAM role.

Select the EC2SecretsAccessRole that you created in the previous step and save it.

This EC2 instance will now have the permissions to access the Secret that you created.

#4 Create your application

The Node.js application will uses the ssh2 library and aws-sdk.

The EC2 instance we are connecting to is in the same VPC as the EC2 host where the Node.js application is running and has the necessary security rules in place.

You can fork the code for the application from the following GitHub repo: https://github.com/nil0blue/SecretsManagerSSHKey

Below is the output from a sample run:

• #1 Create the Secret.
• #2 Create the EC2 Instance role with the Secrets Manager Access Policy.
• #3 Attach the Role to the EC2 Instance host from where you will be running your application.
• #4 Create your application.