Case Study

1 Introduction

1.1 What is Embrasure?

Secrets management is a critical aspect of modern software development, where safeguarding sensitive information is essential for the security and integrity of projects. Effective secrets management ensures that software projects can securely access their secrets, whether they are API keys, database passwords, or something else entirely.

Embrasure is an open-source, self-hosted secrets management tool built on Amazon Web Services (AWS) for small teams seeking simplicity and security. This case study describes the design challenges, architecture, and trade-offs that molded Embrasure into what it is. Before diving into Embrasure, let us first explore what secrets are and why we go through all this trouble to safeguard them.

2 Secrets

2.1 What are secrets?

Secrets in software are similar to secrets in life; they are sensitive information you do not want publicly exposed. A more software-centric definition would be that they are private pieces of data that unlock protected resources or sensitive information such as third-party APIs or databases.

2.2 What problems do secrets solve?

Secrets protect sensitive information from unauthorized access. By using secrets, developers can confidently protect their system's integrity from malicious attacks and potential security breaches, with the assurance that only authorized users and applications can access sensitive data.

2.3 What problems arise from having secrets?

Secrets can be a weak point in a project's security. They are a vulnerability that, if discovered, can be exploited to gain unauthorized access to a system or even steal user information. So, the biggest challenge of having secrets is protecting them, or in other words, keeping them from being revealed. The following sections explore some of the ways that secrets are exposed.

2.4 Hard Coding Secrets

In 2019, ^[1] North Carolina State University researchers scanned almost 13% of Github’s public repositories. They found that "not only is secret leakage pervasive–affecting over 100,000 repositories–but thousands of new, unique secrets are leaked daily." This study highlights the reality that many software projects accidentally expose their secrets.

2.5 Malicious Actors

From phishing to packet sniffing internet traffic, developers need to remain vigilant against attempts to steal their secrets. ^[2]The consequences are compromised systems that can lead to things like data breaches, such as the one at Uber in 2016 that resulted in the exposure of the personal data of 57 million users of the app.

The breach was initiated by hackers who exploited private authentication information inadvertently exposed on GitHub, a widely used platform for code storage. The information had been pushed into source control and was now publicly visible, allowing the attackers to use the information to breach a third-party, cloud-based service.

2.6 Securing secrets

Secrets can be exposed at two main points in their life cycle: at-rest and in-transit. At-rest exposure refers to unauthorized access from a breach in a plaintext database or an accidental commit of a secret to a code repository. In-transit exposure refers to the risk of secrets contained in communication over a network being intercepted and discovered by unauthorized viewers.

2.7 Secret Sprawl

In contrast to traditional credentials, a project’s secrets are shared among developers, applications, and infrastructure systems. The dynamic nature of projects inevitably leads to a rise in the number of secrets employed in a development cycle. This progression of distributed and sensitive information is a well-known phenomenon with a specific title: secret sprawl. The surface area of attack and your overall security risk increases with each place secrets are shared.

2.8 Secret Synchronization

Sharing secrets among a group of developers can pose significant challenges. What happens when someone on your team changes a secret? How can you ensure that every team member has access to the updated secret?

It is crucial to keep everyone updated when changes are made to a secret. Failure to synchronize these changes can result in developers and critical systems not having access to the resources they need to work.

Now that we have explored the importance of secrets management and its potential challenges, let's take a closer look at how those challenges are typically addressed: secrets managers.

3 Secrets Managers

3.1 What are secrets managers?

Secrets managers are specialized software tools enabling developers to store, manage, and distribute sensitive information securely. They provide a centralized repository for keeping secrets, reducing the risk of secret sprawl and minimizing the surface area of attack.

Only authorized users and applications can access the secrets, ensuring the security and integrity of software projects. Secrets managers are a critical component of modern software development, as they help safeguard sensitive information and control access to it.

3.2 Centralized Secret Storage

Secrets managers address secret sprawl by making a single source of truth that houses secrets, thus limiting the proliferation of secrets and reducing the surface area of their exposure. A central database housing all secrets also has the advantage of automatically synchronizing a team's secrets since everyone can only access secrets from that same database.

While having a central database addresses problems associated with secret sprawl, it creates a new problem: a single place that would-be malicious actors can target to access an organization’s secrets. It is essential to protect this single target to mitigate this risk as much as possible.

3.3 Securing Secrets

All secrets managers rely on encryption as a critical part of their security. Encryption is the scrambling of information to become unreadable without the decryption key that can unscramble it.

Secrets managers use this exact technique to encrypt their stored data and only decrypt it for intended users. The information is useless to unauthorized users even if the database is breached. This form of encryption is called at-rest encryption.

Secrets managers also provide in-transit encryption to secure secret information sent over the network and provide a mechanism for authorized users to decrypt the message. This is typically done through a secure communication protocol like HTTPS.

A service that offers both in-transit and at-rest encryption, along with a mechanism for secure decryption, is said to provide end-to-end encryption, ensuring that the secret information remains secure throughout its transmission and storage.

Secure access to secrets is about more than encryption, however. It also requires a mechanism for controlling access to those secrets.

3.4 User authentication and access management

All secrets managers provide some form of authentication to identify users and confirm they can access their requested resources. The most common form of identification is a username and password.

3.5 Principle of least privilege and user access management

Authentication alone is not enough to secure access to secrets. No secure organization gives everyone access to every single sensitive document they own. Instead, they follow the Principle of Least Privilege (PoLP), a security concept that limits user access rights to only the necessary resources required to perform their job functions.

Since each developer’s access to secrets increases the risk of secret exposure, secrets managers allow teams to specify what secrets individual developers do or do not need access to. This helps limit the potential size of a breach if a developer's authentication information is compromised.

However, even with PoLP in place, there is still a risk of secrets being compromised. Audit logs are essential for identifying and responding to potential security threats.

3.6 Access Audit Logs in Secret Management

Audit logs comprehensively record events such as accessing, modifying, creating, and deleting secrets. This gives organizations a clear overview of the characteristics and timing of interactions with secrets.

Through the review of audit logs, organizations can not only identify anomalous access patterns but also engage in effective action monitoring. This dual functionality enables quick detection and response to potential security threats as organizations actively observe ongoing activities in real time.

3.7 Injection Techniques for Secrets

So far, we’ve covered the importance of secure storage, but how do we transfer these sensitive secrets to where they’re needed within a software application? What does this transfer process entail? How can we prevent unauthorized interception during transmission and make the transfer process as seamless and effortless as possible?

The solution to this problem is called secret injection, which can be implemented several ways.

There are three primary implementations for secret injection – HTTP API calls, SDK commands, and program wrappers.

3.8 HTTP API Calls

Some secrets managers expose an API that allows users to perform various operations, such as retrieving and updating secrets. Secrets are securely stored outside an application and are fetched by sending requests to these endpoints, which retrieve the secrets used by the application.

3.9 Software Developer Kit (SDK) Commands

Many secrets managers provide SDKs in many languages for interacting with them.

The advantage of this method is that it abstracts away the complexity of working with the API described above and replaces it with easier-to-use function invocations. The disadvantage is that developers migrating to an SDK-based secrets manager in an existing codebase must replace all references to secrets with the SDK functions, which can be time-consuming. It also means that any significant updates to the secrets manager have the potential to break the apps that are using the SDK.

3.10 Program Wrappers

A wrapper and an SDK have similar functions but differ in implementation. While an SDK provides a complete set of tools for interacting with a secrets manager, a wrapper is a secure gateway that retrieves secrets from storage, decrypts them, and injects them into your application at runtime.

Wrappers work by preloading secrets before your application runs and initiating a child process of your application with the secrets injected. This process allows your application to access secrets using the same syntax as it would if it were using a .env file without worrying about exposing its secrets or requiring direct modifications to the source code.

Let’s shift our focus from application-level concerns to storage and server hosting.

3.11 Self-hosted vs. Managed Solutions

Like many other tools, secrets managers can typically be deployed in two ways: self-hosted and managed. Self-hosting involves deploying the secrets manager within the organization's infrastructure, while managed solutions outsource the management of secrets to a third-party provider. Each option has advantages and disadvantages, and the choice depends on various factors, such as the organization's size, budget, and security requirements.

3.12 Self-hosting

In the age of cloud service providers, self-hosting is no longer synonymous with owning your hardware but can instead mean relying on a cloud provider to host infrastructure you otherwise control. When secrets managers offer a self-hosting solution, users install the secrets manager instance on their environment.

3.13 Managed

Cloud-based, managed secrets managers are services provided by third-party vendors that allow users to offload their maintenance, security, and management responsibilities to the vendor. This option is usually more convenient as users don’t need to worry about managing their architecture. However, managed solutions do not offer the same level of control as self-hosted solutions and can be cost-prohibitive in the long run.

4 Existing Solutions

We’ve covered the essential features provided by all secrets managers, but there are important differences between them.

4.1 Lightweight Solutions

Lightweight Solutions are tailored for quick and easy setups or smaller-scale deployments. These solutions prioritize ease of use and streamlined interfaces focused on essential secret management requirements. Their minimal administrative overhead makes them ideal for smaller organizations seeking straightforward, agile solutions.

4.1.1 Confidant

Confidant is an open-source secrets management service developed by Lyft that is characteristic of lightweight solutions. Its capabilities include secure storage and retrieval of secrets, access control, and encryption. Confidant is designed for smaller organizations and provides simplicity and ease of managing sensitive information.

4.2 Enterprise-level Solutions

Enterprise-level solutions cater to complex enterprise environments and provide advanced functionalities like dynamic secrets, encryption-as-a-service, fine-grained access controls, and automated secret rotation. These capabilities are designed to handle the needs of larger organizations or higher security requirement applications.

Automated secret rotation updates secrets without manual intervention or downtime. It generates and archives new credentials while seamlessly updating the resource that uses them.

On the other hand, dynamic secrets create temporary credentials upon request and provide short-lived access, but they come at a higher cost and require more involved configuration.

Enterprise-level solutions are meant for large organizations looking for a comprehensive suite of tools tailored to their unique security needs. However, these solutions also have a steeper learning curve and require more intricate configuration.

4.2.1 HashiCorp Vault

HashiCorp Vault is an open-source solution for securing secrets across corporate IT infrastructure. It is a comprehensive tool offering a robust framework for securing sensitive information beyond API keys and database passwords. However, the setup and configuration of HashiCorp Vault can pose challenges, such as complexities in managing access controls, defining policies, and integrating with existing infrastructure.

4.2.2 AWS Secrets Manager

AWS Secrets Manager is a proprietary AWS service for securely managing secrets. It offers features like automated secret rotation, access controls, and injection methods designed to enhance security and streamline operations related to microservices, CI/CD, and database credential management within the AWS environment.

AWS Secrets Manager is a good fit for larger organizations that require advanced features and prefer a managed service. However, configuring the service, especially its automation features, can take time and effort. Additionally, there is no free tier.

4.3 Introducing Embrasure

Embrasure is a self-hosted secrets manager for small teams prioritizing simplicity and advanced security features. It follows industry-standard tools and AWS best practices, employs IAM and DB authentication to ensure secure database access, and uses audit logs to document all server requests for potential security threats.

Embrasure is a cost-effective choice for smaller teams that have already integrated AWS into their workflow and want to maintain control over their secrets and infrastructure. All of Embrasure's components are available on the AWS Free Tier.

5 Design Principles

Our primary goal with Embrasure is to provide a highly secure secrets manager without compromising usability or workflow efficiency. Embrasure prioritizes security, authentication and authorization, action monitoring, and user-friendly design while maintaining ease of use. These demos and GIFs walk through common workflows supported by Embrasure.

Principles behind Embrasure’s Design

We followed five guiding principles while building Embrasure. Those core design principles are as follows:

Principle 1: User-Friendly Design

We designed Embrasure to be developer-friendly and accessible to users with varying technical backgrounds.

To achieve this, our CLI has intuitive commands and a simple setup, which automates the process of building everything a user needs to manage their secrets effectively and with minimal complexity.

After just two CLI commands and minimal configuration, Embrasure is set up and ready to use on your AWS infrastructure.

Secret management

Embrasure has the standard CRUD operations (Create, Read, Update, Delete) for interacting with secrets. For example, a user would simply type embrasure addSecret -n NAME -v VALUE to add a new secret for the entire team.

Secret Injection into a file

Embrasure injects secrets into your application using a program wrapper.

By entering embrasure run -file app.js in your CLI, all necessary secrets will be grabbed and injected into your program environment.

Principle 2: Administrative Control and Ownership

Embrasure is fully open-source and self-hosted, ensuring administrators have complete control and ownership of their secrets management instance By making the entire codebase accessible, administrators can deeply understand the system's inner workings. This transparency also allows for comprehensive audits, fostering trust within the user community.

Principle 3: Security and Availability

One of the significant challenges of security is that you can always be more secure. There will always be another adjustment or tool that organizations can implement to increase security, but each addition makes that system more and more complicated to understand or operate. When designing Embrasure, we considered many alternatives and features to balance security, simplicity, and usability.

Embrasure minimizes its attack surface area to provide high-level security and reliable access to your secrets by leveraging established AWS standards for end-to-end encryption and dynamic scaling. Additionally, it relies on trusted AWS tools to scale with variable network demand and achieve high availability.

Principle 4: Authentication and Authorization

Embrasure strictly adheres to the principle of least privilege, ensuring that it authenticates and authorizes all users. In short, our system secures secrets and makes them available only to authorized users. To achieve this, we must ensure that users are who they claim to be and are authorized to access those secrets.

Authentication occurs first in the process and confirms the user's identity. It aims to answer the question, "Who are you, and can you prove your identity?"

Authentication involves verifying an individual's identity, usually using credentials such as usernames and passwords. It aims to answer the question, "Who are you, and can you prove your identity?"

By identifying users of our system, we can prevent unauthenticated access to our sensitive data and protect against malicious activities such as unauthorized changes, deletions, and data theft by malicious actors. Authentication also allows us to see which users use our system and map actions taken by those users. This is important for auditing purposes and investigating security incidents.

If authentication is unsuccessful, the user is denied access. However, the next step is authorization if the user is authenticated successfully.

Authorization determines the resources the authenticated user can access and the actions they can perform. It involves granting or denying access to resources based on the user's identity. In simple terms, authorization answers the question, "What can you do and access based on who you are?".

If the user is both authenticated and has proper authorization, the server will proceed with their request; if not, the request will be rejected.

User management

Embrasure can also create new AWS IAM users, setting their access level permissions, and delete users, all with a few simple CLI commands. To add a new user with the ability to add new secrets, enter the command embrasure addUser -name NAME [-write], and then send them their new AWS credentials through whatever password-sharing service your team uses.

Principle 5: User Action Monitoring

Embrasure records all user actions to provide comprehensive visibility into how secrets are accessed. This feature gives admins the tools to control and oversee their secrets, creating a secure operational environment.

Although we have taken great care to secure our secrets, it is still essential to have a plan to deal with security breaches. Audit logs are critical as they provide valuable information and enable effective incident management.

Audit logs are records of all interactions and events related to a secret manager, such as creating, reading, updating, and deleting secrets. They provide observability of events, enabling teams to monitor and review access to the secrets, see unauthorized attempts to access resources, and investigate security incidents.

Regarding privileges, audit logs provide insight into when a user was given access and what actions the user did with those privileges. Essentially, audit logs provide a way for teams to see what happened and take the necessary steps to maintain accountability for an organization and rectify any mistakes.

6 Architecture

Embrasure's architecture revolves around a straightforward yet powerful concept: a secure and centralized database of secrets accessible to everyone on a distributed team. However, this description glosses over many technical hurdles and design challenges.

With that said, let us begin the deep dive into Embrasure’s architecture by starting with the first technical challenge of its design: how do you allow a distributed team to access their shared secrets anytime and anywhere?

6.1 Utilizing Cloud Services

Building a cloud native tool gives us a reliable way to build an application that is highly accessible to any user with an internet connection. This, however, raises the question of what cloud provider to use in the diverse cloud hosting field, or potentially making a cloud-agnostic tool packaged into containers.

We answered this question by building Embrasure on AWS primarily due to its authentication tools (more on that later) and its popularity. Embrasure deploys itself directly to a user's AWS account as its own, isolated instance for each team.

6.2 Storing Information

Having chosen a cloud provider, we can begin thinking about how Embrasure will store secrets.

6.2.1 Database

The cornerstone of Embrasure's architecture is its database: an AWS RDS (Relational Database Service) instance running PostgreSQL. This database will house all stored secrets and encrypt all information stored (at-rest encryption)

6.2.2 VPC

AWS Virtual Private Cloud (VPC) is the cloud networking service that allows Embrasure to create and manage its own virtual network within the AWS cloud infrastructure. Functioning as an isolated container, the VPC will protect and connect the various pieces of Embrasure's backend, assigning them with private IP addresses for identification and interconnection while isolating them from the broader internet.

6.3 Limiting the threat surface

Embrasure now has a highly accessible database in the cloud, but anyone who knows the database's IP address can access it. This lack of security is unacceptable for a secrets manager. This leads us to the next major technical design hurdle. We must now determine how we can limit access to the database to just the team using Embrasure.

6.3.1 Database Isolation within the VPC

Embrasure's design principles prioritize safeguarding the database against unintended access, a common concern in database management. Addressing the challenge of accounting for various potential ways to access the database, Embrasure takes a proactive stance by configuring its database to be private, which rejects outright any connection attempts originating from outside the Virtual Private Cloud (VPC) (see section VII, Diagram C8 for full architecture diagram).

An additional isolation measure taken by Embrasure is placing all resources in the VPC into private subnets exclusively, which results in all AWS resources having a private IP address accessible within the VPC. These intentional measures effectively seal off the database and Embrasure’s inner workings, ensuring that interactions are limited to specific, secure channels within the controlled environment of the VPC. This strategic approach to database protection aligns with industry best practices.

6.4 Networking to Communicate with the Database

So, we have an isolated database in the cloud that can only communicate with resources inside the VPC and a VPC that only accepts the network traffic we want Embrasure to use, but there is still a gap here.

We must now confront how a client outside of the VPC communicates with the isolated database containing all their information. Utilizing an HTTP API server as a communication middleman within the VPC fits nicely to solve this database access challenge.

6.4.1 AWS API Gateway and AWS Lambda

In order to understand how Embrasure’s backend communication works, we need to quickly refer to its CLI commands.

Under the hood, all of Embrasure's CLI commands send a request to the API server within Embrasure’s VPC that communicates with the database. A custom-built middleware transforms the CLI commands into HTTP requests with the headers required to reach the server.

Embrasure leverages AWS API Gateway and AWS Lambda built during deployment to act as this communications intermediary between a user's Embrasure CLI commands and Embrasure's central database.

API Gateway

The API Gateway is the conduit for secure communication with managed API endpoints. These endpoints, secured by TLS to ensure HTTPS communication, form the bridge through which users interact with Embrasure. It provides a well-organized and standardized approach to API management.

Using TLS (Transport Layer Security) and HTTPS (Secure HTTP) allows all API calls made to Embrasure to be encrypted in transit, guaranteeing that the sensitive information contained in each request-response cycle cannot be intercepted and read in transit.

Embrasure's API endpoints are designed using standard RESTful principles to enhance usability and maintainability. Each endpoint is tailored to facilitate interactions with specific resources, offering a structured interface for managing secrets, user access, and other relevant functionalities.

The RESTful design ensures clarity in the communication protocol, simplifying the process for developers to understand and leverage the API effectively.

API endpoints are utilized for all Embrasure interactions, of which three categories exist. CRUD operations for secrets, CRUD operations for users (admin only), and read operations for access logs (admin only).

Lambda

Requests to Embrasure’s API Gateway are directed to and processed by Lambdas, but what is a Lambda?

AWS Lambda, a serverless and event-driven compute service, eliminates the need to manage servers or backend resources. The Lambda dynamically provision resources, allowing for seamless scaling based on processing requirements.

Now that we understand what a Lambda is, we can discuss the design choices orchestrating Embrasure’s API Gateway to receive secure HTTPS requests from the CLI. The setup involves adding custom-generated user identification headers (more on that in next section) to each request sent to Embrasure’s Lambdas.

Each Lambda assumes the role of a vigilant listener, empowered with the necessary permissions to authenticate incoming requests with AWS IAM and communicate with the database.

Its responsibilities include converting requests into the requisite SQL queries via Sequelize ORM, querying Embrasure's central database, and processing the returned response.

The orchestrated workflow ensures that the Lambda serves as an intermediary, connecting the secure CLI command execution to the operations within Embrasure's central database. The user receives the processed response through this Lambda, completing the transaction initiated by the Embrasure CLI command.

Converting user inputs to API calls while banning direct database access ensures no risk of database exposure, and user queries are all sanitized. These two characteristics of Embrasure combined create a simple yet very secure barrier around an Embrasure team’s secrets. We’ll explore the final layer of Embrasure's security, user authentication, next.

6.5 Identifying and Authenticating Users

Embrasure has now become quite fleshed out. We have an isolated, cloud-hosted database where users can access stored information using an API server. However, a design challenge we alluded to still needs to be answered. Requests made by Embrasure CLI commands are supposed to be authenticated to confirm that the user sending the request is cleared to access whatever they are trying to access. So, let us now break down how exactly Embrasure ensures users are who they say they are.

User authentication is one of the most challenging security aspects to get right. Embrasure handles authentication with AWS’s Identity and Access Management (IAM) service. IAM allows us to create, manage, and authenticate users within an AWS account. Each IAM user has a corresponding user account stored on the database. Through this database user account, we can manage authorization for the user and what they are permitted to do on that database.

As mentioned, when a user runs an Embrasure CLI command, an HTTPS request is sent to the corresponding endpoint on our API gateway connected to our Lambda. This request includes headers that will be used in the user's authentication process and will establish an SSL connection to the database, which is required for IAM authentication. Some of these included headers in the request are the user's IAM username and a generated IAM authentication token.

AWS IAM uses the access keys stored in a user’s.aws folder to generate database authentication tokens, eliminating the need to store database user credentials. These access credentials are the basis for generating an IAM authentication token—a distinct string of characters requested and generated by Amazon RDS.

In place of a password, the IAM authentication token is used and dynamically generated. Because of this, the token is not stored anywhere, forgoing the need to store database user passwords. In addition, authentication tokens have a lifespan of 15 minutes, so you do not need to enforce periodic password changes for your database users.

Embrasure uses the signed IAM Authentication token to verify users rather than a traditional database username and password. Once the user’s request containing their IAM username and IAM authentication token reaches the API server, those credentials are used to connect to the PostgreSQL database. The database checks that the credentials are valid using IAM and that the user has an attached IAM policy that authorizes them to connect to the database.

Once connected to the database, privileges defined on the PostgreSQL database authorize what actions the user can perform on the database, such as creating, reading, updating, and deleting secrets. These permissions are applied to the user by the admin using the Embrasure CLI. Once the database query completes, a response is made and sent back to the user.

6.6 Tracking Access to Stored Secrets

There is now one final technical hurdle Embrasure sets out to solve. That is the challenge of keeping a record of every attempt to access the secrets stored by Embrasure. We considered implementing potentially costly tools provided by AWS, finding an open-source access log tool that fit our architecture, or building an access audit log from scratch.

Eventually, we settled on building our access log tool. Since all traffic to Embrasure goes through a single point of entry - the predefined endpoints on the API server, this provided us with a single place to monitor actions. We simply track and look at the traffic coming in and out of this single point of entry.

When a request reaches our API server, various middleware process it during the request-response cycle. Particular middleware looks at the request headers and uses the information contained in the headers to authenticate the user. If authentication is successful and the user who sent the request has the necessary authorizations, the request is processed by an additional middleware to carry out that request.

A response with the appropriate HTTP status code is sent, and a log is created and stored in a logs table in the database. Alternatively, if authentication or authorization is unsuccessful, that particular request is routed to a different middleware that will send the appropriate HTTP status code and create a log for that request, which is also stored in a logs table.

Each entry in the audit log contains information about the user who initiated the request, the IP address the request was made from, the request type, the resource route, the timestamp, the request authentication status, the request authorization status, and the HTTP status code.

Audit logs are written to the database through a dedicated logs worker. Because our database requires IAM authentication to connect with it, the logs worker itself has to generate IAM authentication tokens like any other non-admin user. Additionally, the logs worker has a corresponding user account defined on the database and the necessary privileges to create records on the database level.

7 Request Lifecycle

Now that we’ve gone over the many pieces of Embrasure’s architecture, let’s look at the complete path an Embrasure command takes from request to response for a non-admin user adding a new secret.

At the start of every request, AWS IAM uses credentials sent over HTTPS to generate an IAM authentication token that gets sent back to the user.

A user enters the command Embrasure addSecret -n databasePort -v 5432. Upon reaching the secure endpoint, the request is decrypted and checked against the firewall to see if it can enter VPC.

The request is then passed along to the Lambda server, which processes the specifics of the request.

A middleware checks headers of network requests for information like the user name and authentication token to validate the identity of the request sender.

Once the request is fully authenticated, it is processed by the Lambda, converted into its associated database query, and sent to the database.

The database performs one final check to confirm the user is authorized to execute that operation on the database, and if validated, the operation is performed.

The completed response is returned to the client’s machine through Embrasure.

The audit log middleware adds a new line to the database with all the request information.

Here is a look at the entire architecture as a unit.

Workflow of creating a new non-admin user

In another workflow, we will see how an admin can use a single terminal command to add a new developer on the team as a user to Embrasure.

Embrasure leverages AWS IAM to create new users, generate AWS access keys, and add the appropriate policies that permit the new user to connect to the database.

The admin runs Embrasure addUser -n bob from the terminal. This triggers a series of steps under the hood to create a corresponding IAM and database-level user.

The request is sent to AWS IAM, and an IAM user, “bob,” is created.

The new user is added to the embrasure-developer IAM user group.

AWS access keys are then generated for the new user.

IAM policies are created and attached to the new IAM user created.

After AWS IAM processes are finished and “bob” is added as an IAM user with the appropriate IAM policies attached to him, a POST request is sent to the API gateway endpoint to add “bob” to the database.

A middleware that is a part of the API server checks headers of network requests for information like the username and password/authentication token to validate the identity of the request sender.

Once the request is fully authenticated, it is processed by the Lambda, converted into its associated database query, and sent to the database.

The database does one final check to confirm the admin is authorized to add a new user. If validation is successful, a new user is created and stored in the database’s users table.

The completed response is now sent back to the admin.

Audit log middleware in the API server now adds a new line to the database, recording that a request was sent to the POST route that handles adding a new user to the database.

8 Trade-Offs

As with all projects, we made some trade-offs that come with the design choices when developing Embrasure.

8.1 Tradeoffs of using an AWS infrastructure

Embrasure heavily relies on AWS tools to simplify the setup and user authentication process for all users except the Embrasure administrator. However, this reliance on AWS means that the administrator must understand AWS services well, particularly AWS IAM.

It's important to note that Embrasure's usage of AWS creates a vendor dependency. This means that even if an organization decides to move away from AWS, Embrasure instances will remain tied to AWS infrastructure. Therefore, it's crucial to consider this reliance as Embrasure may pose as an anchor if an organization tries to migrate away from AWS.

We did consider other cloud providers, such as GCP and Microsoft Azure. We even considered creating a secrets manager that could be used across different cloud platforms and shipped in Docker containers. However, we ultimately decided to build our suite of tools within the AWS ecosystem due to their dominant market share in cloud computing (34% as of 2022) and the additional complexity of building a cloud-agnostic tool.

8.2 Tradeoffs of using a Lambda

During the development phase, we considered using an AWS EC2 (Elastic Compute Cloud) instance to host the API Server. However, we ultimately decided to use an AWS Lambda instead. While an EC2 instance would have been ideal for Embrasure's high availability goals, it would have incurred costs for computing resources during potential downtime and added complexity to managing the EC2's backend infrastructure.

On the other hand, Lambdas are event-triggered and only activate when a request is sent, meaning you only pay for the computing time you use. AWS Lambda runs your code on high-availability infrastructure while performing all the management and maintenance of computing resources. This makes it easier to manage and lowers costs incurred at the scale of small teams.

However, it's important to note that Lambdas automatically spin down when not in use, which means that if demand falls and spikes back up, Lambdas will require spin-up time, increasing latency. Despite this tradeoff, we decided that Lambda's simplicity and cost savings outweighed any temporary latency increase.

8.3 Tradeoffs regarding user authentication

AWS IAM handles user authentication in Embrasure. We chose AWS IAM over other authentication solutions because of the complexities and challenges of securing user authentication. Since our entire infrastructure is built on AWS and relies on an AWS account, it made sense to integrate user authentication seamlessly through AWS IAM into Embrasure.

Opting for a non-AWS authentication solution in our AWS-centric infrastructure would have introduced unnecessary complexities, which we aimed to avoid.

Moreover, the integration with AWS IAM came at no additional cost or changes to our existing infrastructure while being a time-tested, flexible authentication tool.

We also considered implementing Multi-Factor Authentication through AWS Secure Token Service (STS) to enhance security. However, MFA with STS would require two devices to confirm user identity, resulting in a less seamless user experience. Users would need to wait for a one-time password (OTP) for each request-response cycle or Embrasure usage session.

On the other hand, the IAM Auth token offers a more user-friendly experience by automating authentication in the background. However, this comes with the risk of complete account compromise if the user's AWS Secret access key is exposed.

We eventually decided that the comparative security vulnerability tradeoff was acceptable for the gain in user experience since a simple user experience was one of the guiding design principles of Embrasure.

8.4 Tradeoffs regarding audit logs

When considering options for implementing audit log functionality, we evaluated AWS CloudWatch as an alternative to Amazon RDS for PostgreSQL. We carefully considered factors such as pricing, integration with the existing architecture, and the level of added complexity before making a decision.

CloudWatch offers seamless integration with a wide range of AWS services, providing a unified platform for monitoring metrics, setting alarms, and responding to real-time incidents. However, this convenience comes at a cost of both financial and technical complexity.

Although CloudWatch provides the first 5GB of data per month free of charge, there are additional charges of $0.5 per GB and $0.12 per GB for log scanning to eliminate sensitive data.

On the other hand, Amazon RDS for PostgreSQL offers 20 GB of free storage with an additional 20 GB of storage for backups on the AWS Free Tier plan. Furthermore, we plan to add functionality for advanced querying of logs, which a relational database like Postgres is built for. For these reasons, we decided against using CloudWatch to store our logs.

Our second major decision concerning audit logs was whether to use an already-built solution or make our own. After careful consideration, we opted to construct our own audit log tool. This decision was driven by straightforward integration into the existing architecture, lower overall costs, reduced administrative complexity, and heightened security for stored information. Nevertheless, this choice comes with its own set of trade-offs.

Our audit log tool records all requests made to the API server, including invalid ones. However, this design has a potential risk. If someone with knowledge of an API endpoint decides to flood the server with spam requests, it could create a large number of logs, leading to increased storage costs.

Additionally, the audit log functionality relies on monitoring specific entry points into the database. If a team using Embrasure requires additional functionality, they must independently develop and integrate new routes into the API server. This process introduces the need to create and manage additional API endpoints, adding complexity to the overall system.

9 Future Work

We are working on improving Embrasure's functionality and user experience in the following two areas:

9.1 Log Management

We plan to upgrade our log management system to introduce advanced log management features. This upgrade will involve moving the logs to a different type of database better suited for their write-heavy nature and unique read needs. This move will improve the efficiency and management of the growing log records.

As a result of this upgrade, users will have access to advanced features such as pagination, search functionality, and sorting mechanisms. These features will enhance the user experience by making it easier to navigate and access relevant information within the extensive log records.

9.2 Dual Interface

We are developing a graphical user interface (GUI) to complement our existing Embrasure CLI. This dual-interface strategy aims to provide users with a versatile and user-friendly environment. The GUI will offer a visually intuitive platform to enhance the interaction with Embrasure, while the CLI will continue to provide advanced users with quick commands and flexibility.

9.3 Integration for Existing VPCs

Embrasure’s architecture construction currently relies on creating a new, empty VPC to prevent conflicts with existing resources during initialization. We plan to rework Embrasure’s initialization to optionally allow for the creation and networking of all resources required by Embrasure into an existing VPC.