The secret behind making SaaS multi-tenant workloads safe

The beauty of multi-tenancy SaaS is undeniable, offering scalability, cost-effectiveness, and streamlined updates. However, just like any powerful tool, it's all about how you handle it. Failing to properly isolate data in a multi-tenant environment can lead to catastrophic consequences. From OpenAI's incident with Redis, which resulted in users being able to see other active users' chat histories and even payment-related information¹, to the AWS SuperGlue vulnerability which could enable to cross tenant boundaries.

Main Causes for Cross Tenant Leakage

In understanding the main causes for cross-tenant data leakage, it's crucial to acknowledge that even the most well-intentioned implementations can have flaws.

Developer bugs can lead to cross-tenant leaks. Subtle errors in code can cause data intended for one tenant to be accessible to another. A common scenario might involve misconfigurations in the caching layer. For example, an undefined variable or incorrect variable order could result in all tenants accessing the same data².

Then there are third-party or supply chain induced data leaks. These can occur due to bugs in external libraries or services that the application depends on, as was the case in the OpenAI data leak due to a bug in the Redis client open-source library, redis-py¹.

Lastly, bad actors or compromised components also pose a threat. Despite best efforts, there's always a risk of intrusion, and without adequate protections, a single compromised element can lead to widespread data exposure.

Developer Bugs Leading to Cross-Tenant Data Leaks

One of the primary causes of cross-tenant data leakage in multi-tenant SaaS environments is developer bugs. Even the most skilled developers can make mistakes, and in a multi-tenant environment, these mistakes can lead to serious data breaches.

Let's take a look at some concrete examples.

Cache Layer Mishandling

Most architectures have a caching layer (e.g., Redis) in use for various purposes. In a multi-tenant system, it's common to distinguish between different tenants by incorporating the unique tenant id into the cache key. For example, you might use a key like "1234:users" for tracking users of each tenant.

However, subtle coding errors can lead to unexpected data leakage. For instance, consider the following JavaScript code:

let key = `${teantId}:users`; // A subtle typo here

The subtle typo in the variable name (teantId instead of tenantId) means that the key will be undefined:users, which all tenants can access. This small mistake can cause a significant cross-tenant data leak.

Incorrect Variable Order

Another common bug is having correct variables but in the wrong order.

For example, you might have a key like ${accountId}:${tenantId}:users instead of ${tenantId}:${accountId}:users. Spotting such mistakes in a code review can be a challenge.

Third-party or supply chain induced data leaks

In some instances, the leak might not be in your code at all. You might need to guard against an external service having their own leak. If the response from the external call includes the tenant identifier, ensure that it matches what you expect. Otherwise, you risk polluting your database with another tenant's data or worse yet - return somebody else's data to the end-user. (read:cross tenant data leak).

A notable example of an external service leak occurred when a bug in the open-source Redis client library led to a cross-tenant data leak in OpenAI's ChatGPT service**1**. The bug caused a connection to return corrupted data that matched the data type the requester was expecting, even if it belonged to another user. The returned data included account information and titles from chat history of another account. This incident underscores the need to be cautious when interacting with external services and caching layers.

The Need for Decoupling Tenant Isolation Enforcement from Service Developers

While developers play a critical role in building secure multi-tenant systems, it's crucial to decouple tenant isolation enforcement from them. At superTenant we're building a 'Tenant Firwall' - which provides a strong tenant isolation layer, without relying on application modification or code changes.

This approach offers several benefits:

Avoidance of Developer Errors

As the examples above demonstrate, even small developer errors can cause significant data leaks in a multi-tenant system. By decoupling tenant isolation enforcement from developers, we can reduce the likelihood of such bugs causing data leaks.

Improved Security

Decoupling tenant isolation enforcement from service developers can make the system more secure. A dedicated tenant isolation layer can implement more robust security measures, provide better monitoring, and respond more quickly to potential threats.

Easier Compliance

Regulatory compliance is a critical concern for many SaaS providers. A dedicated tenant isolation layer can help ensure compliance with data isolation requirements, making it easier to pass audits and meet regulatory standards.

By introducing a dedicated tenant isolation layer, or a "tenant firewall," we can better protect against developer-induced data leaks, enhance overall system security, simplify regulatory compliance, and improve scalability.

This approach allows developers to focus on building great applications, while the tenant firewall takes care of data isolation.

Conclusion

The adoption of multi-tenancy in the SaaS industry is inevitable and provides immense benefits. SuperTenant's Tenant Firewall can drive this adoption further by adding a robust, developer-independent data isolation layer. This not only enhances overall security but also bolsters customer confidence. Moreover, it provides a safety net to quickly catch and rectify any tenant-isolation related developer bugs.

In a world where data is king, its protection is paramount.

If you are considering how to improve your handling of tenant isolation for co-located shared workload, the SuperTenant team would love to talk to you and offer some of their experience around these challenges.