Fordel Studios
I have shipped a lot of bugs in fourteen years of engineering. Most of them are boring. This one kept me awake for three nights straight.
What is the worst kind of production bug?
It is not the one that crashes your server. Crashes are loud. Monitoring catches them, pagers fire, you fix them. The worst bugs are the ones that look like everything is working perfectly — fast responses, green dashboards, happy metrics — while quietly doing something catastrophic underneath.
In late 2024, we were working with a Series B SaaS company. They had a solid product — a financial reporting platform used by about 300 mid-market companies. Their architecture was clean, their team was competent, and their customers were mostly happy. Mostly.
The complaint was latency. Dashboard loads were hitting 4-6 seconds for customers with large datasets. Their investors were asking questions. Their sales team was losing deals to faster competitors. They brought us in to fix the performance problem without rewriting the application.
How did we decide to fix the latency problem?
We spent two weeks profiling. The bottleneck was obvious: every dashboard load triggered 12-15 database queries, some of them joining across tables with millions of rows. The data changed maybe once or twice a day for most tenants — quarterly financial reports do not update every second. But every page load recomputed everything from scratch.
The fix seemed textbook. Add a Redis caching layer. Cache the expensive aggregation queries. Invalidate on data import. We estimated three weeks of work. The client was enthusiastic. Their CTO had been wanting to add caching for months but never had the bandwidth.
We designed the caching strategy carefully. Or so we thought. The cache key format was: feature:queryHash:dateRange. Query hash was a SHA-256 of the SQL query string with parameters. Date range was the reporting period. We wrote invalidation logic that cleared relevant cache entries when new data was imported. We added TTLs as a safety net — 4 hours for dashboards, 1 hour for reports, 15 minutes for real-time widgets.
We tested it in staging for a week. Performance was extraordinary. P95 latency dropped from 8 seconds to 200 milliseconds. Cache hit rates stabilized around 94%. The client was thrilled. We deployed to production on a Thursday afternoon.
What went wrong with the cache key design?
You have probably already spotted the problem. Read the cache key format again: feature:queryHash:dateRange. Notice what is missing.
The tenant ID.
In the application layer, every database query was already scoped by tenant — the ORM added a WHERE tenant_id = ? clause automatically. So the SQL query string for Tenant A's revenue dashboard and Tenant B's revenue dashboard were different. The query hash was different. In theory, this was safe.
In theory.
In practice, the application had a feature called "template dashboards" — pre-built financial views that the client's product team had designed. These templates used parameterized queries where the tenant ID was injected at execution time, not at query construction time. The ORM's tenant scoping happened inside a middleware that ran before the query cache check. But template dashboards bypassed that middleware because they used a different query path — a raw SQL executor that the original team had built for performance reasons two years earlier.
“The bug was not in our code. It was in the interaction between our code and a decision someone made two years ago for completely valid reasons.”
For template dashboards — which accounted for about 40% of all dashboard views — the SQL query string was identical across tenants. Same query, same parameters (except tenant ID, which was bound at execution time, after the hash was computed). Same hash. Same cache key. Same date range.
Tenant A loads their revenue dashboard. Cache miss. Query executes. Result cached. Tenant B loads their revenue dashboard thirty seconds later. Cache hit. They see Tenant A's revenue numbers.
How did we discover the data leak?
We got lucky. One of the client's customers was a CFO who had been using the platform for three years. She opened her Q3 revenue dashboard on Friday morning and saw numbers that were roughly 10x what she expected. Instead of assuming the platform had a bug — which most people would assume — she recognized the revenue figures as belonging to a company she had worked at previously. A company that also happened to be on the same platform.
She called support. Support escalated. The client's CTO called us at 11 PM on a Friday night.
The conversation was short. He said: "One of our customers is seeing another customer's financial data." I felt my stomach drop. There are categories of bugs, and cross-tenant data leakage in a financial platform is at the very top of the severity list. This was not a latency problem anymore. This was a potential breach notification, a legal liability, and possibly the end of a company's reputation.
What did the first 4 hours look like?
We killed the cache immediately. Full Redis flush, cache layer disabled, application falling back to direct database queries. Latency went back to 4-6 seconds. Nobody cared about latency anymore.
Then we started forensics. We needed to answer three questions: How many tenants were affected? What data was exposed? How long had this been happening?
The answers were not as bad as we feared, but bad enough. The deployment had been live for 18 hours. Template dashboards accounted for about 40% of views. But cache hits only happened when two tenants viewed the same template dashboard within the same 4-hour TTL window. We analyzed Redis access logs and cross-referenced with the application's audit trail.
Twenty-three tenants had been served at least one cached response that originated from a different tenant. For most of them, the data was partial — a single widget on a multi-widget dashboard. For seven of them, full dashboard views had been served from another tenant's cache. For three of them, the exposed data included revenue figures, margin calculations, and customer counts.
Financial data. From competitor companies. On the same platform.
How did we handle the aftermath?
The technical fix took two days. We added tenant ID as a mandatory prefix in every cache key. We added a validation layer that checked the tenant ID in the cached response against the requesting tenant before serving it — defense in depth. We added automated tests that specifically attempted cross-tenant cache pollution. We audited every query path in the application to ensure tenant scoping was applied consistently, not just in the ORM middleware.
The non-technical aftermath took three months.
The client had to notify all 23 affected tenants. Their legal team drafted disclosure letters. Two tenants — the ones whose financial data was most sensitive — escalated to their own legal teams. The client's Series B term sheet, which had been in final negotiation, was delayed by six weeks while investors assessed the incident.
- Full security audit of caching layer and all query paths
- Breach notification to 23 affected tenants with detailed exposure reports
- Legal review and response to two tenant escalations
- Investor confidence rebuilding — delayed Series B close by 6 weeks
- Implementation of tenant isolation testing in CI/CD pipeline
- Third-party penetration test focused on data isolation
Nobody sued. The client kept all 23 customers, though two renegotiated their contracts. The Series B eventually closed, at the same terms. But the cost — in engineering time, legal fees, executive attention, and trust — was enormous.
What would we do differently?
Everything about the cache key design, obviously. Tenant ID should have been the first element in every key. That is the easy answer and the least interesting one.
The real lesson was about implicit versus explicit multi-tenancy. The application relied on middleware to enforce tenant isolation. That works for the happy path. But every system has paths that bypass the happy path — raw query executors, batch jobs, admin endpoints, template engines. If your isolation strategy is "the middleware handles it," you have an isolation strategy that fails silently whenever anything bypasses the middleware.
“If your multi-tenancy is implicit — enforced by middleware rather than encoded in the data model — every new layer you add is a new opportunity to forget.”
We now treat multi-tenancy the same way we treat authentication. You do not assume it is handled upstream. Every layer verifies independently. Cache keys include tenant ID. Queue messages include tenant ID. Log entries include tenant ID. Every function that touches data takes tenant ID as an explicit parameter, not an ambient context variable pulled from a request scope.
The second lesson was about cache key design reviews. We had code reviews. We had architectural reviews. But nobody specifically reviewed the cache key schema for tenant isolation. It was not on anyone's checklist because caching felt like an infrastructure concern, not a security concern. We now have a specific review gate for anything that introduces a new shared state layer — caching, queuing, pub/sub, file storage. The review question is always: "Show me where the tenant boundary is enforced."
Is aggressive caching worth the risk in multi-tenant systems?
Yes. But the risk is not in caching itself. The risk is in treating caching as a performance optimization when it is actually a data distribution system. When you cache a query result, you are creating a second copy of that data in a system that has different access control rules than the primary database. If you do not design those access control rules with the same rigor you apply to your database, you are building a data leak that waits for the right cache key collision.
We still build caching layers for multi-tenant systems. We have done it five or six times since this incident. Every single time, the first design document includes a section called "Tenant Isolation in Cache" and the first test we write is the cross-tenant pollution test. The 40x performance improvement was real and valuable. The bug was entirely preventable.
That is maybe the hardest part of this story. It was not a sophisticated failure. It was not an edge case buried in distributed systems theory. It was a missing string in a cache key. The kind of thing you would catch in a code review if you knew to look for it. The kind of thing that is obvious in hindsight and invisible in the moment.
The CFO who spotted the wrong numbers saved the company from a much worse outcome. If she had not recognized those revenue figures — if the exposure had continued for days or weeks instead of hours — the story would have ended differently. We got lucky. We designed our systems so we would not need luck again.
