SoundCloud Cut Database Waits 84% With Pythian

A music platform has an obvious audience: the person pressing play. Inside the business, there is another audience, less visible but just as impatient. Engineers need answers from other engineers before they can finish their own work. A database problem can leave both audiences waiting, even when the queues sit in completely different places. Pythian customer story ↗

SoundCloud brought in Pythian for on-demand database administration support. Pythian reports that Cassandra tuning lowered 99th-percentile read latency from 250 milliseconds to 40 milliseconds. That is an 84% reduction in the measured wait. The engagement also gave a stretched engineering organization access to specialist help.

The interesting part is the relationship between those outcomes. Better infrastructure needs people who have time to investigate it. Those people, in turn, need a support model that lets them investigate without becoming the answer to every question in the company. SoundCloud’s story offers engineering leaders a way to think about both problems together.

When ownership becomes a queue

SoundCloud’s teams managed their own databases, escalating unusual problems to Production Engineering. That central group also had tools and customizations to build. Requests for help competed with its own work, and limited capacity slowed its response to internal customers. Partnership overview ↗

Distributed ownership has a persuasive appeal: the people building a product stay close to how it behaves. But an organization chart cannot distribute every kind of expertise evenly. Some problems demand knowledge that a feature team rarely needs. Maintaining that knowledge everywhere can be expensive; concentrating it in a few people creates a queue.

For a leader looking at a delayed roadmap, the useful question is where work actually stops. Is a team missing a decision, a diagnosis, access, or implementation capacity? Those are different constraints. Hiring a generalist may help one and leave another untouched. Before deciding how many people to add, identify the questions that repeatedly travel to the same desk.

Expertise with a way in

Pythian offered SoundCloud a subscription to consulting hours over time. A shared Slack channel connected the teams. Jake Maizel, identified in the case study as SoundCloud’s VP of Infrastructure, described the relationship this way: “They’re like an on-tap consulting company.” Original case study, p. 2 ↗

That description is useful because it makes the service easy to picture. Someone encounters a problem; there is an established place to bring it. The practical value of specialist knowledge depends partly on how easily a team can reach it. A contract tucked away in procurement does little for an engineer who does not know where to ask.

For other organizations, the transferable idea is to design access alongside scope. Decide which questions belong with the specialist, who supplies context, and who acts on the answer. A messaging channel provides the doorway. Clear responsibility makes it productive. The aim is a short route from uncertainty to an informed next step, with enough context preserved for the next person who encounters the same problem.

The partnership

“They’re like an on-tap consulting company.”

Jake Maizel · VP of Infrastructure, SoundCloud
As identified in Pythian’s case study

The number worth slowing down for

The reported measurement is a 99th-percentile database read latency. It describes a point near the slow end of the measured reads, rather than their average. The change removed 210 milliseconds from that percentile: subtract 40 from 250, then divide by 250 to get the 84% reduction.

Precision makes the result more useful. It gives another infrastructure team a metric to compare with its own work. Calling a whole application faster would raise a different set of questions about which screens, requests, and workloads were tested. Here, the evidence concerns a particular database performance measure.

A percentile also changes the conversation about what good performance means. Averages can make a system look comfortable while some requests remain slow. Looking toward the tail asks a harder question: what happens beyond the typical experience? For teams preparing a performance review, the practical habit is to write down the metric, its scope, and the baseline before discussing the improvement. A percentage travels farther when its denominator travels with it.

The fix was in the details

The original technical account describes several changes: removing inefficient table compression, switching Java garbage collection to G1 and enlarging the heap, tuning kernel settings, and adjusting read-ahead. Pythian also helped resolve Cassandra Reaper issues so anti-entropy repairs could operate automatically. Technical detail, pp. 3–4 ↗

This is the part of infrastructure work that rarely supplies a glamorous photograph. It requires looking closely at the interaction between a database and the environment around it. A setting can have a sensible purpose and still impose a cost on a particular workload. Diagnosis means establishing whether that tradeoff is paying its way.

The lesson to borrow is the investigation method. Treat configuration choices as hypotheses that need evidence. Identify the workload, measure its behavior, make a reasoned change, and check the result. A successful setting in one environment is a starting point for a question elsewhere. The valuable deliverable is understanding why the system behaves as it does, so the next decision has a firmer foundation.

Availability needed its own work

The engagement also included helping provision another datacenter for a Cassandra cluster under existing topology constraints. Pythian examined consistency settings and produced guidance on the DataStax Java driver. The case study links that work to improved data availability. Architecture detail, pp. 4–5 ↗

Speed and availability deserve separate lines in a project brief. A quick response is valuable when a request can be served; a resilient architecture addresses the ability to keep serving requests when conditions change. Success on one dimension does not answer every question about the other.

That distinction suggests a useful review exercise. Ask the team to explain where data lives, how requests reach it, and which assumptions a new location changes. Then ask how those assumptions will be tested. A diagram should help the next engineer reason about a failure or a deployment. If it merely displays more boxes, it has not yet explained the operational benefit. Expansion becomes more useful when the behavior of the expanded system is understood.

Make room for the next hard question

The broader organizational lesson is about preserving attention. An engineering team can be busy every day and still struggle to make progress on planned work. If specialists spend their available hours answering urgent questions, the projects meant to reduce future urgency may keep slipping.

External expertise creates an opportunity to interrupt that pattern, but the opportunity needs a destination. A leader can name the work that recovered capacity should make possible: a troublesome upgrade, a tool that removes repeated manual effort, or a design review that has been waiting. Making that choice explicit helps turn relief into progress.

It also suggests a better way to evaluate support. Alongside technical outcomes, track how long teams wait for help and whether repeated questions become easier to answer. These are proposed management measures, not additional SoundCloud results. They make the human side of database operations visible enough to discuss, fund, and improve.

What another engineering team can borrow

Start with one recurring obstruction. Record which team encounters it, who can resolve it, and how long the handoff takes. This small exercise can reveal whether the organization needs more hands, a narrow specialty, or a clearer route to knowledge. Avoid making the purchase decision before understanding the queue.

Next, give the engagement a measurable technical objective and a practical working arrangement. Specify how questions arrive, what evidence accompanies them, and how recommendations become changes. Keep the before-and-after measurement attached to the same definition. That makes a successful result useful to the people who must maintain it later.

Finally, decide what the internal team should be able to do with the capacity it recovers. SoundCloud’s partnership shows why database support belongs in a conversation about engineering work. The database has a waiting time. So does the engineer who needs help with it. A good operating model pays attention to both.