Unpacking Project Failures

After successfully re-architecting our core frontend systems in just under a month and handling a traffic surge of nearly 1 million requests, the team decided to celebrate. During the celebration, an intern—just two months into the role—asked me a simple question: How many parties have I missed?

Initially, I couldn’t quantify it. But it got me thinking—not about the parties we’ve had, but the ones we missed. The celebrations we couldn’t have because of late project deliveries or underwhelming outcomes. As I reflected further and revisited past project failures, I noticed a clear pattern. Most projects from our engineering roadmap ended up in the "project graveyard" or did so because of biases or misguided assumptions.

Whose fault Is It Anyway?

Bias can come from anywhere—whether it's the development team, product management, or infrastructure. Anyone involved can contribute to the roadblocks that hold us back. Here are a few examples of how bias/atomistic view(s) has/have impacted our work and, ultimately, cost us a few more reasons to celebrate:

Throughput Estimations

Estimating system throughput can be incredibly complex. Typically, we rely on historical patterns and projections to evaluate throughput and conduct peak benchmarking. In one instance, we assumed a seemingly high threshold as a safe limit for our system. What we didn’t anticipate was a sudden surge in app opens and site visits. We believed the system could handle anything unless we saw a surge of over 100x.

All the key engineering decision-makers were in agreement; everything seemed under control. It was a significant event, and every team was geared to support growth. The marketing team, however, triggered a series of staggered push notifications that coincided with the peak in site visitors. This, combined with an unusually high app open rate (~24% compared to the usual ~11%), created added pressure.

While that alone was manageable, the issue escalated due to a shareable landing page that allowed real-time interaction between users and their friends. This triggered even more app opens and active users. What we failed to account for was the user’s exit from the landing page, which compounded the problem.

The combination of push notifications, back navigation, and increased traffic caused a rise in system latency. With the higher latency came retries due to timeouts, overwhelming the system and causing the gateway service (think BFF) to crash. Fortunately, our load-shedding mechanisms kicked in, and recovery happened faster than expected.

This event forced us to realign our priorities. Although the system has not handled another large-scale event with full user interaction since, it was pushed into the icebox—until recently, when it resurfaced for future consideration.

Learnings: honour Murphy’s Law

ReInventing the Wheel

Our velocity in shipping out features sometimes puts us in situations where a project appears fairly straightforward. There’s no need for fancy algorithms, complex UX, or cutting-edge technology. These projects seem direct and are easy to estimate. However, many of these projects still spill over deadlines.

Upon digging deeper, a clear pattern emerged. Projects in this category often had proof of concepts (POCs) and detailed documentation readily available in our digital workspace. This created an interesting scenario. New engineers, often early in their careers, were usually assigned these tasks by senior engineers since they were clear, well-defined, and had plenty of resources to help the younger engineers execute independently.

But this approach had an unintended consequence: the new engineers often ended up focusing on reinventing the wheel.

Younger engineers, particularly those early in their careers, tend to enjoy building things from scratch. There’s a certain excitement in the creative process, and it helps them feel a sense of ownership over the project. For them, it’s an opportunity to learn and demonstrate their skills. On the other hand, more experienced engineers typically prefer not to build from scratch unless absolutely necessary. They understand the trade-offs involved and often lean towards leveraging existing solutions to save time, reduce risk, and ship faster. Experience teaches them that the end goal is delivery, not reinvention.

This realisation made me start viewing interns and younger SDEs simply as 'engineers,' without the bias of preconceived notions about what they can or cannot do. By removing these preconceptions, I began to see their potential more clearly and understand that, while their approach may differ and sometimes omit nuances due to having less context, it can still be just as valuable in delivering impactful results.

Perfect Product Bias

This journey toward perfection doesn’t just apply to engineers. Product managers, designers, and others in the team are all striving for excellence in parallel, each from their own perspective.

For example, product managers often focus on defining every feature in a way that maximises value for users and business impact. They push for perfection in ensuring the product vision is fully realised, which can sometimes lead to scope expansion as new ideas and possibilities emerge. On the other hand, designers aim for pixel-perfect user experiences, seeking to refine every interaction, animation, and visual element to create a seamless and engaging interface. This pursuit of flawless UX can lead to iterative design improvements that demand more time.

Meanwhile, engineers are often caught in the middle, working to bring the product to life with efficient, scalable code. They may be focused on technical excellence—writing clean, optimised code, handling edge cases, and building a robust system that can handle future demands. This desire for perfection can also lead to scope creep, as engineers see opportunities for technical improvements or better ways to architect a solution.

While each team strives for their own version of perfection, these efforts can sometimes converge in a way that pulls back the project’s delivery. It’s like a tug-of-war between idealism and practicality. Each group, with the best intentions, wants to push the boundaries of what’s possible, but in doing so, deadlines can slip, and the focus on shipping can waver.

There’s always that pivotal moment in projects where reality sets in—when all the teams realise the need to recalibrate. Engineers start to trim the "nice-to-haves," product managers redefine the scope to focus on core features, and designers accept a slightly less fancy version of their vision for the sake of delivery. This convergence of compromise is what eventually brings the project to the finish line.

In hindsight, the tensions and constant iterations between teams are what make the process both rich and challenging. However, balancing the pursuit of perfection with the need for timely delivery is essential to ensure that the project ships on time without losing its core value.

The key to our ability to ship at super speed has been our focus on identifying the core product. We’ve shifted away from micro-optimizing at the implementation level, which previously slowed us down. Instead, we follow a clear set of principles that guide our process:

1. First, solve the problem – Our primary focus is always addressing the core issue.
2. Then, evaluate the solution and impact – We assess how well our solution performs and what impact it has on the product and users.
3. Finally, optimise the offerings – Only once the problem is solved and validated do we refine and optimise, avoiding premature enhancements.

By following this structured approach, we ensure that we ship faster while still delivering impactful, high-quality results. And, of course, there’s always hope for further improvement (and parties!).