Rethinking Analytics

The 'data revolution' never arrived, but we can still build something that matters. Adopting a structured, causal modeling framework presents the most effective solution to our challenges.

The Gartner Hype Cycle

There was a time when data teams embodied the future. From the advent of BI tools in the 1990s to the 'big data' era spanning the late 2000s to early 2010s, and onward to the ascent of data science and cloud data platforms, the data industry had seen a continual renewing of its hype cycle, powered by the investment that comes with being on the up. A data revolution was always just around the corner, lying in wait. A revolution that would be as exciting as it was scary. We just needed the time and money to build it.

Then interest rates went up - and patience ran out. Data teams started coming under scrutiny: can you measure your impact on the balance sheet? Are you able to justify the size of your team? Why are your infrastructure and tooling costs so high? An increased need for accountability in recent years is not a unique challenge for data teams, but it has been especially difficult to navigate for us. The promised revolution simply never arrived, at least not as we said it would.

I started my career at the tail end of this hype cycle and haven’t always been confident I was adding value as a data analyst. Not out of incompetence, but because the underlying operating model didn’t seem to be delivering for data teams or stakeholders. There were even moments where I wondered if the data industry was an elaborate ruse, and if I shouldn’t just pivot away entirely before it contracts in on itself. With the industry in its own ‘disillusionment’ phase, so was I.

When I feel stuck, I find it helpful to write down what’s wrong and address it from first principles. In the words of Charles Kettering, an American inventor and engineer, ’A problem well stated is a problem half solved’. This post is a summary of my observations on what isn’t working, none of which I think are that surprising to those who work in analytics. Less obvious is what we can do to change it. My hope is that by outlining the problems in this way, you will understand why I think that adopting a structured, causal modeling framework presents the most effective solution to our challenges.

Let's start by examining a common assumption that has led many data teams astray.

Data is not the new oil

In 2006, mathematician Clive Humby likened data to oil, coining the famous phrase "data is the new oil." He later clarified this comparison, emphasising that, much like crude oil requires refining to unlock its full potential, raw data must also undergo processing to become truly valuable. But the impression remains that data is intrinsically a store of value, and that realising it becomes practically inevitable once you hoard enough of it and hire a few data engineers and analysts. What if that’s not the case?

Benn Stancil has written extensively on this and one example he provides illustrates how this assumption can break down. Imagine that you are the owner of a convenience store and decide to hire a data analyst from a top tech company. You have access to all the data you can conceive of, and the analyst to help you make sense of it. Unfortunately, it’s unlikely your analyst is going to improve things as much as they could at their top tech company, because your product isn’t intertwined with data in the way that say, Google Search or the TikTok algorithm is. It operates within well understood constraints - labor costs, competitor prices, supply costs and so on. There isn’t that much space to optimise. And in that little remaining space there is to optimise, as the owner you will have a good grasp on it from your years of experience. It’s not to say the analyst you hired couldn’t suggest some moderate improvements, but it’s not data revolution. Frankly, no matter how amazing the data this store had, an analyst trying to generate insights is not going to drive decision making in a transformative way.

As an assumption, this is something we have to just discard and move on from. Data teams have often taken for granted that there is value lying in wait in their database. And probably yes, in modest, non transformative amounts. But it’s dangerous if the work of the data analyst gets reduced to repeating a magic trick again and again, usually in a highly disorganised way. There has to be an alternative.

Assertive data teams

Let’s say that you do have a useful dataset, and you’ve pulled together a few insights that you think could deliver some value. What then? If there isn’t a data driven culture within the organisation, you may struggle to make your case to the stakeholders that matter. Or there may be the appearance of a data driven culture, but you’re really just reinforcing decisions that have already been made, or really want to be made. In the latter case, they might have raised a ticket where they prescribed exactly what they needed. It can often feel more like bureaucracy than valuable work.

This is a very passive way of doing things, which I do not believe will get data teams anywhere. Specifically, I do not believe that analysts sitting behind a ticketing system refining data for stakeholders, abstracted from the questions that are actually being solved for, are ever going to deliver positive value for the business. If anything, we need to embrace becoming a more antagonistic force, or at the very least assertive.

Katie Bauer discusses this in Elbows of Data, making the case that analysts have to insist upon themselves and be proactive in driving change, whether they are invited to or not. She suggests some practical ways of doing this: shadowing people on the frontlines and seeing how they use (or could use) data, making work accessible to everyone and not just primary stakeholders, chiming in on email threads and newsletters, and generally taking anything as a prompt for fact-finding. Ultimately, effective analysts don’t wait for someone to ask, but solve problems even if the company hasn’t realised it’s ‘one for the data team’ yet. There’s probably good and bad ways of doing this, but as an ethos it’s hard to fault.

This leads to another question - how do data analyst solve the problems they find and is there a formalised framework for this?

Professional signal extractors

Decision makers depend on their data teams to address a pivotal question: is the world today significantly different from what I expected it to be yesterday? For example: did the increase in marketing spend have the impact I thought it would? Is that 10% drop in visits worrying and worth investigating, or just normal? Recognising when something has changed implies that we also need to establish the expectation had nothing changed. In other words, we need to be able to disentangle the ‘signal’ that our decision makers are interested in from everything else - the ‘noise’.

If you’ve ever been asked something like ‘what does this actually mean’ you are familiar with the need for signal extraction. And since it’s so common, we have methods for making that question go away. The first method is to pretend that the calendar year is a control group, convert the metric to year-on-year change and move on with your life. The second, perhaps more insidious, is to make the noise work for you in order to reinforce the story you’re telling. Why not give that tiny increase in sales some green conditional formatting if it strengthens your narrative even a little bit? When you don’t have the tools to disentangle signal and noise - and we don’t, at scale - it becomes tempting to just take what you want from it.

One person who has thought a lot about this problem is Cedric Chin. He has written a series on it, becoming data driven in business,  which highlights W. Edward Deming’s principles of statistical process control (SPC) as a possible solution. SPC has been used for decades in the manufacturing industry to isolate exceptional variation from routine variation, usually for monitoring efforts to minimise variation in quality control. But Cedric’s argument is that SPC has applications well beyond manufacturing. With sufficiently refined metrics¹, you can use SPC to understand if variation is meaningful or not; if a change has actually happened.

An example of an SPC method is the XmR chart. This is a simple chart which takes a time series and plots two control limits either side of it². These charts essentially detect the presence of multiple probability distributions in the variation of a series. To change a process, then, is to draw from a new probability distribution. The control limits of the XmR chart - set to three sigma around a centre line - captures with >95% true positive rate a significant change in probability distribution. This empowers data teams to tell decision makers: yes, this is meaningful, it worked. Or yes, this is meaningful, we need to investigate what happened. One caveat is that XmR charts are unlikely to be useful for an extremely volatile series. In such a case, Cedric notes that you should try and find out the exogenous shocks that are driving that volatility and adjust the metric accordingly. And if you can’t, it’s not something you can control so it’s unlikely to be worth focusing on.

XmR charts are just one example of SPC in action, and their simplicity makes them easy to adopt. Cedric adds, however, that many of the pioneers of Deming’s SPC - Amazon, Toyota to name a few - are not necessarily using XmR charts en masse in their business. This is because the XmR chart is a gateway drug: the “quickest, most effective way to create org-wide understanding of variation”. You can think of it as like training wheels on a bike, eventually you shouldn’t need them. Humans are expert pattern matchers and at the top of our game we can internalise variation very well. For example, Amazon executives look at 400-500 metrics every week, even if there hasn’t been a significant change. One of the reasons for this is to better internalise what normal looks like.

Once you adopt the philosophy of SPC, either governed through formal processes or as an informal mental model, it becomes a lot easier to do root cause analysis to understand what drives the metrics you care about. And in so doing, you may realise there’s a lot of metric interdependency in your business. You pull a lever in marketing, it results in a change in sales, which impacts customer support and so on. In other words, you start to conceive of your business as a system of inputs and outputs, a hierarchy of metrics with levers that you can pull that have measurable impact. You have developed a causal model of the business. And this way of seeing things excited me because the more I thought about it, the more I saw it as a natural antidote to some of the challenges troubling our field.

Towards system level thinking

Earlier, we discussed how ineffective analytics teams tend to be disorganised, abandoning codification or constraints for boundless exploration in the hope of generating, again and again, some insight that moves the needle. What structure we do have tends to make us meek and passive, as we wait for permission from the relevant department to answer important business questions that we often lack full visibility into. But in the previous section, we saw how statistical process control can help us to build a causal model of the business almost by accident, and I think this is the most promising framework yet for how analytics can drive growth.

This implicit causal model can be developed into something more explicit. One of the biggest proponents of causal models is Abhi Sivasailam, and I strongly encourage you to watch his talk at Data Council on Metric Trees. Metric trees are just one (very intuitive) method of causal modelling. In this framework, you start with ‘north star’ KPIs and recursively decompose it into its components. For example, a revenue north star KPI can be decomposed into customers x ASP, customers can then be decomposed into leads x win rate and so on. For most metrics, mappings between them will not be mechanical but instead empirical or hypothetical. It is then the role of the analyst to assess the confidence level of these connections. This is an immense responsibility: we become the central operating framework and single source of truth for the business, with stakeholders relying on us to share what worked in the past and what levers they can pull in the future to deliver growth. It’s anything but passive. This codified structure also resolves the problem of analysts engaging in boundless exploration and getting nowhere. Guard rails ensure that analysts are focused always on the growth of the business, whilst still enabling the creative work many of us enjoy.

The process of mapping metrics mitigates another challenge for data teams: dashboard sprawl. By developing a causal model and tying things together, we eliminate many of the reports that are dead ends or duplications. It should also reduce the number of dashboard requests we get. After mapping out a causal model, stakeholders should simply be less confused. With connected rather than fragmented reports, they will have visibility into the true system level dynamics of the business. As an example, imagine that the business has a new strategic initiative to improve customer retention. Currently, departments will workshop separately what that means for them, each having their own interpretation (and likely their own dashboard). With causal modelling, teams should immediately know how their metrics feed into retention and what levers they have to drive it from past experimentation. From this initiative will likely come new experimentation and new levers, all of which can be re-codified into the model.

It’s not necessary to use metric trees. In fact, it could be an entirely mental model (but at scale, it’s worthwhile to hard code it somewhere). It is the core concept itself, of relating input and output metrics through a causal model, that is powerful. This has the power to transform analytics teams from being a supporting function of varied importance into a vital strategic partner for the business. And ultimately, wouldn’t it be satisfying if the data revolution we’ve been waiting for was just… BI.

In any case, we are at an inflexion point. In the hype cycle we still remain - as an industry - firmly in the disillusioned phase. Much of the investor funding has moved on to AI, and the modern data stack has made our work easier but not materially impacted the value of it. Perhaps now that the VC money has dried up and it’s clear things aren’t working, we can collectively engage in the holistic thinking that this moment deserves.

Until we do, the slope of enlightenment will remain out of reach.

---

If you would like to read more about causal models, I would recommend the work of Abhi Sivasailam and Ernest Xheblati. Both of them discuss the challenges of implementing causal models in practice, such as winning exec buy in, where to store the model, where to start from and so on. They also have open-source templates.

1. A whole separate post in itself! But Cedric gives some examples in his article of metric refinement, so I encourage you to read.

2. A mathematical breakdown is available here: The Math Behind The Process Behavior Chart