Proof of Value – How to assess whether to take an AI project forward

    John Godfree

    It may sound obvious, but a starting point for any undertaking is:

    ‘Is it worth doing?’

    When choosing which data projects to progress, companies often start with proof of concepts to assess whether the AI can be built. Proof of concept is about exploring whether you can do something.

    But just because you can, doesn’t always mean you should.

    A more sensible starting point is thinking about proof of value.

    What is proof of value?

    A proof of concept might ask ‘how can we design a system that uses past maintenance data to predict future mechanical failures?’ A proof of value, conversely, asks questions like: ‘How valuable would it be to predict mechanical failures and can we do it accurately with existing datasets?’ and 'If we could predict failures, could we actually do anything about it?'

    The benefit can then be assessed against the cost of doing it and the opportunity cost of using your data scientists to do something else, to decide whether it’s worth taking forward. It’s designed to assess, explore, and take the risk out of projects. Through multiple proofs of value, you can identify a portfolio of the most promising use cases.

    Proof of value in action: AI predictive maintenance

    Let’s look at an example from Tessella’s own experience: a train operator implementing predictive maintenance to warn when components might fail.

    We began by analyzing their available in-service data to identify which faults had the largest impact on maintenance engineers, delays, and customer inconvenience. Surprisingly, we discovered that faults in lavatories had the most negative impact on the travel experience, and were most easily remedied through predictive analytics with existing data.

    The project was pursued and ultimately led to a machine learning algorithm able to predict faults and schedule maintenance, which is now used by several transport providers.

    The proof of value allowed us to identify which of a wide range of projects were possible with existing data, and also use that data to identify what would have the most business value. Had we just said, ‘door faults cause delays, let’s deal with that’, we would have missed a more pressing problem. We may have even jumped headlong into a far more expensive project for which we didn’t have the right data.

    Discover how to build, train, and deploy AI for use in the enterprise. Download  our white paper.

    A check on data science hype

    As well as identifying promising projects, proof of value saves companies from embarking on projects likely to fail, either from a technical or business case perspective.

    The most common outcome of proof of value is identifying holes in datasets that prevent value being realized. Organizations often believe they have relevant datasets, but once they start gathering them for the data scientists, many can’t be found or have fundamental issues like missing time periods or metadata. This is one of the most common reasons data projects become derailed.

    Another outcome might be to spot better ways of approaching a planned project. Too many project managers are determined to deploy costly neural networks when a simple statistical model will do just as well. Data science is no different from any other technology or business project: start simple, assess, and evolve.

    Spotting such problems early means the company can understand how to get data, processes, and teams ready before embarking on proof of concepts and the associated expectations and deadlines.

    While this is highly valuable at the start of a project, it’s often worth repeating at critical points in the progression towards a minimal viable product. This lets you assess whether new learnings have created new opportunities or spotted roadblocks that need to be overcome.

    How to run a proof of value

    We advocate starting with "art of the possible" workshops. These look at a range of suggested use cases and explore the potential for new ones.

    For each project, first agree to what ‘value’ means. This can vary. Some projects may have very clear definitions of value. Others – especially those in R&D - may have clear objectives but unclear solutions, like exploring whether there’s any way to use preclinical data to usefully predict later stage success. Only by starting the journey can you begin to solidify the direction of travel.

    The most promising use cases should be prioritized, and then investigated by data scientists who use samples of available data to explore and visualize insight. This allows a quick assessment of whether proposed value can be delivered, to explore potentially interesting associations, and to understand the limitations of the data.

    Those which are likely to deliver value then feed into a portfolio of projects which are taken forward to proof of concept.

    Valuable AI projects start with proof of value

    Data science and AI projects infamously struggle to get beyond the proof of concept phase. One of the reasons for this is that the project wasn’t worth doing in the first place.

    Proof of value, however, allows companies to identify valuable projects and focus resources on getting there in a realistic way. Such an approach not only takes the risk out of projects, it underpins clear, evidenced business cases that will secure organizational buy-in.

    Three Stages White Paper

    Subscribe to Our Newsletter