International Conference on Libraries 2017

Build-Measure-Learn Project Management

Previous Next

Drawing upon the AGLE software development life-cycles and from Lean Manufacturing processes, Eric Ries, the originator of the Lean Startup management movement, proposed a new model for fast, iterative technological development of software products and services in what he calls a BML loop, short for three project phases Build - Measure - Learn.

Build

The start of a technology project for a new library service or resource starts in Build phase, where taking the requirements driven by three main sources of pull; People, Institutions, and what I believe is becoming increasing important, algorithmic sources of pull.

People

People Pull Source

Institutions

Organizations or Institutions Pull Source

Algorithms

Algorithms Pull Source

Minimum Viable Product (MVP)

From these requirements, a Minimum Viable Product (MVP) is built for initial use by your sources of pull by creating one or more testable hypotheses that you think the project will improve or change. The purpose of an MVP is to only build the least amount of functionality and user interface design to meet but not exceed these requirements and hypothesis and to generate usage statistics and impressions by the people and institutions that are the sources of pull for the project.

A library MVP should deliver an improvement in the resource access or service by patrons or in the operational work-flow for managing the resource or service by the library's staff. As your patrons and staff use the MVP, you move into the second Measure phase of the BML loop.

Measure

The next step in the loop is Measure, identifying key metrics from which you prepare for the final step in the project development lifecycle. These metrics and measurements allow you to test the hypotheses generated in the Build phase.

One method of doing this is using A/B testing of User Interface elements where you randomly give half of your users one version of the user interface, and a variant of the user interface to the other half. You then measure to then see what design preforms better to test your hypotheses that you created in the Build phase.

One area that you may not think of using the BML Loop is in the evaluation of commercial library systems, services, or products. As the Build portion is typically completed (likely through a push process) by the vendor, you can usually schedule a trial period that you can then turn into a Measure phase where you gather usage statistics and impressions by sources of pull before moving the Learn phase.

Look out for Vanity Metrics

When deciding what metrics or measures to track, be careful of Vanity metrics that's value seems to indicate improvement but the underlying process has not changed in the direction you would like from hypotheses. For example, don't confuse click rates on an instruction web resource to indicate learning of the topic, especially if the average time being spent on the webpage is measured in seconds but would count as a click.

In the Measure phase, capturing and acknowledging the participation of your end users of the software service or resource is important for the evaluation that starts here but continues in full force in the final phase of BML Loop, the Learn phase.

Learn

Taking the key metrics and measurements gathered in the previous step, the Learn phase involves analyzing these results and then making one of the following decisions based on the values of the actionable metrics.

  1. Continue

    If the results of the actionable metrics are positive and validate your hypotheses, the decision to continue the project is made and then you prepare for the next Build-Measure-Learn loop.

  2. Pivot

    A pivot occurs in a project when the actionable metrics indicate the current project is not meeting the predictions of the hypotheses of the Build phase. However, the actionable metrics suggest that if the project changed focus or technology called a pivot in the Lean Startup vocabulary, that the project could serve a new or related purpose.

  3. Terminate

    The other action that occurs when either the actionable metrics suggest the project is failing, either the measurements do not support the hypotheses or other external forces, like technology change or administrative decisions, then the project will cease.