Revolutionizing JTBD Research: Evan Shore on AI
Evan Shore is Senior Director of Product Management for Walmart Health & Wellness. His team’s objective is not only to provide easy access to the products and services that customers want but also to help them make progress in their health-related Jobs to Be Done (JTBD).
For example, with an initial focus on health equity, the Health & Wellness team is developing programs to enable all people to live their healthiest lives, as well as reduce the life expectancy gap facing disadvantaged racial and ethnic populations. To do this, they start with the systemic factors — and individuals’ JTBD — impacting food and nutrition insecurity, maternal and infant mortality, and the prevalence of cardiometabolic conditions.
Recently, Evan has leveraged AI to assist his JTBD research. In January 2023, he presented just some of his results using AI at our monthly community call, “JTBD Untangled.” The audience was stunned. You can see a replay of it here.
Evan also spoke with Jim about AI and JBTD, below.
Learn how to use the JTBD framework in the JTBD Toolkit’s live training course. More details and registration here.
JIM: When and how did you get into JTBD?
EVAN: I took Clayton Christensen’s course, “Building and Sustaining Successful Enterprises,” at Harvard Business School. His frameworks, including Jobs to Be Done (JTBD), reframed my worldview, and I devoured every book he wrote. The Innovator’s Prescription gave me a blueprint for my career in health and wellness.
After business school, I decided to work on “Chapter 8: The Future of the Pharmaceutical Industry” by applying telehealth/remote monitoring to decentralize clinical trials at Medidata Solutions. I also became an Adjunct Fellow at The Christensen Institute to support their healthcare efforts.
Then I moved into Value-Based Care at Evolent Health to work on “Chapter 7: Disrupting the Reimbursement System.” At Walmart, I am now on “Chapter 4: Disrupting the Business Model of the Physician’s Practice” and “Chapter 6: Integrating to Make It Happen.”
My wife works in Medical School Admissions at the Association of American Medical Colleges, and our dinner conversations often refer to “Chapter 10: The Future of Medical Education.”
JIM: It always comes back to Clayton Christensen, doesn’t it? Jealous that you were able to have taken his course.
Can you share a specific example of using JTBD in your project work?
At Walmart, my teams use Jobs to Be Done to provide a scaffolding on which to hang features, communicate how our roadmap aligns with the big picture, and target key moments in which to engage customers.
On the Vision team, we mapped the JTBD of improving eyesight (specifically with prescription glasses), quantified the time/effort it takes to move through each step to prioritize where to focus, and established OKRs based on the improvement new features could create. We are also exploring the life contexts in which people hire and fire their glasses to understand how our shopping experience and marketing should be modified to support each situation.
A couple of weeks ago, I discovered a shortcut while helping someone on my team get up to speed on the JTBDs of our vision merchants so she could create a new application for them. Before her first discovery session with them, we asked ChatGPT simple questions like, “What are the JTBDs of merchants of prescription glasses at an omnichannel retailer?” and “What are the job steps for doing that job?”
She received feedback from the merchants that the output of those queries accurately covered 80% of their work. The remaining 20% is unique to Walmart. The result is a much faster discovery process, as the team could focus on filling in the 20% and prioritizing the opportunities versus starting from a blank page.
JIM: Amazing. Can you tell us more? How did you approach combining JTBD and AI?
EVAN: I will let ChatGPT answer that question. When I asked what JTBD it addresses for product managers, it said it can help across the product lifecycle, including “identifying customer pain points and creating solutions, generating new product ideas based on customer feedback and trends analysis, generating conversation simulations for testing product ideas, generating product descriptions and customer-facing content…”
I found ChatGPT to be particularly useful when I have the JTBD of: “When I am starting to work on a new product, help me go from zero to rough draft, so I can shortcut the learning curve and start to test and learn as soon as possible.” It dramatically accelerates onboarding to a new area and provides something to validate and refine instead of starting with a blank page.
It excels at creating human-like text and synthesizing information from across the internet. These capabilities enable it to understand how people talk about and address their JTBD, as well as to fill in the JTBD templates with answers that reflect how people might talk in an interview.
With unlimited questions, I can dive deep into lines of questions without “troubling” customers or research teams. Knowledge builds on itself within a thread for consistency and continuity.
I set up prompts to provide a structured output for different JTBDs. With significant coaching in the prompts, I was able to coax the AI to generate some fairly “thoughtful” JTBD research, product ideas and feature specs, and marketing language.
ChatGPT replicated perhaps 80% of the insights that I have seen in simple examples such as:
- “Do Milky Way and Snickers Compete?” ChatGPT identified that Milky Ways are nostalgic treats while Snickers are filling snacks. The context for eating them differs by degree of hunger, mood, and time of day. Differences in texture and specific ingredients underpin these distinct jobs. Milky Way competes with products like ice cream, while Snickers competes with products like protein bars. The AI listed tradeoffs that impact how people choose among these alternatives.
- “How Would You Beat Google?” I coached ChatGPT to map the Job Steps and associated functional, social, and emotional needs for the JTBD of “arriving to destinations on time.” The AI estimated speed, accuracy, and effort scores of current alternatives such as Google Maps. The steps and scores would need to be validated through additional research. The variables that are slowest, least accurate, and most effortful would be where to focus to beat Google Maps.
- “What is the JTBD of a Sofa?” People have trouble choosing sofas based on product attributes alone. Sofas “bring people together,” “relax and unwind,” “provide a place to sleep,” “maximize space,” and/or “make a statement.” ChatGPT generated an idea for a retailer to create a “showroom organized by these JTBD,” where “sofas would be organized and tagged” to guide people based on their JTBD. Signage might say “hang out on me,” “sink into me,” “sleep on me,” “maximize your space,” and “focal point.” In order for the right sofa to be included in each section, the AI identified the relative importance of different causal factors (comfort, durability, space constraints, aesthetic, support, convertibility, and style) for each JTBD and deconstructed how to map product attributes map to those factors.
JIM: What else have you tried? Anything more work-related?
EVAN: I also went deep into JTBDs around the Flu — specifically focusing on treating flu symptoms. ChatGPT populated your JTBD Canvas 2.0 after some modification of the instructions on your website.
Then it mapped Bob Moesta’s timeline and forces diagram from “Demand Side Sales,” identifying inputs, actions, outputs, and outcomes along the way. It became apparent that people try OTC/retail products like cough drops and humidifiers before escalating to a healthcare service or medication.
I had ChatGPT identify all the products, services, and actions people can take along this timeline so we can see the sequence of dominoes that fall as people make progress. I automated the parallel prototyping methodology in Bob Moesta’s book, “Learning to Build“ to develop contrasting concepts.
ChatGPT suggested we vary the prototypes to test different levels of personalization, reactive vs. proactive engagement, high touch vs. high tech, and type of healthcare delivery (virtual, home, or in-person). ChatGPT then wrote feature specs, marketing language for each prototype contrasting current / future states, and a starter list of search terms to use.
JIM: So do we even need to do JTBD research in the future? Will it all just be a touch of a button? Curious about your thoughts about the future of the field now that AI is in the picture.
EVAN: There should always be a place for human empathy and one-on-one connections, but computers are well-suited for synthesizing large amounts of data and automation of tasks.
We are facing a tradeoff around the JTBD of “When I am starting to work on a new product, help me go from zero to rough draft, so I can shortcut the learning curve and start to test and learn as soon as possible.” For now, doing traditional research is more accurate but slow and effortful. ChatGPT is more than 10x faster but is not as accurate and nuanced.
This tradeoff is a hallmark of disruption. ChatGPT is starting at the low end (not “good enough” by traditional measures) but enables product managers to do more work faster themselves, without relying on research teams or consultants to get a high-level overview of a new area. As ChatGPT gets better, research teams and consultants will be able to move upstream to do more validation and in-depth research.
JIM: With so much to offer already, what’s the outlook on AI and JBTD, then?
EVAN: Over time, ChatGPT could become more accurate and nuanced than researchers. To get there, there are several dimensions along which I would want to see it improve:
- Effort/Expertise Required to Program: ChatGPT needed significant coaching to understand the frameworks and how the output should look. Without automated prompts or a deep understanding of the JTBD framework, it would be difficult for product managers who are less familiar with JTBD to implement this strategy themselves and fine-tune the analysis for their domains.
- Nuance/Accuracy: ChatGPT provides ~ 80% of the common/general insights and ideas I might have otherwise had, but it misses the 20% of nuanced insights and more creative ideas. It could become more accurate by connecting it to scientific evidence / research, social media, internal company data, and other places where humans discuss their jobs and ways to address them.
- Knowledge-Building and Collaboration: Knowledge is siloed to a single thread and only for a single user. It would be wonderful to have a shared experience that facilitates collaboration and builds knowledge across my team.
- Integration with Product Workflows: It was time-consuming to format the output in Powerpoint. There is no integration with my tools to translate insights into OKRs and a roadmap. It cannot design wireframes. It can write code if prompted, but it is not integrated with a dev environment.
- N of 1 Personalization: We want to design products for specific individuals with features that can be generalized to a larger population. ChatGPT has access to population-level data and not the hyper-specific circumstances of individual customers to drive recommendations. We would need ways to gather the right signals and feed that into an AI to inform personalization.
Over the very long term, perhaps AI could evolve to even create its own applications, unprompted. We already know it can, with some degree of accuracy, “identify customer pain points, generate new product ideas based on customer feedback and trends analysis, generate conversation simulations for testing product ideas, generate product descriptions and customer-facing content…” write code, and generate descriptions of designs.
Perhaps one day, when an AI can string such functions together, we will not have any Work-Jobs to do, except enjoying the hyper-personalized experiences that the AI builds for us.
Until then, I will enjoy using my newfound superpowers.
JIM: So in the end, do you think AI is worse or better in terms of accuracy?
EVAN: On the one hand, it’s not as nuanced as human-based research. On the other, it might catch things that a limited set of interviews would miss. Complementary might be the best way to frame things.
JIM: Thanks, Evan! Incredible stuff.
— — — — — — — — — — — — — — — — — — — — — —
For more, see Evan’s presentation on JTBD Untangled
Here are Evan’s slides from the presentation:
Learn how to use the JTBD framework in the JTBD Toolkit’s live training course. More details and registration here.
