Impact of AI on Product Management

Redefining PM in the age of advanced AI models

My organization has been pumped about the new advanced LLMs since the last few months (and with reason!), and that energy has permeated many of our product teams, with leadership encouraging all of us to think out of the box, take the time to experiment, be willing to absorb risk for a while, and ride on this momentum that is diffusing through the entire software (& beyond) industry.

My team being in the Enterprise Search space does provide a lot of specific implications of how my team applies, develops and rolls out these advanced technologies to the benefit of search. This generational shift in search has been forcing function for product managers (& other teams) to embrace the new/updated ways of doing things, become more risk-tolerant, enhance the way we communicate and set expectations with our leadership team, and be more thoughtful around how we create new experiences for our users.

From my experience, the impact of AI on product management can fall into two themes — (a) the way we solve problems using AI & (b) the way we develop AI products. So here goes —

#1 Culture of Learning

One new theme that has emerged with the energy around AI across the org has been the hunger to learn more — about new products and services being developed/launched in the industry, about the basics of advanced ML models (by playing in a toolbox), by taking a training course or by forming ad hoc teams that can generate new applications of AI for a hackathon/ideathon. On the technical side, many others have started experimenting and advancing innovative algorithms, and publishing research papers about these advances/applications.

#2 Ethical and Responsible development

Applying LLMs and other advanced ML models to use-cases needs guidelines, supervision and controlled boundaries to avoid introducing biases in the name of innovation. Being mindful about ethical ways of creating new product experiences is not really an option, but a condition of doing business. Few ways of establishing responsible AI practices might include —

• thoroughly understanding and identifying ethical implications of AI-infused products and services across various dimensions — fairness, bias, transparency, explainability etc.
• using datasets that encompasses diverse perspectives, plus continuously enhancing the ML model to remove biases towards user groups
• being transparent about the assumptions and logic behind AI-powered user recommendations
• upholding high privacy and security standards around user data
• regularly conducting AI risk assessments and audits
• standardizing governance practices across various functions in the organization which might help influence the choices developers make whilst building ML models for the better or help other stakeholders make the right choices whilst implementing/marketing/rolling out these products.

#3 Enhanced Storytelling with AI products

As chatGPT, Bard, Dall-E and other highly publicized applications of AI have demonstrated, a product manager needs to be able to tell the story of the product’s applications, edge-cases, risks and user impact that lets you take control of the story, instead of the other way around.

Internally, you need to be able to ideate, convince, create and communicate value proposition of solving user challenges with this advanced tech, thereby quelling any objections/concerns of nay-sayers with a crisp vision, tangible user impact and long-term user & business value.

Externally, you need to be able to illustrate the user impact as well as the associated risks of using the early iterations of the product with disclaimers and transparent messaging within the user experience, in addition to the dimensions you established internally. How you control the narrative through storytelling, and timing your story right might be more relevant in the wider market as more and more companies are out getting their AI products in the hands of users (& major publications).

#4 Risk Management of AI products

One of the immediate implications of using AI in the world of product management has been risk management — internally & externally. Internally, you’d need to balance your product development budget across low-mid ROI, short term projects (eg. your ongoing features) vs. high ROI, long term projects (eg. AI-powered products). There’s always the risk of long term (business & customer lifetime) value of an AI-powered product not being realized/recuperated, which you can balance with enough /appropriate investment into your “cash-cows”.

On the other hand, externally, you need to handle the risk of your much-hyped AI product/experience not being appreciated, or worse ridiculed and a whole lot of scenarios in between (especially if the experiences are experimental and not exactly ready for public consumption on a large scale). AI products, I find, are quick to be adopted, but can also be dropped like hot cakes in the face of negative publicity or less-than-popular uninviting scenarios. Product Managers need to be better prepared for such products once they hit the limelight and have a risk-mitigation strategy in case things don’t go according to plan.

#5 Managing expectations around AI development / go-to market timelines

Communicating up (the leadership chain) is always a work-in-progress which product managers are more or less accustomed to.

1. Product development & GTM milestones & timelines — with AI products, it becomes even more imperative to showcase consumable data to help set expectations with leadership regarding ML model development milestones and go-to-market (GTM) timelines. Leveraging competitive baseline and market data (or past data of similar products in your org) can help reinforce the point with the key decision makers.
2. Edge cases — you might also want to over-communicate the potential risks & challenges that might arise with as-yet-unknown edge-cases after productizing your AI-powered experience at launch. And have a risk-mitigation plan to boot.
3. ROI impact — AI powered products might get energetic adoption in the first try but take a long time to find retention (or for ROI to breakeven). If this is the case with your products, this point needs to be hammered out and aligned on with your stakeholders and leadership, lest everyone get carried away in the excitement to get on the AI bandwagon. The risk of not achieving near-time (quarterly/semester) ROI targets needs to be over-communicated with leaders, and PMs should actively work to help leaders assimilate the dictum of short-term pains to hopefully achieve long term gains with such products.

#6 Understanding AI Tradeoffs across attributes

Whilst working with data scientists and AI models, product managers need to acutely understand the tradeoffs between accuracy (precision) vs. performance vs. cost vs. size vs. ethics (explainability etc.) and other attributes that might affect how well your AI models function for the user experience.

Depending on the usecase— eg. covid-diagnosis (where the slide might tip towards accuracy) vs. search & traffic-routing (performance & speed are more critical among other factors such as relevance) vs. financial advice (where explainability of results is expected) vs. enterprise workflows (where size of the model and cost of processing humongous amounts of data might be key) — you and your data engineers/scientists might need to decide how and when you want to slide the scale towards any of these attributes (& explain the reasoning behind your decision).

Another best practice might be to think about these considerations right from the start during ideation stage and engage your governance bodies (risk, legal, compliance teams) early in the process to elicit diverse set of feedback to help inform your product decisions. And definitely iterate on it as new information becomes available.

#7 Solving problems differently using AI

As product managers, we want to use new frameworks and optimize the way we solve problems, however with AI/ML based challenges, you still need to continue thinking about customer value (saving time, increasing efficiency, enhancing experience — functional, social, financial, psychological) & business value (reducing costs, generating new revenue streams, increasing existing revenue streams or improving operating efficiencies).

So rather than thinking about building an AI-powered feature, you need to think about how an AI model can potentially help users fast-track time-to-value or find enhanced value in their user experience — help get the job done faster, more efficiently, unlock new delightful solutions to problems they didn’t even know existed or help get a job done with less mental stress.

Other thoughts -

1. Two types of AI companies — (a) building ML from ground up, and (b) leveraging /extending AI to build new applications. The level of technical skills required as a product manager in both these scenarios vary.
2. Stage of the company also impacts how PMs operate — depending on a startup w/ limited constrained funding vs. a behemoth corporate with strong resourcing, product management differs as well with the amount of risk guardrails PMs can operate in.
3. Confidence in AI is low; but returns are many-fold — companies playing the long game (multi-year) without expecting short-term returns might just win the AI race. Hence PMs need to think long-term and convince leadership about the same.
4. Risk/Effort — Most outside unexpected wins are AI-driven wins, but it’s time consuming and risky.
5. Forest and trees — great PMs will look at the broader picture, and the details as well (macro and micro) ; they can seamlessly connect the dots between forests and trees.
6. No standard playbook — as PMs, play to your strengths since companies need people with diverse talents and who can represent differing views in the AI race.
7. When NOT to use AI and ML — be really smart about when to invest and convince how to invest in AI/ML.
8. Eliminating bias in data — monitoring data and results proactively; accounting for existing users having bias in the first place; don’t take anything for granted; ask questions, be proactive.
9. Validating accuracy of ML recommendations — thinking about all ways someone can ruin the results for you, test before you release to get more confident, getting feedback from users (thumbs up, down); set up beta customers and release it to them for iteration.

Overall, a product manager in the AI space still needs to go through the traditional product management lifecycle — identifying a user problem space, building the what/why/who/how around it, working with engineering and design to find the least resistant experience for the user and more. However, whilst working with AI, you as a PM shall also help your data science and engineering teams determine the right AI model/s for the challenge, help find and train domain-specific data, help iterate on the output of the model/s, smoothen out the edge cases, hyper-test and validate the experience and just be prepared to encounter and hack complexities that befit such an innovative ecosystem.

Hopefully this article demystifies your tryst with AI as a Product Manager. Leave a comment if you’ve any other insight that I’ve missed here!