Before AI vs. Now: PM Communication With Engineers
PMs need to define model boundaries, reliability targets, and human approval gates, and treat AI outputs as drafts that still need review.
For years, the handoff between a PM and engineering followed a familiar script. Write the spec, agree on a delivery date, and ship once QA signs off. If the code did what the spec said, the job was done.
AI features break that script. A model can give different answers to the same question, and some of those answers will sound confident while being wrong. A green test suite proves the code runs. It can’t tell you whether next week’s answers will be good enough for your customers.
That changes what PMs and engineers need to talk about. “What will it do, and when will it ship?” is no longer enough. Now the team also has to answer harder questions: What is the model allowed to touch? How good is good enough? How will you measure it? Who decides it’s safe to launch?
This guide walks through how four everyday conversations have changed (scope, requirements, testing, and release) and what to agree on with your engineers in each one.
The short version
- Scope: Rules still matter. Now you also define the model’s limits, the data it can use, and what happens when it can’t help.
- Requirements: Add quality targets, failure limits, response-time limits, and cost caps to the spec.
- Testing: Keep your software tests, and add evaluations that score answer quality and customer risk.
- Release: Tie launch, rollout, and rollback to test results, with a named owner for each call.
- Team skills: Learn to read evaluation scores critically, double-check AI-written work, and turn customer complaints into new test cases.
A good first step: open every AI feature discussion with launch criteria alongside the delivery date. Agree on what the model can access, which actions stay in regular code, and when a person needs to step in. Then build a set of 50–100 test cases that someone reviews by hand. Treat that set as your starting baseline. Passing it won’t prove every future answer is right.
Before AI vs. now at a glance
Discussion | Fixed-rule features | Model-based features |
|---|---|---|
Scope | Define allowed behavior | Also define model and data boundaries |
Requirements | Specify expected results | Also set measured quality targets |
Testing | Verify logic and workflows | Also score varied outputs |
Release | Check tests and launch conditions | Also check model quality and failure limits |
The thread running through all of it: AI adds new checks, and people still own the outcome. You still need clear requirements, verified code, and someone accountable for the decision to ship.
Video: From MVP to GitHub.com: An AI Story of Engineering & Product Collaboration, by Anjuan Simmons
1. Scope: from fixed rules to model boundaries
With a traditional feature, scope meant listing the conditions and the actions the system could take. The code followed explicit branches, so if the rules were well defined, you knew what would happen.
A model-based feature also depends on prompts, retrieval sources, tool access, and whatever customers type in. That’s a lot of moving parts, and some rules should stay in code, outside the model. Access checks and tool permissions are the clearest examples.
This shift goes deeper than a new type of feature. It changes what engineering has to constrain, and it’s part of why people are asking whether product managers will be replaced by AI or whether the role will evolve. Here’s how scope compares on each side:
Scope question | Fixed-rule features | Model-based features |
|---|---|---|
Scope | Explicit conditions and allowed actions | Model capabilities, prompts, retrieval sources, and permitted tools |
Expected behavior | Defined outputs for specified inputs and system state | Variable outputs, including plausible but wrong answers |
Dependencies | App logic, APIs, databases | Those same components plus model quality, data quality, latency, and cost |
Customer risks | Incorrect rules, calculations, or access decisions | Those same risks plus invented claims, biased answers, and changing model behavior |
Write down data access, limits, and fallbacks
Before anyone builds, agree on five things: the model’s job, the sources it can use, the data it can access, the requests it should turn away, and who takes over when it hands off. Put these boundaries in the scope document as well as the prompt, so they get reviewed like any other requirement.
One more thing to watch: connecting a tool to the model doesn’t mean the model should use every permission that tool offers. Decide which actions it can take, one by one.
Example: Picture a support assistant for a small business. It can look up and explain the refund policy, but it can’t issue refunds. Refund requests go straight to a person. If the policy doesn’t cover a customer’s question, the assistant hands off to a teammate so it never makes up an exception.
Connect uncertain answers to customer risk
Test results only describe the cases you tested. Future answers can still be wrong, and a confident wrong answer has real consequences. It can change a customer’s purchase or refund decision, cost money, damage trust, and create extra support work.
So when you discuss failures with engineers, talk about the impact. “This answer could cost a customer $200” gets a very different response than “this answer is incorrect.”
Set limits based on potential harm. Regular code can make expensive mistakes too. Pricing changes, legal disclosures, and account deletions need stricter controls and clear approval steps, whether a model or regular code triggers them. Spell out when the system can act on its own, when a reviewer has to approve, and when it should decline.
Once scope is settled, the next conversation is about how reliable the system has to be.
2. Requirements: from defined behavior to measured reliability
A traditional spec describes what should happen for a given input. For an AI feature, the spec also has to say how reliable the system needs to be, how much variation is acceptable, and what evidence the team needs before approving it. [5][6]
Discussion | Fixed-rule features | Model-based features |
|---|---|---|
Requirements | Specify behavior for known inputs and conditions | Add quality targets, acceptable variation, and failure limits |
Feasibility | Ask whether the logic and integrations can be built as specified | Ask whether data and model capacity can meet the target |
Acceptance | Check expected results and operating requirements | Measure against a reviewed benchmark set |
Diagnosis | Investigate code, data, and infrastructure | Separate model limits from retrieval, prompt, or orchestration failures |
Ownership | Agree on priorities, technical delivery, and risk | Assign evaluation ownership, escalation, and risk acceptance |
Set reliability targets and failure limits
“Accurate enough” isn’t something anyone can test. Pick one primary metric instead. Two common choices:
- Precision: how often the cases the system flags are actually correct.
- Recall: how many of the relevant cases the system catches.
Then write down the details engineers will need: the primary metric, how many false positives you can tolerate, the failure threshold, the requests the system must refuse, and what triggers a human review. Add latency limits (including the 95th percentile), cost limits, what you’ll monitor, and who responds when something breaks.
To start, 50 to 100 cases that someone has reviewed by hand make a solid baseline. [2][4][6] These targets become the checklist your test plan has to prove.
Figure out whether the model or the system failed
When something goes wrong, PMs and engineers each have a job. PMs explain what the customer was trying to do and what happened as a result. Engineers trace the path through retrieval, permissions, tools, prompts, orchestration, and the UI. Together, that tells you whether you’re dealing with a limit of the model or a fault in the system around it, and which fix is worth the effort. [6]
Review AI-written requirements and code
Treat AI-written requirements, code, and tests as drafts that still need review. PMs should check claims against customer research and interview transcripts. Engineers should check the implementation, security, and test coverage.
AI is great at poking holes, so ask it which assumptions are missing. Just don’t let AI-written tests be the only tests. People still own priorities, risk decisions, and release approval, and Security, Privacy, and Legal review the constraints in their areas.
PMs need enough technical knowledge to question the evidence. A machine learning background isn’t required. [1][2][4][6] Once requirements are reviewed, they become the criteria your tests measure against.
3. Testing: from expected outputs to model evaluations
Classic tests check for an exact result: given this input, expect that output. Model evaluations measure quality across many cases, because the same question can have several good answers.
Evaluations sit on top of your standard software tests. Core app logic, such as pricing or account deletion, still needs regular tests. [3][4]
Discussion | Fixed-rule testing | Model evaluations |
|---|---|---|
Test scope | Check code paths, integrations, and complete workflows | Assess output quality across representative tasks and manually reviewed cases |
Required evidence | Expected results, test results, and defect reports | Reference cases, scored outputs, and a pass threshold |
Quality criteria | Correct behavior plus security and performance requirements | Pass/fail thresholds and risk-specific failure limits |
Regression tracking | Rerun automated suites after changes | Compare versioned evaluations after model, prompt, or data changes; monitor drift |
Agree on how you’ll score results before building
Start by listing the customer journeys and risks that matter most. Engineers can help turn those into evaluations you can rerun after every change. Include known edge cases and the customer segments where the system tends to struggle. Then agree on how well each journey is covered, how outputs get scored, and the minimum score to pass. [1][6]
Score answers on intent. Two answers can use different words and both be right. Your scoring guide should spell out what makes an answer correct, incomplete, or unacceptable.
Keep everyday quality issues separate from high-impact failures. An awkward summary and deleting an account without permission shouldn’t be judged by the same rule, so give high-risk actions their own pass criteria. [2]
Look for gaps before trusting the score
A single score is only one piece of evidence. Before accepting a result, ask:
- Which journeys have too few test cases?
- Which expected answers are unclear?
- Where do reviewers disagree?
Clear up any confusing scoring rules before using the results to judge a change. Automated scoring helps with volume, but someone should still read the outliers and every high-risk output by hand. Averages can hide the failures that matter most. [2][3][4]
Over time, track drift, high-risk regressions, and latency spikes next to your quality scores. Compare results by evaluation version, journey, and risk category. For every run, record the model, prompt, dataset, and scoring guide used. That way, an overall improvement can’t quietly hide a serious regression. Recurring failures and coverage gaps show you what to fix or test further before release. [5][6]
4. Release: from delivery dates to quality gates
Once the evaluations are in, the question becomes whether the feature is safe to ship. A delivery estimate and launch approval are separate decisions, and AI releases need launch gates tied to measured quality and risk. [2][6]
Release decision | Fixed-logic products | Model-based products |
|---|---|---|
Schedules | Delivery dates and milestones | Target windows and checkpoints |
Launch gates | Automated tests, compliance, and QA pass | Pass the agreed release bar |
Rollout | Staged rollout based on traffic or load | Gated by performance thresholds, model drift, and measured uncertainty |
Rollback | Bugs, outages, or logic failures | Safety regressions, wrong answers, or model drift |
Share a target window and let evidence set the date
When stakeholders ask when it will ship, share a target window, any open dependencies, and the next checkpoint. Keep external launch commitments marked as draft until the right owners sign off.
For high-risk behavior, a designated owner should approve the release alongside the PM. If reliability falls short, narrow what the feature supports or fall back to regular code, and keep the safeguards in place. Pricing, legal disclosures, and account deletion need explicit human approval. [2][3]
Plan rollout, pause, and rollback rules up front
Before customers see the feature, agree on how you’ll start small. Limited traffic works, and so does shadow testing, where the model runs in the background without customers seeing its answers.
Write down measurable limits for regressions, safety failures, and fallback behavior. If latency or cost goes over the agreed cap, rollout stops. Also decide how long you’ll watch the results and what evidence you need before expanding to more users.
Finally, name who can pause the rollout, turn the feature off, or roll back to the last approved version. Dangerous failures need an immediate pause and escalation, even if the next review is days away. [2][6]
Turn customer feedback into new test cases
After launch, review production logs (with privacy protections in place) alongside support escalations. Usage numbers alone won’t tell you if it’s working, so judge quality through human review or reliable labels.
Give each issue an owner. Add every confirmed failure to your evaluation set, and rerun the affected workflow before expanding to more users. Those failures then become part of the next release gate. [2][6]
Putting it all together
Before AI, most PM–engineer conversations centered on features, fixed rules, and delivery dates. Now they center on model boundaries, measured reliability, and whether the evidence supports a launch.
The fastest way to get comfortable is practice. Read evaluation results with your engineers, and connect each system limit to the customer risk it creates. Start with a small set of hand-reviewed test cases, then work with engineers to find what those cases miss. [2][3][4]
Questions to bring to your next planning meeting
- Does this problem actually need a model?
- Which behavior has to stay in regular code?
- Which evaluation best reflects what customers value?
- What has changed since the last release?
Use the answers, along with what you know about model quality, operational risk, and customer impact, to decide whether to expand, pause, or roll back.
FAQs
How should PMs choose between AI and fixed rules?
Use fixed rules when you need predictable logic, accuracy, and compliance, especially in high-stakes, safety-critical, or regulated processes. Keep strict approval gates that a person controls.
Use AI to pull data together, model scenarios, or run quick experiments. Treat its output as a starting point for a human decision. Before handing over a task, plan how a person will check the work, for example by comparing results against 50 to 100 reference cases.
How do we set realistic AI quality thresholds?
Think in percentages. Before handoff, agree on evaluation criteria, success metrics, and requirements like precision or latency. Spell out how often the feature needs to work, for which customer segments, and under what conditions.
Keep a golden set of 50 to 100 test cases for manual checks, outlier review, and spotting failure patterns. Set clear stopping points and checkpoints that need human approval. People stay accountable for outputs that affect safety, privacy, or trust.
How do we resolve disagreements about launch readiness?
Make the decision together, but name one owner for the final call. Base choices on business value, customer needs, and feasibility. Bring stakeholders in early, write down the criteria everyone agrees on, and use a prioritization framework like RICE or MoSCoW so everyone can see how decisions get made.
Review progress, dependencies, and delivery dates regularly, and adjust the release plan based on measurable evidence.
Sources
- Changing roles: how teams work with AI agents (Working Surface)
- Product Managers Shouldn’t Use AI for These 10 Things (Product Management Society)
- Future of Product Management Beyond 2026 (Product Management Society)
- What Can Product Managers Expect in 2027? How to Stay Ahead (Product Management Society)
- AI for Product Managers: 2026 Guide to AI-Native Strategy (Stoa)
- What Does an AI Product Manager Actually Do (Compared to a Regular PM)? (TechCerted)
If you’re finding this blog valuable, consider sharing it with friends, or subscribing if you aren’t already. Also, consider coming to one of our Meetups and following us on LinkedIn ✨ And check out our official website.
Connect with the founder on LinkedIn. 🚀
About the Product Management Society
The Product Management Society is an international community for product managers, founders, designers, and career-switchers, with 2,400+ members across active chapters in Lisbon, Berlin, Frankfurt, and Mexico City. The community runs more than 50 in-person meetups per year, a Slack network, an invite-only WhatsApp group, a blog, and a growing suite of free tools for product leaders. More information is available at www.productmanagementsociety.com.
About Gabriela Naumnik
Gabriela Naumnik is an AI product leader and the founder of the Product Management Society. A Staff Product Manager working at the intersection of AI and enterprise product, she focuses on AI-powered platforms serving Fortune 500 companies. She is a regular speaker at product conferences, publishes on product management at the Product Management Society's blog, and has built the PM Society into one of the most influential product communities in Europe and Latin America. She holds a B.S. from NYU/NYU Shanghai and an M.S. from Columbia University. More information is available at gabriela-naumnik.com.
