Quality Control (What does “good quality” mean for this summariser?) - 1. Zero Hallucinations (Grounding): We need strict guardrails on Grounding. If Sid didn't explicitly say it in the transcript, it doesn't exist in the summary. The model absolutely cannot invent new frameworks or 'best practices' just to fill space. We want a faithful mirror of the session, not a creative writing exercise so we might have to set a high accuracy score to start. - 2. Signal Over Noise (Data Strategy): The model needs to be smart enough to separate the 'signal' from the 'noise.' It should be able to ignore the casual chit-chat—like the weather or Zoom logistics—and zero in on the core mental models like the Deterministic vs. Probabilistic shift. If it captures the fluff but misses the framework, the data strategy failed. - 3. Actionable Intelligence (The "So What?"): A good summary doesn't just parrot back definitions; it captures the practical implications. We need to make sure the output connects the dots for the user—specifically, how these concepts change their day-to-day work in scoping, measuring, or designing features. If someone reads this and doesn't know what to do differently tomorrow, the tool missed the mark. - 4. User-Centric Formatting (Solution Design): Let’s treat the summary format as part of our Solution Design. We need to optimize for scannability—bold headers, clean bullets, and tables for comparisons. The user experience should allow a student to refresh their memory on the entire session in under two minutes. If it's not digestible, it's not useful. --------------------- Measuring Success: Beyond the Thumbs Up - 1. Track "Edit Distance" (Implicit Feedback): It shouldn't be enough to look at whether the user kept the summary; we should look at how much they changed it. If a user saves the summary without touching it, the model’s 'best guess' was a win. But if they have to rewrite half the text or delete whole sections, our quality is missing the mark. While supporting adjustable output, we want low edit rates—that means we actually saved them time. - 2. Look for the "Content lifecycle" (Action Taken): The strongest signal isn't a survey response; it's what users do with the content. If a student actually copies a code snippet or a comparison table out of the chat and into their notes, that’s 'super feedback.' It proves the content was accurate and useful enough to live outside the tool." - 3. The "Recovery" Metric (User Agency): We all hate hitting a dead end. We could track how effectively users can fix a bad result. If they use a 'Regenerate' or 'Focus on the 7 Questions' button and then accept the output, that’s a success story. It means our recovery tools work, allowing the user to steer the AI back on track without rage quitting. - 4. Benchmark Against a "Golden Set" (Evals). We need a reality check. We could compare the AI's output against a 'Golden Summary'—a manual version written by an expert or a pool of higher performing models. We can rate the AI on a 1–10 scale based on how well it captures the same key insights the expert or pool of models did. This gives us a concrete baseline to adjust our prompts and measure if we're actually getting better.

@Sid Arora @Sid Arora Thanks Sid, this is really useful 🙌 1. On actionable intelligence, I see the tension you are pointing at: we want the summary to be useful, but the moment we ask for a “so what,” we risk pushing the model into interpretation instead of reflection which is probably the hardest constraint to get right. We may need to redefine “actionable” as surfacing implications already stated, not generating new ones. This can prove a fine tuning challenge for the model to grasp the instructions subtleties. Dunno how to solve it 🤷‍♂️ 2. On success metrics, your questions highlight a real risk of designing things that sound good but are hard to monitor or scale. We could pressure-test each metric for objectivity and automation early. For the Golden Set, I was thinking a small, stable expert baseline with clearly defined key points, so as to compare coverage and grounding rather than subjective writing quality. Again not sure how to 🤔 Realising now I went full “classic PM mode” here: I set the what and the why and expected R&D to figure out the how. But in an AI PM world, the data and evals too are now on US 😅 It’s a whole new game for me 🤣

@Bahar Malekshahi that’s an amazing journey you got yourself into. You’ll see there are quite many similarities when building a client or a product case 😉 Would love to hear more about the law you practiced? All the best

Hi everyone, I am Manu and currently living nearby Stavanger, Norway. I am a product manager with experience in delivering platforms for Edtech and EHS industries. I am really excited to be part of the program and look forward to learn from you all.

My biggest challenge is that I have been sending hundreds of applications but don’t get any replies despite having what I was told is a strong CV. I don’t get it, I never had any problems before but now I don’t even get a screening call… Desperate to understand what needs to improve and how…