Ori Eval is live on OpenRouter. If you already route models through OR, this is the systematic version of "should we switch." Public benches and Twitter recs measure a fixed task set. They tell you how a model behaved on someone else's prompts, someone else's tools, someone else's budget. With 500+ models on OpenRouter, "just switch to the new one" is expensive guesswork. Same leak as picking a CMS because a roundup said it was number one. OpenRouter shipped Ori Eval for that. It is not a leaderboard. It is a way to score models on your app. What it actually does: - Scans the repo for every place a model gets called - Asks what you care about: accuracy, latency, cost, tool-call correctness, whatever you name - Writes a *.eval.ts file. You do not have to be an eval person - Runs your agent against candidate models through OpenRouter, so the comparison can cross labs - Pins the harness and the model for the run. If the score moves, it is the model, not the furniture shifting - Checks whether tools were called or not, and grades open-ended answers with an LLM judge - Can sit in CI so a worse model does not ship. Re-run when a new one drops The sample they show is a table: catch rate, p50 latency, dollars per PR, pass/fail against your criteria. Then a recommendation, plus a cheaper "value" pick if volume grows. I have not run this on my agents yet. Treating it as garage until someone in here drops a real table. Cost is real. It calls live models. OpenRouter's own skill says 10 to 30 minutes and it can spend more than the credit on the key. First pass lands in a throwaway workspace. The eval is not yours until you decide to keep the file. Try it Tell your coding agent: run curl -fsSL https://openrouter.ai/skills/spawn-ori-eval and follow the instructions in its output to get started Already on the OpenRouter MCP? /spawn-ori-eval By hand: curl -fsSL https://openrouter.ai/labs/ori/install.sh | bash

This is happening to me all the time now, and either I wasn't catching it before, or something changed. I do realize that this might be operator error, which is why I'm asking for help. Have you been able to wrangle the "hallucinations" which let's just call them what they are, fiction and/or lies. One request: please don't blindly copy/paste outputs if you run this post with AI. I'm not saying don't do that, just politely asking if you could give it a read and if the output makes sense to you, drop it in the comments. Here are two examples: --------------- Example #1 --------------- Had Claude draft this up, and in the process it created a rule for it without me asking. What's frustrating about this is that now I have to read through another markdown file to ensure accuracy and consistency to how I do things, and I still have doubt that it'll even honor the rule it wrote for itself. The admission: 1. What I did. Credited three voice rules to NAME, and wrote that he flagged them in a comment on your Voiceprint LinkedIn post. The comment doesn't exist. I checked his comments feed, his posts, Gmail, Slack, and Skool. The earliest trace is my own /session capture on 8/29, with nothing upstream of it. I can't reconstruct where the name came from, and I didn't invent a story for how I invented it. 2. It happened twice. Yesterday I told you the attribution had reached the public repo. It hadn't. I repeated the session log's "added all three to..." line without opening the file. That second one is the more useful failure, because it shows the actual mechanism: an unsourced sentence in a workspace file gets read as a source by whoever comes next, including me. 3. Six guardrails were loaded and all six missed. constraints.md bans invented attributions by name. Its final scan says every claim traces to a source. Shared rule 0 forces that file to load every EOD run. Root map rule 5 says verify the live system. Digirol bans "I think" and demands a real search before calling a contact new.

Your operating system is one of those quiet decisions that shapes every tool after it. I'm curious what this room actually runs. 1. What OS are you on day to day? 2. Why that one over the alternatives? 3. How long have you been on it? I'll go first: I've been a die hard Mac girl my entire life from a ui and creative perspective they always gave me the best experience. However now with the age of Ai I'm experimenting with Omarchy Linux, It is a Distro that has a skill for the OS... So yes that does mean we are in the Vibe-Code your own OS modification era. Just remember... With great power... Comes great hallucinations... No wrong answers. Dual boot counts. Distro names welcome. The "I switched last month because X" stories are the useful ones. OMARCHY: https://omarchy.org/ //A<3

Simon Sinek's Golden Circle asks why you do what you do. You probably already have an answer for your own system. It's in an ICM somewhere. A values doc, a mission file, the folder that says what it's for. A COO I work with asked a different question on a call last month. He wasn't asking why his team ran performance reviews. He was asking what would happen if they stopped. Nobody in the room could answer. Those are two different tests. Sinek's question checks whether you can articulate a belief. The COO's question checks whether that belief is load-bearing. You can pass the first one in a conference room with nobody watching. You can only pass the second one by pointing at what actually breaks. That COO had a month of build-versus-buy behind him. Platforms compared, discovery calls stacked up. Then he asked one question, and the room went quiet. Nobody had ever tested whether their reasons for running reviews would hold up under questioning. Take whatever you've documented about why a process exists and ask what specifically breaks if you stop doing it. Name a consequence you can check: a number that moves, a decision someone makes differently, a person who acts differently on Monday. If you can name it, you have a claim to test. Then look for evidence that the process produces that result. If you can't, you haven't established what the process does in practice. @Jake Van Clief's ratio describes database queries, logic, and AI calls. I apply it as an analogy to people and process. The ratio is his. A client forced the load-bearing test on me, not the other way around. If you want to run it against your own 60%, I'm happy to talk it through. The full article, The Tool Is Ten Percent, is in David's classroom.

Built a guided way to turn work you know by heart into something you and your AI can actually use. generates a HUGE packet for you. Workflow Audit Package - The Vault · Clief Notes This is inspired by something I do in my Corporate workshops and also wanting to build something sustainable to help out the Premium and VIP members. Start with a copy-and-paste interview prompt: your AI asks one question at a time about what comes in, what you do, where things get messy, what goes out, and the decisions you want to keep. It works for business owners, consultants, subject matter experts, creators, or anyone with a recurring task they want to understand and improve. You can answer in rough words or by voice; no technical background is needed. Once you have checked the interview, give it to the included builder skill with the blank HTML template. Your AI audits the answers and creates a detailed, editable workflow package with a map of your work, evidence for where improvement could matter, open questions, practical prompts, and a recommended folder structure based on ICM: ordinary folders and readable files that give the AI the right context at each step. Open the package in your browser, correct anything it got wrong, download your copy, and use the first-test plan on a completed example you can check. Everything you need to start is in the kit; the start guide explains both copy-and-paste and file-upload options. Workflow Audit Package - The Vault · Clief Notes