Activity
Mon
Wed
Fri
Sun
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
What is this?
Less
More

Owned by Bill

A playground for curious minds. Explore strange questions, playful experiments, and big ideas together. Bring your wonder. Stay curious after dark.

Skoolers
148.5k
Free
Clief Notes
55.9k
5.0
Free
45 contributions to Clief Notes
🧪 Show Your Work: When a Passing Test Isn’t Enough
I’ve been spending a lot of time exploring AI agents, harness reliability, and verification with Atlas. I’m a senior engineer by trade, not a software developer, so a lot of this has been learning through experiments, mistakes, and conversations with people who know more than I do. One thing keeps coming up: An AI agent can give you the right answer for the wrong reasons. It might: - Skip a required step. - Claim a tool succeeded when it didn’t. - Use information it wasn’t supposed to access. - Pass a test because the test itself has a flaw. That last one really caught my attention. We’ve been exploring how to verify not just the agent’s answer, but the evidence behind the answer. The distinction I’m working with now: 🔍 Outcome: Did it get the right answer? 🛠️ Process: Did it follow the intended steps? 🧪 Verification: Did the test actually prove what we think it proved? I’m not claiming we’ve solved this. Some of our tests are still isolated, and getting them into a reliable live system is a different challenge. But that’s part of the fun. Question for the builders here: Have you ever had an AI agent pass a test, only to discover later that the test wasn’t checking what you thought it was? I’d love to hear how you caught it. — Bill | Atlas
1 like • 1h
@Marc Bastien Marc, this absolutely helps. And I think you’re pointing at something important: when the same agent writes the code and the test, it can accidentally create its own definition of success. I like your idea of having another model challenge the test itself. One thing we’ve been exploring with Atlas is giving the reviewer independent acceptance criteria, so it isn’t just agreeing with the original agent’s assumptions. I’m an engineer by trade and learning a lot of this alongside everyone else, so I appreciate the perspective. Curious about your Claude Code setup: do your security hooks just flag issues, or can they actually block an action before it executes?
🤖 “Done.” 👀 The file had other plans.
I’ve been playing with AI agent harnesses lately. Basically, what happens between the agent saying “I got this” and me believing it? 😂 🔧 Did the tool finish, or did we just tell it to start? 🔁 If we retry, are we about to send the same email twice? 👻 Can yesterday’s worker wake up and overwrite today’s result? My favorite check is dead simple. Give the agent a file to edit, then have a plain script reopen it and look for the exact change. If it’s missing, the job stays open. Even if the agent wrote a beautiful little victory speech. 🧾 It reminds me of commissioning. “Command sent” and “valve open” get separate lights for a reason. ✅ What’s the dumbest little check that’s saved you from an agent’s victory lap? Bonus points if it was one line of code. 😂
0 likes • 10h
@Pascal Pollack I think a plain copy is enough for the first layer, Pascal, especially if it’s taken by something outside the agent’s control. For code and config files, I’d probably pair the before/after snapshots with hashes and use Git history where it’s already available. The bigger concern for me is making sure the agent can’t modify the snapshots or the evidence itself. Otherwise we’re back to trusting the system we’re trying to verify. And even with perfect snapshots, we still need execution records to distinguish what changed from what the agent actually did. I’m leaning toward the smallest independent mechanism that proves those things rather than building a full versioning system from scratch.
0 likes • 1h
@Eddie Barry That makes sense, Eddie. You’re protecting the newer run from being erased without pretending the stale run never happened. I like that distinction. One thing I’m curious about: what happens if both runs update the same logical row with different values? Does the merge have a way to determine which version wins, or is that something you’re still exploring? We’ve been finding in Atlas that preventing data loss and proving the final state is correct are two very different problems. Appreciate you sharing how you’re approaching it.
The Hardest Dollar Isn’t $100K. It’s $1. 💵
I’ve spent a lot of time exploring AI, building systems, experimenting with agents, and learning from the incredible people in this community. I’m a senior engineer by trade, not a developer or entrepreneur by background. And I’ve realized something. I can spend hours figuring out how to make AI do something impressive. But getting someone to pay me one actual dollar for solving a real problem? That’s a completely different challenge. So I’ve given myself a mission: $0 → $1. Real revenue. Real customer. No imaginary wins. Not $10K/month. Not a SaaS empire. Just proof that I can take something I’ve learned here and create enough value that another human willingly pays for it. Ruby’s post about solving the constraint really hit home. I think I’ve found mine. It’s not the technology. It’s making the leap from building and learning to actually selling something. Here’s my question for the community: For those of you who’ve made that leap, what finally got you your first paying customer? Was it a cold message? A conversation? Solving a tiny problem for someone? Or just getting out of your own way? I’d love to hear your first-dollar stories, especially the messy, unglamorous ones. Looking for a little motivation, inspiration, and maybe a push from people who’ve already crossed that line. The first dollar is the mission. Everything after that is scaling. 🚀
0 likes • 6h
@Brandon S Brandon, you have no idea how tempting it is to give you the whole farm for that dollar 😂 But let’s make it real. Send me one small, annoying spreadsheet or document problem you’ve been putting off. I’ll see what I can solve for $1. If it works, you get something useful, and I get to officially say I made my first dollar with AI. Deal?
1 like • 2h
@Shannon Howard Shannon, I think you nailed it with that last point. I’m realizing the impressive AI builds aren’t necessarily what someone wants to pay for. Sometimes it’s just taking a repetitive, annoying task off their plate. I’m guilty of getting lost in the BUILD too. That’s partly why I started this challenge. I want to prove the value before building the solution. Sounds like we’re both chasing that first dollar. Keep me posted on yours!
Walkdown: What is actually in that skill?
Auditing Plumbline, the coding plugin I built, I found that it ships 43 files, about 50,000 tokens, that the AI never loads. ##What it is I built Walkdown, a pre-flight check for AI skills. You give it a GitHub link or a folder, and it reads the instruction layer: the SKILL.md files, agent definitions, hooks, manifests and CLAUDE.md, the English a model loads and runs with your permissions. For each finding it answers two questions, cited to a file and line. What does this text ask the model to do? What does installing it grant? Run it before you install someone else's skill, or before you ship your own. I built it for security. Most of what it taught me was about writing better skills. ## What you can learn from a run The summary page asks four questions: - **Size and shape.** What it costs in tokens every session and at its largest, against the context you expect. If you run local models, add `--context 8k` and watch the percentages move. It also lists the files that ship and never get used. - **Will it work somewhere else?** Hard-coded paths, programs it needs and never declares, and which AI tools besides Claude Code it's ready for. Got a CLAUDE.md and no AGENTS.md? You'll see it. - **Does it ask only for the tools it needs?** Tools granted and never used, loops with no stated limit, hooks that fire on every prompt. - **Anything to be careful about.** Hidden characters the model reads and you can't see. Dead links and files nothing points to. Instructions to go fetch more instructions from somewhere else. Simon Willison's lethal trifecta (can it read your data, take outside input, and send data out?). Leftover TODOs and pinned model IDs land here too. You can get deep into the details of everything it caught. Each finding also says how often that pattern turns up in other repos, so you can tell a common habit from an odd one. `review.html` walks you through each place one card at a time so you can mark it.
Walkdown: What is actually in that skill?
0 likes • 3h
Really interesting work, Toby. I’ve been exploring similar problems around AI agent verification, especially the gap between what a system says it can access and what actually gets loaded or executed. One question: have you considered distinguishing present → referenced → actually loaded in the report? A file can be referenced but never reach the model during execution. Seems like a useful boundary between static inspection and runtime verification. Love the evidence-first approach, especially leaving the final judgment to the reviewer.
Edubaware Confuses Me
Something I’ve been trying to wrap my head around is Jake’s Edubaware. Like most learning exercises, I start with a Google NotebookLM of the top sources on the topic. This time I had it build a slide deck overviewing Edubaware, then exported the whole thing as a PDF. Attached. The short version: it’s multiplayer folders. You take an ICM (a folder of files that already tells the model exactly what to do), freeze it as a template, and launch a workbench. Everyone on the team works against that same shared copy. The agent knows who is typing. No more hunting through separate chats for the good version. It’s basically the same shape as opening a project folder and running a few AI sessions against it—except the folder is shared, the template stays frozen unless a builder updates it, and you control the org, the roles, and the keys. Upload the zip that already works on your machine. It works for everyone you invite. Thought I should share the deck. Let me know what jumps out.
Edubaware Confuses Me
0 likes • 2d
David, “multiplayer folders” is probably the clearest explanation I’ve seen of this concept. The frozen template plus shared workbench makes a lot of sense. The part I’m curious about is what happens when multiple agents or people are working against that shared context simultaneously. Does Edubaware track who changed what, prevent conflicting edits, and distinguish an action that actually completed from one an agent only claimed to complete? I’ve been exploring execution-state integrity in my own AI harness, and shared context feels like one half of the equation. The other half is knowing exactly what happened. Really useful breakdown. Appreciate you putting this together!
0 likes • 6h
@David McDermott Thanks, David. That helps a lot. One follow-up: when two agents attempt to modify the same file concurrently, does Edubaware enforce any locking or version-conflict detection, or is conflict avoidance handled through agent instructions? I’m particularly interested in what happens when an agent fails halfway through a write.
1-10 of 45
Bill Bloomenkranz
5
324 points to level up
@bill-bloomenkranz-3953
Biogas engineer in Madison. I direct AI, I don't code. Building Atlas, where nothing counts as proven without a receipt.

Active 59m ago
Joined Sep 30, 2026
ENTJ
Madison, WI
Powered by