Trust once lost is hard to earn back. How are you keeping yours safe?

I’ve noticed there are some fairly distinct opinions about AI in here (and that’s a good thing - I appreciate diversity of thought).

There’s a view - and I’m certainly hearing it *everywhere* right now - that success is end-to-end from a model. No human involved. And I have some very strong human in the loop opinions. And so do some others in here. (And if I change that “in”’ to “on”, we get one more).

So I have been talking to

@Gabriel Azoulay

in here about *stakes*. How I tackle things and consider them to figure out what does and doesn’t need my brain and why. Here is some food for thought about my perspective in this area.

I think about this in layers. And some of them are orthogonal. But ultimately, we start big and go small. (And then we get to the orthogonal piece at the end, if I haven’t lost you).

The big gates:

Domain - right at the top for a reason. Is this a high stakes, sensitive domain or a regulated industry? I’m talking medical, legal, financial services? If so, you have to make sure that you are meeting the high bar that the domain requires *and* the regulatory requirements. If yes, tread cautiously and involve experts. If no, continue on.
What is the potential for harm? How much could this hurt someone if it goes wrong? - could this have financial consequences for them? Could it hurt children? Genuinely, I stop and ask myself this. There are areas where many are happy to use AI - measuring job fit for example - where I am pretty selective. My view: this could affect whether or not someone gets a job. That could have massive potential harm on someone who is overlooked not because they are not right for the job but because a model didn’t have the right context to see it (and yes, baked in bias, I am looking squarely at you). Everyone has got to make their own decisions on where this sits that is right for them, but you should be thinking about it upfront in two ways: should this mean you don’t do it at all? If you do proceed because the level of harm exposure is within tolerance, *what are you doing to REDUCE and MITIGATE the potential for harm exposure?*. This should be decided upfront, not baked in later.
Reputational harm potential. If this goes wrong and you or your company end up hammered in public for it, what does that look like? Does that look like “they engineered badly” or “they happily gambled with the safety, privacy and wellbeing of minors with no forethought?”. Not the same thing. Worth pausing to consider.

After the big gates, we get into what I think of as “the functional stuff”:

Scale - how much information/data/volume is going through it and how many people have to use it? This is functional, because it’s going to have design implications. But it’s also important when considering stakes because AI is an accelerator - if the way you are heading means you’re going to make a mistake, AI is going to mean you make that mistake faster, louder and much more intensely. The greater the volume, the more likely this will happen. It’s not just a requirement consideration, but also a stakes one. It moves the bar you need to hit.
What could go wrong - the mini version of the above. Before I was talking whole domain, potential for harm (big scale), reputational harm. Now I’m talking about the functional side. Is there money involved? Is an AI deciding where that should go? Is a *client’s* reputation or satisfaction riding on this? If these emails start firing out with no one looking at them, are all our customers going to think we’re mad and stop buying our products? Genuinely, this is the point to stop and think about it.
Security. Are you using an MCP connector from a reputable source that gives you a nice auth method baked in? Or are you handing your credentials to a model? (DON’T HAND YOUR CREDENTIALS TO A MODEL. PLEASE). This is a time to be responsible. If that means you have to click some buttons or copy and paste across, maybe you should be sucking that up rather than rolling the dice with access to your systems. 🤷🏾‍♀️

By the time you get past these, you get to what I think of as “the details”. Where do *you* belong and does it matter to be fussy versus what can you hand to a model or automate:

Is the final product going external to your system or staying within it? (The amount of effort and what you let a model do unsupervised changes based on the dial here - if this is producing content you post or work that’s going out to clients, you are going to want some decent review gates. If it’s writing some logs for a 1 person system - and yes this is a real example - it doesn’t matter that much, stakes are very low).
How much value do you add? Is this something that humans are good at (emotion, interpersonal understanding, based on lots of memory or experience) or something a model is good at (miniscule detail at scale, discovering patterns in a compressed time period over significant amounts of data, shaping/structuring raw ideas into themes).
How up to date does it need to be and how fast is it moving? The “general knowledge” of models that they walk in with is locked the moment they are shipped and they are no longer training. If your domain is moving fast (hey, AI, I’m looking at you), they won’t come with that knowledge out of the box - you’re either going to have to provide it *or* ensure that it’s out searching for it. Food for thought.
How much do generalisations versus specifics matter here? Models collapse nuance and grey area and they do it *early* - they do not hold it open for long, they decide and keep moving. Even if you ask them to do it (and trust me, I’m playing with this), in some cases it feels like fighting gravity. Kind of pointless. If these are important, maybe keep it away from here. Or at least go in with your eyes open.

Orthogonal to all of this is what only you can bring. And that checklist should be there every time:

1.Ethics - how does doing this line up with your values and beliefs? This is a gate you should run every time.

2. Expertise - how well do you know the process you’re automating, the data you’re working with, the topic you are looking into? If you don’t know it blind and backwards and forwards, you need to go in eyes open about how much critical thinking you can bring to the mix and how you’re going to compensate for that. And if you don’t know it blind, you should pause and consider whether you should be proceeding. Genuine examples: I am not a dev (hobbyist coder at best), so any time I get models to write actual code for me, I have a whole model versus model testing process I go through. Lots of iterations, lots of fresh context, a minimum of 2 model providers. If I can’t bring that, I want to make sure something is. And also, the worst that can happen if my scripts go wrong - I have some drift in my ICM or the count of items in my to do list is wrong. Or maybe I buy the wrong things at the supermarket and have to go for a second trip (hasn’t happened by the way). I’m not shipping actual code, this is for me. Risk is worth running and I mitigate it. Different example - FMCG is a new industry for me. I’m 5 months in. I refuse to ask AI about it. It’s too dangerous. I could get advice that’s right for offshore and not here or misses nuance or is out of date. That’s what Google is for. Know it, use it. Cognitive sovereignty.

3. details - the other side. Are there fine-grained cultural or social nuances at play here? Are any of those sensitive? Again, time to think about it from the soft, ethical, lens. Is this in line with your beliefs down in the details as well as the big picture part of this section?

4. is there value in doing it yourself? Are there muscles you need to build? Learning it by doing, understanding it fully, before handing it over. Let’s take ICM as an example- this was new to me. I did a lot more building in a really hands-on way to start off with because I was learning. Now I’m starting to build in protocols to delegate it. Not because I can’t do it, but because I have built that knowledge and I want to move on to other knowledge (the next phase) and scaling that gives me more opportunity to learn the next phase. It’s a choice.

5. How good are you going to be at it versus a model? I presented my strategy - 5 months in the making - last week and defended funding it this week. I got my system to help me create a rubric and score potential KPIs for it against what I developed and the list I had achieved. That was really helpful. I didn’t let it help develop my strategy at all or come up with the initial list of KPIs we were scoring to winnow them down. That’s something where I can’t hope to teach a machine how to do it and I’m better off doing it myself. Think about how much you should be owning and the model should be owning - this combines not only the stake of the deliverable, but the level of expertise and thinking in the mix.

I almost feel like I should be apologising for this post. It’s long. It’s about the boring stuff. Building and making it happen, that’s the fun part. But maintaining the right boundaries to keep yourself safe? That’s invaluable. And your cognitive sovereignty - once you lose it, I’m not sure there’s always a road to recovery.

How are you looking at stakes?

37 comments

Trust once lost is hard to earn back. How are you keeping yours safe?