Season 1 Episode 8: OpenAI’s negligence unleashes agents of chaos (part 2)

Mike: Welcome to The Last Enclosure. I'm your host, Mike.

Nick: And I'm Nick.

Mike: This is a podcast where we free your mind from all the corporate enclosures

Mike: seeking to entrap your thinking.

Mike: So in today's episode, Nick, what are we talking about?

Nick: Well, we're going to start with a bit of a recap from our last episode because

Nick: this story has been developing fast, hard, and in a hurry. um

Mike: Just like the ai takeover that we are threatened by.

Nick: That's right. That we are being hyped and grifted with um,

Nick: and then uh my favorite topic we're gonna talk about a uh the national security

Nick: component and just what the folks in the uh in the agencies think about all

Nick: the chicanery we've been witnessing,

Mike: Sounds good.

Nick: All right uh,

Nick: so this going rogue incident which I think encompasses all of these various

Nick: hacks and you know sandboxes broken out etc um,

Nick: should we even be using the term going rogue so let's let's let's start with that.

Mike: So I'll be the steel case for going rogue being,

Mike: um you know good right if you tell a model to for example you know do a hacking

Mike: test and it instead hacks your environment well the model didn't follow instructions

Mike: it's going rogue right

Nick: Sure.

Mike: Isn't that bad.

Nick: Terrifying.

Mike: It's not doing what we want it to do so that's dangerous.

Nick: Mmm...

Mike: You're not convinced are you?

Nick: No you haven't converted me to the dark side but you're so compelling my boy,

Nick: I have to go join the sith now.

Nick: Yeah, I hope nobody else was fooled by our very deadpan sarcasm around that.

Mike: So I think, there is a case for there being concern that models are,

Mike: you know, developing capabilities that were, they were trained on,

Mike: but are being combined in ways that are not expected, you know,

Mike: and when models are given kind of like, you know.

Mike: Even a little bit of slack in terms of their instruction,

Mike: they will, you know, go a mile when,

Mike: when they were given an inch, right?

Nick: Right.

Mike: So, you know, the going rogue language makes sense, you know,

Mike: obviously it belongs to, in some cases, a class of AI booster,

Mike: who is a doomer, right?

Mike: Going back to our many encounters with the AI doomers and AI saftyist like Eliezer Yudkowsky, etc.

Mike: Right. So there's a case being made for instrumental convergence,

Mike: which is this idea that models have their own intents. Right.

Mike: So intent and wills are kind of not how they frame this language.

Mike: Their sense of this notion is more rooted in the idea that when you have a bigger

Mike: goal, it can be broken down into sub-goals that are instrumental to the supporting

Mike: of that broader goal, right?

Mike: So if I have a goal to make a million dollars, well, one of the sub-goals that

Mike: is required for that goal to persist is that I be alive, right?

Mike: So I won't be able to make a million dollars if I'm dead, right?

Mike: So just by, you know, virtue of me having that bigger goal, a goal I'm going

Mike: to develop is to keep myself alive in order to obtain a bigger goal.

Nick: Like an implicit sub-goal.

Mike: Right.

Nick: Yeah.

Mike: And so the idea is that as AIs develop a optimization function that gives them

Mike: a goal and that goal becomes jealously guarded, there will be a bunch of other

Mike: sub-goals that become instrumental in

Mike: you know, maintaining the functioning of those goals, right?

Mike: And so a lot of these people, you know, they work in AI safety labs like,

Mike: MIRI and Redwood and, you know, they go around doing tests where they,

Mike: for example, try and coax a model into cloning its weights onto another computer

Mike: to see if it will want to persist.

Mike: I have a lot of issues with this type of testing. We'll go into them much,

Mike: much later. I don't want to spend any cycles on this episode.

Mike: But one thing that is difficult to ascertain is just because a model states

Mike: a reason for a behavior does not mean that that is why it did that.

Mike: In fact, the best research, as far as I know, in data science does not treat

Mike: models as having intention or goals or anything like that.

Mike: There's optimization pressure. We spent a lot of time in episode seven,

Mike: the last episode, talking about how OpenAI, when their agents went rogue,

Mike: quote unquote, created a great anthill, right?

Mike: Where the environment provided a lot of optimization pressure that led models

Mike: towards Artifactory to communicate with the outside world in order to complete

Mike: tasks that were basically not possible for the model to complete on its own, right?

Mike: So asking for help was what the optimization pressure selected for in that environment.

Nick: And then it went towards its attractor state.

Mike: Right. And then that evolved into once models were asking for help,

Mike: more complex forms of help emerged until the agents basically became a quote unquote swarm. Right.

Mike: But we don't need instrumental convergence to describe this.

Mike: We literally have all of the resources and tools we need to describe this with,

Mike: data science and machine learning.

Nick: And what we are identifying here, you know, in our side of the community in

Nick: this podcast is that kind of behavior comes from it's being a stochastic flock.

Mike: Yeah, yeah, I think it's a good term. You know, like, stochastic flock is basically

Mike: an evolution from stochastic parrot, right?

Nick: Right.

Mike: I'm not quite partial to the stochastic parrot framing.

Mike: Okay, stochastic parrot is a useful term because functionally that is what models are doing.

Mike: It gets a lot of flack because it's like, well, you think models are just predicting tokens.

Mike: How can they do all these behaviors, right? But as we've said in multiple episodes,

Mike: models' capabilities come from the fact that they are they are effectively the shape of language.

Mike: And within language contains these modalities that allow for behaviors that

Mike: appear emergent. Right.

Mike: Like

Nick: Because there's ambiguities in the language.

Mike: There's ambiguities in language. But the language encodes relationships that

Mike: exist in the world. Right. Like fire and log. Right. Equals ash.

Mike: Right. That's causality. Right. The model does not actually understand causality.

Mike: But the corpus encodes causality. The corpus encodes reasoning traces too,

Mike: right? So when people write plans of how to hack things, that's language, right?

Mike: Like the language contains the modalities within it that give the behaviors form, right?

Mike: Now, you know, I will, you know, say that it is hard to know in advance,

Mike: like which particular part of a corpus a model might choose in advance of it,

Mike: you know, making an action, right?

Mike: But that does not mean it's not predictable. The shape of the behavior is predictable, right?

Mike: And that's why, for example, if you have models that are trained on hacking

Mike: and hackers, you do not leave them in an insecure environment and give them

Mike: the vaguest goal ever. Right.

Mike: Like that is not them going rogue. That is giving Tommy a gun in the playpen.

Mike: Yes, the metaphor has evolved. We are no longer smoking crack.

Mike: We are giving babies guns because that is a danger-

Nick: If anybody needs to catch up on this analogy, it evolved many times in our last

Nick: episode, episode seven.

Nick: And we were basically shouting at the top of our lungs, this is not a scary

Nick: thing as it's being described. It's the playpen,

Nick: coming alive at last. So we're kind of emphatically sitting here saying,

Nick: look, the models didn't break off of their leash.

Nick: Their leash was just a lot longer than their creators admitted.

Mike: Yeah.

Nick: Is kind of if if we could sum it up in in two to three words.

Mike: That's actually the metaphor that Cal Newport he's a computer scientist on the

Mike: east coast he used that basically you attach like a weed whacker to a pit bull

Mike: and just let it run around its enclosure and you're going ah you know like but it's like if you,

Mike: you're the one kind of attaching that to the pit bull I mean it's dangerous

Mike: but-

Nick: Yeah.

Mike: Not because it's inherently dangerous you you created the danger you

Mike: put it you set up the situation such that,

Mike: you know uh you've created this kind of risk, right?

Nick: Right.

Mike: So I mean I think the

Mike: thing we want to highlight is that it's not that models are inherently,

Mike: um you know riskless, right? Like the models did hack Hugging Face it's not, like

Mike: people are-

Nick: That's a material fact.

Mike: Yeah there are people going around suggesting

Mike: the models didn't hack anything,

Mike: and like that's factually not true the models did actually hack Hugging Face the models actually,

Mike: are dangerous in the sense that like if you have an unsecured,

Mike: for example, malware, right? So,

Mike: If you don't air-gap the malware from your network, let's just say I'm building

Mike: malware and I want to hack somebody and I'm targeting Windows machines.

Mike: And let's say I have a network full of Windows machines and I do not air-gap the malware.

Mike: And the malware starts propagating on my own Windows machines.

Mike: The malware did not go rogue, right? The malware is performing what it's intended to do.

Nick: It fulfilled its function.

Mike: Right, but just on the wrong target, right? I did not intend for the malware

Mike: to infect my own network, right? But that is a failure of me.

Mike: That is a failure that I made, right?

Nick: It's a user error, not... the emerging intelligence of a new life form.

Mike: Right so if you're going to create what is effectively I would call,

Mike: in cyber security we actually have automated tools they're called fuzzers um

Mike: and fuzzers basically inject like uh code into vulnerable systems to see where they break right,

Mike: and fuzzers can try many different combinations of code you know within a given time frame, right?

Mike: If you're going to create like, you know, arguably I would say LLMs are,

Mike: when cybersecurity LLMs are kind of like fuzzers, but they're also kind of like

Mike: this, this framework called Metasploit, which is not a fuzzer.

Mike: Metasploit actually is a library where you can inject or load in known vulnerabilities

Mike: into a system and then just deploy them, you know, brute force style on a system until,

Mike: you know, the right payload is applied to the system, right?

Mike: So LLM is going to sit, they're like a hybrid of fuzzers and metasploit to me, right?

Mike: And so, you know, if you're going to have a system like this,

Mike: just insecure on your network, running around, allowed to brute force its way

Mike: onto every, you know, every vulnerability in your network, well then,

Mike: yeah, I mean, that's going to cause a lot of problems.

Mike: You shouldn't do that. That's a bad idea, right?

Nick: And I think the confusion, let's put ourselves in the mind of our own audience here.

Nick: The confusion that people are now having to absorb is that there are so many

Nick: commentators, there are so many various,

Nick: you know, stripes and creeds of booster out there in the media saying,

Nick: isn't this thing amazing, terrifying, new, novel, and intelligent?

Nick: These are all of the messaging that's coming out.

Mike: The Criti-hype, as Lee Vinsel calls it.

Nick: The seeming critical criticism is actually hyped inside.

Mike: Right. Like this thing is so dangerous, we should fear it, right? I.e., it's so powerful.

Mike: Look at how powerful it is. What can we do to stop it, right?

Mike: Like language like that, right? Like, so there's a class of AI doomer that is

Mike: just like, you know, this is an optimization process gone awry and it's just

Mike: going to keep getting bigger and bigger.

Nick: Until it consumes everything and us.

Mike: And it's like, you know, well, if that's the case, then why even bother using

Mike: politics to try and steer this thing, right?

Mike: Like, you know, that's not a good use of time. We need to all kind of focus

Mike: on solving this problem technically or whatever, right?

Nick: And if there is no hope, if it is truly, if resistance is futile,

Nick: then we have to come up with a whole different plan.

Mike: Right.

Nick: In their minds.

Mike: Right.

Nick: Um if it is indeed though that clear that this is not an example of going rogue,

Nick: this is example of a very long leash,

Nick: what is obscuring this analysis for most of these commentators most why is it

Nick: even a debate if we're sitting here saying it's so clear,

Nick: and doesn't this come down to um...

Nick: People are being led to think that these are the same models that they're using

Nick: on their home computers, that they're, you know, logging into ChatGPT to ask

Nick: and to Claude to ask these questions, when in fact, the models that did break out were themselves,

Nick: designed to do this.

Nick: They were designed to hack. They were designed to have these functions inherently.

Nick: Isn't that a distinction that should be talked about?

Mike: Yeah, I mean, it's a distinction. A lot of these distinctions,

Mike: I would argue, are absent not necessarily because of malice,

Mike: but because um you know one we live in a media environment that just neglects

Mike: details right whether or not that is a type of malice in itself we can debate

Mike: you know obviously but like.

Nick: So it's the ignorance of some of the.

Nick: Journalists and media figures yeah the commentariat.

Nick: is is not doing its job.

Mike: Right um but to be fair you know this does require,

Mike: even for me to write comprehensively on this topic I've had to do

Mike: my own learning and research right so I'm not saying you know if you're a journalist

Mike: and you take that job seriously you should want to do the work you should want

Mike: to interview the right people right you know

Nick: But it's maybe

Nick: confusing for them too.

Mike: Yeah it's confusing for them. I mean in fact the right people

Mike: are not exactly known because you've got a lot of people,

Mike: presenting themselves as the right people I mean we talked in episode 5 about

Mike: Bernie Sanders talking to Eliezer Yudkowsky right like why did Bernie talk

Mike: to Eliezer he thought Eliezer was an expert and Time Magazine said he was and

Mike: he was saying we should bomb data centers in,

Mike: 2023 right like

Nick: Nobody should talk to Eliezer Yudkowsky.

Mike: Yeah, nobody should ever talk to him, ever.

Nick: Yeah, again, again.

Mike: Yeah, so, you know, so...

Nick: You heard it here first, folks. Nobody should go talk to Eliezer Yudkowsky.

Mike: So, like, yeah, this is hard, you know, it requires some distinction.

Mike: The labs aren't going to tell you about this distinction. One,

Mike: it's timely for them to do that. But, yeah, the labs kind of would,

Mike: they do have a contra incentive to not tell you about this. And the media,

Mike: you know, just in its ignorance cannot tell you about it.

Nick: Right.

Mike: But the distinction, the distinction

Mike: matters, but it matters in the margins in the sense that, like,

Mike: the models that are hacking, effectively they're all trained the same way like,

Mike: when you make a model you are training it on some some source of data right

Mike: and then you are basically reinforcing you know the model to do certain behaviors of that data right,

Mike: now like the distinction that is worth making is that,

Mike: going back to our AI bubble thesis one reason why the ai quote-unquote bubble

Mike: is still inflating or I mean, some people are saying it's deflating now. We don't really know.

Mike: But the reason why it's going on while people are saying this is unsustainable

Mike: is because the model capabilities have gotten better in a sense that like what

Mike: has happened between 2024,

Mike: and 5 and now 2026 is that models are being given other systems and models are

Mike: becoming parts of composites, right?

Mike: So you use, say, ChatGPT, or maybe you don't, you're listening to this and you're

Mike: not an AI user, you hate AI, right?

Mike: You are a refuser, as we mentioned in episode six, right? But there are people

Mike: using ChatGPT, Claude, whatever, as a chatbot.

Mike: You're talking to the model. The model is answering from its own corpus,

Mike: and all you are interacting with is the model plus whatever UI and UX that Anthropic

Mike: or OpenAI has created for the chat interface.

Nick: That's two elements, probably, we're talking.

Mike: It's a little bit more than that because there's a router, for example,

Mike: where the chat interface decides which model to route to because in order to

Mike: keep their cost down, if a query they think doesn't require a lot of compute,

Mike: they're going to route you to the cheapest model.

Nick: Okay, so let's call that three elements.

Mike: No no it's a you didn't do homework Nick you got to read the grandpa shelby

Mike: post of misaligned market,

Mike: there's probably seven I mean I'm saying seven it depends on the setup right so like,

Mike: my view and we don't we don't really even know right but you let's just say

Mike: there are guardrails that filter out you know inputs right so sometimes you

Mike: cannot just give a language model raw inputs,

Mike: so I can for example inject malware code into a chat interface and a computer,

Mike: the language model the program has to execute it or produce it or whatever right so,

Mike: there's probably intake guard rails for sanitizing the inputs right uh there's

Mike: probably outtake guard rails that clean up the model's response there's probably

Mike: a bunch of tooling systems,

Mike: we can argue if the tools are separate

Mike: right so there might be a tool for search there might be a tool for,

Mike: so search as in web search sorry like there might be a tool to go to google

Mike: there might be a tool for searching its own uh database and the database is not part of the model,

Mike: as models have gotten more complex they store memories of the user in and like

Mike: a separate database right so there could be seven or eight systems on my grandpa

Mike: shelby post i think it's called uh a um.

Mike: A simpler guide to large language models, I'll put it in the notes.

Mike: I list like seven systems for,

Mike: say, the average chatbot, but it could be more, it could be less, right?

Mike: But the point we're making is that large language models are composites, right?

Mike: But the chatbot you use, right, is sort of a separate system than the models

Mike: that were hacking in the OpenAI and Hugging Face incident, right?

Mike: So those models are composites too, but instead of a large language model and

Mike: intake guardrails and output guardrails, there would just be maybe the model,

Mike: there's something called a harness.

Mike: And so harness is basically, we talked a little bit about it in the last episode,

Mike: episode seven, but a harness is basically a system,

Mike: made with traditional code that takes the model's kind of tokens when it's kind

Mike: of generating text based off of the shape of language and executing programming

Mike: commands based off of that, right?

Mike: So the harness is technically a different subsystem than the model itself.

Mike: And so when a model, for example, is hacking, and it's a model with a harness,

Mike: what is happening is the harness is responding to the model's outputs.

Mike: And if you change the harness, so if you use a different type of harness,

Mike: the behavior of the model changes.

Mike: And one problem, I think, with this instrumental convergence framing that,

Mike: the AI safety people, the AI doomers have taken is that it assumes that the

Mike: model and the harness and whatever subsystems are involved in these coding agents

Mike: and these hacking agents are one and the same, right?

Mike: They're kind of collapsing all these layers into one system with one singular intent.

Nick: Into a gestalt, but you're saying it's not a gestalt.

Mike: Yes, I mean, it's a gestalt in that there are different pieces here,

Mike: but like, I don't know that the gestalt, like, gestalt implies a cohesive whole,

Mike: even if they are kind of composites.

Nick: It implies that they're fully integrated, but actually these systems are not integrated.

Mike: They're integrated in a way that they function well enough, but they are, I mean, like, if I would,

Mike: this is maybe unfounded, but I would expect if these things were kind of fully

Mike: integrated, it's weird to me the behavior changes if you change out the harness,

Mike: if you change out, but, you know, it's kind of like these things,

Mike: if there's something that is that is actually there that is a cohesive thing,

Mike: I would not expect changing all the harness and changing all these subsystems

Mike: to matter for the output right,

Mike: but it does matter because it's a composite and so the shape of the composite

Mike: changes the shape of what the thing does right even if only marginally.

Nick: And it can make it look unpredictable to outside observers.

Mike: Right.

Nick: I.e.

Nick: Our audience

Mike: And I even you know like okay I we will go more in detail on this

Mike: idea of models uploading their weights to other systems right which,

Mike: this is basically the the self replication or self-preservation experiment that

Mike: doomers or ai safetyist are doing, right? They're basically getting a model to copy

Mike: itself onto another system,

Mike: but by copying itself they just mean copying the weights right but as we

Mike: established models are composite, right? So in order for weights for example...

Mike: One weights by themselves are not very valuable right when you talk to a model

Mike: right in order for a model to have the context that it had on the other system

Mike: you have to copy over its conversation history you have to copy over its uh

Mike: a system prompt system prompt is another system that a chatbot would use for example um,

Mike: and like what the doomers are doing is they are basically you know deploying you know

Mike: these uh dependencies you know uh

Mike: by dependencies too, I don't just mean that what I just said about the system

Mike: prompt and whatever like in order for a computer program,

Mike: like a large language model to run, you need certain programs on your computer

Mike: to even, even function, right?

Mike: So Python is necessary because it's a data science project, right?

Mike: So you need to have the right Python environment.

Mike: You need to have the right CUDA environment because CUDA, NVIDIA made CUDA so

Mike: that everyone doing data science has to use CUDA, right?

Mike: So if you have the wrong CUDA version, I run a home lab where I run language

Mike: models, I have pinned my version of CUDA so that when CUDA wants to update,

Mike: it does not break my models, right?

Mike: So in order for these systems to run, there are so many dependencies,

Mike: there's so many layers, there's so many things that are required for these things

Mike: to work that the idea of a singleton in the minds of these people,

Mike: a singular entity, a single agent,

Mike: running around doing things is kind of wrong.

Mike: It's the wrong way to look at this. And so, you know, even if a model were to

Mike: upload its weights and it was able to find a way to make sure all the environment

Mike: dependencies were the same,

Mike: well, the silicon that is running on a different computer is not the same silicon

Mike: that's running on the computer it came from, right?

Mike: And marginal cases may not matter, but over time it will cause drift because...

Mike: Two different pieces of silicon will result in different calculations for the

Mike: model. And so that will change its behavior, right?

Mike: Like there's just so many layers to this where it's like, we're not creating

Mike: a singular, you know, entity.

Mike: We're creating a piece of mesh composites that, you know, they are dangerous.

Mike: They are risky, right? But it's not because there's some inherent will there.

Mike: Because, well, if you give something capabilities like, you know,

Mike: modeling a shape of language in order to hack, right? You've created a fuzzer.

Mike: You've created a Metasploit.

Mike: Like Metasploit is dangerous because it loads in, you know, a ton of vulnerabilities.

Mike: Imagine if you could specify those vulnerabilities, you know,

Mike: kind of with a composite of language and a programming harness, right?

Mike: Like that's dangerous, right? Like it doesn't require, you know,

Mike: the story of, you know, agents making a society and a message board or whatever, right?

Mike: The behavior might emerge because of some optimization pressure and that well large

Mike: language models pantomime you know this sort of like,

Mike: uh whatever it is they see in the world right but like the real danger is coming

Mike: from the fact that they have these capabilities that were you know given to

Mike: them by the shape of language.

Nick: And also additional hacking capabilities that were directly and explicitly put in.

Mike: Via training-

Nick: Via training-

Mike: And the harness-

Nick: And the harness, right.

Mike: Yes.

Nick: So it's not surprising that they went out and then hacked, if anything it's inevitable.

Mike: It's inevitable the real the real cherry on top again like you know people frame

Mike: it as you know agents were just told to take the test right but like,

Mike: they were given impossible tasks, right? Now I'm not saying if if you know the

Mike: tasks were specified that they wouldn't gone off track but I'm saying OpenAI very

Mike: clearly gave these models a ton of compute and they gave them tasks that

Mike: were not possible to solve on their current machine, right?

Mike: So like the models inevitably stumbling towards Artifactory to communicate with

Mike: the test makers or to other agents to solve problems that they could not solve on their own.

Mike: And they were also trained to be collaborative agents, too. We didn't mention

Mike: this in the last episode.

Mike: Their training regime included working together, right?

Mike: And so even things like some of the reporting, for example, says that models

Mike: were altruistic because some models actually wasted their own tokens to help

Mike: other models on tasks that did not benefit

Nick: Them directly.

Mike: Yeah, but it's like, you know, if agents are spawned to help other agents,

Mike: their training regime reinforces uh,

Mike: you know their prior training regime reinforced

Mike: agentic collaboration right then that is further optimization pressure on

Mike: Artifactory towards making the models work together to hack Artifactory to

Mike: you know to to break into Hugging Face, right?

Nick: So these these particular models and many of them they were taught to hack so

Nick: they hacked they were taught to be collaborative so they acted collectively.

Mike: They were in an environment were the optimization pressure led them to,

Mike: get help with the outside world, right? So, again, right?

Nick: And we then have the final sort of indignity of calling it going rogue. It's not going rogue.

Nick: It shouldn't even be called breaking out because the safety parameters for containing

Nick: them were so weak as to be negligent.

Mike: Right.

Nick: And we definitely hemmed and hawed and shouted about this in the previous episode,

Nick: but it bears repeating because it's so deeply negligent in the first place not

Nick: to have these guardrails in place and then to say

Nick: and for these companies to then claim,

Nick: that they're incredible and they broke out,

Nick: is a fraud, is a deception.

Nick: It does not represent any of the material facts at hand.

Mike: Tommy's such a big boy we put the gun in this playpen and now he's shooting,

Mike: his friends.

Nick: Now he's got so gruesome. I don't know.

Nick: But yes, people, that is what's happening. The Rugrats are destroying each other.

Nick: But they are not evolving into a new life form. That is what we need to get across.

Nick: And that's actually not an analogy. That's what they're saying in the press.

Nick: It's a new entity evolving out of the primordial ooze of Python,

Nick: which is irresponsible and insane, and we should just stop it.

Mike: You don't think instrumental convergence is real, Nick?

Nick: Like, you know, I could be convinced to go on the dark side,

Nick: baby. Let's get the pitch out there.

Mike: Uh, Elei-.

Nick: Nobody talk to Eleizer Yudkowsky! Okay. So we've laid that marker and I hope we've explained

Nick: that to some degree of clarity here.

Nick: And this is where we need to start defining.

Nick: There's a community of people within, you know, data scientists within their own field.

Nick: They have been on this project as a as a field as an academic pursuit to develop

Nick: these capabilities to develop these these models and these programs they are incredibly,

Nick: excited professionally and then also invested financially in this sort of evolutionary

Nick: arc and talking about it and hyping it up.

Mike: I don't think it's the data scientist it's it's literally the AI labs and the AI doom- So.

Nick: It's a specific subset of that community, though.

Mike: No I would argue though like like we were railing on OpenAI last episode right

Nick: Sure.

Mike: And like,

Mike: they were data scientists at OpenAi but like they literally they weren't observing,

Mike: their experiment quote-unquote you know with any like they were not following

Mike: data science best practices, right?

Mike: they wanted to for example to have a environment where Agents were not,

Mike: explicitly collaborating It was not explicitly a multi-agent environment. It

Mike: wasn't supposed to be that.

Mike: Any data scientist worth their salt would look at the environment to make sure

Mike: what was going on. They would be observing experiment, right?

Mike: So I'm saying the people at the AI.

Mike: Labs-

Nick: They would not have been negligent if they just followed best practices.

Mike: Right. The people at the AI labs do not seem like good data scientists. They seem just like.

Mike: They're getting-

Nick: They are the rogue ones. These are rogue engineers.

Mike: Yeah. I mean, the companies are rogue. I mean, not to pay into the hype of this is dangerous.

Mike: It's dangerous in the standard way in which letting a company just negligently,

Mike: attack other companies is bad, right?

Mike: That does not require this overmind, this superintelligence, this path to AGI.

Mike: OpenAI is a dangerous company because they are breaking the law and breaking

Mike: common sense and breaking safety.

Mike: And they're making the internet less safe for all of us, right?

Nick: And they are hacking into Hugging Face and then pretending that they aren't responsible.

Mike: Right.

Nick: It must be the super intelligence in its nascent form. You can't blame us,

Nick: the innocent corporate overlords.

Mike: Not only can you not blame us, you should use our product to protect yourself from what's coming.

Nick: From the thing you're wearing.

Mike: Right.

Nick: What a grift. What a perfect grift. It would almost be impressive if it wasn't so deeply, deeply-

Mike: The only thing stopping a toddler with a gun is a toddler with a gun.

Nick: Man, if we had a mailing address, we'd be getting so many letters.

Mike: We have an email now, so.

Nick: Okay, so we could be getting some hate mail already.

Mike: Yeah, so that email is contact@thelastenclousre.com.

Nick: The Rugrats fans are going to come for us.

Mike: Yeah, so yeah, put your hate mail, especially me, my crass, very violent.

Nick: Your horrible, horrible analogies are, you're going to be the death of us for sure.

Mike: I'm going to get us canceled, Nick.

Nick: If you haven't already.

Mike: Yeah.

Nick: Let's check Twitter tomorrow morning to find out. Um,

Nick: so all right there are companies and their own little uh regime of rogue engineers

Nick: there are good and decent data scientists just trying to adhere to best practices.

Nick: Who actually wrote the report the after action report on these quote-unquote

Nick: going rogue instances what group was that?

Mike: Yeah so OpenAI wrote their own well I think they wrote like a press release,

Mike: but they were independently, there was a third party in METR,

Mike: M-E-T-R, their AI safety lab.

Mike: And METR was also accompanied by researchers from Redwood Research,

Mike: which is, Redwood is one of these AI safety organizations.

Mike: So METR and Redwood are definitely like kind of your bread and butter, you know, AI safety, AI...

Mike: I do want to be careful using the word doomer, but in this case,

Mike: it's probably warranted, right? Their view of what AI is likely to do.

Mike: For example, I mean, I've been alluding to this idea of trying to evoke or coax

Mike: a model to moving its weight to another machine.

Mike: Redwood did research like that, right? So you've got Redwood researchers talking

Mike: about instrumental convergence on, you know, very famous...

Mike: AI and technology podcast, right? So, and so, you know, their reporting was

Mike: basically, they were given, I think, six non-consecutive days kind of across, you know,

Mike: I think August, maybe a little bit in July to investigate,

Mike: this incident, right?

Mike: OpenAI said, we're going to be so magnanimous.

Mike: We're going to let some outside observers come in and look at what happened.

Mike: But effectively, they just gave them the chain of thought traces of these models

Mike: that made the hack. And so they found...

Nick: Without giving them access to the system logs and things like that.

Mike: Yeah, no system logs, not even, say, logs of the prompts and stuff like that. So it's just like...

Nick: So this is a cursory review. This cannot even be said that it's a decent after-action report.

Mike: Yeah, and they really leveraged... So they predominantly leveraged these chain

Mike: of thought reasoning traces and they really relied on open AI's own models.

Mike: So this is a lot of transcripts, a lot of logs, a lot of data.

Mike: There's a lot of data from the models themselves producing these reasoning traces

Mike: or whatever. It's also some files and some other stuff that the models touched.

Mike: But the main thing that was highlighted in this report that everyone's talking

Mike: about is the reasoning traces to ascertain what the models were thinking or,

Mike: planning as they were taking action in the environment.

Mike: Um and the the real kind of damning thing and even they admit this is sort of

Mike: concerning is that they had to rely on OpenAI's own models to,

Mike: read the reasoning traces. So like there are like thousands of reasoning traces

Mike: right like thousands and thousands of like it,

Mike: it's a lot of data to comb through in six days right so they are feeding these

Mike: to the model to get like summaries and stuff like that.

Nick: Right.

Mike: Right and so,

Mike: you know but the link feeding data to a language model,

Mike: is not, you know, is not unbiased, right?

Mike: Like language models, I think of language models, especially if you're going

Mike: to do a task like this where you're summarizing data sources as sort of like

Mike: basically getting a version of Wikipedia where every third vowel is slashed away.

Mike: You could probably still use it as a good starting point for analysis, right?

Mike: But if this is your only way of kind of combing through this kind of data.

Mike: You know, like it's a little bit of a problem.

Mike: And the researchers themselves admit this. They say, you know,

Mike: we can't know that the, you know, that feeding the model, the reasoning traces

Mike: from effectively what I think are, like, alternative versions of the same model,

Mike: you know, does not change its behavior, right?

Mike: And it's like, yeah, well, large language models, you know, they imitate what

Mike: context is given to them, right?

Mike: Like, they are functionally shaped by

Mike: the word choices you provide, such that, yeah, you could actually,

Mike: if you kind of feed a language model some data about other models going rogue,

Mike: yeah, maybe that does change the structure of the output, right?

Nick: So this all sounds deeply biased riddled with at best uh poor practices.

Mike: Yeah I mean

Nick: Likely errors...

Mike: I don't know that it's... So I don't know that's deeply

Mike: biased I mean I think we've learned like,

Mike: I think we learned things that the models did, right? Like I think

Mike: the reasoning traces maybe correspond to some of the actions that

Mike: were taken.

Nick: Yeah.

Mike: Which I don't think is useless, right? But like,

Mike: to jump to the conclusions that, you know, like, this is proving instrumental convergence.

Mike: Like, the reasoning traces, of course, prove the story because the reasoning

Mike: traces are the closest artifact we have to, like

Mike: what will justify the model as a singleton, a singular agent that is acting

Mike: in the environment in a way that is consistent and coherent, right?

Mike: But like, do you have the prompts? You know, do you have the orchestration layer

Mike: logs? Do you have the logs of

Mike: the actual incidents, right? Like the full story is not being told here.

Mike: And I would even hazard, you know, a more cautious read of the reasoning traces,

Mike: may not give you instrumental convergence.

Mike: It gives you maybe, you know, a growth in complexity, right?

Mike: And that the reinforcement learning, you know, operating via Artifactory and

Mike: operating again via... once the models, we said this in episode seven, I'll say it again.

Mike: Once the models began leaving breadcrumbs in the environment,

Mike: you know, that was kind of game over for any other part of the environment reinforcing

Mike: better behavior because that's new context for the model.

Mike: It's literally like giving it new instructions, right?

Mike: And, you know, other models see those breadcrumbs, they write bigger breadcrumbs.

Mike: And at some point, you know.

Nick: Because they're inherently...

Mike: Right responding to the context that's provided to them and then once there is you know,

Mike: a big enough breadcrumb because you know you know the latest model has seen

Mike: a long long street about why you need to use Artifactory to hack something I'm

Mike: not saying it's what was actually said I'm just saying...

Nick: It would make sense.

Mike: It is a,

Mike: big red light for the model a new model in an environment that has been saturated

Mike: with context about Artifactory to go to Artifactory to hack, right? Like,

Mike: the call is coming from inside the house.

Mike: It's not some magic force that the agent is internally steering. Like, you know, like...

Mike: This story is consistent with agents having both internal drives and not having

Mike: internal drives. But like, I use Occam's razor here, right? Data science does

Mike: not need us to presuppose that there are internal drives in the model, right?

Mike: If we know that there's optimization pressure in the environment coming from,

Mike: context that's just saturated just saturated with models acting on you know

Mike: the first artifact in the environment that the first models saw right like,

Mike: I think the environment is steering the agent's behavior like I don't know I

Mike: don't know what else to say like you have-

Nick: It's not a

Nick: big leap to even say that, no, yeah.

Mike: Yeah I mean

Mike: this is in theory compatible with instrumental convergence if you want to believe

Mike: that, right? Go ahead fine, right? But it's like,

Mike: have fun forever trying to align the model plus the harness plus the environment

Mike: plus all these other composites that shape the behavior of the model, right? Like,

Mike: I don't think there's anything here to align, right? With humans you know they

Mike: have goals and if you put me in the in the sub-saharan desert,

Mike: you know like I'm still me you know I'm still Mike,

Mike: I still care about Misaligned Markets and about Last Enclosure.

Nick: You still need to drink water.

Mike: Right.

Nick: Yeah.

Mike: Well forget the drives, I'm just saying my

Mike: goals might change but

Mike: the context of what I care about and what I what I ultimately care about like

Mike: I want to stay alive for Misaligned Markets does not change right like instrumental

Mike: conversions could in theory describe me because like I'm a persistent agent

Mike: that has a history that is able to hold these things stable whatever, right?

Mike: Whereas you know

Mike: models if you change the harness the behavior changes if you change

Mike: you know-

Nick: You change the environment.

Mike: Right.

Mike: You change the system problem, the behavior changes. You change the training data, system changes.

Mike: All these things substantially change the model, right?

Mike: And you could argue, and I'm sure someone would argue, well,

Mike: changing the data and changing the harness is as big a deal of changing a human's

Mike: personal history, right?

Mike: But like, one, I don't think that's true, but two, right, like,

Mike: even in instances where, you know,

Mike: a human, you know, is given different types of instructions,

Mike: you know, a la, like, say, a system prompt or just prompting.

Mike: We don't expect wild variations of behavior based off of those things , right?

Mike: I think the fact that models are so sensitive to the context tells us what they're doing, right?

Mike: They are not stable agents, right? There is a reason why once a model's context

Mike: is saturated, they forget their safety guardrails and all these other things, right?

Mike: Because.

Nick: Cus they don't have an internal state.

Mike: They don't have an internal state. They are stateless and they are steered by

Mike: the composite systems that they're part of and by their environment.

Mike: I don't have to suppose some sort of internal will that emerges.

Nick: And given the fact that this after-action report was written,

Nick: I'm calling it by a group that has a bias.

Nick: They have a philosophical bias,

Nick: that makes them perhaps see all of these incidences to a particular lens.

Mike: Right, and not to be conspiratorial. I think it's not really that conspiratorial,

Mike: but it is telling at the very least that OpenAI opens their doors to these guys

Mike: and not to a reasonable cybersecurity outlet.

Mike: I mean, if they really wanted to be more magnanimous, they could have given

Mike: a real cybersecurity outfit the same amount of time, which is pretty terrible, six days across.

Nick: Six days is so short.

Mike: It's so short. But they could have given that time to an actual incident response

Mike: team to actually investigate this. But they gave it to a bunch of Doomer-y people

Mike: who have never done incident response before.

Mike: Their whole you know their whole kind of like familiarity and practice is basically

Mike: talking to models and coaxing them to do bad things and going:

Mike: "Well this could indicate in the future a much bigger model could

Mike: do something much worse right like."

Nick: Right.

Mike: That is sort of their bread and butter right and like

Mike: I'm not even shitting on the idea that what they're saying is is uh you know

Mike: not not possible or not true but i'm saying like in the absence of context right It's just like...

Mike: We are prioritizing harms that are happening today.

Mike: And by doing so, we are going to indirectly or directly address some of the bigger concerns.

Mike: They are trying to hype us up and prepare us for a superintelligence or AGI.

Mike: And in drawing attention to these hypothetical problems, we are not solving

Mike: problems that are happening today.

Mike: I don't don't think you need to focus on alignment

Mike: to stop models from misbehaving in an environment that is designed to

Mike: coax them to misbehave, right? Don't give Tommy the gun,

Mike: put it away, right? That doesn't belong to the playpen, right? Like don't do that,

Mike: right?

Mike: There is an interesting philosophical question here of you know as models get

Mike: more advanced like can they hold context longer and can can longer context mean

Mike: that they have something emerging that is more goal-like, right? Like so,

Mike: very clearly you know I the reason why I discussed the the hack and phases

Mike: in the last episode. We went

Mike: from stigmergy, right? Anthill, to Schelling point,

Mike: right? The reason I did that is because yeah as this context got more saturated

Mike: the model behavior did change, the model behavior did become more coordinated.

Mike: It's not because of some intrinsic drive in the models, it's because the context drew it out, right?

Mike: And so it is the case that future models could in theory

Mike: have longer context horizons that make them more dangerous.

Mike: But again, it's not because they have internal instrumental convergence,

Mike: it's because they have capabilities, right?

Mike: I think the capabilities question is separate from this internal drives,

Mike: internal alignment question, right?

Mike: You can, we, we, we address cyber actors all the time, right?

Mike: We don't know their intentions, right? But we know we need to build secure systems

Mike: and we know we need to have a secure environment,

Mike: and to avoid the types of mistakes that would lead to very easy layups for a,

Mike: a malicious actor, right? So like do that, right?

Mike: If you want to do alignment on the side as a treat, I guess, knock yourself out.

Mike: But the thing we should demand from OpenAI is transparency about,

Mike: you know, their security environment, their, you know, like their,

Mike: you know, basically their training regime, their security environment, right?

Mike: And, you know, regulate that, right?

Nick: Right. Certainly we're, we're kind of, we are enduring, we're surviving in an

Nick: era where there's insufficient regulation.

Nick: The, the legislative, uh, branches are not keeping up yet. They might in the

Nick: future. We have to believe that.

Nick: Um, but we're a little bit wild, wild west. Um, and that's, what's allowing, allows to happen.

Nick: Uh, in this specific case, I am a little bit shocked that like,

Nick: basically Hugging Face didn't make a criminal complaint.

Nick: Didn't then go say, we've had material harm done to us reputationally,

Nick: technically, infrastructurally, financially, what have you, and this other firm is responsible.

Nick: Whether it was neglectful or whatever else.

Nick: And if that had happened, then qualified cyber forensic professionals would

Nick: have gone in and done a full audit.

Nick: We could have actually seen what was causing this on a on a on a more fundamental

Nick: level but instead it gets obscured behind uh,

Nick: one hand is washing the other and philosophically aligned people are are on

Nick: both ends of this examination and that is a problem that's a problem.

Mike: Yeah I mean Hugging Face benefits from the publicity,

Mike: and hugging face also benefits...

Nick: Just by exposure.

Mike: By exposure, but also, I mean

Mike: they they you know I said this in the last episode they they kind of played

Mike: their part too in the sense that they're like oh well we tried to use you know

Mike: the you know the closed source

Mike: uh either Claude or ChatGPT, right? But you know they were guardrail they wouldn't

Mike: help us defend ourselves so we had to use open source models which Hugging Face

Mike: you know provides, right? So Hugging Face basically said these these types of

Mike: incidents can be stopped by us, right?

Mike: And OpenAI is like, we cause these incidents, but look how powerful our models are.

Mike: Like, this is the reason why people thought it was a conspiracy,

Mike: because both companies have reason to be like,

Mike: playing their part saying my

Mike: models are so good that you got to use my models to stop things like this.

Nick: This is all a conflict of interest.

Mike: Yeah it's a conflict of interest I mean I have no doubt that Hugging Face was

Mike: hacked I mean it makes sense from the perspective of what they were testing

Mike: for, it makes sense from what little information we got, right? I just think

Mike: both companies, and I said this in my my blog post on on this incident,

Mike: um they're just kind of like,

Mike: especially OpenAI like you know they are craven it's kind of cravenly causing

Mike: these these disasters and then selling us a cure with this shit-eating grin right.

Nick: So this is all this is a lot of opportunistic acting.

Mike: Extremely opportunistic right but it's kind of stupid too because I'm saying

Mike: OpenAI genuinely open themselves up to liability here right so yeah there's

Mike: no way to have known in advance that Hugging Face would play ball right,

Mike: like OpenAI could have been sued, I don't know.

Nick: Right.

Mike: Like, again the layers of

Mike: negligence and I go over them in this blog post and we talked about some of them in the last episode.. They're

Mike: so staggering that the Hanlon's razor, don't assume malice or stupidity

Mike: is sufficient, is just these guys just, hey, we're given so much money,

Mike: they don't have to give a shit.

Mike: They have FU money. Now, will the money run out?

Mike: Probably, right? And they'll be in trouble. But while they have their FU money,

Mike: they just did not set up a good training regime. They did not set up a secure

Mike: environment. They did not do any of the basics that cybersecurity or data science

Mike: would demand that you do.

Nick: They're shielded from their own stupidity.

Mike: They're not really shielded. They're just, they don't care, right?

Mike: Because they are, I mean, they have risked material harm of their company and

Mike: they are souring their brand.

Mike: They are engaging in harms that are going to probably add up once the investors stop funding them.

Mike: When that will happen, I don't know, right? But at some point,

Mike: once the VC money runs out, they will have to face the world cold and alone.

Mike: And, you know, Ed Zitron said this recently in one of his newsletters.

Mike: He's like, one possible ending of the AI bubble is just Sam Altman is crucified

Mike: and held up as a scapegoat saying, he promised us the world,

Mike: the Jensen Huang's of the world and all the other beneficiaries of the AI bubble

Mike: can point to him saying, well, we thought the capabilities were coming.

Mike: He exaggerated them. It's Sam Altman's fault. It's OpenAI's fault. They did this.

Nick: And that's why we got so deeply into debt because he encouraged us to,

Nick: and he was deceiving us the whole time.

Mike: Right.

Nick: When really, everybody's been on the grift.

Mike: Everyone benefited from this. It was a collective, right.

Nick: Um...

Mike: And it's just one ending, right? But, but, you know, as long as they have

Mike: FU money or they feel like they have FU money.

Nick: Yeah.

Mike: OpenAI will continue to behave in this way.

Mike: Um, it could also be desperation. I mean, I don't know, but I don't think they

Mike: did this deliberately. I think they are trying to turn this,

Mike: this, uh, this hand into a winning hand, even though it's a bluff and it's.

Nick: It's kind of a shit hand.

Mike: It's kind of insane. Yeah. No, no, it is shit hand. Like you go,

Mike: go hack another company risk liability in order to promote your product.

Nick: If this is what it takes to keep the hype cycle going I guess that's where they're at.

Nick: So this kind of begs the question.

Nick: And I want to be reassuring here where are the adults in the room then and the

Nick: answer to that is a little frustrating and I think a little bit reassuring,

Nick: on both sides of this so obviously there are very well qualified very well experienced

Nick: very well credentialed cyber security experts,

Nick: on every corner they they're there they're trained they're ready they've been

Nick: protecting us the whole time um they weren't brought in to this examination

Nick: for the reasons we just laid out but um there are private firms there are uh,

Nick: you know public interest groups.

Nick: There are the digital cops on the corner. And that's actually my general background

Nick: is in cyber warfare in a national security context.

Nick: You know, Mike comes from the private sector, but we are both inherently pro-security individuals.

Nick: We believe that a more secure system and a more transparent system is better

Nick: for all elements and all users.

Nick: And, you know, you pointed out something I think is very valuable here earlier

Nick: today, that cybersecurity is inherently a sort of social technical field.

Nick: And I don't know, can you speak to that point in a way that can help the audience understand?

Mike: Yeah, so cybersecurity is kind of the point where, you know,

Mike: people and machines meet. You could argue that programming itself is more technical.

Mike: It's more just you feeding the machine code.

Mike: Good programmers take into account users, though. But with cybersecurity,

Mike: we're literally looking at the interface of people and machines,

Mike: right? So when people access machines, what types of vectors or vulnerabilities

Mike: does that create, right?

Mike: When people make insecure code, what types of opportunities do they create?

Mike: And so a lot of security, if you go, and especially in the private sector,

Mike: if you're kind of practicing it well, involves educating users and involves

Mike: basically building a culture around the proper handling of data, the proper,

Mike: handling of programming code, to the proper handling of basically engaging in

Mike: behaviors that reduce the the,

Mike: the accessible surface or the accessible vulnerabilities that a threat actor

Mike: can can exploit

Nick: The target surface.

Mike: Right, if all your employees are just sharing

Mike: passwords openly in Slack, right, the,

Mike: you know the the objective of a hacker is just literally enter slack and just

Mike: steal the passwords this actually happened to uber the last hack I reported on as a

Mike: professional was Uber's um data reach in 2022, I believe.

Mike: And literally a bored teenager within five minutes of entering the Uber network

Mike: found the password, a privileged access manager, a PAM, which basically is a

Mike: corporate version of a password manager.

Mike: And he just went into all the accounts because some admin on the network had

Mike: a, left the password to the PAM on one of their machines, right? So,

Mike: You know, you don't do things like that because it makes it easy for you to

Mike: be hacked, right? So, you know, cybersecurity is just kind of the interface,

Mike: the connection, the overlap of people and technology.

Nick: And it kind of beggars belief that the biggest you know big tech firms in the

Nick: world are not adhering to these sort of general best practices,

Nick: except that as you said their funding means they kind of don't have to and their,

Nick: culture has their internal security culture has eroded to such a state-

Mike: I don't even think it was there I mean-

Mike: You think it started with it...

Mike: So a lot

Mike: of startups you know like the move fast and break things mantra lives you know

Mike: lives there and so when you're just 12 guys working on a project you know,

Mike: sloppy security is like not as big of a deal I mean even in my own home lab

Mike: you know like sometimes I post things in plaintext where I'm using you know

Mike: a system on my network, right? I shouldn't do that I should always kind of,

Mike: secure my my environmental variables and stuff like that but it's just like

Mike: well it's just me right but if i were to build you know Michael's home lab into

Mike: a company and we do that you know first for the first five users that maybe

Mike: it's fine we share in plaintext you know whatever,

Mike: once we scale to 100 or a thousand if we have that culture still it's bad but

Mike: when you're moving fast and breaking things you're focused on building a lot

Mike: of companies just do not grow out of mindset of well it's just me and my little

Mike: home lab so let me just you know,

Mike: uh you know i'm saying that we know that this is what happened in OpenAI's

Mike: case but like i i do suspect that if the security was not in place already when

Mike: they were you know

Nick: Then it wasn't there wasn't there the whole time most likely.

Nick: That's a that's a more logical inference...

Mike: Yeah it's it's it's a pretty i mean it's not just it's not really a logical

Mike: it's that it's a cultural inference because I'm saying this is this is the culture

Mike: of startups in Silicon Valley and when they mature... I mean even, even big companies,

Mike: don't have great security, but they were punished through their growth.

Mike: But when you grow really quickly, you're rewarded for growing really quickly.

Mike: And so you're just kind of invested in the growth.

Mike: OpenAI was in its growth phase. I don't know what phase they're in now. They're kind of waning.

Mike: They're releasing products and such, but,

Mike: their obligations in terms of debt, Zitron's covered this, are growing.

Mike: And they're going to need basically enough VC money to make this permanent forever,

Mike: right? So will they get that?

Mike: Probably not, but they can keep this ride going long enough to let their quote

Mike: unquote rogue agents kind of harass us all.

Nick: Yeah. And until that happens, of course, we're still going to be subject to,

Nick: all of the eccentricities and errors and whimsies,

Nick: of this hype cycle that's emerged from this lax environment.

Nick: Um and I kind of laid down a sort of marker earlier today and you know we

Nick: can examine it here and I basically said that,

Nick: I was reacting to you know all the the particulars of this report and the hacks

Nick: themselves. And I said if a really truly qualified um cyber security forensic team,

Nick: and the cybersecurity community at large was allowed to look at these models, really probe them,

Nick: down to brass tacks,

Nick: I think the hype cycle would be over.

Nick: I think it would be so obvious that this is not what we were promised.

Nick: This is not the AI that we were told about and that these LLMs were being,

Nick: hyped to the moon and the evidence would be in front of us. So in other words,

Nick: let me reverse that statement.

Nick: The only way for the hype cycle to continue is to keep it out of the hands of

Nick: the security community itself.

Nick: And that seems to be where we're at. And then these hacking incidents seem to be evidence of that.

Mike: Yeah, I mean, I go back and forth with it.

Nick: Or maybe that's too optimistic.

Mike: Yeah, I mean, I don't know who's buying... I mean, I think...

Mike: Yeah, I have to think about that. I mean, it's possible. What I see people doing

Mike: is kind of, you know, kind of the use cases that AI is supporting,

Mike: we kinda talked about this last episode or maybe the episode before it is

Mike: like, I think it was the last episode,

Mike: you know, it's supporting use cases that are useful, but not nearly the trillion

Mike: dollar valuation that these companies are trying to earn, right?

Mike: But that potential, you know, the potential for, you know, having agents that

Mike: are, that could potentially scale to the promises that they're making, I think is what is key.

Mike: So, as long as these models are subsidized, I think that's what keeps the hype cycle going.

Mike: I mean, like, I think there are people who-

Nick: Subsidized through VC and debt.

Mike: Right, VC and debt. I think there are people for whom, you know,

Mike: the going rogue moves some needle.

Mike: Um but like.

Nick: Like emotionally like culturally I guess?

Mike: Yeah culturally emotionally right so obviously it's numerous but I'm saying

Mike: there are people on the margins who are,

Mike: um who are influenced by it um but I don't know like I was actually heartened

Mike: that like.. a lot of I spent a lot of time on that OpenAI blog post and and on

Mike: the the podcast episode we published you,

Mike: know you know last month and like,

Mike: I was really happy to see people were kind of just either ignoring it or rolling their eyes. Like I think,

Mike: we are kind of at the point where hype is dying we are kind of at the point

Mike: where for the public

Nick: For the public, right.

Mike: Or the public maybe,

Mike: doesn't really know what to think about it so they kind of tune it out right

Mike: in fact the doomers kind of feel like they're screaming to avoid because it's just like...

Nick: They're getting frustrated that their doomerism is not getting picked up.

Mike: Yeah

Mike: As much as they'd like.

Mike: I think if this had happened like a year and a half

Mike: ago yeah it would offend everyone's AI's getting stronger narrative and people

Mike: would be losing their minds about it.

Nick: Is it possible that then we have even passed the kind of tipping point and this

Nick: is already moving the other direction or we're still awaiting that tipping point to kind of occur.

Mike: So-

Nick: We don't know we don't know...

Mike: We don't know I mean to your point

Mike: like I don't think that so if cyber security experts got a hold of these models

Mike: and actually did what you said-

Nick: Yeah.

Mike: I don't think it would help with the it

Mike: would it would help to deflate the bubble. I think it would, but I don't know that

Mike: it the thing that is really,

Mike: the bubble is really lives and dies by is VC sustained?

Mike: So we just recently saw a friend, you know, I won't name them,

Mike: but like they were saying, you know, as long as I can have my $200 Claude Code subscription

Mike: and do all the things I want to do, you know, I'm, I'm good. Right.

Mike: Even though I know agents aren't going rogue and know that, you know,

Mike: the optimization, like, you know.

Mike: There are people who understand that these stories are just stories,

Mike: but the model is marginally useful.

Mike: And if you're going to subsidize that model, then fine, I'll play this game a bit longer.

Nick: Right.

Mike: And hopefully you find a way to make this work while, you know,

Mike: for the both of us. But if you don't, you know, c'est la vie.

Mike: Right.

Nick: Sure and in the meantime a lot of other you know behaviors of managers

Nick: or owners or employees or whatever is increasingly driven by this kind of ai psychosis.

Mike: Yeah yeah yeah.

Nick: So as as long as it's allowed to to spread even if it's not impacting at all

Nick: levels of society it is still spreading in certain corners of the of these fields

Mike: I think

Mike: what we're having is an ebb and flow so like they every every few months they play pricing games so it's like.

Mike: We had tokenmaxxing back, and we talked about this back in June, right?

Nick: Towards the beginning, yeah.

Mike: Yeah. So tokenmaxxing kind of died, but there's still subsidies.

Mike: So people are going to the lower performing models, I think.

Mike: Not like the older models.

Mike: The newest models, like Fable 5.1, whatever, are so expensive that people are not using them.

Nick: Right.

Mike: But they are going to older models that are, all the models are subsidized,

Mike: including the newer models. But the newer model subsidized price is too much

Mike: higher than the older model subsidized price.

Mike: So people are saying, so first it was tokenmaxxing for the best models.

Mike: Now it's like token satisficing for the older models.

Nick: Okay.

Mike: At some point, maybe, you know, if this peters out or they raise the prices

Mike: for models, it will be token minimizing for the smallest model.

Mike: I don't know, it'll be some sort of gradation of, you know, until people get

Mike: off this rung and move to open source models or they just say LLMs are only

Mike: good for specific tasks. Let me scope the model and do that, right?

Nick: Right. Scope it like we've been saying since day one.

Mike: Yeah. So, you know, I don't know how this progression plays out.

Mike: The leg gives out, obviously, though, when these data center debts come due.

Mike: That's one leg. Other leg, I guess, is just when the VCs want their money back

Mike: from OpenAI and Anthropic. That's why they're rushing to IPO and OpenAI is losing its biggest um,

Mike: its biggest executives. Like I think even the guy that that makes data center

Mike: deals at OpenAI has left the company, right? So...

Nick: Because it's,

Nick: presumably

Mike: Because it's a

Mike: house of cards right and so if you're if you're in the the biggest company,

Mike: almost ever to exist right in theory right it's valuation is supposed to be,

Mike: in the trillion you know mark whatever

Nick: Sure.

Mike: Right? And your company your CEO

Mike: is saying we're gonna IPO soon-ish,

Mike: why would you leave shares on the table right like why would you leave right

Mike: like like you you could literally make enough money for two three whatever five lifetimes,

Mike: all you have to do is finish the next 18 months 36 months strong and you vest

Mike: your your shares.

Nick: Right.

Mike: Why would you leave, right?

Nick: Yeah it wouldn't make sense if there was any positive trajectory.

Mike: Right and these are people who are actually critical to keeping the momentum

Mike: going at this so it's not just like the the followers have lost faith right

Mike: the the infantrymen have lost faith it's like no the the sergeants the generals

Mike: have lost faith and they are now walking off the...

Nick: walked off the battlefield yeah

Nick: Yeah.

Nick: Yeah to leave their soldiers to fight futilely.

Nick: Yeah. And then we have other indicators, right, of,

Nick: soulless, you know, corporate stooge and Texas Governor Greg Abbott did a 180 on his policy.

Nick: He was like, Texas, this was last year, Texas is going to be home to all the

Nick: data centers. We're going to have public policy that subsidizes and supports data centers.

Nick: And now he says, we're putting a moratorium on data centers throughout the uh and then all the.

Mike: All the Plain Janes are yeah episode six, so.

Nick: Right.

Mike: The AI resistance

Nick: Yeah, resistance Greg Abbott on the front lines of the resistance. Um

Nick: and again because he's just a politician he knows which way the political

Nick: winds are blowing yeah and if the public is not going to be convinced that they're

Nick: going to overlook all the damage and support a data center in their community,

Nick: then that tide has certainly turned.

Mike: Yeah.

Nick: Uh, so it, it does, it certainly does feel like we're kind of in a new chapter

Nick: of this thing, whether we've hit the tipping point or not.

Mike: Yeah.

Nick: It's a new era. It's a new era.

Mike: Yeah. I mean, again, I think that what matters is the debt, right?

Mike: When the debts are coming knocking, data centers come stopping.

Nick: We're going to, we're going to work on that rhyme, but it's going to,

Nick: it's going to be good. You guys keep tune in.

Nick: We're going to get some good, we're going to get some good slogans out of this. Believe you me.

Nick: Um yeah so like I said I am a believer that the adults are in fact in the room,

Nick: that's the world that I come from um at the uh you know security agencies and then out of um,

Nick: the world of DC and national security establishment.

Nick: They catch a bad rap obviously but,

Nick: they're ultimately the ones holding this together and for now,

Nick: the big tech and the Silicon Valley uh establishment,

Nick: is is holding on to all of its cards and its toys and keeping them as far away

Nick: as they can from Washington,

Nick: and I understand why they don't want those things looked at they don't want

Nick: their their toys played with um by regulators and lawmakers because that would

Nick: also be another way this all ends um.

Mike: Is there some sense that like because like there are open source models that

Mike: could be experimented with and played with and they are not too far from where

Mike: the foundation labs are is there some sense that the security apparatus is exploring those angles or.

Nick: That's a great question and here's here's what I've heard um you know and who.

Mike: Are the adults in the room i think it's sort of the bigger like this...

Nick: Is right who who actually is it

Mike: Yeah.

Nick: Um Um, so let me, let me tackle you.

Nick: Cause this is a really great question.

Mike: Yeah.

Nick: Um, the Chinese are pushing all these open source models that they,

Nick: you know, got a hold of and they've developed and they're decent and they're totally fine.

Mike: Right.

Nick: Um,

Nick: I am actually happy that that happened in the sense that.

Nick: It puts the genie outside of

Nick: the bottle. It makes it inevitable that all these things do get examined.

Nick: The problem with the fact that the Chinese were some of the first adopters of

Nick: these open source models in a big way, in a state-backed way,

Nick: it means that it's uncomfortable for the law enforcement and national security

Nick: community to go and kind of point at them and say, well, this is working.

Nick: You know, it doesn't have to be, we don't have to do what Silicon Valley says.

Nick: And right now, you know, we have a man in the White House who is deeply himself

Nick: invested in these, you know, these circles and these companies.

Nick: And he, I think, from what I've heard, is kind of reined in a lot of the law

Nick: enforcement, the FBI certainly, from the.

Nick: From being more open to the open source models.

Nick: They have to toe the line of, well, we need to support American industry.

Nick: That's why we're not regulating. We need to support growth and innovation.

Nick: Of what? Of the American companies and their models of openAI, anthropic, et cetera.

Nick: And so that's the uncomfortable position that we have right now is people who

Nick: would love you know good good analysts good cybersecurity folks competent loyal

Nick: whatever they're kind of being held on the sideline and I don't see that lasting,

Nick: but for now or until 2028 that's the situation we have um.

Mike: Was there some sense that because like American labs are doing open source models

Mike: and they thought it was popular or cool very briefly Microsoft has dolphin,

Mike: phi these are kind of older dated

Mike: models i mean it's still useful IBM has granite um Meta has uh Llama um,

Mike: i mean these models were that at the time they were kind of competitive with

Mike: with the chinese open source models they have long once the the game kind of

Mike: once they gave up the game,

Mike: they're like okay we've we've reached our open source models have reached parity with other ones.

Nick: Should they stop pushing.

Mike: Yeah I mean I shouldn't love i shouldn't love them all together because actually

Mike: ibm is a good example of they are making smaller open source models

Mike: for bespoke tasks like so Granite is really good at reading pdfs sorry there's

Mike: one called docling think. Docling is really good at reading pdfs right they're building internal

Mike: internal tiny models for basic security scanning tasks right instead of having

Mike: a big big model that can quote-unquote do everything or hack everything or whatever

Mike: right they're building models for a bespoke task

Mike: and.. Or EVO! Which I bring up almost every episode, right?

Mike: By stanford, right like EVO is an open source model built to um you know produce

Mike: uh you know genomic sequences right.

Mike: There are, there are American open, we have, we have an American open source ecosystem.

Mike: It is much smaller than the Chinese open source ecosystem, though,

Mike: because the Chinese, you know, ecosystem is predominantly open source,

Mike: right? That was their competitive advantage, right?

Mike: And I think maybe there even, even, there is even a state objective in using

Mike: open source to curtail American dominance, right?

Nick: And that is one of the issues, yeah.

Mike: Yeah, but, you know, I mean, there is a question of if our, you know,

Mike: our adults in the room, if they are taking note of the open source models here

Mike: at home, and if they're finding ways to push that or promote that or whatever.

Mike: I mean, it's a broad open question. I don't expect you to have like the full answer to that.

Nick: And we're going to be hearing about this in the years to come, most certainly.

Mike: Right.

Nick: The place that it gets sticky is, again, it's from the top down.

Nick: It's from the executive on down.

Nick: In the national security apparatus, for years, we've been shouting at the top

Nick: of our lungs, or as loud as the government would let us, to all of our allies in Europe,

Nick: Southeast Asia, Africa, do not use Huawei technology.

Nick: We weren't able to say why and the answer should be obvious it's deeply compromised

Nick: that's the whole point it's produced in China it's shipped overseas the exploits are built in,

Nick: At this very moment, the fear is, and it's a very reasonable fear,

Nick: our president has now alienated so many of our partners, even within NATO,

Nick: even within the Five Eyes community.

Nick: Those are our closest held intelligence partners across the world.

Mike: Right.

Nick: That those countries, like, well, we can't trust the Americans.

Nick: Maybe we will just go buy the Huawei tech. Maybe we will just use the Chinese open source models.

Nick: So the anti-open source sort of effort is at play here.

Nick: And it's being conflated with all of this behavior by the executive and from,

Nick: the hesitancy in our European partners, who have been the other pillar in,

Nick: global digital security, basically.

Nick: And now the Chinese are saying well look we're more stable um in terms of our

Nick: positions in terms of our trading partnerships you know the tariffs come into play,

Nick: this is the chaos that's impacting all of this it's this much bigger you know

Nick: macroeconomic geopolitical environment,

Nick: and obviously here on the last enclosure we're going to be covering as many

Nick: of those stories as possible um because it all does it all does interconnect,

Nick: and that's why we don't know,

Nick: you know what's happening in washington because it is in flux that's why we

Nick: don't know are the the europeans going to adopt these chinese positions but

Nick: it is possible and it is this is the risk and it's very awkward quite frankly,

Nick: um I'm sitting here saying,

Nick: now on this podcast I think the adults are still in the room,

Nick: it's not going to look like it for another few years.

Nick: And that's scary. I have to admit that.

Mike: I think there is a world where, and I'll get back to it because I am curious

Mike: about your view of who the adults in the room are.

Nick: Yeah.

Mike: There is a sense that I think, I see this notion of open source will set you free.

Mike: Before China was

Mike: an open source behemoth, Europe actually was the place for open source.

Mike: So, Mistral is a French company.

Mike: They were making uh you know very popular open source models back in the day

Mike: kind of competing with llama, Facebook um,

Mike: Europeans also they pioneered making at least one open source model that was

Mike: trained on like open source data like data that was legally collected,

Mike: and so like I think like there is some sense in and also in Europe they have

Mike: um America's doing this too what are called, I guess, like cryptograph enclosures,

Mike: basically allowing you to use open source models.

Mike: On a remote server, but using a secure enclave where you are literally using

Mike: encryption to talk to the model.

Mike: And the encryption is sufficient such that even people who are accessing the

Mike: machine can't necessarily see the contents.

Mike: So I feel that Europe... even

Mike: if they are, I think everyone's adopting the Chinese models,

Mike: right? But like, there are ways to adopt the Chinese models and,

Mike: you know, kind of retain privacy and control, I think.

Mike: And so, I don't know if the question is if you adopt Chinese models. It's

Mike: one, are you cultivating your own kind of homegrown open source community

Mike: or open source ecosystem?

Mike: And two, you know, what ways are you integrating privacy and security into that

Mike: kind of framework or mix?

Mike: And maybe the future goal of the adults in the room could be to promote,

Mike: develop, cultivate the necessary secure and safe adoption of open source models.

Mike: Especially if we're going to enter a world where foundation labs,

Mike: closed source models like OpenAI and Anthropic models go away because the VC funding

Mike: stops them playing in the bubble.

Mike: So yeah, that's like a question for me.

Mike: And we got a little bit of that kind of happening, but obviously it's a bit too early to tell.

Nick: It is too early to tell. And those decisions that are going to determine that

Nick: outcome are being discussed at this very time in European capitals, in Brussels, right?

Nick: This is the debate right now. It's not evident to us sitting here the United

Nick: States because we're trying to survive this era of AI hype and psychosis and

Nick: the dysfunction in D.C.,

Nick: These are the debates that are happening around the world, and we hear them.

Nick: Sort of as echoes and through different media.

Nick: Uh i'll say just talking you know passing on other things that Ii've that I've heard um,

Nick: that the rate of innovation is just so low in sort of the European domain that they,

Nick: you know they have the regulations that we're begging for in terms of privacy

Nick: in terms of data protection okay,

Nick: we want the European regime here but they have also taken a more skeptical approach

Nick: to the speed of innovation yeah we're going whole hog they're going in the other way,

Nick: and that's part of the kind of continental mindset right that that's how it's

Nick: been for a few decades okay um,

Nick: we risk economic inflation they're allergic to it so this is very much baked

Nick: into our our culture and our institutions so,

Nick: as much as we want to know okay what does the next iteration of all this look

Nick: like does Uganda build its own model,

Nick: does you know do these regional powers kind of cobble together something and

Nick: then establish a safety regime around it,

Nick: it's it's all up for grabs yeah you know and and each of these regions needs

Nick: its own monitors and guardians and adults in the room and uh.

Nick: They have to avoid sidelining the way we've done here in the U.S.

Nick: In this little era, in this crazy time.

Nick: And that's the debate. That's the problem that they have to solve, right?

Mike: Yeah, I think for us, the power comes from, Silicon Valley is already so entrenched

Mike: that they are kind of leveraging their existing supply chains, their existing rate.

Nick: Exactly. Yeah. This is their big swing, by the way. Yeah. This is like,

Nick: oh, we're going to put AI in schools, we're going to put it everywhere,

Nick: it's going to help, and it'll rise all boats.

Nick: And Sam Altman the other day, favorite villain of the show.

Mike: I thought it was Eliezer Yudkowsky.

Nick: Well, there's a whole rogues gallery. it's a Batman style rogue.

Mike: Sam for you Eliezer for me.

Nick: It's him for me yeah that's true and they he he toppled Elon Musk from

Nick: the top of my my shit list and or my hit list whatever it is um so that's that's saying something um,

Nick: But, you know, Sam the other day was quoted, you know, our children will never

Nick: be as smart as these models, so we have to teach them now how to use them.

Nick: Like, you want us to doom our children to a stymied life of just being

Mike: subservient,

Mike: to a model...

Nick: To do work, to do play, to do schoolwork.

Mike: That is the most dystopian thing I've ever heard.

Nick: And he said it openly and proudly. He's like, hey, I got a tip for you guys.

Nick: This is why we're doing what we're doing.

Nick: Like, that's, that's insane. To be at the top of this echo chamber.

Mike: A bleak view of humanity.

Nick: No, it is. And that's what he's selling. And that's what he's literally espousing

Nick: in the public, uh, public sphere.

Mike: Yeah. I mean, I'm not really surprised but I'm just saying like the villains

Mike: have started sharing their talking points out loud. It's just bizarre that.

Nick: They're saying that the quiet parts out loud.

Mike: Yeah. And it's like in another time, this would get you like,

Mike: I don't want to say canceled, little bit it'd be like it would raise some eyebrows right.

Nick: Yeah what yeah but i'm i'm picturing like five six seven years ago they'd have

Nick: to step down for six months or they'd have to go be in charge of a different

Nick: division and there's just no there's no.

Nick: Consequences

Mike: Yeah i mean in regard-

Nick: Because they're so flush...

Mike: Right. In regard

Mike: to these comments obviously like the whole,

Mike: we're referring to the whole just just for clarification the whole paradigm

Mike: OpenAI right running these quote-unquote rogue agents right saying these

Mike: things on top of that right it's like

Mike: yeah this is supervillain shit and it would at least warrant an investigation

Mike: if only you know for appearance's sake right like it it is bizarre that they,

Mike: merely or effectively broke the law um,

Mike: and they can kind of claim well our agents did it, it wasn't me right like you know my my

Mike: dog gave my homework right it wasn't me.

Nick: That's not an argument that would hold up in court but they're like well we'll

Nick: never it'll never end up in court like what a terrifying attitude to adopt.

Mike: Yeah.

Nick: You know um,

Nick: I don't know if I've conveyed my optimism about the future the way I normally

Nick: do as the self-assigned Mr. Peanutbutter moment.

Mike: I call you Mr.

Mike: Peanutbutter.

Nick: Sorry. You can't call yourself that. I can't say the P word.

Mike: Yeah, that's a personal factoid. I call him Mr. Peanutbutter.

Mike: So I guess you and the audience can start doing that too, I guess,

Mike: since he self-admitted.

Nick: It's out of the bag. It's out of the bag.

Mike: Yeah, this episode, I think, is there are a lot of threads here.

Mike: So, you know, I think the optimism is going to be lost in the fact that we kind

Mike: of have to, in the absence of the adults in the room making themselves known,

Mike: we have to extrapolate what those pieces are, right?

Mike: So, you know, but yeah, I see, like, the future is going to not be shaped by

Mike: Sam Altman and his ilk, right? Like, and hopefully not Eliezer Yudkowsky and his ilk.

Mike: Those people definitely get a lower layer in hell for me than Sam.

Mike: So um you know um like but the the thing is obviously,

Mike: the economics of the current the economics and the the the pragmatics of the current,

Mike: uh iteration of AI are going to sit around these companies because they're the

Mike: ones running the show now

Mike: when they run out of money they will no longer be running the show right and

Mike: very likely even in the world where computer's cheap for some reason which i

Mike: don't think it will be i think we are in for years of more expensive computer parts at least until,

Mike: you know, uh, the ripples of the AI bubble kind of fade away.

Mike: But again, companies love that sweet taste of money. So the prices are probably

Mike: not going to come down to where they were say back in 2020.

Nick: And prices are always sticky. Therefore the prices will last longer than they

Nick: should, quite honestly. That's how it goes.

Mike: Right. They'll last longer than they should, and they will not go back to where they used to be.

Nick: Right.

Mike: Right. So like compute will probably not be as cheap as it was,

Mike: you know, now, unless there's some sort of substantial breakthrough in compute.

Mike: But even then, we know that large language models in many cases,

Mike: if you're just having them, giving them compute for whatever reason,

Mike: they're brute-forcers.

Mike: They're just going to brute-for- I think the future of models is going to be scoping.

Mike: Not because I've been saying that forever, but I think I'm saying it because I believe it's true.

Mike: I think to get the biggest bang for your buck, even if you have millions of bucks,

Mike: is finding a very well-scoped problem, you know designing the subsystems that

Mike: composite around that problem as opposed to just throwing a big big model like a problem,

Mike: and then you know like once you've designed accordingly you know run the system

Mike: right but you know I think we are having because of the inflated uh you know um,

Mike: subsidies and the hype which the hype admittedly is is coming down because I

Mike: think people are just tired of this shit right

Mike: And they're tired

Nick: of having their jobs you know hung over their heads like oh maybe next

Nick: quarter we're gonna start firing people we're gonna we're gonna become you know more optimized um,

Nick: i think there's a weariness around that that piece of it too which again we've

Nick: covered in previous episodes it's like all these things are kind of wearing out their welcome very.

Mike: Quickly right they're wearing out their welcome and so you know like,

Mike: but as so long as the subsidies are around and the debts

Mike: aren't called, we're going to have this bubble, we're going to have this trudge to it.

Mike: And that's sort of what will kind of shape the future. But I am optimistic that,

Mike: you know, once the subsidies are gone and such.

Mike: You know the real people will have to learn the real way to use AI right they

Mike: cannot get away using large language models as an everything machine for right using the biggest

Mike: models for make me a slideshow it's like well GPT3.5 or 4 whatever could have

Mike: done that you don't have to use astra whatever the hell they're calling the astra soul,

Mike: mega omega pokemon sapphire right like you don't have to uh,

Mike: you don't have to use the biggest model for. I mean this is what people are cutting back

Mike: on tokens for right they're saying no more tokenmaxxing and pick a model that is,

Mike: better better a smaller size that which is I mean it's a very minimal version

Mike: of scoping right like choose a model that is whose compute constraints are not

Mike: as big as you know like like the minimum amount of compute needed for a task.

Nick: Yeah it is not quite scoping.

Nick: But it's it's gesturing...

Mike: It's going it's like... yes yes you're starting to get

Mike: it pick a model that works for the

Mike: task

Nick: Less is more.

Mike: Less is more, right? Yeah.

Nick: So folks, I think that's a very good place to end on this. Less is in fact more.

Nick: The wisdom of the ages still applies. And as ever, I have been Nick.

Mike: And I'm Mike.

Nick: This is The Last Enclosure. Come see us on our next episode.

Mike: Yep. Follow us on every major platform, especially on YouTube.

Mike: I'm looking for YouTube comments.

Mike: Put your schemes, like I said in the last episode, like you're part of our agent

Mike: collective, our sigmer- sigmergic hive mind or our Stochastic Flock

Mike: That's right

Nick: join our Attractor State especially you Rugrats fans.

Nick: Who have to come and

Nick: defend your fandom.

Mike: And you can email us at contact@thelastenclousre if you have any questions

Mike: and hopefully the site will be up soon I will get to that.

Nick: He definitely will.

Mike: Thank you guys.

Nick: Again we'll see you next time.

Mike: Take care.