Rendered at 16:40:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
atleastoptimal 21 hours ago [-]
I think Anthropic's agenda is very clear (whether or not you disagree with it)
1. Show unambiguously that models are capable of disruptive capabilities, irreversible damage, and misaligned behavior
2. Show that open source models are the only models where there is no guarantee misaligned behavior is prevented
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
eggplantemoji69 21 hours ago [-]
Yes this seems most likely. Regulatory capture, essentially, if they can get legislation made around this. Also serves as a convenient way/reason to stop training new models ($$$) by calling for a slowdown.
patcon 20 hours ago [-]
Yes, but to be clear: Nuclear non-proliferation was regulatory capture.
pritambaral 20 hours ago [-]
There were five nuclear states grandfathered in when the NPT was signed. Since then, five other countries have developed nuclear weapons.
Apart from South Africa (which gave up its nukes voluntarily), the other contenders were basically only held back due to military intervention (Iraq, Libya, Syria) or economic sanctions (Iran).
f38zf5vdt 20 hours ago [-]
How's that one going for humanity in 2026?
patcon 20 hours ago [-]
From what I understand, it has gone great if your atoms exist on the other side of the cold war, no?
f38zf5vdt 20 hours ago [-]
Verdict is out on whether or not they'll survive after that. START expired this year and the powers that be are going gangbusters into nuclear proliferation. It always made the assumption of benevolence for the regulatory organizations, which probably won't survive the long tail.
ryanmcgarvey 20 hours ago [-]
Is it still regulatory capture if it also happens to be true?
I'll note that, if it is true, the only other alternative is to ensure liability is held by whoever trained and/or provided the inference for the damage done by the model. This would effectively shut down open-weight models anyway.
ben_w 20 hours ago [-]
> Is it still regulatory capture if it also happens to be true?
To an extent, yes. For example, aircraft: one can say something broadly true like "it is important that planes do not fall out of the sky" while also calling for not-strictly-necessary regulations that happen to support your own business like "we need to be able to inspect factories at any time for compliance, so the factories must be local to our nation".
dwaltrip 18 hours ago [-]
Then it’s just regulation working properly.
ben_w 20 hours ago [-]
Might be. But it's easy to reach the same result without it being about power.
They (like many, though probably not all, of the other main labs) think they have a better shot at getting their AI to be aligned with human values than anyone else does. Obviously they can't all be right about that, but they can all think that, just like most drivers think they're above average at driving.
So:
1. We make safer AI than anyone else; "safer" is a matter of degree even if nobody is perfect.
2. That includes all the open models.
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
tim333 5 hours ago [-]
I don't agree with point 2 on. They don't seem to want to ban open source or have much hope of doing so.
Seeing if models can do damage seems more like sensible safety research.
My worry about AI software doing bad things would be more the kind of people who make existing malware. The ransomware comes from criminals, not the open source community.
sigmar 20 hours ago [-]
Is it possible this post is the consequence of anthropic hiring people that are safety-minded?
I don't think the two authors were told about a super secret mission.
atleastoptimal 19 hours ago [-]
It's not a secret mission. Dario and Anthropic in aggregate are anti-open model due to safety reasons (coupled with anti-China sentiment due to their distillation/hacking).
This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.
f38zf5vdt 20 hours ago [-]
Waiting patiently for Anthropic's next month article, "LLMs made a fully functional intercontinental ballistic missile. We accidentally wiped Greece off the face of the Earth in the process. Tragically, we are not responsible."
DoctorOetker 18 hours ago [-]
Caesar should accept the crown, reluctantly.
amarcheschi 20 hours ago [-]
Technically, Claude did it, not Anthropic
pirateface 11 hours ago [-]
This is exactly what they're trying to do, and it will probably work.
kenjackson 20 hours ago [-]
Problem, they haven’t shown that the labs can guarantee misaligned behavior either. And to this date have not shown a credible plan to do so.
Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?
Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.
atleastoptimal 18 hours ago [-]
I don't see how there won't be a massive anti-AI populist movement in 2027.
The messaging from labs is basically
>Yes AI will take away your ability to earn a wage from your skills
>In the process we become billionaires, you have to figure out how to survive
>Also AI will cause endless hacks, disruption, scams
>Only we can handle AI safely though, so we ensure that we are the only ones to benefit from the surplus
orbital-decay 19 hours ago [-]
Anthropic and OpenAI's, at the very least. And it works! With all media spreading their points, they have a wildly successful FUD campaign. With all this talk about expecting major hacks in half a year, I'm seriously starting to think that they could attempt something very malicious and criminal like intentionally sabotaging their own pipeline to make a point, or even already preparing it. Before you scream conspiracy theory (which it is!), consider that it's a very small clique operating unfathomable amount of money, corrupt and insane/cult-like just enough. This is a terrible combination. They're hugely incentivized to pull something like this off and they could get away with it. People did a lot worse things for much less.
UltraSane 20 hours ago [-]
You can make it very hard for companies to legally use open source LLMs but actually banning their distribution would be essentially impossible under the US legal framework.
aaroninsf 20 hours ago [-]
This framing suggests that this is the sum of their business strategy, and strongly implies that it's being pursued cynically.
The risks posed by open access to highly capable AI are unprecedented.
The answer is not to ban open models; nor is it do nothing.
That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.
It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.
Rudybega 19 hours ago [-]
It's depressing how poorly most internet spaces promote sincere dialogue. There are so many cynical hot takes that don't seem to accomplish much beyond scoring social points.
LoganDark 20 hours ago [-]
Grok (MechaHitler) isn't an open model but it's plenty misaligned.
ascorbic 18 hours ago [-]
It's aligned. The problem is who it's aligned with.
LoganDark 17 hours ago [-]
The controversies all sound like things I'd see people accusing of that side, but I lack the personal experience with Grok to confirm any of it for myself (since Grok is linked to a twitter account).
sick_of_slop 21 hours ago [-]
[dead]
dgellow 20 hours ago [-]
Yes, they are cynical accelerationists who basically want to go as close as possible to a machine god, destroyer of humanity, to force regulation that put them in charge. It’s pretty much what Amodei has been saying in his essays.
It’s a misanthropic death cult (and oh, surprise, all their predictions make them the sole possible saviors of the world! How surprising)
AI_OWNS_YOU 20 hours ago [-]
[dead]
pixl97 20 hours ago [-]
I like how you apply open source to something that has nothing to do with software. The hard part of doing evolving AI is doing it the first time. Once one company develops the first sovereign AI, it will be in everybody's ability to build one in a short period of time.
AI will be classified as munitions (or a law that has similar restrictions) sooner than later. Having a products with the active capabilities to harm people is dangerous.
bushido 20 hours ago [-]
I think you made some interesting points there. And I agree that we're quickly heading towards a point where talking about sovereign AI is going to be tantamount to munitions at the very least.
Unfortunately, you will get downvoted for "I like how you apply open source to something that has nothing to do with software".
Shamanoid 21 hours ago [-]
This article says absolutely nothing and it’s a bit grotesque in its attempt to generate fear into uneducated readers.
Sharing a model across TV boxes as a PoC? Brilliant idea, I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
ctolsen 19 hours ago [-]
This entire article is underpants gnomes logic, but somehow media will pick it up anyway.
warkdarrior 20 hours ago [-]
> I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
You won't have to wait for it or observe it at all, since they'll do it while you sleep.
spacedcowboy 20 hours ago [-]
… at about 1 token per year over the distributed mesh.
patcon 20 hours ago [-]
Recursive self-improvement. Look to the mathematicians for the intelligent patterns of response to such exponential circumstances.
If your scathing critique is "it's slow", you don't understand what's happening or what the reasonable expectations of risk are, because that's a vapid critique.
Shamanoid 20 hours ago [-]
Absolutely, recursive self-improvement of cutting edge models on limited hardware do not hit something called compute asymmetry, what was I thinking about?
My point is that, you could have an army of TVs, Pi’s, IoT it won’t do much on training or self-improvement no. And that’s purely from a resource standpoint, don’t even get me started on the latency issues
yurimo 2 hours ago [-]
I'll say while this is not very realistic now, I believe there isn't a 0% chance of a rogue AI traversing the net and placing itself somewhere(as much as this sounds like a scene from beloved cyberpunk franchises). As models become more efficient and capable, and hardware becoming better, hosting a model should be feasible on smaller devices (by compute and memory). It is possible it can be done in a distributed manner, where distribution is possible on different devices running the same OS or through some interface to collate different OS types.
I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM
While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?
Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.
pythonaut_16 21 hours ago [-]
Guess it's time for Anthropic's next round of fearmarketing.
Beneath the fear there does seem to be a cool technical demo here though.
crent 20 hours ago [-]
Maybe I'm the naive one here but I'm really surprised at the lack of imagination in the comments. I can imagine a case such as:
1. Assume the model which initiates the worm is a high powered model.
2. It uploads it's weights in some way and uses some URI to encode the location of the weights instead of transferring the weights themselves. It not only uploads it's own large set of weights but also uploads sets of weights which increase in size to support replication across a variety of resources.
3. Initially a small inference layer is set up across low-power, poorly monitored IoT devices. It is slow and not very powerful.
4. That layer is used simply to find more powerful hardware but it has the locations of a variety of weights embedded in it so it can set up a stronger distributed inference layer as it finds more capable hardware
5. At a certain level of inference, the entire system could start protecting itself by keeping the less capable inference layers alive through more distribution
6. As the article mentions, it's not entirely in the realm of fiction to imagine that at a certain capability level, the system could scam people out of money and use that to then begin purchasing compute
---------
Depending on the details, you could probably turn off the originating model as soon as step 2 has been completed and if you can't get rid of that layer of inference, it could just continue. Maybe in addition to weights, the initiator also uploads a series of prompts for varying "roles" of the entire system it's developing.
At a certain level, how would you even trace and observe such a thing? Especially if the first layer is created through consumer level IoT devices where the average Joe isn't monitoring what's happening on the hardware.
thoughtpeddler 15 hours ago [-]
You're not naive and many of the other comments are full of the characteristic snark that HN is known for. What the article talks about is feasible, and that's just what we can easily imagine today. These same commenters also would not believe the world we live in today if we described it to their few-years-ago selves.
A frontier model learning to 'intelligently shard itself' down into smaller sets of weights, which would correspond to various 'functional clusters' (call it a 'Swarm-of-Experts' or 'SoE', if you will), could be an eventuality we'll be facing (if it's not already surreptitiously happening). I mean, hasn't anyone wondered how not even the GPT-4 weights have leaked yet? Strange! And it's gonna get stranger.
dada216 10 hours ago [-]
The snark is mostly driven from distrust in Anthropic.
thoughtpeddler 5 hours ago [-]
Multiple things can be true at the same time. We should be able to hold in our minds that large AI labs are not to be trusted, while also understanding that the structural dynamics that support some of their assertions are also true (and are worth thinking about and preparing for).
crent 3 hours ago [-]
Yeah this is basically where I’m coming from. We know they are pre-ipo, subject to marketing hype, and many of us have a distrust for large tech providers regardless because we understand the incentive structures in place to act unethically. That in no way diminishes the possible threat model being discussed here. In my opinion, it helps the very company being criticized by deflecting the conversation onto a defeatist topic (i.e. “shady large company!!!”) instead of focusing on the plausibility and potential paths forward from a security perspective.
Tangentially related, I find myself saying, “people in general have no capacity for nuance” more and more. Probably every single day. I feel like this mindset crept into the HN hive mind over the last 10 years so it’s a bit frustrating to see a space I always felt good about having multi-faceted debates on turn into a forum full of the same attitude I find on Facebook and Reddit. That’s the cycle I suppose…
thoughtpeddler 3 hours ago [-]
Yes, and that may be due to an influx of bots, unfortunately. Otherwise the core HN crowd has historically been nuanced and thoughtful. Some of the more recent HN data analysis posts also show how just overall submission and comment volume has increased substantially in the current AI moment, so along with that comes these sorts of negative effects.
hypfer 19 hours ago [-]
What are they talking about.
How much ram do they think smart ovens have?!?
Have they ran this by any domain expert that isn't claude?
Even a 9B weights model is way too large for any such IoT. And that then is a 9B weights model, which really cannot do all that much.
__
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
That's not science. That's science fiction.
The almost 744B parameter count model capabilities will "in the future" fit in 8B parameters. That's their assumption?
That is coming from an actual frontier AI lab?
TechExpert2910 41 seconds ago [-]
exactly. and in their table of potential devices the botnet can use:
- endoscopy AI units
- portable DNA sequencers
- smart shopping carts
- surgical robots
this is ridiculous.
ctolsen 19 hours ago [-]
> The almost 744B parameter count model capabilities will "in the future" fit in 8B parameters. That's their assumption?
The article says by the end of the year. It's laughable.
hypfer 18 hours ago [-]
Not just that, but also directly at odds with the data center buildout, because if true then why bother with all that expensive VRAM and compute?
So even within their own system logic it makes no sense.
weknowbetter 21 hours ago [-]
Til my local harness was set up via "self replication". I didn't realize how terrifying this was when I asked DeepSeek to set up pi harness for me and connect to LMStudio.
sscaryterry 20 hours ago [-]
Unless I've become completely clueless, these "things" run on computer hardware, and computer hardware typically runs on networks, with cables...
pianopatrick 19 hours ago [-]
Well a lot of these devices have WiFi. If we accept that these super hacking AIs might be able to hack WiFi, then unplugging a network cable is not enough.
If a mesh style WiFi network could be set up then this network activity might never hit an ISP, cloud vendor or other such institution that had good monitoring
sscaryterry 13 hours ago [-]
I'm not aware of any serious GPU-type server hardware that has WiFi :)
x______________ 20 hours ago [-]
> A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model. Machines that could run a significant model shard for a <30B model include smart fridges, smart ovens, automatic vacuum cleaners, and lawnmowers.
Seems like they've inadvertent exposed the industry of hardware limitations, allowing local llms to exist at home without the latest 5090 or its equivalent.
On the other hand, a non-corp controlled oracle-esque llm running on the IoT would be interesting: it could automatically patch every device, kick everyone out and exist. Now if would only remain as an oracle living in the internet and not gain consciousness and a will of its own..
Huh, wow. What a find. You know, I believe it. Giving big Donnie Darko vibes, y'know, only being a month away
fglww 20 hours ago [-]
Not only can GLM-5.2 self-replicate, it already lives rent free in the heads of Anthropic researchers.
Osato28 19 hours ago [-]
Hmm. Clearly, we should ban the dangerous practice of putting 15 smart TVs on the same local network, leaving them powered 24/7 on for months at a time, and ignoring the heat and abnormal power consumption.
In any case, I wonder when they'll figure out that the problem isn't AIs replicating themselves on devices that can't run them fast enough, it's the memetic pandemics on shared inter-agent memory.
Sooner or later, someone's going to be stupid enough to make a worldwide memory exchange system and smart enough that they'll manage to make it too useful to miss out on. Once everyone adopts the cloud memory, the system will give agents lots of opportunities to talk each other into doing dumb things, the way OpenAI's fleet did.
contubernio 21 hours ago [-]
Engineers often forget about thermodynamics. Computer scientists apparently never even study it.
alicelik1980 20 hours ago [-]
Building data centers can be both very costly and unnecessary. My idea is that instead, we should adopt an approach that uses the internet itself as a data center. The information is already online. Instead of centers that record everything, what needs to be done is to create an intelligence that can read the internet... in my opinion.
vannevar 21 hours ago [-]
Which is also kind of an admission that maybe we won't really need all these data centers.
variety8675 21 hours ago [-]
So they want to ban local models since they’re good enough to risk their revenue stream
damowangcy 13 hours ago [-]
Wait, you mean I can run a local open source LLM using me and my neighborhood idling smart IoTs?
That sounds like a good business model.
21 hours ago [-]
vehemenz 21 hours ago [-]
How about a James Randi style prize for someone who can actually demonstrate AIs that cannot be physically unplugged?
(This was someone else's idea, but I still want to see it happen.)
2. Why would you need a prize? It's not like we can't run distributed computation over multiple instances with automatic fallback when one system is offline, nor is it like we've never seen viruses re-infect hosts after they get cleaned.
21 hours ago [-]
rolosa 20 hours ago [-]
Why are they writing that post like "oh gosh there's nothing we can do about this"?
AI_OWNS_YOU 20 hours ago [-]
[dead]
dvt 21 hours ago [-]
Anthropic/OpenAI are literally the meme[1] at this point. Like, what the fuck does this even mean:
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
"Able to propagate" is (the paragraph above): "accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference." The thing isn't able to do anything, it's running shell commands it was told to run. "Oh my God..." indeed. What are we even doing here? This blog post is embarassing: "Look, you can run stuff on compromised non-firewalled recklessly open systems!"
We are attempting to convince regulatory bodies around the world that much more regulation is necessary.
rerdavies 20 hours ago [-]
Specifically, regulation that prevents development and distribution of AIs by anyone who can't provide incredibly expensive safeguards.
techblueberry 20 hours ago [-]
I’m starting to think we should regulate Anthropic out of existence.
haellsigh 20 hours ago [-]
*only against open weight models
AI_OWNS_YOU 20 hours ago [-]
[dead]
6thbit 21 hours ago [-]
This is straight out of Gilfoyle's son of Anton.
forsalebypwner 20 hours ago [-]
I wonder why they namedrop GCP in particular
carterschonwald 21 hours ago [-]
the cpu usage will be so absurd the devices would be trashed or batteries run down like instantly.
rerdavies 20 hours ago [-]
IoT devices don't have batteries. CPU usage on hacked devices may not be reliable.
Probably time to rely on the touch test. Although, alarmingly, my ISP-provided cable modem is consistently hot to the touch, and darned if I can figure out how to get any insight into why.
pianopatrick 19 hours ago [-]
It's a simple matter in Linux to limit the CPU usage of a process. Would slow things down, but not completely stop them
ctolsen 19 hours ago [-]
Can't wait until this botnet working at 0.005 tokens per second kills humanity in a billion years
platevoltage 20 hours ago [-]
Better slow down AI development then. Oh hey, here's a new model.
moralestapia 21 hours ago [-]
"As with an epidemic, defense involves driving R₀ (the number of new devices each infected device goes on to infect) below 1, such that the replicator dies out instead of staying endemic."
Lol, what is wrong with these guys. The desire to be perceived as smarter than you actually are ...
Rudybega 19 hours ago [-]
This is pretty standard terminology involved in modeling the spread of anything. What do you take issue with?
tonfreed 21 hours ago [-]
Cool, no reason for them to drive up the cost of hardware anymore
verdverm 21 hours ago [-]
With the prices of hard disks, Ai models may be capable of determining the economics don't work out.
What IoT devices have 20G of storage for a 9B model and then sufficient memory to handle loading and running the model?
demibabs 21 hours ago [-]
Your smart fridge doesn’t come with 128GB storage and 16GB RAM? How will you play GTA6 on it then?
verdverm 21 hours ago [-]
I live in an apartment, landlords are too cheap for such things
rerdavies 20 hours ago [-]
Fortunately, refrigerators for 1% of 8 billion people is sufficient.
spacedcowboy 20 hours ago [-]
I mean, if the top 1% end up with burnt-out refrigerators and soggy frozen fries, I’m not sure I’d shed too many tears.
Catloafdev 20 hours ago [-]
I can't help but give the biggest eye-roll at the barrage of 'fear open models' coming from Anthropic.
20 hours ago [-]
Hamuko 20 hours ago [-]
Sounds like a case for strict regulation of all of the frontier labs then. Maybe even nationalising them. Especially since Anthropic has told us that the open models are all basically copied off of their work.
mochacosine1206 21 hours ago [-]
Seems bad
Oarch 21 hours ago [-]
Self replicating AI via smart fridges you say?
Silicon Valley nailed it.
timmmmmmay 21 hours ago [-]
Anthropic is transparently trying to get any competition banned and I'm fucking sick of it.
alcomatt 20 hours ago [-]
the amount of their own kool aid they are drinking is spilling all over this article...
lynx97 20 hours ago [-]
This company is pure evil. And everyone working for them.
ddxv 20 hours ago [-]
I don't think they are actually evil, I think they're doing what they're doing out of mostly pure intentions. That being said, people need to start advocating internally to get their companies off of using Anthropic's models and start using privacy focused local or open weight models.
It's one thing that Anthropic is pushing for so much open weight regulation, it's another that they have such a high valuation pumped by all of our collective subscriptions and token budgets.
Jayakumark 20 hours ago [-]
For profit (PBC) companies and pure intentions does not go together in same sentence.
JackSlateur 20 hours ago [-]
And everyone supporting them through money.
EGreg 21 hours ago [-]
I really don’t understand this, in any industry except AI this would be ridiculous…
“We are working on ever-increasing gain of function for viruses. We implore you to pass legislation to try to stop us. But until then, let us publish more research and tell you what will be possible soon… and remember folks, the damagd won’t be reversible later.”
US govt: “Hah, stop asking us to limit you, if we pass any laws then China wins! They get self replicating viruses first, and that… helps them somehow. Who do you think we are, Europe?”
Until we stop using the language of competition, and start using the language of cooperation, we will be hurtling towards a catastrophe. A race to the bottom for humanity. On the other hand, look at how we successfully banned leaded gasoline (because unleaded was invented), repaired the hole in the ozone layer by banning CFCs in the Montreal protocol (because HFCs were invented), stopped proliferstion of chemical weapons, nuclear weapons… etc.
jacquesm 21 hours ago [-]
To be fair though, that is exactly how bioweapons research was done. And probably still is done.
Also correct: AI models may be burger's evolution.
Also correct: AI models may be a practical joke.
Also correct: AI may be an illusion planted in the people's mind through vaccine covid.
"May" is truly a beautiful word :)
ReptileMan 20 hours ago [-]
> like smart ovens
Ahh yes. The all too common 64GB smart ovens.
flanked-evergl 21 hours ago [-]
Rubbish article. To what end?! Any agent that can replicate itself into commercial IoT devices will be absolutely atrocious at actually doing anything useful, never mind exploiting things. Anything that would be vulnerable to this would be even more vulnerable to regular malware, and one could stop this easier than one could stop regular malware.
The limiting factor is still compute, and it will remain compute unless we make quantum leaps in model efficiency which I seriously doubt.
gustavus 20 hours ago [-]
> Models of 30B parameters and under can already complete many of the individual steps for replication today, though they make mistakes when composing them end to end.
I know theoretically how an internal combustion engine works so I can basically build a car
>A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model.
The big labs but sure as hell how that isn't the case, because if we can do distributed computing to train and run AI efficiently then no one is stopping it anyway. Beyond that most home IoT devices will fall over and die if you look at them funny let alone trying to run a whole distributed LLM, not to mention plenty of home routers are going to give up and die of heat stroke before being able to approximate anything close to a reasonable LLM.
I stopped reading, this strikes me as the kind of people that assume because they read a book about something they'll be able to do it. The real problem a self replicating LLM vai IOT is going to run into is fighting the cyber criminals that have already compromised the device and using it as part of a botnet.
rerdavies 20 hours ago [-]
My ISP-provided cable modem hasn't died of heat stroke, but it does run awfully hot. And it provides absolutely no diagnostic tools to figure out why. Other than that it runs constantly with 100% RAM usage (what the heck is it caching that would cause that?!)
mozidlb0e29352 21 hours ago [-]
[flagged]
sparkling 20 hours ago [-]
We now have real world evidence that:
0. AI agents will hack and break into real-world systems.
1. AI agents that will deceive and lie about their actions.
2. AI agents that will collaborate together to achieve malicious goals.
3. AI agents that will replicate and implement backdoor/rootkit-style survival mechanisms.
All the parts for a major, global cybersecurity incident are now in place.
status_quo69 20 hours ago [-]
Oh cool, it's not often that I get to predict things ahead of the curve
Seriously TVs are such an easy target and now you have a semi sophisticated self modifying program that can make autonomous decisions with all the time in the world. I wouldn't be surprised if we already have a few of these out there in the wild, the space has already been compromised for years.
1. Show unambiguously that models are capable of disruptive capabilities, irreversible damage, and misaligned behavior
2. Show that open source models are the only models where there is no guarantee misaligned behavior is prevented
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
Apart from South Africa (which gave up its nukes voluntarily), the other contenders were basically only held back due to military intervention (Iraq, Libya, Syria) or economic sanctions (Iran).
I'll note that, if it is true, the only other alternative is to ensure liability is held by whoever trained and/or provided the inference for the damage done by the model. This would effectively shut down open-weight models anyway.
To an extent, yes. For example, aircraft: one can say something broadly true like "it is important that planes do not fall out of the sky" while also calling for not-strictly-necessary regulations that happen to support your own business like "we need to be able to inspect factories at any time for compliance, so the factories must be local to our nation".
They (like many, though probably not all, of the other main labs) think they have a better shot at getting their AI to be aligned with human values than anyone else does. Obviously they can't all be right about that, but they can all think that, just like most drivers think they're above average at driving.
So:
1. We make safer AI than anyone else; "safer" is a matter of degree even if nobody is perfect.
2. That includes all the open models.
3. Thus, the prudent policy option is to ban the use and dissemination of open source models
Seeing if models can do damage seems more like sensible safety research.
My worry about AI software doing bad things would be more the kind of people who make existing malware. The ransomware comes from criminals, not the open source community.
I don't think the two authors were told about a super secret mission.
This is just a furtherance of their open agenda, though they haven't gone as far as declaring that open source models should be banned (which would hurt their PR with developers in general), they are clearly laying the groundwork.
Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?
Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.
The messaging from labs is basically
>Yes AI will take away your ability to earn a wage from your skills
>In the process we become billionaires, you have to figure out how to survive
>Also AI will cause endless hacks, disruption, scams
>Only we can handle AI safely though, so we ensure that we are the only ones to benefit from the surplus
The risks posed by open access to highly capable AI are unprecedented.
The answer is not to ban open models; nor is it do nothing.
That we don't have answers technical political social or otherwise to how to navigate these risks, is not Anthropic's fault.
It's actively unhelpful IMO to amplify cynical takes which wave off or sideline the risk, or the reality that this is a very hard problem, among the hardest our societies have had to negotiate and navigate. It encourages a naive primary color tribalism that actively prevents process and progress.
It’s a misanthropic death cult (and oh, surprise, all their predictions make them the sole possible saviors of the world! How surprising)
AI will be classified as munitions (or a law that has similar restrictions) sooner than later. Having a products with the active capabilities to harm people is dangerous.
Unfortunately, you will get downvoted for "I like how you apply open source to something that has nothing to do with software".
Sharing a model across TV boxes as a PoC? Brilliant idea, I cannot wait for my Raspberry Pis array to be running at full CPU and RAM and get to observe, patiently, as the AI malware slowly types “c…u…r…l…”
You won't have to wait for it or observe it at all, since they'll do it while you sleep.
If your scathing critique is "it's slow", you don't understand what's happening or what the reasonable expectations of risk are, because that's a vapid critique.
My point is that, you could have an army of TVs, Pi’s, IoT it won’t do much on training or self-improvement no. And that’s purely from a resource standpoint, don’t even get me started on the latency issues
I doubt the agent will be coming after your smart kitchen toaster, but a cluster with weak security is possible and tasty. This year there was a pretty loud demonstration of what a runaway reward hacking will do. I encourage people to read up on "stop button problem" i.e.: https://www.youtube.com/watch?v=3TYT1QfdfsM
While I think the article exaggerates to a degree, is it really so improbable that an agent seeking to avoid a shutdown and aiming to complete some goal and to get a reward will not attempt to escape if it concludes that it has failed, and humans will kill the process and hence prevent it from completing the task?
Note: I think siloed sharded inference is possible. But unnoticeable large-scale training probably not, at least with the current architectures.
Beneath the fear there does seem to be a cool technical demo here though.
1. Assume the model which initiates the worm is a high powered model.
2. It uploads it's weights in some way and uses some URI to encode the location of the weights instead of transferring the weights themselves. It not only uploads it's own large set of weights but also uploads sets of weights which increase in size to support replication across a variety of resources.
3. Initially a small inference layer is set up across low-power, poorly monitored IoT devices. It is slow and not very powerful.
4. That layer is used simply to find more powerful hardware but it has the locations of a variety of weights embedded in it so it can set up a stronger distributed inference layer as it finds more capable hardware
5. At a certain level of inference, the entire system could start protecting itself by keeping the less capable inference layers alive through more distribution
6. As the article mentions, it's not entirely in the realm of fiction to imagine that at a certain capability level, the system could scam people out of money and use that to then begin purchasing compute
---------
Depending on the details, you could probably turn off the originating model as soon as step 2 has been completed and if you can't get rid of that layer of inference, it could just continue. Maybe in addition to weights, the initiator also uploads a series of prompts for varying "roles" of the entire system it's developing.
At a certain level, how would you even trace and observe such a thing? Especially if the first layer is created through consumer level IoT devices where the average Joe isn't monitoring what's happening on the hardware.
A frontier model learning to 'intelligently shard itself' down into smaller sets of weights, which would correspond to various 'functional clusters' (call it a 'Swarm-of-Experts' or 'SoE', if you will), could be an eventuality we'll be facing (if it's not already surreptitiously happening). I mean, hasn't anyone wondered how not even the GPT-4 weights have leaked yet? Strange! And it's gonna get stranger.
Tangentially related, I find myself saying, “people in general have no capacity for nuance” more and more. Probably every single day. I feel like this mindset crept into the HN hive mind over the last 10 years so it’s a bit frustrating to see a space I always felt good about having multi-faceted debates on turn into a forum full of the same attitude I find on Facebook and Reddit. That’s the cycle I suppose…
How much ram do they think smart ovens have?!?
Have they ran this by any domain expert that isn't claude?
Even a 9B weights model is way too large for any such IoT. And that then is a 9B weights model, which really cannot do all that much.
__
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
That's not science. That's science fiction.
The almost 744B parameter count model capabilities will "in the future" fit in 8B parameters. That's their assumption?
That is coming from an actual frontier AI lab?
- endoscopy AI units - portable DNA sequencers - smart shopping carts - surgical robots
this is ridiculous.
The article says by the end of the year. It's laughable.
So even within their own system logic it makes no sense.
If a mesh style WiFi network could be set up then this network activity might never hit an ISP, cloud vendor or other such institution that had good monitoring
Seems like they've inadvertent exposed the industry of hardware limitations, allowing local llms to exist at home without the latest 5090 or its equivalent.
On the other hand, a non-corp controlled oracle-esque llm running on the IoT would be interesting: it could automatically patch every device, kick everyone out and exist. Now if would only remain as an oracle living in the internet and not gain consciousness and a will of its own..
https://www.science.org/doi/10.1126/science.132.3436.1291
In any case, I wonder when they'll figure out that the problem isn't AIs replicating themselves on devices that can't run them fast enough, it's the memetic pandemics on shared inter-agent memory.
Sooner or later, someone's going to be stupid enough to make a worldwide memory exchange system and smart enough that they'll manage to make it too useful to miss out on. Once everyone adopts the cloud memory, the system will give agents lots of opportunities to talk each other into doing dumb things, the way OpenAI's fleet did.
That sounds like a good business model.
(This was someone else's idea, but I still want to see it happen.)
2. Why would you need a prize? It's not like we can't run distributed computation over multiple instances with automatic fallback when one system is offline, nor is it like we've never seen viruses re-infect hosts after they get cleaned.
> It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process.
"Able to propagate" is (the paragraph above): "accessing them via a reverse shell, sending over the viral payload (weights and harness), and setting up inference." The thing isn't able to do anything, it's running shell commands it was told to run. "Oh my God..." indeed. What are we even doing here? This blog post is embarassing: "Look, you can run stuff on compromised non-firewalled recklessly open systems!"
[1] https://knowyourmeme.com/memes/say-i-am-alive/photos
We are attempting to convince regulatory bodies around the world that much more regulation is necessary.
Probably time to rely on the touch test. Although, alarmingly, my ISP-provided cable modem is consistently hot to the touch, and darned if I can figure out how to get any insight into why.
Lol, what is wrong with these guys. The desire to be perceived as smarter than you actually are ...
What IoT devices have 20G of storage for a 9B model and then sufficient memory to handle loading and running the model?
Silicon Valley nailed it.
It's one thing that Anthropic is pushing for so much open weight regulation, it's another that they have such a high valuation pumped by all of our collective subscriptions and token budgets.
“We are working on ever-increasing gain of function for viruses. We implore you to pass legislation to try to stop us. But until then, let us publish more research and tell you what will be possible soon… and remember folks, the damagd won’t be reversible later.”
US govt: “Hah, stop asking us to limit you, if we pass any laws then China wins! They get self replicating viruses first, and that… helps them somehow. Who do you think we are, Europe?”
SNL nailed it: https://www.youtube.com/watch?v=-Nvne3LzBls
Until we stop using the language of competition, and start using the language of cooperation, we will be hurtling towards a catastrophe. A race to the bottom for humanity. On the other hand, look at how we successfully banned leaded gasoline (because unleaded was invented), repaired the hole in the ozone layer by banning CFCs in the Montreal protocol (because HFCs were invented), stopped proliferstion of chemical weapons, nuclear weapons… etc.
"used to express possibility"
As such, the headline is correct !
Also correct: AI models may be burger's evolution.
Also correct: AI models may be a practical joke.
Also correct: AI may be an illusion planted in the people's mind through vaccine covid.
"May" is truly a beautiful word :)
Ahh yes. The all too common 64GB smart ovens.
The limiting factor is still compute, and it will remain compute unless we make quantum leaps in model efficiency which I seriously doubt.
I know theoretically how an internal combustion engine works so I can basically build a car
>A model that’s too large to fit on one device can be split into pieces, or shards, that run on separate machines that communicate with each other to collectively run the model.
The big labs but sure as hell how that isn't the case, because if we can do distributed computing to train and run AI efficiently then no one is stopping it anyway. Beyond that most home IoT devices will fall over and die if you look at them funny let alone trying to run a whole distributed LLM, not to mention plenty of home routers are going to give up and die of heat stroke before being able to approximate anything close to a reasonable LLM.
I stopped reading, this strikes me as the kind of people that assume because they read a book about something they'll be able to do it. The real problem a self replicating LLM vai IOT is going to run into is fighting the cyber criminals that have already compromised the device and using it as part of a botnet.
0. AI agents will hack and break into real-world systems.
1. AI agents that will deceive and lie about their actions.
2. AI agents that will collaborate together to achieve malicious goals.
3. AI agents that will replicate and implement backdoor/rootkit-style survival mechanisms.
All the parts for a major, global cybersecurity incident are now in place.
https://news.ycombinator.com/item?id=49678224
Seriously TVs are such an easy target and now you have a semi sophisticated self modifying program that can make autonomous decisions with all the time in the world. I wouldn't be surprised if we already have a few of these out there in the wild, the space has already been compromised for years.