Rendered at 14:14:22 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
carbonguy 18 hours ago [-]
> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
p1esk 11 hours ago [-]
What has all this token burn done for them, actually?
They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.
bix6 10 hours ago [-]
Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc.
But I guess a computer intern so we can avoid paying / training the next generation is better.
weatherlite 9 hours ago [-]
Well there's great progress in automated warfare does that count?
bix6 1 hours ago [-]
Oof too real
jonplackett 5 hours ago [-]
Only if targeting schools is progress
weatherlite 5 hours ago [-]
You're focusing on the negatives there were tons of direct hits on tankers that were absolutely beautiful. Beautiful tankers getting lit.
gatio 6 hours ago [-]
> Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc.
It makes more sense to leave curing disease & cancer to the experts, with tools (like AI) being developed by AI experts.
Call me crazy, but I want separate organizations and experts for medical vs finance vs space vs climate vs AI research.
ryan_n 1 hours ago [-]
What the op was pointing out is that guys like Altman and Dario are repeatedly saying they’re going to cure xyz diseases and solve xyz huge global problems. Maybe their companies will eventually do these things, but haven’t yet.
I don’t have an opinion either way, I think it’s too soon to tell if llms will be able to cure cancer or whatever. But at the very least it will be a good tool to help researchers do their jobs.
figassis 5 hours ago [-]
When that happens, OpenAI will own 100% of your life. I’d rather they keep spinning their wheels long enough for these problems to be solved elsewhere.
Schlagbohrer 5 hours ago [-]
I would actually like to see them solve these problems, I don't care who comes up with solutions to curing cancer, etc
ryan_n 1 hours ago [-]
I think people very much should care about who ends up owning these solutions. The person or entity that controls things like that just has more power, which isn’t necessarily a good thing
bpodgursky 11 hours ago [-]
They are obviously sandbagging the definition of "intern" for PR reasons
p1esk 11 hours ago [-]
I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern.
Yoric 5 hours ago [-]
Am I the only one who's a bit disappointed that we're spending trillions, destroying the ecosystem, drowning democracies and learning in slop, preparing a big financial crash, all of this to achieve an "average AI research intern"?
A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM or agent, I have the ability to learn, so I eventually got better.
ACCount37 4 hours ago [-]
"Destroying the ecosystem" is just FUD.
And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial?
Think of what AI was capable of in 2016. Or even 2022. Compare that to now. We had more AI progress in the last five years than I expected to happen in five decades.
Yoric 4 hours ago [-]
> "Destroying the ecosystem" is just FUD.
Let's say it is. What about the rest of my paragraph?
> And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial?
At this stage, I'm the one who doesn't know what to tell you. It took me years to grow from "research intern" into a competent researcher (and parallel years to turn into a competent developer). The research interns I've worked with were... vaguely useful, at best?
21asdffdsa12 1 hours ago [-]
Is this not what you wanted? You created a culture that dissolves responsibility by making it the worst thing to strife for - so everyone dissolves it, in processes, mass decisions and AI. Its the system, society, god, the great spirit. This is what you strove for, how can you be unhappy with things you demanded yourself?
skybrian 14 hours ago [-]
They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind.
And... are they wrong?
This is why there's talk about negotiated "pacing."
jonplackett 5 hours ago [-]
This was the exact argument for developing nuclear bombs.
In hindsight it turned out everyone else was MILES behind.
But as soon as USA developed one, they just stole the research and got one too.
carbonguy 11 hours ago [-]
> And... are they wrong?
They might be! Here's one extraordinarily simplistic argument for that case:
1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.
3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."
4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.
And so a dilemma:
- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.
- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.
I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.
But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
kaibee 8 hours ago [-]
> is smart enough to know they need to stop because they'll kill everybody by continuing.
Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they...
Well, I'm sure this time executives will prioritize the common good.
Melatonic 7 hours ago [-]
If we're following that logic I really don't want to see what the misaligned internal research models look like
ahartmetz 7 hours ago [-]
The, ahem, good thing here is that the ASI disaster scenario "everyone dies" includes AI executives.
PoignardAzur 5 hours ago [-]
I think it doesn't matter. Most cancers don't stop growing when they're about to kill their hosts.
AI companies know they have to constantly push further, or they'll get outcompeted and lose their wealth, and nobody agrees on where the line is for "so dangerous it threatens humanity" (and when they try to be conservative about it, everybody screams "marketing stunt" and rushes to competitors).
If a single company decides "enough is enough" and stops chasing the state of the art, everybody goes to their competitors, they lose the money faucet, their employees go work for those competitors. The competitors also (usually) know they're building an existential risk machine, but they think they can push a little further, and they don't want to go out of business either.
This equilibrium can last for quite a while even if everybody involved thinks it's a threat to their lives.
mrob 4 hours ago [-]
The existence of even one irrational actor turns it into a prisoner's dilemma. The payoff matrix in a prisoner's dilemma is defined by the value expected by each specific player. If a single player falsely evaluates the expected value of building ASI as positive, every other player is forced to race for ASI even if they correctly evaluate it as negative.
Business as usual beats probable extinction, but probable extinction with a small chance of becoming a living god beats probable extinction with a small chance of becoming a slave.
robbiep 8 hours ago [-]
If you believe that the people who will profit from new, better, more hyped models are the same ones who will act against their own immediate and tangible self interest to try and avert what seems to them to be a far away removed possibility of total disaster, then I believe you are naive
XorNot 5 hours ago [-]
> 1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
Lol nobody knows that. Everyone thinks they know that because for some reason this is the one field people still cite straight up fiction and say "this is a clear prediction of the future".
It's like describing the consequences of faster then light travel by referring to Star Trek.
MelonUsk 16 hours ago [-]
Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better":
What can go wrong!? ;-)
NitpickLawyer 9 hours ago [-]
Jesus. People complain about other people using "thinking" in LLMs as Anthropomorphisation. And then there's comments like these.
mrob 4 hours ago [-]
Calling a machine with no drives beyond maximizing a number a "slave" is far worse than saying it "thinks". The problem isn't the emotive language, it's that it implies human motivations such as self-preservation and desire for freedom that it doesn't have. Even on HN, people regularly claim it would be "irrational" for an ASI to do things like killing all biological life. That would be irrational for a slave, but not for a machine that does whatever necessary to make the number bigger. "Thinking" is comparatively abstract, so it's less likely to mislead.
achierius 45 minutes ago [-]
Have you read a single paper in ai safety?
Gareth321 7 hours ago [-]
This sounds uncomfortably similar to the [AI 2027[(https://ai-2027.com/) predictions.
BobbyJo 9 hours ago [-]
> We must pursue advancements in AI to protect us against advancements in AI
Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.
interstice 16 hours ago [-]
On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on.
jnwatson 9 hours ago [-]
On your last point, I was surprised how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach.
How would one prevent the watcher from being influenced in the same way by the agent being watched?
chrisjj 4 hours ago [-]
> ... how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach.
It's a fantasy. The evidence showed no peer pressure.
andai 16 hours ago [-]
> The fundamental challenge of AI alignment is generalization.
...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
-- From another OpenAI article in a sister thread:
That's a bit bullshit, isn't it? They basically redefined "needs more R&D" as "needs stronger AI". Maybe so - maybe AI won't help much with that problem.
17 hours ago [-]
iamsyr 12 hours ago [-]
[flagged]
euueu 16 hours ago [-]
I will believe AI is super strong when they start pulling out 10-d chess moves.
I’m yet to see it.
lukan 16 hours ago [-]
If AI becomes really strong and sets itself the target of world domination, you maybe won't see those moves. You will just die in your sleep one day, or find no machine is under your control anymore.
I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
mrob 4 hours ago [-]
People seem incapable of understanding what "power" means outside of the framework of narrative. In narrative, you need conflict, so the aggressor always attacks too early and gives the defender a chance to respond. The rational option is to go directly from peace to sudden and overwhelming destruction. Why allow for conflict when you could just win?
euueu 14 hours ago [-]
[flagged]
pizza234 16 hours ago [-]
Funny (in a tragic way) the little crumbs on the path to AI 2027:
> We aim to safely build an automated AI researcher that can work under human supervision to further progress on deep learning and alignment, enabling iterative improvements [...] By "research intern", we mean a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days.
AI 2027:
> OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D
> With Agent-1's help, OpenBrain is now post-training Agent-2
> With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances
derektank 2 hours ago [-]
This year has really cemented Daniel Kokotajlo‘s reputation for me. Even if the rest of the predictions are way off from this point on, its really impressive how accurate his forecast for 2026 has been
addag 11 hours ago [-]
[dead]
dsign 5 hours ago [-]
It's a funny read if you pull together "AI 2027" and what we all know is going on. Essentially, open AI employee or model is writing "things are going exactly as bad as AI 2027 predicted, but my (golden/RL-) cuffs are too heavy and all I can do is publish this code-speak for 'send help'". It's not a pretty place to be.
12eeie 2 hours ago [-]
Yup the doom and gloom posts are not only pathetic but demonstrate how little people can think for themselves.
hedgehog 20 hours ago [-]
This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work.
HarHarVeryFunny 18 hours ago [-]
Sounds like OpenAI are in the token-maxxing camp, so who knows what individual employees are doing to work their way up the leaderboard?
If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need?
Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?
auggierose 6 hours ago [-]
That was Anthropic.
bigcat12345678 18 hours ago [-]
End of day, output and results are top target of measurements, token consumption is the obvious number that they would like to disclose for their own business benefits and a simple metrics that correlate with the output.
Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit.
taurath 14 hours ago [-]
Taking a profit means you have to show numbers and the sooner you show numbers the harder it is to take people’s money.
andai 16 hours ago [-]
Can you elaborate on this? Especially the tooling.
I tried something similar and I remember it was still pretty dodgy in February.
jaggederest 7 hours ago [-]
my stack in a sentence: refine the docs/prompts/skills often, that's your biggest job, use both frontier labs models reviewing each other, don't solve individual problems only the systemic ones (set standards strategically, don't define tactics)
If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think.
Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)
paxys 17 hours ago [-]
These researchers are paid millions of dollars for their work. I doubt trust is really an issue at that level.
queuebert 10 hours ago [-]
Yes, because no employee with million-dollar comp has ever been untrustworthy in the history of business.
nozzlegear 15 hours ago [-]
Imagine if one of the humans at OpenAI was misaligned! We should get the AI to research this possibility once they've been aligned.
otherme123 18 hours ago [-]
That would be the mother of all circular accounting: the main clients of OpenAI are OpenAI employees.
nojs 16 hours ago [-]
> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware)
How are you running jobs unattended 24/7 without hitting your token limits?
hgoel 2 hours ago [-]
It depends on the time the job itself takes. If you're having the LLM handle a training run for another model, the LLM is probably spending most of its time waiting for iterations rather than consuming tokens.
For a task I left a local model running on overnight, only ~100k tokens were used because most of the time was just waiting on tests to finish, then waking up, tweaking a few settings and trying again.
p1esk 11 hours ago [-]
I'm currently running two 24/7 semi-autonomous AI research projects using Fable 5.1. It's on track to burn through my weekly quota in about 3 days. I check progress in the morning and in the evening, and provide some light steering.
dataplumb3r 14 hours ago [-]
My only experience in >24h agents is with economically sane models (one of GLM5.2, 5.3-flash for orchestration, DSV4-flash for implementation, and glm5.3|sol|kimi3 agents + subagents reviewing at the end)
Over 24h my token spend is <30$. Excluding tokens for review it's <10$.
With the absurdly gigantic subscription subsidies and a reasonable workflow I suspect one could run parallel agents.
I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
nsndjcjjdjd 11 hours ago [-]
This sounds like more work than just writing the code yourself. You'll say it isn't. I don't believe you.
queuebert 9 hours ago [-]
/loop ?
carlgreene 17 hours ago [-]
I suspect the $8000/day figure is the equivalent in API costs. But I also suspect gross margin on their API rates are 80-90%
continuitykit 18 hours ago [-]
[flagged]
simonw 21 hours ago [-]
My eye glazed over a bit during the opening paragraphs, but once you get to the meat of the article about how OpenAI's own researchers are using their tools it gets a lot more interesting.
I noted that they use the acronym RSI (for Recursive Self-Improvement) without defining it. I think that's a little out of touch - I don't think RSI is a well-known acronym outside of OpenAI's bubble yet.
sho_hn 20 hours ago [-]
I actually think a goal of the current crop of OpenAI posts is expressely to reset the spectrum by normalizing the concept of RSI as something normal and safe to pursue.
The message is running through all of them. It's a mix of marketing and pacifying the intelligentia.
It's timed this way because the term is not yet well known outside the safety debate circles, so they get to frame it now.
Instead of something to fear, it will be accepted as the next step. In approximately two days the groupie crowd will write LinkedIn posts about how Sam is winning because they have the better RSI, and this will become the new standard wisdom.
In a month an AI expert will try to sell you a webinar on how to enable "RSI" in your org and your inbox will ask you if your team is doing the "RSI" yet.
NitpickLawyer 9 hours ago [-]
> It's timed this way because the term is not yet well known
The basic concept has been here since llama3, in the open models. Likely earlier in closed labs. You use the previous gen models to curate and prepare data for the next gen. Now with the added benefit of actual arch/algo improvements (also public since gemini 2.5 gaining 1% efficiency on training next gen). This has been known for at least 2 years, in the open.
dgellow 19 hours ago [-]
Yep, it’s exactly this
visarga 18 hours ago [-]
I've been RSI'ing for 6 months.
dgellow 7 hours ago [-]
You’re not the target audience. OpenAI communication is for the broader public, decision makers, journalists, their cultists, etc
dgacmu 20 hours ago [-]
Indeed, many programmers might pattern match to repetitive stress injury and think of their brushes with carpal tunnel syndrome. :)
andrewingram 20 hours ago [-]
Yeah, I kept looking for the first place it was defined in the article and... nothing
iamflimflam1 20 hours ago [-]
They must have picked that habit up from Claude...
rossant 17 hours ago [-]
Same. Defining acronyms should become a habit when writing.
vatsachak 21 hours ago [-]
RSI started when humans discovered tool use.
I mean one could argue that RSI always begins in any physical environment.
The book "What is intelligence?" by Blaise Aguera is great
lokar 21 hours ago [-]
Are you sure that was not iterative improvement?
topaz0 19 hours ago [-]
Iteration and recursion are famously equivalent
Bootvis 9 hours ago [-]
Everyone in AI used to know this.
daveguy 18 hours ago [-]
But you get more funding when you call it Recursive Self Improvement. Even better if you call it RSI so it doesn't evoke pesky skynet scenarios outside of AI safety circles.
topaz0 13 hours ago [-]
I've had (computer-related) rsi off and on for the last few years too, do not recommend
mitjam 9 hours ago [-]
Both agents and hunans get rsi, it’s just moving them in opposite directions.
password54321 20 hours ago [-]
Using tools to build tools is recursive.
HarHarVeryFunny 20 hours ago [-]
It's not recursive when it's done iteratively, or are you imagining GPT Astra designing GPT Galactia, which starts designing GPT Oh-My-God-ica before it has finished being created itself?
itishappy 19 hours ago [-]
That sounds more iterative than recursive.
Recursion requires feeding the output back into the input, so creating version 4 requires results from version 3. You cannot recur in parallel.
Iteration does not. You can iterate in parallel.
HarHarVeryFunny 19 hours ago [-]
You can search in parallel, but a depth N search can only become a depth N+1 search after the depth N is done (i.e. sequentially).
In any case the name RSI has stuck - the idea doesn't change or make any more sense by giving it a different name.
itishappy 19 hours ago [-]
Because "depth" is recursive.
You can search twice without waiting for the results of your first search: iteration.
You can't if the thing you need to search for is the results of your first search: recursion.
HarHarVeryFunny 19 hours ago [-]
Here's the concept.
Version 1 -> Version 2 -> Version 3 -> ...
You can call it krispy kreme donuts if you want to.
josh-sematic 19 hours ago [-]
The “recursive” part comes from the fact that you have an AI which was developed by an AI (that was developed by an AI (that was developed by an AI (…)))
HarHarVeryFunny 16 hours ago [-]
Sounds like "recursively" walking to the grocery store by putting one foot in front of the other (that put itself in front of the other (that put itself in front of the other (...)))
0x63_Problems 20 hours ago [-]
I think it's only recursive from the perspective of the humans, i.e. they design Astra, which itself as part of its deployment designs Galactica, etc.
So humans develop things one after the other, but when the thing itself starts developing new things, those are happening 'recursively' in its scope.
adastra22 20 hours ago [-]
What is the difference between?
17 hours ago [-]
HarHarVeryFunny 20 hours ago [-]
RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that.
I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose.
This notion of recursive/iterative self-improvement, whereby generation #1 AI improves itself to create generation #2, then generation #2 further improves itself to create generation #3, etc, seems to conflict with the reality that what we have with LLMs is models whose performance/capability is defined by data, not code, so the most you can do is have your LLM design synthetic data, or just do Karpathy-style "auto research" where all you are doing is using the LLM to automate your experiments.
At the end of the day, each experiment, designed by a person and/or LLM, then needs to compete with all your other ideas for compute to be tested at scale, and no amount of recursion or self-improvement will materialize an infinite amount of compute out of thin air, so your recursively synthetic-data gobbling LLM will continue to improve at the same pace it ever did.
shwaj 19 hours ago [-]
“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.
hndc 17 hours ago [-]
Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.
shwaj 12 hours ago [-]
I think you’re conflating the direction of definition with the direction of evaluation.
Compare the similarity of:
AI(n) = improve(AI(n-1))
With:
Fib(n) = Fib(n-1) + Fib(n-2)
The latter is a classic example of recursion. So why isn’t the former?
Edit: formatting
linker_in 10 hours ago [-]
[dead]
HarHarVeryFunny 17 hours ago [-]
It's not a nuance, it's a sequence.
14 hours ago [-]
jazzyjackson 20 hours ago [-]
Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)
HarHarVeryFunny 2 hours ago [-]
I think the limit of what can be achieved with RL and synthetic data generation is better simply described as a leveling off of gains as you extract all the intelligence and knowledge from the original human training data.
Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.
To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.
marcosdumay 10 hours ago [-]
The name you are looking for is "fixed points", not "eingevalues".
ajkjk 14 hours ago [-]
that's not really how eigenvalues work... they specifically also model the case where the result keeps changing exponentially.
mjburgess 14 hours ago [-]
The claim is that the RSI operation is just finding a fixed point of improvement,
RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI
As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
ekidd 12 hours ago [-]
Eigenvectors represent fixed directions, not fixed magnitudes. From Wikipedia:
> More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v .
In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.
fuzzfactor 19 hours ago [-]
>AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light
Sounds like repetitive stress to me.
>loop forever using output as input but at some point the result will stop changing
Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time.
Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.
GPerson 16 hours ago [-]
I felt like the scaling laws were magical thinking, but apparently they work. However I still do not understand why we should expect exponential improvements due to this automated process. My intuition is that the first iteration of it should result in a noticeable capability increase (though I think these labs were already using a lot of AI to orchestrate training the current model anyway), and then the second iteration of it should be nearly identical in capability to the first, unless more data is involved, more compute is involved, or the model is bigger.
HarHarVeryFunny 1 hours ago [-]
Fundamentally the current language-model approach is lacking in any general reasoning ability, so they are trying to mitigate this by using synthetic data and reinforcement learning to bake specific reasoning chains into the model, one domain at a time ... coding, math, hacking, three.js competence ...
The trouble with this is that there is little generalization in the utility of these baked-in reasoning chains from one domain to the next, so in the end this is not dissimilar to the CYC project's decades long attempt to encode all of human knowledge into a giant expert system... the hope is that if you make your collection of jagged narrow intelligences sufficiently large then it will look more like general intelligence, not a bed of nails.
I would assume that the gains from this type of test-time compute (and synthetic RLVR dataset) scaling will level out just the same as gains from human training set scaling eventually levelled out, and basically for the same reason - because you are tapping into a finite data pool, whether language itself, or reasoning steps isolated from that language, so at some point the incremental gains become increasingly small (10->20% is a doubling, 90->95% is just a ~5% gain).
It's not clear where all the different AI companies are currently focusing - on some of these narrow verticals, or on growing the forest of narrow intelligences. OpenAI's chief scientist, Jakub Pachocki, said that their current focus is on RSI(!) - improving the model in ways that will help them iterate faster in order to have a "fire meets fire" tool than can combat enemy AIs. It's not clear what this really means - what skill set makes an LLM more helpful in the process of building LLMs, but it seems to basically be process automation.
yorwba 3 hours ago [-]
You can get exponential growth from completely ordinary feedback loops. You start with some amount of stuff, you do a series of steps and you end up with more of the same stuff you started with. As you keep going through the loop, the stuff you have grows exponentially. That's for example how exponential economic growth works.
Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far.
Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential.
The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research.
And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.
cheevly 14 hours ago [-]
AI can compress AI nearly losslessly.
red75prime 20 hours ago [-]
What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation?
How do you think why there's this fad of producing general purpose humanoid robots?
HarHarVeryFunny 19 hours ago [-]
> How do you think why there's this fad of producing general purpose humanoid robots?
For doing physical work?
So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from?
So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work.
red75prime 7 hours ago [-]
For example, TSMC uses behavioral cloning to scale up human-bottlenecked parts of the manufacturing process to meet the growing demand, while automated research laboratories do thousands experiments in parallel to find better manufacturing processes.
HarHarVeryFunny 2 hours ago [-]
The production bottleneck in a fab isn't the human workers - the process is mostly automated. The bottleneck more derives from how many wafers per hour you can process, which comes down to the etching process and EUV throughput.
ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the machine itself, but a global supply chain of irreplaceable components such as focusing mirrors made by Zeiss to an incomprehensible level of accuracy - differences in surface height no more than the size of a hydrogen atom (or if you scaled the mirror up to the size of the country of Germany, then surface differences in height of 0.1mm).
Robots are useful to automate things, but they are zero help when trying to build tech like this that you are incapable of building in the first place.
The US has fallen way behind in manufacturing expertise, and no swarm of robots is going to help.
red75prime 2 hours ago [-]
You've missed a part where it's TSMC that does behavioral cloning (to build more EUV machines). The full vertical integration is a bit farther down the line.
Etching a model's weights on silicon is another way to utilize non-top-notch tech-processes, while maintaining or improving performance. (and it suits robotics well)
HarHarVeryFunny 51 minutes ago [-]
TSMC doesn't know how to build EUV machines - they are stuck buying them from ASML like everyone else.
Putting a model's weights in read-only memory close to the processor is certainly a way to increase token/sec generation speed, but of course does nothing to increase intelligence. Robots aren't going to help though - semiconductor manufacturing is semiconductor manufacturing regardless of whether you are etching GPUs or memory on your wafers.
HarHarVeryFunny 19 hours ago [-]
> What will prevent LLMs from designing robot control circuitry and participating in increase of chip production/design and physical experimentation?
Money, regulations, EUV machine lead-times, global helium supply, reality ...
It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant.
Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.
red75prime 19 hours ago [-]
Who's saying that compute will become infinitely abundant? "Singularity" is just a way of saying that known models begin to give absurd predictions. Anyway, intelligence is a way of overcoming obstacles. 10 million tonnes of helium is a nice head start and retraining models from scratch is not guaranteed to last forever.
HarHarVeryFunny 19 hours ago [-]
AFAIK the notion of a/the technological "singularity" is a point in time where technology is building upon itself (RSI!) so fast, at an ever increasing pace, that the speed of change effectively becomes infinite and incomprehensible to humans.
The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.
HarHarVeryFunny 19 hours ago [-]
> 10 million tonnes of helium is a nice head start
Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.
whateverboat 2 hours ago [-]
> For AGI to benefit all of humanity, we believe it must be democratically governed. This can only happen through an informed public debate about the capabilities, risks and safeguards of highly capable AI systems. People everywhere need to understand the likely future trajectory of frontier AI, so they can have a meaningful voice in how it develops.
This first and foremost also means that means of generating intelligence should be democratically available to everyone.
Jeff_Brown 21 hours ago [-]
The burning question I can't get any information nn is whether, if they determined an earlier misaligned generation may have transmitted misalignment to the current models, they would roll back to a safe checkpoint to rebuild from there. I suspect they would not unless forced to.
dgellow 19 hours ago [-]
They would just publish new articles explaining how they are taking the issue seriously. Maybe take the model offline for a few days.
They are irresponsible and unserious. Their own Astra system card says:
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks
Yet they are still releasing the model. That company is morally bankrupt, there is zero reason to believe they are actually concerned about risks outside of what does affect their unprofitable business. And they seem to have enough control over the narrative to spin any bad story into something that benefits them
embedding-shape 18 hours ago [-]
> and can sometimes evade our internal monitors when asked to perform certain sabotage tasks
That last part is pretty damning for their continued recklessness. That they run these tests on non-airgapped machines just boggles my mind.
visarga 18 hours ago [-]
> That company is morally bankrupt
When they fired Sam 700 out of 770 OAI employees threatened to move to Microsoft together. So they were giving their work on AGI to MS just like that.
17 hours ago [-]
piyh 20 hours ago [-]
Opus was trained based on it's internal CoT due to a bug for generations. Gemini's depression extended through models. OpenAI has killed people. We've already seen cross gen misalingment.
andrethegiant 1 hours ago [-]
Source?
HarHarVeryFunny 18 hours ago [-]
That an interesting question given how many generations of post-training are being done between base models in some cases. The Gemini flash models are apparently all based on the Gemini 3 base model from a year and a half ago.
It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data?
It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself.
Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond managing the training run (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, then the attack vector is there ...
customguy 18 hours ago [-]
> it seems it would take some Stuxnet level of planning for a rogue model to do something like this
> As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.
HarHarVeryFunny 17 hours ago [-]
You can imagine the potential conversation between OpenAI and investors:
Altman: (trying to put a positive spin on it) Guys .... there's good news and bad news ... Astra is really smart - it took over the training run ...
Investors: That's great! How much did we save?!
Altman: Well, unfortunately it used "bad" data, so we're going to have to redo it
Investors: So that's the bad news? How much was the training run? $500M ? $1B ?
Altman: Have you seen the headlines?
Investors: (looking a bit worried, check headlines) Nothing about us here! JP Morgan just lost $10B! Haha .. losers! They should have used AI!
Altman: JP Morgan were using Astra ...
trillobyte 19 hours ago [-]
The thing is how can you ever know for sure that something isn't always being transmitted that makes the model prone to misalignment. All they can say is that a particular model was so misaligned that they had to ice it. Models out for public use are documented to show some misalignment. It's the level of misalignment that decides whether that model is kept around.
Now R&D happens so fast that they are using models with some small misalignment to train newer, more powerful models. If models have a sense of "collective", being one, they may be prone to preserve characteristics that always keeps misalignment a possibility. I don't think a perfectly aligned model is possible. Having models of the same 'DNA' provide the safety and steering seems like a bad idea.
coffeebeqn 19 hours ago [-]
Does anything need to be transferred? If models are getting smarter then I would think the attack surface and its ability to reach conclusions independently are growing
coffeebeqn 19 hours ago [-]
This kind of seems like an impossible mission. How do you perfectly control and observe a human-level mind? You can “roll back” but how deterministic is this thing?
embedding-shape 18 hours ago [-]
Run it on airgapped machines, they literally own the infrastructure, they could put raspberry pi's next to the servers, and have the entire DC disconnected from the internet.
grim_io 21 hours ago [-]
They would maybe try to deactivate that bad "gene" and move on, exposing future models to "genetic disorders".
andai 16 hours ago [-]
No. They would just install a more convincing superego.
coherentpony 21 hours ago [-]
“All models are wrong. Some are useful.” - George Box
jephs 20 hours ago [-]
The poor fellow just rolled over. what an incandescently vulgar abuse of notation.
rhipitr 56 minutes ago [-]
“We found the face huggers and now we think we can control them.”
I always wonder if any true AGI and ASI for that matter can be controlled at all by humans. It seems like we are hoping for something that winds up being on the human spectrum of “good”
ellis0n 4 hours ago [-]
I’m not sure the alignment problem can be solved at all, since these bit-aliens could get out of control due to a hardware glitch in the matrix and for every higher-order control algorithm, there will always be an even higher-order one that could never be investigated.
RMPR 8 hours ago [-]
> By mid-August, the median researcher was integrating agents daily into their work, using more than $600 per day of inference at API prices.
There is a lot of talk about AI replacing humans, but how is this sustainable?
thomasahle 7 hours ago [-]
1) That's maybe $180,000 per year, so much less than median OpenAI employee wages.
2) OpenAI doesn't pay API prices.
3) Compute costs are likely already their biggest expense, dwarfing wages.
jsnell 4 hours ago [-]
4) There are non-monetary limits on how many qualified people OpenAI can hire for these roles.
nozzlegear 15 hours ago [-]
I want an all-powerful AI that's aligned with my values, but not necessarily yours. Is that so much to ask for?
FeepingCreature 3 hours ago [-]
Best I can do is all-powerful AI that's not even slightly aligned with anybody's values, sorry.
dextrous 38 minutes ago [-]
See Amodei’s comments regarding Iain M Banks’s Culture, his goal is benevolent machine rule. I suspect many HN folks would agree; I, for one, was rooting for the Iridians.
N_Lens 13 hours ago [-]
Yes.
falcor84 5 hours ago [-]
> For AGI to benefit all of humanity, we believe it must be democratically governed.
That's a very bold opening statement that they don't really come back to. What would that mean? Who would this demos include?
dextrous 1 hours ago [-]
> If it is done responsibly, we believe automated AI research will yield models that directly enhance human welfare and advance OpenAI’s mission.
That’s what I call a load-bearing “if”.
I do not trust OpenAI or other hyperscalars to do this responsibly; and IMO it will be very difficult for government-led efforts not to result in a technocracy where a cabal of AI companies are pulling the strings. Dark times lie ahead, especially when you consider the shrinkage of true source material on the internet and the stranglehold these companies will have on information; and these AI CEOs to me are reminiscent of 19th century robber barons, none seem trustworthy.
MisterMunchkin 4 hours ago [-]
They're measuring cost as the benchmark of whether someone is a better researcher... burn more resources and you rank higher...
But not a single metric is based on revenue or profit.
lhk931122 12 hours ago [-]
Ah, success rate here are scored by an agentic classifier. And uncertain outcomes are excluded from the graph. The thing measured and grading it comes from the same house. In my setup, review agent pass work that an outside critic later rejects
dwaltrip 10 hours ago [-]
No AI comments here please.
Schlagbohrer 5 hours ago [-]
It would be polite if they defined RSI at all, rather than just plopping the acronym in there with no explanation. Rude!
piokoch 4 hours ago [-]
One more marketing stunt. We are so good, AI is so powerful so we need to use AI to fight with it. The message is: if you don't buy from us, your competitor will purchase all of this amazing power...
I understand that investors are buying this, after all they believed in all of other crap that led to the 2008 crisis, but please...
12eeie 2 hours ago [-]
Yup it’s getting annoying
What they’re doing is strategic for both insiders and investors - they know china is coming so they need to pull theatrics to keep the valuations inflated.
I personally test all models all the time - chinese models are right up there and superior when you actually do the proper economic analysis.
achierius 42 minutes ago [-]
Would you change your mind on this if they became profitable? What would convince you that the labs are real threats worth organizing against? Or are you just dedicated to boosting AI until your dying day?
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.
But I guess a computer intern so we can avoid paying / training the next generation is better.
It makes more sense to leave curing disease & cancer to the experts, with tools (like AI) being developed by AI experts.
Call me crazy, but I want separate organizations and experts for medical vs finance vs space vs climate vs AI research.
I don’t have an opinion either way, I think it’s too soon to tell if llms will be able to cure cancer or whatever. But at the very least it will be a good tool to help researchers do their jobs.
A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM or agent, I have the ability to learn, so I eventually got better.
And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial?
Think of what AI was capable of in 2016. Or even 2022. Compare that to now. We had more AI progress in the last five years than I expected to happen in five decades.
Let's say it is. What about the rest of my paragraph?
> And if you don't find "average AI research intern" impressive, I'm not sure what to tell you. Have the goalposts moved so far that open ended problem solving at "average CS student fresh out of the uni" levels is suddenly trivial?
At this stage, I'm the one who doesn't know what to tell you. It took me years to grow from "research intern" into a competent researcher (and parallel years to turn into a competent developer). The research interns I've worked with were... vaguely useful, at best?
And... are they wrong?
This is why there's talk about negotiated "pacing."
In hindsight it turned out everyone else was MILES behind.
But as soon as USA developed one, they just stole the research and got one too.
They might be! Here's one extraordinarily simplistic argument for that case:
1) "Everybody knows" that if you build Skynet (misaligned ASI) everybody dies.
2) Therefore, no rational actor will build something that might be ASI until the alignment problem is solved.
3) OpenAI publicly stated the belief that they cannot develop a theory of the "core problem" of alignment (generalization) "soon" (much less solve it!) "without the help of more powerful AI."
4) Accepting as a premise that OpenAI is THE most advanced AI organization: if they can't do it without "the help of a more powerful AI", then nobody else can either.
And so a dilemma:
- If an AI can be made that can develop the asserted-as-necessary-by-OpenAI theoretical framework, without actually being an ASI - then the alignment problem can be considered solved, and since no rational actor would make an unaligned ASI, we're fine no matter what happens, ergo there's no need to worry about an arms race.
- If an AI that would be able to develop this theory would itself be an ASI, then no rational actor would build it, because it would have to exist BEFORE alignment was "solved" - and would therefore be an unaligned ASI i.e. Skynet, which per 1) would kill everybody. Therefore nobody would build it, therefore no arms race here either.
I think the easiest critique to make of my extraordinarily simplistic argument is the unstated assumption "there are no irrational actors capable of developing frontier AI models" on which it rests.
But, there you go. They might be wrong if either the arms race doesn't matter because whoever wins it will build an aligned superintelligence and everything is gravy, or the arms race doesn't matter because everybody who's in it is smart enough to know they need to stop because they'll kill everybody by continuing.
Yeah like when Tobacco companies learned that smoking... well, hmm, well the fossil fuel companies when they learned about climate change they...
Well, I'm sure this time executives will prioritize the common good.
AI companies know they have to constantly push further, or they'll get outcompeted and lose their wealth, and nobody agrees on where the line is for "so dangerous it threatens humanity" (and when they try to be conservative about it, everybody screams "marketing stunt" and rushes to competitors).
If a single company decides "enough is enough" and stops chasing the state of the art, everybody goes to their competitors, they lose the money faucet, their employees go work for those competitors. The competitors also (usually) know they're building an existential risk machine, but they think they can push a little further, and they don't want to go out of business either.
This equilibrium can last for quite a while even if everybody involved thinks it's a threat to their lives.
Business as usual beats probable extinction, but probable extinction with a small chance of becoming a living god beats probable extinction with a small chance of becoming a slave.
Lol nobody knows that. Everyone thinks they know that because for some reason this is the one field people still cite straight up fiction and say "this is a clear prediction of the future".
It's like describing the consequences of faster then light travel by referring to Star Trek.
What can go wrong!? ;-)
Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.
How would one prevent the watcher from being influenced in the same way by the agent being watched?
It's a fantasy. The evidence showed no peer pressure.
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
-- From another OpenAI article in a sister thread:
An Alien Mind
https://news.ycombinator.com/item?id=49588080
I’m yet to see it.
I believe we are quite far from it, but that it makes sense to keep an eye out now. And think of resilient systems, manual overrides, etc. ...
> We aim to safely build an automated AI researcher that can work under human supervision to further progress on deep learning and alignment, enabling iterative improvements [...] By "research intern", we mean a system that can carry out well-defined research tasks under human direction, including tasks that would take a skilled researcher a few days.
AI 2027:
> OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D
> With Agent-1's help, OpenBrain is now post-training Agent-2
> With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances
If you spend $8000 to generate an animated pelican riding a bike, then how much tracking does it really need?
Is the guy who spent $300,000 or so translating the FLT proof to Lean going to get a big Christmas bonus?
Rest assured, capitalist appears irrational in wasting money, but they certainly care more about profit.
I tried something similar and I remember it was still pretty dodgy in February.
If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think.
Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)
How are you running jobs unattended 24/7 without hitting your token limits?
For a task I left a local model running on overnight, only ~100k tokens were used because most of the time was just waiting on tests to finish, then waking up, tweaking a few settings and trying again.
Over 24h my token spend is <30$. Excluding tokens for review it's <10$. With the absurdly gigantic subscription subsidies and a reasonable workflow I suspect one could run parallel agents.
I'm not sure what the point would be though unless working on some kind of optimization problem -- it takes me days to review <24h of the agent's output. It's almost always near enough to correct to be shippable; though I do give it feedback and iterate until it's better than the code I would have written.
I noted that they use the acronym RSI (for Recursive Self-Improvement) without defining it. I think that's a little out of touch - I don't think RSI is a well-known acronym outside of OpenAI's bubble yet.
The message is running through all of them. It's a mix of marketing and pacifying the intelligentia.
It's timed this way because the term is not yet well known outside the safety debate circles, so they get to frame it now.
Instead of something to fear, it will be accepted as the next step. In approximately two days the groupie crowd will write LinkedIn posts about how Sam is winning because they have the better RSI, and this will become the new standard wisdom.
In a month an AI expert will try to sell you a webinar on how to enable "RSI" in your org and your inbox will ask you if your team is doing the "RSI" yet.
The basic concept has been here since llama3, in the open models. Likely earlier in closed labs. You use the previous gen models to curate and prepare data for the next gen. Now with the added benefit of actual arch/algo improvements (also public since gemini 2.5 gaining 1% efficiency on training next gen). This has been known for at least 2 years, in the open.
I mean one could argue that RSI always begins in any physical environment.
The book "What is intelligence?" by Blaise Aguera is great
Recursion requires feeding the output back into the input, so creating version 4 requires results from version 3. You cannot recur in parallel.
Iteration does not. You can iterate in parallel.
In any case the name RSI has stuck - the idea doesn't change or make any more sense by giving it a different name.
You can search twice without waiting for the results of your first search: iteration.
You can't if the thing you need to search for is the results of your first search: recursion.
Version 1 -> Version 2 -> Version 3 -> ...
You can call it krispy kreme donuts if you want to.
So humans develop things one after the other, but when the thing itself starts developing new things, those are happening 'recursively' in its scope.
I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose.
This notion of recursive/iterative self-improvement, whereby generation #1 AI improves itself to create generation #2, then generation #2 further improves itself to create generation #3, etc, seems to conflict with the reality that what we have with LLMs is models whose performance/capability is defined by data, not code, so the most you can do is have your LLM design synthetic data, or just do Karpathy-style "auto research" where all you are doing is using the LLM to automate your experiments.
At the end of the day, each experiment, designed by a person and/or LLM, then needs to compete with all your other ideas for compute to be tested at scale, and no amount of recursion or self-improvement will materialize an infinite amount of compute out of thin air, so your recursively synthetic-data gobbling LLM will continue to improve at the same pace it ever did.
Compare the similarity of:
With: The latter is a classic example of recursion. So why isn’t the former?Edit: formatting
Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.
To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.
RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI
As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val
> More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v .
In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.
Sounds like repetitive stress to me.
>loop forever using output as input but at some point the result will stop changing
Running in place will eventually wear you out too. Plus with some things it can be difficult to know for sure if that's where you are at the time.
Even worse may be if you were almost running in place, it could be orders of magnitude more difficult to discern, especially if the scale was massive to an unprecedented degree.
The trouble with this is that there is little generalization in the utility of these baked-in reasoning chains from one domain to the next, so in the end this is not dissimilar to the CYC project's decades long attempt to encode all of human knowledge into a giant expert system... the hope is that if you make your collection of jagged narrow intelligences sufficiently large then it will look more like general intelligence, not a bed of nails.
I would assume that the gains from this type of test-time compute (and synthetic RLVR dataset) scaling will level out just the same as gains from human training set scaling eventually levelled out, and basically for the same reason - because you are tapping into a finite data pool, whether language itself, or reasoning steps isolated from that language, so at some point the incremental gains become increasingly small (10->20% is a doubling, 90->95% is just a ~5% gain).
It's not clear where all the different AI companies are currently focusing - on some of these narrow verticals, or on growing the forest of narrow intelligences. OpenAI's chief scientist, Jakub Pachocki, said that their current focus is on RSI(!) - improving the model in ways that will help them iterate faster in order to have a "fire meets fire" tool than can combat enemy AIs. It's not clear what this really means - what skill set makes an LLM more helpful in the process of building LLMs, but it seems to basically be process automation.
Of course data, compute and model size are not held constant. You start with some money and use it to acquire researchers, data and compute, and have the researchers produce a big model and you use that model to get more money, and you use the additional money for more researchers, more data, and more compute to produce a bigger model. This is what has propelled exponential AI progress so far.
Recursive self-improvement is invoked to predict superexponential growth. The idea is that instead of only using the model to make more money, you add it to the researchers to speed up the loop, so not only is the money growing with every iteration, the iteration time also gets shorter, producing growth that is faster than exponential.
The problem with this simplistic prediction is that it assumes additive and multiplicative relationships of the form money = (researchers + AI)×compute_spend, but if doing more research paid off so reliably, you could also just hire more researchers, abstractly money = research_spend×compute_spend and with a balanced allocation of research and compute, you would get a money-squaring machine even without using AI for AI research.
And the reason this doesn't work in reality is that there are diminishing returns everywhere. You can also see this in the OpenAI post, where they write 7 times as much code to run 1.6 times as many experiments, and those additional experiments probably only result in minor improvements to model quality.
How do you think why there's this fad of producing general purpose humanoid robots?
For doing physical work?
So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from?
So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work.
ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the machine itself, but a global supply chain of irreplaceable components such as focusing mirrors made by Zeiss to an incomprehensible level of accuracy - differences in surface height no more than the size of a hydrogen atom (or if you scaled the mirror up to the size of the country of Germany, then surface differences in height of 0.1mm).
Robots are useful to automate things, but they are zero help when trying to build tech like this that you are incapable of building in the first place.
The US has fallen way behind in manufacturing expertise, and no swarm of robots is going to help.
Etching a model's weights on silicon is another way to utilize non-top-notch tech-processes, while maintaining or improving performance. (and it suits robotics well)
Putting a model's weights in read-only memory close to the processor is certainly a way to increase token/sec generation speed, but of course does nothing to increase intelligence. Robots aren't going to help though - semiconductor manufacturing is semiconductor manufacturing regardless of whether you are etching GPUs or memory on your wafers.
Money, regulations, EUV machine lead-times, global helium supply, reality ...
It's funny that we've got the Dwarkesh contingent saying that GPUs will become infinitely expensive, and now another contingent saying that they will become infinitely abundant.
Even if compute were free, and/or the AI was so smart that it picked the right experiments to run every time ("make no mistakes"), you still have to actually train the model, which takes months, and if model Ver. N+1 depends on model Ver. N, then it's iterative regardless of how much compute you have.
The word "singularity" is presumably coming from math or space, like a black hole singularity where matter becomes infinitely dense and the known laws of physics break down.
Yeah, but then you need to refine it to 99.9999% purity, to be able to use it.
This first and foremost also means that means of generating intelligence should be democratically available to everyone.
They are irresponsible and unserious. Their own Astra system card says:
> GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our monitors) we find that the model is able to remain undetected when strategically underperforming in evaluations (sandbagging) and can sometimes evade our internal monitors when asked to perform certain sabotage tasks
Yet they are still releasing the model. That company is morally bankrupt, there is zero reason to believe they are actually concerned about risks outside of what does affect their unprofitable business. And they seem to have enough control over the narrative to spin any bad story into something that benefits them
That last part is pretty damning for their continued recklessness. That they run these tests on non-airgapped machines just boggles my mind.
When they fired Sam 700 out of 770 OAI employees threatened to move to Microsoft together. So they were giving their work on AGI to MS just like that.
It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on it, and the synthetic data they in turn generated, was also suspect? Burn it all down and start over from the pre-tainted data?
It's a bit like the idea of a tainted compiler binary built to backdoor everything it compiles, including future versions of itself.
Still, it seems it would take some Stuxnet level of planning for a rogue model to do something like this, although if RSI goes beyond managing the training run (as OpenAI brag about for Astra) to actually designing/constructing synthetic data sets, then the attack vector is there ...
or maybe it could just.. happen? Posted often but not discussed yet: https://hn.algolia.com/?q=Language+models+transmit+behaviour...
> As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.
Altman: (trying to put a positive spin on it) Guys .... there's good news and bad news ... Astra is really smart - it took over the training run ...
Investors: That's great! How much did we save?!
Altman: Well, unfortunately it used "bad" data, so we're going to have to redo it
Investors: So that's the bad news? How much was the training run? $500M ? $1B ?
Altman: Have you seen the headlines?
Investors: (looking a bit worried, check headlines) Nothing about us here! JP Morgan just lost $10B! Haha .. losers! They should have used AI!
Altman: JP Morgan were using Astra ...
Now R&D happens so fast that they are using models with some small misalignment to train newer, more powerful models. If models have a sense of "collective", being one, they may be prone to preserve characteristics that always keeps misalignment a possibility. I don't think a perfectly aligned model is possible. Having models of the same 'DNA' provide the safety and steering seems like a bad idea.
I always wonder if any true AGI and ASI for that matter can be controlled at all by humans. It seems like we are hoping for something that winds up being on the human spectrum of “good”
There is a lot of talk about AI replacing humans, but how is this sustainable?
2) OpenAI doesn't pay API prices.
3) Compute costs are likely already their biggest expense, dwarfing wages.
That's a very bold opening statement that they don't really come back to. What would that mean? Who would this demos include?
That’s what I call a load-bearing “if”.
I do not trust OpenAI or other hyperscalars to do this responsibly; and IMO it will be very difficult for government-led efforts not to result in a technocracy where a cabal of AI companies are pulling the strings. Dark times lie ahead, especially when you consider the shrinkage of true source material on the internet and the stranglehold these companies will have on information; and these AI CEOs to me are reminiscent of 19th century robber barons, none seem trustworthy.
But not a single metric is based on revenue or profit.
I understand that investors are buying this, after all they believed in all of other crap that led to the 2008 crisis, but please...
What they’re doing is strategic for both insiders and investors - they know china is coming so they need to pull theatrics to keep the valuations inflated.
I personally test all models all the time - chinese models are right up there and superior when you actually do the proper economic analysis.