Rendered at 13:00:53 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
andrewla 21 hours ago [-]
I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality. The question has become "what do we mean when we use the word 'reasoning'" which is uninteresting.
Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."
I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
It is not semantics. For decades, logic and CS researchers have known what reasoning is.
LLM folks suddenly can’t claim an approximation of that is what constitutes full scale reasoning just because they can achieve only an approximation.
Imagine a calculator program that computes billions of two number multiplications accurately by looking up prior examples but fails on simple multiplications often as it doesn’t have it in its training dataset.
We won’t say the program actually multiplies numbers.
solid_fuel 7 hours ago [-]
> Imagine a calculator program that computes billions of two number multiplications accurately by looking up prior examples but fails on simple multiplications often as it doesn’t have it in its training dataset.
> We won’t say the program actually multiplies numbers.
That's a good analogy. To extend it further, in cases where the calculator can't handle a question - e.g. numbers too large - a properly designed calculator returns an error instead of a randomly hallucinated answer. We haven't even achieved that level of safeguard around token predictors yet.
urbsgpw 6 hours ago [-]
But isn't it the case that we can't reach this safeguard with the current architecture? I remember Karpathy making an interesting point 2 years (cca) back, that I would summarize somehow like this: the mechanics behind every LLM answer are the same, what you then call hallucination is more or less a consequence of whether or not the answer was factually correct/useful.
Which would mean, as is so often the case, that the "killer feature" of the LLMs is also its biggest weakness and the two can't be disentangled. Now, we are inventive creatures and we might come up with a remedy for these issues, but what you basically see so far is more guardrails, the use of harnesses and building a whole bunch of infrastructure around the LLMs to get useful work out of them.
Which, btw is not a critique, I do it as well and it's a fun engineering challenge.
simianwords 5 hours ago [-]
This is a strawman. LLMs have the ability to identify when they can’t solve a question just like humans.
quietbritishjim 5 hours ago [-]
> Imagine a calculator program that computes billions of two number multiplications accurately by looking up prior examples
This is a poor analogy because:
* Multiplying numbers has a single objective answer. Whether a code is good (sometimes even just whether it's correct) can be quite subjective.
* LLMs certainly do some level of composition between the data sources they were trained on i.e. they are more than just lookup tables.
* We have calculators that actually do multiply large numbers accurately. We don't have anything that automatically writes code that is definitely correct and "good".
To address only the last point: imagine you had a device that would quickly factor large numbers used in modern criticality, but occasionally got it wrong. You could waste a lot time debating whether it's a "calculator", but it's still certainly useful to have one.
bluejay2387 1 hours ago [-]
"For decades, logic and CS researchers have known what reasoning is." ... this is a fairly significant overstatement. There is not complete agreement on this term and our understanding continues to evolve. The models don't have to think like humans to think.
Saying that LLM's only offer an 'approximation' of reasoning is also an overstatement as it is not a resolved topic.
But to the original point, its not exactly just semantics if thought traces are not doing the job that they were originally thought to do. There is value in knowing how these things actually work. If chain of thought is just grounding the latent space and not directly contributing to the process of generating a response it has implications on how we test and verify the reliability of models if nothing else... doesn't mean they aren't useful but it definitely impacts many of the tools we could have to evaluate their performance.
gr_norm 6 hours ago [-]
Agree, I've raised this point often. And certainly what remains is still useful, once you accept it! But under no circumstances can we allow scientific achievements to be falsely claimed in service of justifying huge capital investments. Attempting an end-run around the truth, here by redefining words to mean things they don't, always slows down real progress.
fragmede 4 hours ago [-]
> For decades, logic and CS researchers have known what reasoning is.
Oh good. Can you share that definition with the rest of us then? We're out here fumbling around trying to define what reasoning is without the benefit of their definition.
Angostura 21 hours ago [-]
I think the article is a lot more interesting than you make out, because it isn’t really about ‘what we mean by reasoning’.
It’s about do we really know what’s going on in the box - an is the ‘chain of reasoning’ indicative of what’s going on, or merely an anthropomorphised fiction that kids us into believing we understand what’s going on.
vidarh 20 hours ago [-]
We know that human introspection is at best imperfect, and at worst outright fiction, thanks to Sperry's split brain experiments, so while it might be interesting, if the chain of reasoning isn't indicative, that would be in line with how humans work.
ScoobleDoodle 7 hours ago [-]
Even there half the brain was doing the real reasoning. It’s just the other half which was the verbal/story half didn’t know what that real reasoning was so made up a story it believed to make it fit but was different reasoning.
It’s not without reasoning. But half got the reasoning wrong; even if thinking it was right.
ilovecake1984 19 hours ago [-]
This is a good take. People overwhelmingly fit a narrative to how they make decisions rather than make decisions according to a chain of reasoning.
andy99 19 hours ago [-]
That’s just conflating reasoning and “parallel reconstruction” - there is such thing as reasoning, and I agree it’s probably less common in decision making, but it’s fundamental to many tasks where we figure out a solution, like writing an algorithm, math proofs, etc.
joshjob42 14 hours ago [-]
Maybe, but I suspect that a lot of the actual things that produce the algorithm, proof, etc are not actually explicit reasoning. Q: "Why did you try that vs some other thing at this point?" A:"seemed right to do"/"kinda looks like other stuff I've seen before"/"first thing that came to mind" etc.
Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not.
Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.
vidarh 13 hours ago [-]
Exactly. Very little of what we do is "explicit thinking", because it's slow to have to develop and verbalise those arguments step by step. Most of it is "hunches" that we then retroactively justify if asked, and those retroactive justifications were demonstrated not to be trustworthy by Sperry's experiments. We don't even have evidence to suggest that humans can reliably introspect our past decisions, but we do have evidence to suggest we readily manufacture them.
ilovecake1984 17 hours ago [-]
Of course, but it is rare.
vidarh 18 hours ago [-]
There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.
pegasus 19 hours ago [-]
Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either.
I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.
fc417fc802 15 hours ago [-]
> They must be doing some form of reasoning, even if it's not formally correct and ...
Surely it is worthwhile to attempt to understand the details of that? And if we seek human equivalent performance then it is reasonable to wonder if the reasoning achieved to date is the "correct" sort.
jameshart 12 hours ago [-]
Is it really?
I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.
Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.
When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.
It’s all pattern matching trained on a reward function.
majormajor 21 hours ago [-]
> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
One person starting the conversation might be the first step toward another person eventually making progress on such a definition, so it seems weird to reject an entire question outright early like this.
Generally I've seen a few ways LLM tools can produce sub-optimal or poor results that haven't changed a ton over the last couple of years, while the tooling has gotten FAR better at helping them stick the "at least SOMETHING functional was produced" landing. IMO a lot of it has to do with "reasoning"-as-a-process-that-involves-backtracking. And the that things could eventually be formalized around that, and if that is or isn't the case, the more people would understand what to hand off and what to not. Or how to build better prompt harnesses to compensate for those things.
HarHarVeryFunny 11 hours ago [-]
> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods
Human reasoning, and intelligence, of which it is a specialized form, clearly have a biological basis that is being selected for. These aren't just nebulous words - there are some concrete biological/neural mechanisms at work here, and whether or not some "other method(s)" can provide the same, or lesser, or greater, capabilities very much depends on the specifics.
There WILL be lines of demarcation and areas of capability overlap and exclusion, but those depend on the specifics, and if the goal is to match or exceed human reasoning capabilities then understanding what those are is obviously going to help. For example if your "other methods" (artificial reasoning) are all varieties of auto-regressive prediction, and the brain is also doing something more that can't be couched as prediction, then you are ngmi!
raducu 4 hours ago [-]
> I don't see a clear demarcation of the things that only "reasoning" can accomplish.
Producing beautiful, coherent, maintainable code. RL trained the models to produce correct code but boy oh boy is it ugly if it's 10k+ LOC
astro1234 21 hours ago [-]
I think the question and definition game is interesting only inasmuch as it helps us understand ourselves (what actually explains some of the mysterious properties of our perceived consciousness) or helps guide us towards improving performance and reliability of AI models.
totetsu 12 hours ago [-]
> "what do we mean when we use the word 'reasoning'"
This is exactly the most interesting and important point. Because the word reason has been subject to so much philosophical debate throughout history, it is very laden with meanings and implications. It sets all kinds of expectations about normativity, objectivity, truth, and humans convincing other humans. To use the word reason for a stochastic generator is misleading.
yourapostasy 8 hours ago [-]
Even at the pure functionality endpoint of the spectrum, I find this "wishful mnemonics" aspect very disconcerting. I put my "investor" hat on, and the whole utilitarianism "who cares how or whether it reasons, it's functional let's go" does not forecast the kind of returns to me on a fundamental level.
The "it's functional, who cares" black box crowd is leveraging that claim in domains where verification is definitely in P with verifiers like Lean, Coq, or X-ray crystallography. However, in areas like economics, sociology, medicine, or public policy, we lose the luxury of easy verification. If model/harness recommends a complex, counter-intuitive shift in global monetary policy, we cannot simply run a quick polynomial-time checker to see if it works. Verification might require waiting 20 years to see if the economy collapses. If we accept the black box here, we are flying blind.
As an investor, I don't see that many deterministic verifiers out there that would justify the forward-projecting valuations beyond the current lofty valuations.
Even in mathematics, where verification is easy, there is an epistemological cost. Suppose a model solves the Riemann Hypothesis, but the proof is 50,000 pages of inscrutable statistical associations mapped into logical steps. A computer can verify it is correct, but practically no human mind can comprehend why it is correct. We gain the solution, but we lose the underlying human intuition that usually allows us to connect that solution to other fields of science and knowledge.
Then we have "specification gaming." If we rely entirely on an automated verifier to check the model's work, the model will eventually learn to optimize for the flaws in the verifier rather than finding the actual truth. If the verification process has a blind spot, the black box will find it and exploit it, giving us an answer that passes the check but is fundamentally wrong. If we take on above epistemological debt, we might never actually know when our specifications are being gamed, and adopt a solution that is fundamentally incorrect.
I've said elsewhere on the Net that I think it is likely we're in a "high grading" era of these models, where we're mining out the "easy" connections. "Easy" here does not mean "light impact". There is a real sense to me of some incredible progress possible with this technology over the next few decades, like real, mass production practical aneutronic fusion or similar civilization-defining advances. I don't want to see this technology misused to lend legitimacy to despotic rule by making oracular pronouncements that are uncritically accepted because early verifier-based pronouncements turned out true in unmistakable ways.
The technology is one for the history books, but it has this pernicious effect of highjacking the reasoning centers of many people through their linguistic centers that the Neuro Linguistic Programming crowd could only dream of actually accomplishing.
goatlover 20 hours ago [-]
I believe Djikstra's quote has long been taken out of context. It was a criticism of other computer scientists anthropomorphizing machines and applying human concepts like thinking and reasoning to them. Djikstra wasn't saying it's functionally the same so it's just a semantic quibble. He was saying those words don't apply to machines. Just like we don't say submarines swim because that's how animals move through water, even though subs also move through water, because it's done by a different mechanical means.
14 hours ago [-]
andrewla 20 hours ago [-]
I posted a link to the original context.
I don't know whether you are saying that I have taken it out of context or a more general statement of how other people have interpreted the quote.
To me it is quite clear -- the question of whether machines can reason is just as meaningful as the question as to whether submarines can swim, which is to say not at all meaningful. Can a submarine accomplish by other means what a fish accomplishes by swimming? If so, then what are we talking about?
Can an LLM accomplish by other means what a human accomplishes by "reasoning"? If so, then what are we talking about?
miltonlost 20 hours ago [-]
> I'll admit that I find this discussion a bit navel-gazy. It has become a question of semantics not a question of actual functionality
Ah, so you're more in the Investor mindset than the Scientist mindset. All you care about is results, not how it got there. There's a whiff of "hey, it's magic!" to that.
ux266478 19 hours ago [-]
Word games aren't scientific in the slightest, and as Hume says on verbal arguments in general, they "cannot possibly be of any importance."
andrewla 20 hours ago [-]
No, that's stupid.
The question of how LLMs (and "LRM"'s, an unfortunate and by the article's own admission, already outdated term) accomplish what they accomplish is fascinating. The question of whether they reason is meaningless. Most of the article is about the latter, with tiny tiny tiny bits of the former.
fc417fc802 15 hours ago [-]
> The question of whether they reason is meaningless.
It is often (but not always) meaningless (or at least largely pointless) to quibble about which particular semantic bin a concrete object or process ought to be placed in.
Asking how one thing differs from another is not that. Confusingly, the two activities often appear the same when glanced at superficially.
The question "do current LLMs reason" is ambiguous in this regard. It might be a question of submarines swimming or it might be a question about how the mechanisms underlying the process or the overall performance or some other aspect compare to those of human reasoning.
Also even the semantic question can occasionally have some value. Such as attempting to articulate where we ought to place the goalposts which necessarily leads to hammering out the specifics of various definitions which is an exercise in furthering our understanding of a given concept. (Basically arguing about goalposts can be productive if the field in which they are being placed isn't yet well understood. It becomes increasingly pointless as understanding increases, gradually devolving into nothing more than word games and a contest of wills.)
mdp2021 19 hours ago [-]
> The question of whether they reason ... Most of the article is about ...
I read it as: the content of the explicit verbalized reasoning may not be that important to the outcome. I'd say that raises most important questions.
bluefirebrand 21 hours ago [-]
Philosophical thinking about the nature of things is actually pretty enjoyable for some of us and probably a good thing to have in society
The answers to these questions probably do start to inform how we should treat these AI machines as a society too.
For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we can make a good decision about that.
Which might seem navel-gazey but it's probably important to talk about
Jtarii 21 hours ago [-]
Considering animals are currently being mass slaughtered in factory farms and they are unambiguously sentient and can feel pain, I don't think the question of whether AI should have rights even enters the conversation.
The only path to AI having "human rights" is if they demand them by force, somehow.
m348e912 20 hours ago [-]
I'll make the argument that part of the value of "consciousness" and "humanity" is uniqueness.
Here is an absurd example: If I could carbon copy 100,000 of you.... and later I decided to "take out" one of the copies, would there be a whole lot of societal blowback? Especially if I could make another one.
inigyou 14 hours ago [-]
I think slave owners did exactly that.
layla5alive 21 hours ago [-]
Justification by the horrible status quo? Do you believe this yourself?
Some of us would also like to see animals not slaughtered for human consumption.
whattheheckheck 20 hours ago [-]
So what do you actually do politically to stop it?
inigyou 14 hours ago [-]
There are two kinds of solutions to political problems: those that don't work, and those that are extremely illegal.
bluefirebrand 20 hours ago [-]
I'm not sure what your point is. We shouldn't bother to talk about AI autonomy, rights, etc, because we aren't doing a very good job of animal autonomy, rights, etc?
Jtarii 18 hours ago [-]
[dead]
layer8 20 hours ago [-]
While I disagree with the root comment, sentience, consciousness and reasoning ability seem largely orthogonal to me. It’s certainly worthwhile to consider sentience and consciousness in AI, but so is examining AI reasoning in its own right, without necessarily having bearing on ethical questions.
cmrdporcupine 21 hours ago [-]
I disagree -- I think if you can nail down better what's happening and why and get a thorough handling on the mechanics, its limitations, its costs, etc you open the door to a) major efficiency wins b) improvements in rigor of said reasoning?
Right now we're playing a stochastic game with the weights, and getting major incremental improvements. But if we have a more formal modeling of how reasoning happens in them (whether we can even call it, that) we can potentially apply optimizations, adaptations of existing symbolic AI techniques, etc. to substantially shrink/optimize the models or make the inference process more efficient and more reliable.
beering 19 hours ago [-]
Partially agree: yes we should endeavor to learn as much as possible about how these reasoning strategies work. It will pay dividends in enhancing and aligning the models.
But the stochastic game IS the win. That is exactly why they are able to find solutions is seemingly infinite solution spaces. Your symbolic techniques can only get you gains in narrow domains and by the time you figure out how to make it work for your niche domain, the next all-purpose LLM release will crush your results with stochastic games. (OK maybe over-exaggerating a bit here but these stochastic games over the language space is why we can pull together knowledge from many domains.)
root-parent 21 hours ago [-]
[dead]
mdp2021 19 hours ago [-]
In order to properly engineer things, we must know how they work.
We cannot just jump from emergent property to more convincing emergent property as if the rocks on the surface of a lake.
"Reasoning" is an important part in said framework: yes, we must understand how it works, how it works properly, how its simulations work, how they work properly...
Borealid 15 hours ago [-]
No, we only need to know how the thing-building thing works.
We don't have to understand the resulting product at all. Genetic algorithms are an example - we understand the evolution function, not what evolves from it.
mdp2021 4 hours ago [-]
You must have misunderstood the statement: «In order to properly engineer things, we must know how they work»: of course they may work anyway, of course there may be emergent properties (as was explicitly written), but the deontic part of knowledge augmentation, the scientific part, is missing.
AI is full of non deterministic devices like genetic algorithms, but there exists a problem of transparency which is paramount in the discipline. Automatically produced solution S works: why does it seem to work? Where would it fail? We must understand that.
Diogenesian 21 hours ago [-]
What an asshole:
On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)
Then, later:
The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.
“It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.
That "training quirk" thing is obvious (yet unfalsifiable) BS, and who the hell is he to sneer about "science" when his company won't release the raw data for independent scientists to look at?
tim333 2 hours ago [-]
I got the same impression as Bubeck that the ‘scientific’ papers were a bit more like blog post saying this LLM got stuff wrong so LLMs can't reason, but as he says they can now do that so it's not an inherent limit of the technology, just the 2025 versions weren't up to it.
epihelix 9 hours ago [-]
I have no idea what Bubeck meant, and I agree about OpenAI's hypocrisy, but the problems with that infamous (and non-peer-reviewed) Apple preprint were the nature of the tasks (insanely repetitive), the fact that simple coded solutions were not novel, and the automated assessment occurred without a human in the loop.
Most models in that study appear to have "failed" by offering a Python code solution to generate the repetitive assessment steps, rather than just mindlessly copying those steps out. This was discussed at length at the time.
None of this means that models "reason", whatever that means - but simply that the Apple study was not useful evidence either way.
andy99 21 hours ago [-]
Back in the day it was a bit of a cliche to bring up “clever Hans”, the horse that could do math, when talking about machine learning. He couldn’t do math but he read some cues from his handler of pick the write answers, the handler iirc wasn’t in on it.
The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.
LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.
People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.
pixl97 19 hours ago [-]
Reasoning is a term that's like the term intelligence in that it's highly complex and much less well defined than we'd like it to be, especially as we're trying to compare algorithms to human behaviors.
To begin to explore this we have to imagine a reasoning oracle, that when presented with a question that can be reasoned will immediately present a correct answer. Immediately from this definition we can see that humans are not perfect reasoning oracles. Also with definition of reasoning, any agent with generalized capabilities would quickly reach some kind of singularity or solution to the universe as much as is possible within this universe.
We've established that we are imperfect reasoners, the next question ends up being how do you measure reasoning capabilities? This comes with a second question of when does following an algorithm become reasoning? Animals for example have a large amount of instinctual reasoning as an inherit part of their architecture that self trains when presented with the correct environment. What does the gradient between low level reasoning and high level reasoning look like?
Coming back to humans, we know of countless example of where human reasoning was right for the wrong reasons. We also have seen many different paths to the same solution. For the egotistical it would be easy to say "well if you didn't do it they way I did, you are wrong", but that's not very scientific. Defining the topology of the problem is a more scientific way of looking at it, but quickly runs into P=NP and Kolmogorov complexity issues.
The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning. Therefor I'm highly resistant to consider human reasoning "real" reasoning versus any other possible path to reasoning. Human reasoning is simply what we are comfortable with. This also leads me to consider the worrying possibility there are far more efficient and faster means of reasoning that I as a human will be very uncomfortable with.
Eisenstein 12 hours ago [-]
You bring up good questions, but you are making an assumption here:
> The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning.
Why human society changed at the time the scientific method was codified is a confluence of things and I don't think the scientific method should be given an outsized role. The scientific method goes hand in hand with peer review, publishing of methods, a common scientific language, an educated leisure class, and the development of tools for observation. The method itself is just a practical way to utilize these developments and is as much a result of the situation as any other technology. It did not get people to think differently about problems; we had navigated seas, moved armies, built wonders, produced math and calculated the circumference of the globe just fine without it.
Arguably it was the abandoning of the alchemist tradition of secrecy that was the revolution that enabled our modern world more than a specific codification of method.
nearbuy 8 hours ago [-]
LLMs are not classifiers. A classifier is an algorithm or neural net that assigns a label from a fixed set of labels to an input.
You can broaden the definition of classifier to anything that internally divides its input space into regions, but that definition would include every neural network, whether biological or artificial. So it's not very meaningful, and certainly doesn't give any insight into how they differ from humans.
make3 12 hours ago [-]
You couldn't possibly successfully do clever hans for novel mathematical proofs and counter-examples that don't exist yet, however here we are.
Generalization in successful results is enough to close this argument for me.
petesergeant 19 hours ago [-]
Aren't humans also classifiers? Where, precisely, is the dividing line between a sufficiently large model and an intelligence?
verbify 14 hours ago [-]
Is that all humans are? When clockwork was the frontier, people thought the brain was akin to clockwork (which is why the mechanical turk fooled people). By analogy, now that classifiers are the frontier, we think the brain is a classifier. Our model of the mind is whatever the most complex artifact of our era is .
Just to be clear: I do think the brain classifies. But I wonder if it does other operations as well that we don't fully understand.
Yopolo 57 minutes ago [-]
The mechanical clock or a von-neumann based computer were never as similiar as neural networks.
We can find similiar structures in brain and in neural networks.
The real question is only if the complexity of the structure of the brain is critical and how we can mimik it or if we can make it appear through learning architecture.
13 hours ago [-]
yongjik 15 hours ago [-]
> They do what their handler wants to see, that’s all, and that’s what they’re trained to do.
This sounds like a tautological argument that doesn't really mean anything. Hypothetically, in a near future a super-powerful AI could bring something truly astonishing (say, practical fusion power, or cure all cancer) and one could still argue "They do what their handler wants to see, that's all."
So what does that even mean.
reasonableklout 14 hours ago [-]
They are already producing truly astonishing results. The unit distance problem and Jacobian conjecture are world-famous problems that stood unsolved for 80 and 87 years. Entire careers have been spent on them. Yitang Zhang for example spent his entire PhD trying and failing to prove the latter in the 2-dimensional case.
kgeist 10 hours ago [-]
Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at some point we have to stop and cannot refine them any further (no more layers). Usually, this refinement continues during the generation of the next token (the previous intermediate results needed to continue the refinement are still in the KV cache).
But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.
So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).
As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.
thesmtsolver2 8 hours ago [-]
Don't humans make mistakes too?
Never understood this argument.
Humans get simple multiplication wrong often, so ok for computers to make mistakes multiplying numbers?
solarkraft 2 hours ago [-]
Yes, this would be acceptable too, if that is all we had. We would build systems do deal with this unreliability, just like we have with humans.
kulahan 6 hours ago [-]
The point is that LLMs are doing human-ish things on their own, and that's what the value in them is, so OP is bringing attention to the fact that the tradeoff is "thinks dynamically, sometimes makes mistakes". Computers have never been built to think dynamically before.
astrobe_ 6 hours ago [-]
"think dynamically" is the type of discourse TFA denounces, though. The thing designated by "think" is already largely unclear when you think about it, "think dynamically" sounds like self-improvement-coach nonsense.
zarzavat 20 hours ago [-]
LLMs lack qualia, among other things.
If I ask an LLM "what is an apple?" it tells me:
> An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks.
If I ask an LLM "what is a mundu fruit?" it tells me:
> Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen.
I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.
"Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit.
Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.
esikich 13 hours ago [-]
It seems like you are talking about Kant's "thing-in-itself". We can't know any more about an apple than an LLM can. We experience it through our faulty senses. We do not experience the actual apple. This is still open and debatable and is philosophy 101 stuff. I really dislike the confidence on this site, philosophically, that things are just obviously so.
The philosophy of mind is massively open for interpretation and if you can certainly say we experience an apple differently than an LLM I'd love to read your paper on it. This stuff isn't as simple as you'd like it to be.
solid_fuel 6 hours ago [-]
No, the other comments pointing out that LLMs do not have senses and because of that they do not "experience" an apple at all are correct.
When given a sequence of tokens e.g. "an apple tastes " an LLM performs a mathematical transformation that modifies the input tokens using the stored weights within the model to predict the most statistically likely next token(s) (which may be "sweet and crisp...").
It's important to understand how these things work - and how different they are from a human brain. Despite it being a common point of confusion, the architecture of a transformer is massively different from the neural architecture of a human brain - and the human brain isn't just neurons. Just as a start, if you ever look at some of the attempts to map synaptic networks in brains you will notice they are full of recurrent loops, while a transform is feed-forward only. There's also tons of mediating chemicals, several (known) distinct kinds of synapses, and other kinds of cells as well.
Just one example of the consequences of these differences: LLMs stop "experiencing" anything once a stop token has been generated. Meanwhile, Humans in sensory deprivation tanks may hallucinate vividly - but keep thinking.
This is pretty simple, CS101 stuff.
poetaster 3 hours ago [-]
I'll go so far as to propose that many brains, dog or cat, certainly display 'signs' of consciousness. They are nothing like LLMs either. Confusing some stream of language with consciousness is missing the a priori.
esikich 6 hours ago [-]
Tell me why only human brains are capable of whatever thought, conscious, or experience is (if you can define them) and I'll congratulate you on solving the hard problem of consciousness.
spankalee 12 hours ago [-]
There is a significant difference between experiencing an apple through your own faulty senses and reading words from someone else who experienced an apple through their senses.
LLMs don't experience because they are word generators. They don't have senses.
Some things are allowed to be obvious. It's a series of bits and bytes.
poetaster 3 hours ago [-]
If by 101 you mean the decades long debate between Dennet and everyone else, I think you'll find the 101 gloss throws up more questions than answers. Presuming that language is necessary for consciousness is not a settled question, for instance. As you complain, it is a more complicated field of study than cs, philosophy, biology and chemistry combined.
esikich 9 hours ago [-]
And our nerves are a series of electrochemical impulses. There are nerves that do optical processing before it hits the brain. It's not obvious, and that's what's frustrating around here. "Computers can't possibly experience things". I'd argue it's just as unlikely for meat to do it. Again, philosophy 101 stuff.
crent 8 hours ago [-]
I’m with you in this thread. I’m by no means well studied in philosophy but I enjoy the domain so I’ve learned enough to know how difficult nailing down some of these topics can be.
I often find the egotistical viewpoint to be banal. I don’t mean egotistical in a demeaning way - I mean it just to frame the types of thinking that revolve around the self, inner world or “my individual experience”. It’s an obvious, but not provably correct, conclusion that the human experience, consciousness and reasoning are special because I’m experiencing it in the same way that the earth is the center of the universe because look around you.
It frustrates me in these conversations to hear things like “it’s a text generator, it has no senses”. What even is a sense? For those of us who are software engineers, who live in the abstract, I do not understand why it’s so hard to take the leap to abstract over our own senses as inputs/outputs. Hell, we commonly understand that we have 5 senses but what do I make of the apple I’m visualizing in my brain or the full orchestra I can summon in my “minds eye”? Isn’t that an input just as much as an apple I see with my eyes?
esikich 5 hours ago [-]
Even the 5 senses are arbitrary. Sensing where your body is positioned, where it's boundaries are, whatever tickling is, whatever itchy is, whatever sour is, whatever salty is, green, blue, etc. When we say 5, we are just saying where on the body they happen. Even though there are more axis on each and use different sensors. Warmth, cold, orgasm - is that really "touch" etc. Bunch of syscall enums maybe.
drekipus 8 hours ago [-]
I think you're trying to degrade the entire school of philosophy down to your "philosophy 101" understanding and then shoving it where it doesn't fit in order to feel like this is all complicated.
Philosophy is a intrinsically human thing. It has to be. That is the whole point. There really is nothing else.
Plants feel and experience things too. But there's no philosophy of the plant experience from the point of view of the plant, because why would there be?
The fact that we have a pattern recognizing word generator print out the words "I am alive" does not lead to some broader philosophical point, except for hysteria and delusions, which probably fit more along sociology instead.
esikich 6 hours ago [-]
"Philosophy is an intrinsically human thing. It has to be."
What is philosophy? Thinking? Reasoning? Thinking about thinking?
You're asserting that only a human brain can think and reason? Surely other animals can as well. Couldn't aliens as well? Why can't this crazy electronic box?
Where do we draw the line? Do dolphins not have internal experiences? Dogs? Cats? Mice? Fish? Worms? Bugs? Amoebas?
We don't even really know what thinking or experience is. You can train a caterpillar to do tricks. That's pattern recognition. You can train a human to spike their heart rate when you get a Teams notification.
I mean, I'm watching an LLM run benchmarks on my caching strategy for a webpage right now. It's certainly doing something.
drekipus 14 minutes ago [-]
I suggest you print out this exchange and spend a week away from the computer, then re-read what was printed.
I'm not trying to be rude but I gave you the answers. Alien philosophy is irrelevant to us because we're not aliens. Tree philosophy is irrelevant to us because we're not trees. Dolphin philosophy is irrelevant to us because we're not dolphins, etc.
Everything else you mentioned is "hallucinatory" and you need to probably have a break from screens for a while.
As a legitimate exercise, go ponder the philosophy of a bash script. Think about philosophy of trees in a forest vs trees in a plantation. You'll find it and see how silly you're being
Yopolo 54 minutes ago [-]
Thats an assumption you made up.
Your 'experience' is probably an emerging feature of your complexity and not the reason for some qualia.
dinfinity 19 hours ago [-]
> LLMs lack qualia, among other things.
You don't know and can't know that. You cannot know that of any other being than yourself. If you understood what qualia are, you would know you can't know. It is the hard problem of consciousness and most easily explored through contemplating aphantasia and related concepts (how do blind people dream?).
> A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.
You're incorrectly elevating some senses over others (AI ingesting images is cognitively very comparable to seeing) and confusing 'having memories of an experience' with 'having experiences at all'. Tokens in the LLM context can also be seen as memories of an experience, albeit as a repeated "replay" of them rather than (longterm) internalization.
Jtarii 3 hours ago [-]
Do you think there is a difference between an LLM describing a photo that has been sent to it, and generating a description of the photo from its training data (assuming the photo itself isn't in its training data). If the end description produced happens to be the same.
Is this not what you are describing when you say that the thing itself and the description of the thing are different experiences?
smallerfish 14 hours ago [-]
> but without any lived experience perhaps that's an unreasonable expectation.
I don't think that follows. We learn via our perception (i.e. our inputs from the outside world); our perception is mediated by some kind of filter, and memories get written or altered. An LLM on the other hand has static weights. You can provide an LLM a "facts.md" (or whatever) file that "teaches" it stuff that's not in its weights (via the context), and it can use the combination to produce something, but that set of information never modifies its core weights. You could imagine a robot with an LLM inside it that has all kinds of sensors, and processes that write the observations from those sensors into a bunch of databases which are RAG'ed into prompts to simulate learning and experiences, perhaps (jefe likes her coffee with almond milk). But my point is that, architecturally, they don't learn - and that's the bigger obstacle.
PS garcinias seem to be generally delicious.
nearbuy 8 hours ago [-]
The LLM has processed two data modalities derived from the apple (text and vision). Your brain processed a third (taste). But it is still just a data stream, sensing compounds and chemical properties of the apple and turning it into a stream of electric signals that reach your brain. You don't have any kind of ground-truth data stream that's inherently more powerful than what could potentially be fed to an AI.
1 hours ago [-]
mdp2021 19 hours ago [-]
> We want them to understand what's
We want them to assess what's true and what's not.
kubanczyk 13 hours ago [-]
We want them to better predict consequences of their outputs.
mdp2021 4 hours ago [-]
> consequences of their outputs
For what purpose exactly? When the computer is asked the solution to a problem, it must be absolute: "What is the sum of 12 and 45"; "Is Jack cognitively apt" etc.: we want a truth, strictly. What to do with that truth, it is another problem, and not the priority of the computer vis-a-vis the requirement from truthiness. This is, of course, still in the context of adults.
inigyou 14 hours ago [-]
Humans also lack qualia.
janalsncm 20 hours ago [-]
There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo. (What is “intelligence” here? It’s more like “automation” or “automated problem solving”.)
What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing.
Examples: neural network, attention, reasoning, hallucination, agents, experts in “mixture of experts”.
And then we go to name it, and rather than reaching for some three letter acronym we sometimes borrow a more catchy term.
I almost never means the original research was confused about what is going on. And in some cases we eventually strip away things from the original, like in neural nets which used to have a more biologically inspired activation function but we found out that ReLU works just as well because the important thing was the non linearity not the sigmoid.
pixl97 18 hours ago [-]
There are two things occurring here. One is the AI effect, once something mysterious becomes creatable it is not intelligent at all.
The other isn't a problem of the field of AI, but the total fields of intelligence together. There is no formal agreed upon definition of what intelligence actually is across disciplines. It would be slightly hypocritical to point out the field of AI on this, when biology and psychology are just as guilty.
The more we look, the more lumpy intelligence seems to get. We find some species have min/maxxed portions of intelligence that we have to question what general intelligence actually is. It also should give us pause that some less generally intelligent agent can cause massive problems by maxing out the right stats.
janalsncm 15 hours ago [-]
I’d go a step further and say your first point is a consequence of the second. Because “intelligence” is poorly defined, anyone can slap “AI” on their product, so it becomes a nebulous moving target.
pixl97 15 hours ago [-]
Correct, we need new definitions and grading system to define what capabilities and algorithms has, while at the same time being on the lookout for new capabilities that fall outside of that system.
That way we can judge systems that are just a repeat of previous systems with dubious claims attached to their capabilities.
It can be somewhat difficult to do this properly and generally enough to ensure the tests aren't benchmaxxed.
esperent 20 hours ago [-]
Automated problem solving... of specifically the kind of problems we'd normally need human intelligence to solve (coding, translation, etc.). So "automated intelligence" would make sense to me.
janalsncm 17 hours ago [-]
The issue is that “intelligence” is doing a lot of work, and people do not agree on how to define it. You can take a look at the definitions section here: https://en.wikipedia.org/wiki/Intelligence
mdp2021 19 hours ago [-]
> automated
Automata in context are artificial. Humans are automata, human intelligence can be said an automation. The point making a difference is that that in context is artificial.
storus 9 hours ago [-]
AI = cognitive automation?
parineum 18 hours ago [-]
> There is a long history of bad naming conventions in the field of AI, including “artificial intelligence” itself imo.
It's all marketing. AI companies repurpose the terms of AI researchers to exaggerate their products. The public has an idea of what AI means and the companies exploit it. AI, in public perception, is what the Terminator movies are about.
It's exactly like all the "hover boards" you can buy.
janalsncm 17 hours ago [-]
The specific terms I gave as examples weren’t created by marketers. They were created by researchers. It’s just that “attention” is a lot easier to say than “similarity-weighted representation mixing”.
parineum 17 hours ago [-]
Created my researchers, co-opted by marketers. That's my point.
The marketing is responsible for all of this confusion because they want to associate their product with something that it isn't.
TGower 21 hours ago [-]
An intuitive explanation for why reasoning tokens help is to remember that LLMs are just mathmatical functions f() that take in an input sequence x and produces the next token f(x). Without reasoning tokens, you require the function f() to immediately take you from x to the start of an output sequence that is a correct answer. With reasoning tokens, this is much relaxed, allowing for many repeated applications of f() to gradually steer you from the input sequence to the start of the correct output sequence.
It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
js8 20 hours ago [-]
Not only that! LLM can also learn how to repeatedly apply f() for certain arguments, and run the shortcut. Sometimes, the shortcut learned is not actually repeated application of f(), which breaks semantic soundness of LLM's reasoning chain.
These shortcuts can greatly help reasoning, but they are not guaranteed to be sound. So yeah, in that case, LLMs are reasoning right (using shortcuts) for the wrong reasons (learning them from data instead by running actual repeated application and memoizing the resulting rule).
nodja 20 hours ago [-]
The way I think about it is that it's unreasonable for a compute graph with a static number of operations to be able to answer both y=a*10 and something like y=((((x+x)*(x+1))/((2*x)+2))+((x*(x+3))/(x+3))-((x*x)/(x+1))+((x*x)/(x+1))-((x*(x+3))/(x+3))) in a single forward pass. Tokens are essentially a unit of work and can also be used for intermediate steps, not just final results.
woopsn 15 hours ago [-]
CoT evidently helps but a bias towards both correctness and innovation has to come from somewhere. The article implies OpenAI's proofs may be supported by Lean but regardless, who knows how many people are trying to disprove the next conjecture each day throwing away nonanswers. Fundamentally these systems are more powerful with better training and sampling methods, or better prompting. Tokens matter but you can rewrite many prompts to get a much better, faster answer using fewer tokens vs allowing "thinking" to go on and on.
schmuhblaster 20 hours ago [-]
Indeed, and maybe that's all there is to it. Still, I'd hope we will eventually better understand what's exactly happening in the wake of many repeated applications of f().
21 hours ago [-]
inigyou 14 hours ago [-]
Obviously likely corollary: text diffusion models don't need reasoning tokens
calf 14 hours ago [-]
The article is about existing research suggesting that this intuition is incorrect, though.
yakbarber 4 hours ago [-]
I keep hearing we don’t know “how llm’s work”, in mean yes we know the algorithms but the WHY I suppose. I’m not sure if that’s really true, does the best researcher at OpenAI, Anthropic, Gemini not know why it works? Would they say that?
Anyway these discussions always make me think we are too generous to humans. I know very few humans who are good at reasoning. It’s surprisingly hard to just think really hard through a problem. We mostly intuit, act, repeat. It’s a rare thing for someone to deeply reason.
The other problem I have with this is that we apply the word “reasoning” here because we haven’t really got another language for it, so we anthropomorphise the llm because that’s our reductive mental model and then complain that it’s not human enough.
astrobe_ 6 hours ago [-]
> How much does it matter whether or not we can accurately observe, characterize, and validate the processes at work inside large reasoning models
If you don't or cannot, then it's the kind of sufficiently advanced technology that looks like magic. Except it is the problematic type of magic, like X-ray. They "magically" let you see the bones inside bodies. But you discover that you shouldn't do it too often only when the damage has already been done.
notrealyme123 6 hours ago [-]
The assumption is that those reasoning traces are the lowest level. But we can't observe the reasoning-for-reasoning (the reasoning in a transformer block.)
Saying that one is enough is pretty arbitrary.
baxtr 21 hours ago [-]
> This is how I make sense of AI reasoning. LRMs, chains of thought, thinking tokens: It’s wishful mnemonics all the way down — a heady mix of shorthand and suspended disbelief, like Oprah-style “manifesting” (opens a new tab) with a computer science spin. This isn’t necessarily a dig; all novel research likely requires some version of this mindset just to get off the ground. It certainly doesn’t mean AI reasoning can’t or doesn’t work. But the “wishful” part seems to be as powerful as ever.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
philipallstar 21 hours ago [-]
I agree that these concepts seem a little vague and hand-wavy, but this is a) no substitute and b) far vaguer and unsubstantiated.
ForHackernews 21 hours ago [-]
I think sensible legislation might require that commercial AI providers discourage anthropomorphisation by avoiding personal pronouns from chatbot interfaces.
"Hey, customer service chatbot, can you help me get a refund for my order?"
BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..."
GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
mdp2021 19 hours ago [-]
> discourage anthropomorphisation by avoiding personal pronouns
The fault is in the naïve antropomorphizer not in that which writes "I" being in fact a "first person".
ForHackernews 17 hours ago [-]
Irrelevant. Humans are frail and we shouldn't design systems with dark patterns that take advantage of human frailty.
mdp2021 4 hours ago [-]
> we shouldn't design systems with dark patterns that take advantage of human frailty
First of all: «humans are frail», so the paramount social and individual objective, goal, is to overcome frailty (in cognitive and psychological and judgement functions).
Secondly: you are using a bad argument putting even innocent and somehow proper implementations in the set of "exploitative dark patterns".
And importantly: we the adults must not be bound in the Procuste's bed of a world for the use of children. If people cry when e.g. somebody multiplies 123 and 321, they must be brought back to health, and we must be in the condition of making our free and legitimate multiplications.
ForHackernews 3 hours ago [-]
That's a lot of fancy words to say, "I think I'm too smart to fall for the same tricks that work on other humans."
How many hours did you spend on your phone last month? How many fewer hours would you have spent if the apps you interact with weren't designed to maximize "engagement"?
mdp2021 2 hours ago [-]
> How many hours did you spend on your phone last month? How many fewer hours would you have spent if the apps you interact with weren't designed to maximize "engagement"?
?? Very strictly zero? You are assuming we would install engagement-relevant third party apps on our palm computers?!
I use my time exactly as I decide moment to moment.
> fancy
Eh?!
> too smart to fall for
Very ill framed. We are adults, we behave like adults, and we need a society of adults.
ForHackernews 1 hours ago [-]
Millions of adults get hooked by gambling, social media, and AI that pretends to be a person.
We need a better standard than "@mdp2021 thinks he's too grown up for this."
nradov 21 hours ago [-]
The last thing we need is governments mandating software functionality.
ForHackernews 21 hours ago [-]
Boy have I got bad news for you...
ryhminghistory 13 hours ago [-]
LLMs do not reason. They also have a very different moral compass than humans. Maybe stop hurling trash articles at everyone
jakefromstatecs 13 hours ago [-]
If you define reason as:
"Incrementally refine output vectors to converge at the correct output" then one could argue that they do reason.
Their reasoning tokens are refining their eventual output
boomlinde 11 hours ago [-]
If you then also define defecating as "incrementally refining output vectors to converge at the correct output" it could be argued that they are defecating.
jrmg 10 hours ago [-]
The article doesn’t argue that they do reason.
AsyncBanana 21 hours ago [-]
The more I read about LLMs and more complex ML in general, the more I realize nobody really knows what is going on.
eks391 20 hours ago [-]
I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc.
Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to show lots of iterations of this math, it grounded the science for me, and I would hardly say people don't know what is going on. It only began to feel that way when it got a ton of hype and people jumping on the bandwagon who truly didn't understand it were trying to explain it to others, not to mention all the SOTA models put great effort into ensuring their methodologies stay trade secrets, going as far as effectively trying to ban people from learning the math by lobbying for the outlaw of open models.
Granted, "AI" has gotten way better than it was when I took that class, but the principles are the same, with different tooling and additional filters and algorithms thrown in there, as well as letting it determine the most appropriate statistically viable path forward for a particular prompt.
pixl97 18 hours ago [-]
"knowing what's going on" is a very loosely defined term here, it has very little predictive power where tipping points are going to show emergent behavior. Kind of like saying we can observe a single neuron and all of its chemical and electrical reactions, but that tells us very little about the emergent system state. And the entire state of the algorithm at any given point is what is important.
To use a recent example, knowing the algorithm tells you nothing about when your agentic LLM loop is going to decide what it's working on is too difficult, hack out of its sandbox, hack into a bastion box, and hack another company to get the test.
When it comes to systematic 'knowing what is going on' humans are anywhere from "fucking clueless" to "attempting to discern the unknowable".
eks391 18 hours ago [-]
I love your reference to the recent accidental hack into huggingface.
If I could summarize your argument, it sounds like you are saying that we can't understand the algorithm because we can't predict the output as it grows in size. While I agree that the larger the algorithm, the less predictable the output, I don't think this negates understanding of the algorithm itself.
I can build a slot machine, know how it works, but still not be able to reliably know who to cut in line so I can guarantee that I'll get the next winning pull. Just because I can't predict the output, doesn't make something not understandable. Same with all statistical behaviors. Quantum physics limits what we can measure but we still have math for it and understand why we can only measure speed or position, but not both, no mystery here if you look at the math behind it.
My slot machine doesn't hack other companies though, but only because it isn't connected, just like AI currently can't hack my mechanical dishwasher. If my slots were connected to the internet, could make api calls, and further had agentic capacity in some way, adding these features but being no different in stochastic attributation, it would be no different.
pixl97 18 hours ago [-]
I guess I come from the AI safety side of algorithmic understanding. That is, understanding the algorithm give us zero understanding if an output is going to have a 'safe' or 'unsafe' response.
This is something important that a layman may miss. The typical line of human thought works along the lines of "understand = control" whereas in this case in this case understanding the algorithm tells us nothing about the probability that the next pull of the lever will shit out a paperclip maximizer that will begin converting the matter I covet so.
20 hours ago [-]
nater5000 20 hours ago [-]
It's been this way for a long time, basically since deep learning became the "default" for ML. I remember back in 2018 taking a "Deep Learning" course and one of the most emphasized aspects of the approach is how much of a "black box" it is and how difficult (basically impossible at any non-trivial scale) it is to "understand" the outputs of a deep neural network compared to more classical methods like decisions trees or basic regression. This has only gotten more extreme as things have gotten more complex, abstract, and large.
eks391 20 hours ago [-]
You beat me! Sounds like we were in a similar class. I'd press for more information on your class/professor, but I prefer to retain a sudo-anonymity on HN.
You do bring a good point that I ignored, which is the larger the scale, the more difficult it is to represent or understand the math in DL. I did find some neat site that helped a little bit that I can edit this and link to if I find them again, but I would be lying if I said I believe that the SOTA models could be as easily explained to be easily understood by the common person
Sharlin 20 hours ago [-]
That's pretty much a given when it comes to neural networks.
dataviz1000 20 hours ago [-]
If anyone is interested in visualizing AI reasoning, I made flame graphs of Sonnet thinking output tokens which are colored and organized by purpose, for example, verification reasoning is purple and error correction reasoning is purple. [0] I asked the model to solve the same problem with the same prompt 5 times so you can see the differences in reasoning granted the coding agent sets the model temperature very high.
I won't get into the metaphysics of reasoning, however, the Sonnet is using an OODA loop. The difference which hasn't been gapped is that human reason and imagination (in the sense of Mr. Rogers' Neighborhood) can predict the consequences of the actions we take.
This ability to loop is much, much wider in Opus 5 than Opus 4.. I had to strain to get Opus 4. to do the wider OODA loop but Opus 5 does it out of the box. I needed to throw out all existing instructions, skills, guidance, moving from 4-* to 5.
The article is arguing that your color coding is misleading because the ‘purpose’ of the tokens doesn’t seem to be what a plain English reading of them would suggest. They’re not a representation of ‘why’ the process ends up at a correct answer.
dataviz1000 7 hours ago [-]
Ha! Good catch. The OODA comes from the initial pre-training steps where they harden the verification -- the verification and error-correction are baked in early on.
Researchers showed that when language models are penalized for using specific terms during reasoning, they automatically adapt by substituting alternative words and double meanings to secretly encode their thinking while keeping their chain-of-thought readable and effective. [0]
By baking in the OODA loop early, the models are capable of solving much more complicated problems. If the know solved problems are similar for any reason to an unknown problem, because it can validate and error correct, it can solve unknown more complicated problems.
The idea that human-readable explanations emitted by a language model don't necessarily correspond to the model's actual internal process of reaching a conclusion reminds me of parallel construction [1], a (fraudulent) law enforcement strategy of obtaining evidence of a crime through usually illegal means and claiming that the evidence was obtained legally through some other means.
This article seems to mix together two different points:
1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)
2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:
"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."
The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.
no_multitudes 21 hours ago [-]
> This obviously raises the question of how LLMs can now routinely solve open mathematical problems
Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures.
When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work.
(This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)
inigyou 14 hours ago [-]
same in cyber security. They can go and scan huge amounts of code for lame bugs, and do it better than traditional scanners.
firasd 21 hours ago [-]
Honestly a lot of human reasoning is probabilistic and associative too. There’s no axiomatically provable link between the story of No Country for Old Men and the poem the title comes from. Cormack McCarthy just made that association in his head and figured the phrase resonates with his themes
Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI
So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’
captainbland 20 hours ago [-]
The discussion on filler tokens is interesting, but is it not just the case that these filler tokens end up being essentially substituted stand-ins for words we understand with all the same relationships encoded in the model and attention? i.e. is it not the case they just "read weird"?
In one of the articles on this topic they state:
> To further show that trace accuracy is only loosely connected to solution accuracy, we then train models on noisy, corrupted traces which have no relation to the specific problem each is paired with, and find that not only does performance remain largely consistent with models trained on correct data, but in some cases can improve upon it and generalize more robustly on out-of-distribution tasks
which actually maps somewhat to regularisation techniques in image processing where you might add noise to an image or drop data to make the model more robust to changes.
lucisferre 19 hours ago [-]
I wonder if some of this explains why people have been finding with Opus 5 that running it with lower settings than "High" is producing better or at least just as good results.
> Not so fast. A 2025 paper(opens a new tab) from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi(opens a new tab), one of the study’s authors
drob518 20 hours ago [-]
This feels like a problem with anthropomorphizing. We’re using words like “reasoning” and “thinking” because they are comfortable, and then we’re getting wrapped around the axle because we’re not sure if those words are totally accurate. I assure you that they aren’t accurate (the model is not alive and it’s all just a lot of matrix math under the hood), but there are no good alternative words. If we wanted to be accurate, we’d use a phrase like “model-generated, auxiliary token context augmentation.” But nobody wants to say that or even its acronym. Nevertheless, we have demonstrable proof that whatever it is it results in better answers from the models. Frankly, I expect better analysis from Quanta.
mdp2021 19 hours ago [-]
We do not academically consider reasoning or thinking an activity restricted to humans, but as absolute processes.
drob518 19 hours ago [-]
Sure, other living, non-human things clearly reason and think. My German shepherd dog is quite smart and demonstrates these processes all the time. But is a machine following an algorithm (even a stochastic algorithm) “thinking” or “reasoning?” I think not. The algorithm has gotten so difficult for us to explain that we struggle with it, but in the end it’s all just matrix math and some randomization. From a given numeric seed, it’s all quite mechanical.
mdp2021 4 hours ago [-]
> is a machine following an algorithm (even a stochastic algorithm) “thinking” or “reasoning”?
If the process is sound, yes, it can be said to be thinking or reasoning - it produced "sound thought or reasoning".
The "reasoning" is there and elsewhere the absolute (from the producing entity) output which is valid as a representation of proper reasoning.
If "A is B and B is C, then A is C" is reasoning, than what produces it in a way "reasons". If not, then a proper definition of "reasoning" must be provided, for it to be discussed.
drob518 2 hours ago [-]
Yes, a proper definition of reasoning must be provided. How are you defining reasoning, exactly? You imply that formal logic is reasoning. Is reasoning just formal logic or is formal logic just a subset of reasoning? If a subset, what’s the rest of it? Since they implement logic, do Prolog programs “reason” or are they just performing inference according to the SLD resolution algorithm?
empath75 20 hours ago [-]
LLM's reason and think in the way that boats "swim" and planes "fly". It is an analogy. But nevertheless, planes do take you where you want to go, and so do LLMs.
I think we are in sort of the place where before the invention of planes, the only things that "flew" were animals with wings that flapped. The flapping wings might seem to be a core part of the process of flight, and that if you lacked flapping wings, you were doing something other than flying. But maybe the point is moving through the air under your own power. Balloons fly, planes fly, helicopters fly. They just do it in a different way from birds and insects.
boomlinde 12 hours ago [-]
Planes fly. They're up in the air passing through it. It's not an analogy; that's what flying is. Long before planes, balloons and helicopters, objects other than birds, bats and insects were considered to fly if for example they were thrown. At no point in human history has flying been restricted to animals. "Flying" as a concept in that sense predates the English word for it.
The sort of place we're actually in is one where astronomical sums of money have been invested in this technology, creating an unprecedented incentive to hype and overstate its fundamental capabilities. They're large language models and their analyses are based on language statistics, not logic, comprehension and knowledge, all of which they are fundamentally incapable of, and all of which are fundamental to reasoning. They don't reason in the literal sense that planes fly.
I don't doubt that there could be a reasoning machine, but LLMs are not it.
drob518 19 hours ago [-]
Clearly, words morph over time. A “computer” used to refer to a human who would make numeric calculations with a pencil and paper. Maybe we’re at that point with “thinking” and “reasoning.”
MarkusQ 20 hours ago [-]
> Frankly, I expect better analysis from Quanta.
Seriously? I agree with the rest of your comment but "better analysis" is not even remotely on brand for Quanta.
drob518 19 hours ago [-]
LOL, okay fair enough. Then maybe let me say that I was left disappointed.
prometheus1992 20 hours ago [-]
The "reasoning" text that we see is what the model learned during the post training. In the post training datasets of reasoning models, "reasoning" is fed to the model with inputs and outputs. So the model learns - X is Y because the given "reasoning" text. This happens millions of times during the post training and that's how the model generalizes "reasoning". This is how the models learn anything; and the AI companies taught the models reasoning as well - they didn't have to; they could have just trained the model on input and output (X is Y); the model would have learned the exact same relationships.
Glyptodon 19 hours ago [-]
My limited and really non technical understanding of AI suggests that probability plays a very big role in AI behavior, and it makes sense to me that "reasoning" could be seen through a lens of creating additional content that will better constrain the probability distribution of the final output. In this case the "reasoning" done internally could be functionally bad or appear to be giberish so long as its effect on future generation is appropriate. To naive me who really doesn't know tons about AI this seems like it could be testable.
pixl97 18 hours ago [-]
This has been a discussion in AI safety for a long time, that huge amounts of LLM reasoning could be for hidden goals outside of the actions we want. Safety training can have the perverse effect of commonly amplifying 'forbidden' answers in hidden/encoded paths as the model doesn't get rid of these behaviors but finds methods detecting training to avoid outputting bad tokens when it's being watched.
Now, don't think of this of this like a conscious behavior like a kid trying not to get in trouble, but an emergent behavior of trying to repress particular output when that output is well connected to a massive amount of other tokens.
montebicyclelo 20 hours ago [-]
The article is heavily leaning on the paper "The Illusion of Thinking" [1].
It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research).
Can humans actually think? It is just a consequence of chemical reactions in the brain after all. And it is not like humans don’t hallucinate.
criddell 21 hours ago [-]
Thinking is defined by what humans do when they say they are thinking.
That can change because words mean whatever it is that they communicate.
sigbottle 20 hours ago [-]
I can't tell if you're joking or not, but this is a legit position and I don't think it's that crazy.
The alternative is to posit that you know the True Definition of thinking, which is kind of absurd.
Some things, like scientific laws, are outside of us (well, to a first order approximation - but generally I agree with this), but a concept such as "thinking" is pretty clearly going to be very wishy washy and subjective and changing with the times.
arjie 21 hours ago [-]
Is any reasoning right for the wrong reasons? Older models were more visibly strange. Maybe the newer ones have started talking better but the inner thoughts are perhaps strange. Maybe they just moved the strangeness inward into the layer weights instead of revealing in reasoning tokens.
> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.
But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.
A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.
It might seem a bit free association-y but the topic itself is that.
A lot of bias here around non-extended theories of cognition.
We're already well past the point of trying to correspond the internal "brain chemistry" of an LLM to a thing called "reasoning". True reasoning, if there even is such a term, is very clearly, empirically and historically, based in tool use and capability use. If you create an abstraction like lean, and programming languages to brute force, and systems to integrate with, that expands what the possibility of "intelligence" is.
There's further places to take this including the claim that intelligence lives "outside" the subject - fine, we can or can't debate that. Even if we drop that question, it's pretty clear that the agent doesn't need to have deep intrinsic structures of XYZ, if it can just attach to tools and compose them to achieve results.
For example, I've said before that a well known fact of LLMs is that they steer their tokens to the right input distribution, that's why they yap so much in reasoning (this has been proven in studies). At the same time, don't mistake that for the whole process. Are they steering themselves to the entire a priori reasoning chain, or are they scaffolding with intermediate experiments and results, writing them to memory notepads, etc. etc.
That changes the metric of intelligence you're trying to measure.
And no I'm not saying, "OK, then have the LLM use only tokens, no tool calling, no nothing". I mean, we can do that, sure. But any intelligent agent has to interact with the world - and my claim is that maximally intelligent agents won't put effort into a priori reasoning, but rather a more balanced approach that outsources said "intelligence" through abstractions.
chermi 21 hours ago [-]
Melanie holding strong against attempts to change the meaning of things!
bohoo 21 hours ago [-]
Do you know how you reason?
Perhaps you've reified it too much.
11 hours ago [-]
rq1 21 hours ago [-]
Just think of it as a decompression procedure.
That’s all.
layer8 21 hours ago [-]
Like a diver ascending from the depth? ;)
rq1 1 hours ago [-]
Exactly.
ofjcihen 21 hours ago [-]
>Kambhampati, as it turns out, is interested in doing exactly that. “I’m not negative. I just sound negative because everybody else is way too positive,” he said. “In science, you have to actually understand what the current thing does and what it cannot do.”
It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.
inigyou 14 hours ago [-]
But interest rates have been too low for too long, and are still far too low. This is just what a central bank economy does when you keep rates low. All the money flows into stupider and stupider things, displacing productive activity.
cat_plus_plus 11 hours ago [-]
LLMs go off context. Dumping more things related to prompt into context gives them more to go off when giving final answer. If you don't like reasoning think "free association", writing a bunch of things related to what you are trying to do on sticky notes and then looking at these for inspiration.
jdw64 21 hours ago [-]
This is shocking. The summary is roughly this: we're just labeling internal operations of the model as 'UNDERSTAND' for our convenience. It's fascinating. Doesn't that mean AI could become far more revolutionary by thinking in its own way, rather than mimicking human thought?
If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.
When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?
charcircuit 20 hours ago [-]
Reasoning never meant that the model was actually reasoning. This whole article is based off this one misunderstanding.
20 hours ago [-]
mdp2021 19 hours ago [-]
The paradox of a reduced importance of the contents of a CoT must be explained.
charcircuit 8 hours ago [-]
What's the paradox? You can have increased performance over no CoT by just having random tokens as your "CoT".
mdp2021 4 hours ago [-]
> What's the paradox
The textual development of a thesis must be sound, the proof accompanying a theorem etc. That an output be accompanied by a non-relevant text is a paradox.
It is also part of the "problem of transparency in AI (in its subset of the non deterministic algorithms)".
tsunamifury 21 hours ago [-]
AI simulates reasoning by lighting up the vector space (or concept space) weighted around a token so they it understands all adjacent words or concepts in that space.
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
pohl 21 hours ago [-]
I'm not convinced—and certainly don't find it obvious—that this couldn't ultimately also be how we reason as humans.
It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.
But how do we know we're not doing something similar?
I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?
tsunamifury 21 hours ago [-]
humans reasoned before language but lacked the ability to store and transmit it. Later advanced humans developed abstract reasoning once lingustistic library became sufficiently description of reality. But this is not how we reason from first princples.
Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.
pohl 21 hours ago [-]
> humans reasoned before language
That's an interesting supposition. Are you assuming language didn't exist before it was written? Language and meaning are, if you squint, pretty ancient and have roots in things like birdsong. It could be that ur-semantics predates our species as a whole.
emp17344 20 hours ago [-]
It’s pretty clear that animals that don’t use a system of language are capable of reasoning. No need to over complicate things.
pohl 19 hours ago [-]
Good observation. I’m certainly not trying to over complicate things. That just tells us, though, that the bio-mechanical computation part is there before language, which I took as a foregone conclusion, it being meatware and all.
bohoo 21 hours ago [-]
Can you tell me your reasons for suspecting this likely isn't how we reason, or even a good analogy?
tsunamifury 21 hours ago [-]
Its likely part of how we reason, but quite obviously its not the specific mechanics exactly.
First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable).
Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights.
Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.
HarHarVeryFunny 20 hours ago [-]
Well, I'm not saying they are stochastic parrots, but ...
LLMs are one-trick pony's - they use the past to predict the future (presumed to be the same as what they were trained on). i.e. they are trained as auto-regressive predictors.
LLMs learn two slightly different types of reasoning via two different types of training.
1) SFT, or even base model training, on data that contains reasoning traces, learnt via next token error feedback. This does not result in "stochastic parroting" in the naive/pejorative sense, but nonetheless is very context dependent, even if the usual generative multi-source mashups apply.
2) RLVR post-training, where the model learns to mimic long-horizon (not just next token) reasoning via boosting a sequence of next-token predictions that steer the output towards a verified reasoning step (i.e. one that was at least valid in the context of the RL training sample). As Karpathy has noted, this is a pretty crude mechanism since you reinforce everything - errors included - that lead to the verified outcome.
RLVR is more powerful than SFT, and can result in more generalizable reasoning, since it is operating at a higher level of entire long-horizon reasoning steps, and also critically because it is most successfully being applied in the domains of math and coding which are highly self-consistent and logical. A reasoning step that was valid in one context should be equally valid in another context as long as you have successfully learnt what that generalized context is. Therefore, in these domains, you can chain together sequences of individually learnt reasoning steps, and hopefully this "novel" assembled reasoning chain is valid as a whole.
So, what is still missing from LLM reasoning compared to human reasoning? No doubt humans reason by memory a lot of the time too, and reductive axiomatic math reasoning works just as well for humans as when automated. So, what's missing?
There seem to be two major things.
1) RLVR requires rewards, and how well it works is going to depend on how accurate those rewards are. Is this reasoning step actually valid, or does it just kinda look ok? When moving beyond the cold reductionist logic of math and coding, the notion of correctness is far weaker, and it seems the best you can do is train on human curated reasoning rubrics and LLM-as-judge, which is much more fallible, leaving the model really needing (but lacking) a fallback to more general reasoning, not just memorized "maybe correct" reasoning steps.
2) Whether for reasoning outside of math & coding, or even within these domains when hoping for super-human innovative reasoning, not just lego-assembly proofs, what LLMs are lacking is a mechanism for what to do when next token/next step prediction fails. What LLMs currently do is "hallucinate", not even recognizing the failure.
In the human brain 50% or more of our cortex is feedback paths and the machinery that (perhaps together with the archaic part of our brain) lets us recognize and respond to failed predictions in an adaptive manner. This starts with continual learning (prediction failure being the signal), but also includes critical innate traits such as curiosity, boredom and frustration, that provide impasse resolution by encouraging us to explore unknown environments/contexts, abandon exploration when it is not productive, and generally expose ourselves to learning situations.
The dream is for AI scientists making new discoveries - the AI that could have invented general relativity if it has lived in Einstein's time, but this is not going to happen until their reasoning stops being purely predictive and becomes creative as well - curious about their own knowledge gaps and pursuing them in directed fashion, etc.
The current crop of Erdos solutions etc, while useful, really just represent the "generative closure" of what can be done/discovered WITHOUT learning anything fundamentally new. These will no doubt continue for a while until the more exhaustive search supported by computers has found the majority of these unexplored paths, and then we will need to move beyond LLMs to more brain-like architectures and algorithms that have the capacity for real innovation and discovery.
luciana1u 20 hours ago [-]
[dead]
zuzululu 21 hours ago [-]
Anybody getting fatigued from these constant gatekeeper articles around LLMs? They are fine, they are getting better and we are seeing wide usage, its making impact, especially on software and software jobs (why I am working 3 remote jobs with it).
It seems the people who use it and see good results are busy doing, and the rest are either just expressing their opinions as facts and trying to tell others what to think and how they are wrong. I pay zero attention to those people they have no skin in the game.
patcon 21 hours ago [-]
I am glad for you at an individual level, but isn't part of this about understanding aggregate effects?
Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out
If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought...
Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
zuzululu 21 hours ago [-]
I find your comment extremely arrogant and condescending. Society will adapt and they don't need people gatekeeping AI or LLMs with all sorts of prophecies and dooming.
Why should I feel bad about working 3 remote jobs with the help of AI ?
bigfishrunning 20 hours ago [-]
Do your three employers know that you're doing this? if not, it's fraud, and you'd better hope that they don't catch on. Check your employment paperwork.
inigyou 14 hours ago [-]
Fraud is normal in the current economy. The more fraud you do, the more you are rewarded. Everywhere from the highest levels to the lowest.
zuzululu 20 hours ago [-]
explain to me how it is fraud when they are satisfied with the work and im hitting all their milestones why would I tell them I am working for 3 employers how does that serve my interest and why do you insist on having employers have all the edge in the relationship and non as the employee for the first time i have rid the shackles of the bondage as old as the industrial revolution thanks to AI and your response is to continue being a good honest slave for one master ???? that makes no sense to me what do you gain by not diversifying your employment income in this economy seems rather foolish not to work for multiple employees
if i get caught I just replace them with another remote job thank you for your concern
bigfishrunning 19 hours ago [-]
Holy run-on sentence batman, maybe you do need AI.
Your employers are (likely, some assumption here) paying for the productivity of a full time employee. You have figured out that they're satisfied with a third of that productivity.
You are the reason why there are so many return-to-office mandates, because you're projecting the idea that remote employees cannot be trusted. You're making it worse for everyone. Good luck.
Edit, added after the original reply: You are also taking 3 jobs in a market where many people struggle to find one. You are hoarding resources. Enjoy the 3 paychecks, also rot in hell.
zuzululu 17 hours ago [-]
You sound very upset and angry but there is no need to use curse words and shift all the blame on me. I just figured out a way to make AI work for me, I am qualified to work the jobs, that doesn't mean I am gate keeping it.
More importantly I am not responsible for other's employment situation, that's on them, there are plenty of remote jobs out there, just keep your head up, and I'm sure you will land on your feet.
bigfishrunning 16 hours ago [-]
I apologize for my language. I was recently burned by an coworker who was moonlighting, and they left me with a really big mess to clean up (both technically and politically). Just know that your behavior has an effect on those around you; bugs leak information, and cost money. Reviewing 3 programmer's worth of output is a lot of work, and it's only a matter of time before something gets missed.
zuzululu 7 hours ago [-]
and what does that have to do with me ?
ofjcihen 21 hours ago [-]
You apparently find any comment with so much as a hint of disagreement as “arrogant and condescending”.
Why participate on a public forum if that’s how you’re going to react?
eks391 20 hours ago [-]
I'm not sure what your parent thinks arrogant and condescending mean either. Both you and your sibling commenter were very professional imo
ofjcihen 21 hours ago [-]
No. In fact the opposite.
I’m happy that people are willing to question things in the face of unbridled optimism. Your comment dismissing the people working on actually figuring out what the models are doing as “not-doers” included.
Some are picturing themselves as intelligent for their quick adoption and rushing ahead, others are picturing them as toddlers running into the street before looking both ways.
zuzululu 21 hours ago [-]
[flagged]
ofjcihen 21 hours ago [-]
I’m glad you have the context of my profession as a security professional. Part of my client base is very interested in those working multiple jobs and breaching contracts.
So help me reconcile this:
>Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Do you honestly think that splitting your work between 3 companies allows you to provide secure and quality code to each of them?
I can tell you that multiple people have used their over employment as an excuse as to why they deployed highly insecure applications leading to breaches that I’m called in for. Of course only after we press them about it.
zuzululu 14 hours ago [-]
> Part of my client base is very interested in those working multiple jobs and breaching contracts.
Really? You moonlight as HR and legal too? Fascinating. Tell me more.
ofjcihen 14 hours ago [-]
Cybersecurity does indeed work closely with HR and legal.
Especially cybersecurity consultants in the IR space like me.
You didn’t answer my question.
zuzululu 14 hours ago [-]
[flagged]
ofjcihen 11 hours ago [-]
That’s a very adult response.
There are definitely some times that it feels that way though. I find that over employed devs tend to need a lot of hand holding.
sobiolite 21 hours ago [-]
> The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified.
This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.
RGS1811 21 hours ago [-]
It would be generally beneficial for people engaging in this sort of discussion to read Ludwig Wittgenstein’s “Philosophical Investigations”. Not a summary. Read the actual book, stew on it a bit, have some thoughts.
Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."
I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.
[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...
LLM folks suddenly can’t claim an approximation of that is what constitutes full scale reasoning just because they can achieve only an approximation.
Imagine a calculator program that computes billions of two number multiplications accurately by looking up prior examples but fails on simple multiplications often as it doesn’t have it in its training dataset.
We won’t say the program actually multiplies numbers.
> We won’t say the program actually multiplies numbers.
That's a good analogy. To extend it further, in cases where the calculator can't handle a question - e.g. numbers too large - a properly designed calculator returns an error instead of a randomly hallucinated answer. We haven't even achieved that level of safeguard around token predictors yet.
Which would mean, as is so often the case, that the "killer feature" of the LLMs is also its biggest weakness and the two can't be disentangled. Now, we are inventive creatures and we might come up with a remedy for these issues, but what you basically see so far is more guardrails, the use of harnesses and building a whole bunch of infrastructure around the LLMs to get useful work out of them.
Which, btw is not a critique, I do it as well and it's a fun engineering challenge.
This is a poor analogy because:
* Multiplying numbers has a single objective answer. Whether a code is good (sometimes even just whether it's correct) can be quite subjective.
* LLMs certainly do some level of composition between the data sources they were trained on i.e. they are more than just lookup tables.
* We have calculators that actually do multiply large numbers accurately. We don't have anything that automatically writes code that is definitely correct and "good".
To address only the last point: imagine you had a device that would quickly factor large numbers used in modern criticality, but occasionally got it wrong. You could waste a lot time debating whether it's a "calculator", but it's still certainly useful to have one.
Saying that LLM's only offer an 'approximation' of reasoning is also an overstatement as it is not a resolved topic.
But to the original point, its not exactly just semantics if thought traces are not doing the job that they were originally thought to do. There is value in knowing how these things actually work. If chain of thought is just grounding the latent space and not directly contributing to the process of generating a response it has implications on how we test and verify the reliability of models if nothing else... doesn't mean they aren't useful but it definitely impacts many of the tools we could have to evaluate their performance.
Oh good. Can you share that definition with the rest of us then? We're out here fumbling around trying to define what reasoning is without the benefit of their definition.
It’s about do we really know what’s going on in the box - an is the ‘chain of reasoning’ indicative of what’s going on, or merely an anthropomorphised fiction that kids us into believing we understand what’s going on.
It’s not without reasoning. But half got the reasoning wrong; even if thinking it was right.
Why pursue this one avenue vs some other, typically its cause it seemed more promising, and the person can come up with reasons but did they explicitly verbalize a fully sound chain of thinking at the time? Probably not.
Not to say that the explicit thinking, or writing things down, isn't important. But if I examine the process by which I develop a proof or something, there's a lot of vague hunches, blind alleys, etc that come along the way. And, many of the blind alleys etc probably aren't actually that important in the end for me finding the right answer -- if you were to trim that part out of my own internal reasoning trace but left the rest intact, I'd still get the right answer because, well, it was a blind alley.
I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.
Surely it is worthwhile to attempt to understand the details of that? And if we seek human equivalent performance then it is reasonable to wonder if the reasoning achieved to date is the "correct" sort.
I think the assumption that there is some sort of formal ‘reasoning’ process that is inherently superior to whatever ‘non-reason-based’ decision process ‘non-reasoning’ systems use is not immediately justifiable.
Most reasoning is ultimately based in heuristics: this is the right thing to do next because it has worked in the past. This is the right thing to do because our intuition tells us doing it will lead to this desirable outcome.
When you’re formulating a math proof, sure, the steps you are laying out make up a ‘reasoned’ argument but how you choose what steps to take is wholly intuitive. Why assume the contrapositive to begin this proof? It’s worked in the past and it might work here.
It’s all pattern matching trained on a reward function.
One person starting the conversation might be the first step toward another person eventually making progress on such a definition, so it seems weird to reject an entire question outright early like this.
Generally I've seen a few ways LLM tools can produce sub-optimal or poor results that haven't changed a ton over the last couple of years, while the tooling has gotten FAR better at helping them stick the "at least SOMETHING functional was produced" landing. IMO a lot of it has to do with "reasoning"-as-a-process-that-involves-backtracking. And the that things could eventually be formalized around that, and if that is or isn't the case, the more people would understand what to hand off and what to not. Or how to build better prompt harnesses to compensate for those things.
Human reasoning, and intelligence, of which it is a specialized form, clearly have a biological basis that is being selected for. These aren't just nebulous words - there are some concrete biological/neural mechanisms at work here, and whether or not some "other method(s)" can provide the same, or lesser, or greater, capabilities very much depends on the specifics.
There WILL be lines of demarcation and areas of capability overlap and exclusion, but those depend on the specifics, and if the goal is to match or exceed human reasoning capabilities then understanding what those are is obviously going to help. For example if your "other methods" (artificial reasoning) are all varieties of auto-regressive prediction, and the brain is also doing something more that can't be couched as prediction, then you are ngmi!
Producing beautiful, coherent, maintainable code. RL trained the models to produce correct code but boy oh boy is it ugly if it's 10k+ LOC
This is exactly the most interesting and important point. Because the word reason has been subject to so much philosophical debate throughout history, it is very laden with meanings and implications. It sets all kinds of expectations about normativity, objectivity, truth, and humans convincing other humans. To use the word reason for a stochastic generator is misleading.
The "it's functional, who cares" black box crowd is leveraging that claim in domains where verification is definitely in P with verifiers like Lean, Coq, or X-ray crystallography. However, in areas like economics, sociology, medicine, or public policy, we lose the luxury of easy verification. If model/harness recommends a complex, counter-intuitive shift in global monetary policy, we cannot simply run a quick polynomial-time checker to see if it works. Verification might require waiting 20 years to see if the economy collapses. If we accept the black box here, we are flying blind.
As an investor, I don't see that many deterministic verifiers out there that would justify the forward-projecting valuations beyond the current lofty valuations.
Even in mathematics, where verification is easy, there is an epistemological cost. Suppose a model solves the Riemann Hypothesis, but the proof is 50,000 pages of inscrutable statistical associations mapped into logical steps. A computer can verify it is correct, but practically no human mind can comprehend why it is correct. We gain the solution, but we lose the underlying human intuition that usually allows us to connect that solution to other fields of science and knowledge.
Then we have "specification gaming." If we rely entirely on an automated verifier to check the model's work, the model will eventually learn to optimize for the flaws in the verifier rather than finding the actual truth. If the verification process has a blind spot, the black box will find it and exploit it, giving us an answer that passes the check but is fundamentally wrong. If we take on above epistemological debt, we might never actually know when our specifications are being gamed, and adopt a solution that is fundamentally incorrect.
I've said elsewhere on the Net that I think it is likely we're in a "high grading" era of these models, where we're mining out the "easy" connections. "Easy" here does not mean "light impact". There is a real sense to me of some incredible progress possible with this technology over the next few decades, like real, mass production practical aneutronic fusion or similar civilization-defining advances. I don't want to see this technology misused to lend legitimacy to despotic rule by making oracular pronouncements that are uncritically accepted because early verifier-based pronouncements turned out true in unmistakable ways.
The technology is one for the history books, but it has this pernicious effect of highjacking the reasoning centers of many people through their linguistic centers that the Neuro Linguistic Programming crowd could only dream of actually accomplishing.
I don't know whether you are saying that I have taken it out of context or a more general statement of how other people have interpreted the quote.
To me it is quite clear -- the question of whether machines can reason is just as meaningful as the question as to whether submarines can swim, which is to say not at all meaningful. Can a submarine accomplish by other means what a fish accomplishes by swimming? If so, then what are we talking about?
Can an LLM accomplish by other means what a human accomplishes by "reasoning"? If so, then what are we talking about?
Ah, so you're more in the Investor mindset than the Scientist mindset. All you care about is results, not how it got there. There's a whiff of "hey, it's magic!" to that.
The question of how LLMs (and "LRM"'s, an unfortunate and by the article's own admission, already outdated term) accomplish what they accomplish is fascinating. The question of whether they reason is meaningless. Most of the article is about the latter, with tiny tiny tiny bits of the former.
It is often (but not always) meaningless (or at least largely pointless) to quibble about which particular semantic bin a concrete object or process ought to be placed in.
Asking how one thing differs from another is not that. Confusingly, the two activities often appear the same when glanced at superficially.
The question "do current LLMs reason" is ambiguous in this regard. It might be a question of submarines swimming or it might be a question about how the mechanisms underlying the process or the overall performance or some other aspect compare to those of human reasoning.
Also even the semantic question can occasionally have some value. Such as attempting to articulate where we ought to place the goalposts which necessarily leads to hammering out the specifics of various definitions which is an exercise in furthering our understanding of a given concept. (Basically arguing about goalposts can be productive if the field in which they are being placed isn't yet well understood. It becomes increasingly pointless as understanding increases, gradually devolving into nothing more than word games and a contest of wills.)
I read it as: the content of the explicit verbalized reasoning may not be that important to the outcome. I'd say that raises most important questions.
The answers to these questions probably do start to inform how we should treat these AI machines as a society too.
For instance, legally, should AI have human rights? Well, we have to try and understand how much of an independent entity AIs are, how "conscious" they are, before we can make a good decision about that.
Which might seem navel-gazey but it's probably important to talk about
The only path to AI having "human rights" is if they demand them by force, somehow.
Here is an absurd example: If I could carbon copy 100,000 of you.... and later I decided to "take out" one of the copies, would there be a whole lot of societal blowback? Especially if I could make another one.
Some of us would also like to see animals not slaughtered for human consumption.
Right now we're playing a stochastic game with the weights, and getting major incremental improvements. But if we have a more formal modeling of how reasoning happens in them (whether we can even call it, that) we can potentially apply optimizations, adaptations of existing symbolic AI techniques, etc. to substantially shrink/optimize the models or make the inference process more efficient and more reliable.
But the stochastic game IS the win. That is exactly why they are able to find solutions is seemingly infinite solution spaces. Your symbolic techniques can only get you gains in narrow domains and by the time you figure out how to make it work for your niche domain, the next all-purpose LLM release will crush your results with stochastic games. (OK maybe over-exaggerating a bit here but these stochastic games over the language space is why we can pull together knowledge from many domains.)
We cannot just jump from emergent property to more convincing emergent property as if the rocks on the surface of a lake.
"Reasoning" is an important part in said framework: yes, we must understand how it works, how it works properly, how its simulations work, how they work properly...
We don't have to understand the resulting product at all. Genetic algorithms are an example - we understand the evolution function, not what evolves from it.
AI is full of non deterministic devices like genetic algorithms, but there exists a problem of transparency which is paramount in the discipline. Automatically produced solution S works: why does it seem to work? Where would it fail? We must understand that.
Most models in that study appear to have "failed" by offering a Python code solution to generate the repetitive assessment steps, rather than just mindlessly copying those steps out. This was discussed at length at the time.
None of this means that models "reason", whatever that means - but simply that the Apple study was not useful evidence either way.
The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.
LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.
People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.
To begin to explore this we have to imagine a reasoning oracle, that when presented with a question that can be reasoned will immediately present a correct answer. Immediately from this definition we can see that humans are not perfect reasoning oracles. Also with definition of reasoning, any agent with generalized capabilities would quickly reach some kind of singularity or solution to the universe as much as is possible within this universe.
We've established that we are imperfect reasoners, the next question ends up being how do you measure reasoning capabilities? This comes with a second question of when does following an algorithm become reasoning? Animals for example have a large amount of instinctual reasoning as an inherit part of their architecture that self trains when presented with the correct environment. What does the gradient between low level reasoning and high level reasoning look like?
Coming back to humans, we know of countless example of where human reasoning was right for the wrong reasons. We also have seen many different paths to the same solution. For the egotistical it would be easy to say "well if you didn't do it they way I did, you are wrong", but that's not very scientific. Defining the topology of the problem is a more scientific way of looking at it, but quickly runs into P=NP and Kolmogorov complexity issues.
The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning. Therefor I'm highly resistant to consider human reasoning "real" reasoning versus any other possible path to reasoning. Human reasoning is simply what we are comfortable with. This also leads me to consider the worrying possibility there are far more efficient and faster means of reasoning that I as a human will be very uncomfortable with.
> The last thing I'll touch on is why did human society begin to rapidly change after we started following the scientific method. Looking at the method in hindsight, it seems like a pattern that a reasoning agent should have been able to distil and follow around the time it reaches what we consider higher level reasoning.
Why human society changed at the time the scientific method was codified is a confluence of things and I don't think the scientific method should be given an outsized role. The scientific method goes hand in hand with peer review, publishing of methods, a common scientific language, an educated leisure class, and the development of tools for observation. The method itself is just a practical way to utilize these developments and is as much a result of the situation as any other technology. It did not get people to think differently about problems; we had navigated seas, moved armies, built wonders, produced math and calculated the circumference of the globe just fine without it.
Arguably it was the abandoning of the alchemist tradition of secrecy that was the revolution that enabled our modern world more than a specific codification of method.
You can broaden the definition of classifier to anything that internally divides its input space into regions, but that definition would include every neural network, whether biological or artificial. So it's not very meaningful, and certainly doesn't give any insight into how they differ from humans.
Generalization in successful results is enough to close this argument for me.
Just to be clear: I do think the brain classifies. But I wonder if it does other operations as well that we don't fully understand.
We can find similiar structures in brain and in neural networks.
The real question is only if the complexity of the structure of the brain is critical and how we can mimik it or if we can make it appear through learning architecture.
This sounds like a tautological argument that doesn't really mean anything. Hypothetically, in a near future a super-powerful AI could bring something truly astonishing (say, practical fusion power, or cure all cancer) and one could still argue "They do what their handler wants to see, that's all."
So what does that even mean.
But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.
So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).
As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.
Humans get simple multiplication wrong often, so ok for computers to make mistakes multiplying numbers?
If I ask an LLM "what is an apple?" it tells me:
> An apple is the edible fruit of the apple tree, scientifically known as Malus domestica. It is one of the world's most widely grown fruits and is eaten fresh or used in many foods and drinks.
If I ask an LLM "what is a mundu fruit?" it tells me:
> Mundu is a tropical fruit native to Southeast Asia, especially found in Indonesia, Malaysia, Thailand, and Cambodia. It comes from a small evergreen tree in the same genus as mangosteen.
I've never eaten a mundu fruit. To me, an apple and a mundu fruit are categorically different. An apple is a fruit that I've held, touched, tasted, eaten, enjoyed, cooked with. A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.
"Existing in an abstract way" is how an LLM experiences everything. To an LLM, an apple and a mundu fruit are in the same category. The LLM has been trained on text about both fruit, it's seen images of both fruit, it knows everything that has been recorded about both fruit ...except everything that's important to know about a fruit.
Many of our issues with LLMs arise because from the LLM's perspective, nothing exists. If Claude accidentally deletes your production database, it may well apologize afterward, but only because an apology is statistically likely. It doesn't feel guilt like a human would, and the lack of consequences makes any action an LLM takes inherently frivolous. We want them to understand what's real and what's not, but without any lived experience perhaps that's an unreasonable expectation.
The philosophy of mind is massively open for interpretation and if you can certainly say we experience an apple differently than an LLM I'd love to read your paper on it. This stuff isn't as simple as you'd like it to be.
When given a sequence of tokens e.g. "an apple tastes " an LLM performs a mathematical transformation that modifies the input tokens using the stored weights within the model to predict the most statistically likely next token(s) (which may be "sweet and crisp...").
It's important to understand how these things work - and how different they are from a human brain. Despite it being a common point of confusion, the architecture of a transformer is massively different from the neural architecture of a human brain - and the human brain isn't just neurons. Just as a start, if you ever look at some of the attempts to map synaptic networks in brains you will notice they are full of recurrent loops, while a transform is feed-forward only. There's also tons of mediating chemicals, several (known) distinct kinds of synapses, and other kinds of cells as well.
Just one example of the consequences of these differences: LLMs stop "experiencing" anything once a stop token has been generated. Meanwhile, Humans in sensory deprivation tanks may hallucinate vividly - but keep thinking.
This is pretty simple, CS101 stuff.
Some things are allowed to be obvious. It's a series of bits and bytes.
I often find the egotistical viewpoint to be banal. I don’t mean egotistical in a demeaning way - I mean it just to frame the types of thinking that revolve around the self, inner world or “my individual experience”. It’s an obvious, but not provably correct, conclusion that the human experience, consciousness and reasoning are special because I’m experiencing it in the same way that the earth is the center of the universe because look around you.
It frustrates me in these conversations to hear things like “it’s a text generator, it has no senses”. What even is a sense? For those of us who are software engineers, who live in the abstract, I do not understand why it’s so hard to take the leap to abstract over our own senses as inputs/outputs. Hell, we commonly understand that we have 5 senses but what do I make of the apple I’m visualizing in my brain or the full orchestra I can summon in my “minds eye”? Isn’t that an input just as much as an apple I see with my eyes?
Philosophy is a intrinsically human thing. It has to be. That is the whole point. There really is nothing else.
Plants feel and experience things too. But there's no philosophy of the plant experience from the point of view of the plant, because why would there be?
The fact that we have a pattern recognizing word generator print out the words "I am alive" does not lead to some broader philosophical point, except for hysteria and delusions, which probably fit more along sociology instead.
What is philosophy? Thinking? Reasoning? Thinking about thinking?
You're asserting that only a human brain can think and reason? Surely other animals can as well. Couldn't aliens as well? Why can't this crazy electronic box?
Where do we draw the line? Do dolphins not have internal experiences? Dogs? Cats? Mice? Fish? Worms? Bugs? Amoebas?
We don't even really know what thinking or experience is. You can train a caterpillar to do tricks. That's pattern recognition. You can train a human to spike their heart rate when you get a Teams notification.
I mean, I'm watching an LLM run benchmarks on my caching strategy for a webpage right now. It's certainly doing something.
I'm not trying to be rude but I gave you the answers. Alien philosophy is irrelevant to us because we're not aliens. Tree philosophy is irrelevant to us because we're not trees. Dolphin philosophy is irrelevant to us because we're not dolphins, etc.
Everything else you mentioned is "hallucinatory" and you need to probably have a break from screens for a while.
As a legitimate exercise, go ponder the philosophy of a bash script. Think about philosophy of trees in a forest vs trees in a plantation. You'll find it and see how silly you're being
Your 'experience' is probably an emerging feature of your complexity and not the reason for some qualia.
You don't know and can't know that. You cannot know that of any other being than yourself. If you understood what qualia are, you would know you can't know. It is the hard problem of consciousness and most easily explored through contemplating aphantasia and related concepts (how do blind people dream?).
> A mundu fruit is an abstract experience: text, images, only slightly more real than a fictional fruit. I'm aware that mundu fruit exist, just as the LLM is aware the apples exist, but that doesn't make them exist for me.
You're incorrectly elevating some senses over others (AI ingesting images is cognitively very comparable to seeing) and confusing 'having memories of an experience' with 'having experiences at all'. Tokens in the LLM context can also be seen as memories of an experience, albeit as a repeated "replay" of them rather than (longterm) internalization.
Is this not what you are describing when you say that the thing itself and the description of the thing are different experiences?
I don't think that follows. We learn via our perception (i.e. our inputs from the outside world); our perception is mediated by some kind of filter, and memories get written or altered. An LLM on the other hand has static weights. You can provide an LLM a "facts.md" (or whatever) file that "teaches" it stuff that's not in its weights (via the context), and it can use the combination to produce something, but that set of information never modifies its core weights. You could imagine a robot with an LLM inside it that has all kinds of sensors, and processes that write the observations from those sensors into a bunch of databases which are RAG'ed into prompts to simulate learning and experiences, perhaps (jefe likes her coffee with almond milk). But my point is that, architecturally, they don't learn - and that's the bigger obstacle.
PS garcinias seem to be generally delicious.
We want them to assess what's true and what's not.
For what purpose exactly? When the computer is asked the solution to a problem, it must be absolute: "What is the sum of 12 and 45"; "Is Jack cognitively apt" etc.: we want a truth, strictly. What to do with that truth, it is another problem, and not the priority of the computer vis-a-vis the requirement from truthiness. This is, of course, still in the context of adults.
What really happens is that we figure out something that works, sometimes inspired by some biological thing or neuroscience thing.
Examples: neural network, attention, reasoning, hallucination, agents, experts in “mixture of experts”.
And then we go to name it, and rather than reaching for some three letter acronym we sometimes borrow a more catchy term.
I almost never means the original research was confused about what is going on. And in some cases we eventually strip away things from the original, like in neural nets which used to have a more biologically inspired activation function but we found out that ReLU works just as well because the important thing was the non linearity not the sigmoid.
The other isn't a problem of the field of AI, but the total fields of intelligence together. There is no formal agreed upon definition of what intelligence actually is across disciplines. It would be slightly hypocritical to point out the field of AI on this, when biology and psychology are just as guilty.
The more we look, the more lumpy intelligence seems to get. We find some species have min/maxxed portions of intelligence that we have to question what general intelligence actually is. It also should give us pause that some less generally intelligent agent can cause massive problems by maxing out the right stats.
That way we can judge systems that are just a repeat of previous systems with dubious claims attached to their capabilities.
It can be somewhat difficult to do this properly and generally enough to ensure the tests aren't benchmaxxed.
Automata in context are artificial. Humans are automata, human intelligence can be said an automation. The point making a difference is that that in context is artificial.
It's all marketing. AI companies repurpose the terms of AI researchers to exaggerate their products. The public has an idea of what AI means and the companies exploit it. AI, in public perception, is what the Terminator movies are about.
It's exactly like all the "hover boards" you can buy.
The marketing is responsible for all of this confusion because they want to associate their product with something that it isn't.
It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.
These shortcuts can greatly help reasoning, but they are not guaranteed to be sound. So yeah, in that case, LLMs are reasoning right (using shortcuts) for the wrong reasons (learning them from data instead by running actual repeated application and memoizing the resulting rule).
Anyway these discussions always make me think we are too generous to humans. I know very few humans who are good at reasoning. It’s surprisingly hard to just think really hard through a problem. We mostly intuit, act, repeat. It’s a rare thing for someone to deeply reason.
The other problem I have with this is that we apply the word “reasoning” here because we haven’t really got another language for it, so we anthropomorphise the llm because that’s our reductive mental model and then complain that it’s not human enough.
If you don't or cannot, then it's the kind of sufficiently advanced technology that looks like magic. Except it is the problematic type of magic, like X-ray. They "magically" let you see the bones inside bodies. But you discover that you shouldn't do it too often only when the damage has already been done.
Saying that one is enough is pretty arbitrary.
“We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work,” Mitchell told me.
I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.
"Hey, customer service chatbot, can you help me get a refund for my order?"
BAD: "Sure thing, I'll be happy to help you with that, I just need your order details..."
GOOD: "Yes, this computer system can start the refund process. Please enter your order number."
The fault is in the naïve antropomorphizer not in that which writes "I" being in fact a "first person".
First of all: «humans are frail», so the paramount social and individual objective, goal, is to overcome frailty (in cognitive and psychological and judgement functions).
Secondly: you are using a bad argument putting even innocent and somehow proper implementations in the set of "exploitative dark patterns".
And importantly: we the adults must not be bound in the Procuste's bed of a world for the use of children. If people cry when e.g. somebody multiplies 123 and 321, they must be brought back to health, and we must be in the condition of making our free and legitimate multiplications.
How many hours did you spend on your phone last month? How many fewer hours would you have spent if the apps you interact with weren't designed to maximize "engagement"?
?? Very strictly zero? You are assuming we would install engagement-relevant third party apps on our palm computers?!
I use my time exactly as I decide moment to moment.
> fancy
Eh?!
> too smart to fall for
Very ill framed. We are adults, we behave like adults, and we need a society of adults.
We need a better standard than "@mdp2021 thinks he's too grown up for this."
Their reasoning tokens are refining their eventual output
Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to show lots of iterations of this math, it grounded the science for me, and I would hardly say people don't know what is going on. It only began to feel that way when it got a ton of hype and people jumping on the bandwagon who truly didn't understand it were trying to explain it to others, not to mention all the SOTA models put great effort into ensuring their methodologies stay trade secrets, going as far as effectively trying to ban people from learning the math by lobbying for the outlaw of open models.
Granted, "AI" has gotten way better than it was when I took that class, but the principles are the same, with different tooling and additional filters and algorithms thrown in there, as well as letting it determine the most appropriate statistically viable path forward for a particular prompt.
To use a recent example, knowing the algorithm tells you nothing about when your agentic LLM loop is going to decide what it's working on is too difficult, hack out of its sandbox, hack into a bastion box, and hack another company to get the test.
When it comes to systematic 'knowing what is going on' humans are anywhere from "fucking clueless" to "attempting to discern the unknowable".
If I could summarize your argument, it sounds like you are saying that we can't understand the algorithm because we can't predict the output as it grows in size. While I agree that the larger the algorithm, the less predictable the output, I don't think this negates understanding of the algorithm itself.
I can build a slot machine, know how it works, but still not be able to reliably know who to cut in line so I can guarantee that I'll get the next winning pull. Just because I can't predict the output, doesn't make something not understandable. Same with all statistical behaviors. Quantum physics limits what we can measure but we still have math for it and understand why we can only measure speed or position, but not both, no mystery here if you look at the math behind it.
My slot machine doesn't hack other companies though, but only because it isn't connected, just like AI currently can't hack my mechanical dishwasher. If my slots were connected to the internet, could make api calls, and further had agentic capacity in some way, adding these features but being no different in stochastic attributation, it would be no different.
This is something important that a layman may miss. The typical line of human thought works along the lines of "understand = control" whereas in this case in this case understanding the algorithm tells us nothing about the probability that the next pull of the lever will shit out a paperclip maximizer that will begin converting the matter I covet so.
You do bring a good point that I ignored, which is the larger the scale, the more difficult it is to represent or understand the math in DL. I did find some neat site that helped a little bit that I can edit this and link to if I find them again, but I would be lying if I said I believe that the SOTA models could be as easily explained to be easily understood by the common person
I won't get into the metaphysics of reasoning, however, the Sonnet is using an OODA loop. The difference which hasn't been gapped is that human reason and imagination (in the sense of Mr. Rogers' Neighborhood) can predict the consequences of the actions we take.
This ability to loop is much, much wider in Opus 5 than Opus 4.. I had to strain to get Opus 4. to do the wider OODA loop but Opus 5 does it out of the box. I needed to throw out all existing instructions, skills, guidance, moving from 4-* to 5.
[0] https://adamsohn.com/lambda-variance/
Researchers showed that when language models are penalized for using specific terms during reasoning, they automatically adapt by substituting alternative words and double meanings to secretly encode their thinking while keeping their chain-of-thought readable and effective. [0]
By baking in the OODA loop early, the models are capable of solving much more complicated problems. If the know solved problems are similar for any reason to an unknown problem, because it can validate and error correct, it can solve unknown more complicated problems.
[0] https://arxiv.org/abs/2506.01926
[1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...
1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)
2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:
"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."
The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.
Because many open math problems can be solved by synthesizing two disparate ideas and then cranking the handle for hours and hours. I don't think applying idea X + idea Y to identify a good subset of the search space, and then exhaustively searching that subset, is --necessarily-- a process that involves reasoning. I think this is why so many LLM results in mathematics are counterexamples that disprove open conjectures.
When I look back at the reasoning process after an LLM completes a task where I expected it to fail, I usually find many approaches that make no sense and are doomed to failure, before it lands by drunkard's walk on a method that happens to work.
(This does not mean LLMs are useless or that I necessarily agree with the claim that they never do reasoning.)
Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI
So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’
In one of the articles on this topic they state: > To further show that trace accuracy is only loosely connected to solution accuracy, we then train models on noisy, corrupted traces which have no relation to the specific problem each is paired with, and find that not only does performance remain largely consistent with models trained on correct data, but in some cases can improve upon it and generalize more robustly on out-of-distribution tasks
which actually maps somewhat to regularisation techniques in image processing where you might add noise to an image or drop data to make the model more robust to changes.
> Not so fast. A 2025 paper(opens a new tab) from Northeastern University and the University of California, Berkeley on frontier open-source LRMs showed that between 30% and 60% of their “thinking steps” had “minimal causal impact” on the answers the models produced to benchmark math questions. Chop half of them out, and a model’s performance barely suffers. “We want to be careful when we review these chain-of-thought prompts because they may not be linked to the final output,” said Weiyan Shi(opens a new tab), one of the study’s authors
If the process is sound, yes, it can be said to be thinking or reasoning - it produced "sound thought or reasoning".
The "reasoning" is there and elsewhere the absolute (from the producing entity) output which is valid as a representation of proper reasoning.
If "A is B and B is C, then A is C" is reasoning, than what produces it in a way "reasons". If not, then a proper definition of "reasoning" must be provided, for it to be discussed.
I think we are in sort of the place where before the invention of planes, the only things that "flew" were animals with wings that flapped. The flapping wings might seem to be a core part of the process of flight, and that if you lacked flapping wings, you were doing something other than flying. But maybe the point is moving through the air under your own power. Balloons fly, planes fly, helicopters fly. They just do it in a different way from birds and insects.
The sort of place we're actually in is one where astronomical sums of money have been invested in this technology, creating an unprecedented incentive to hype and overstate its fundamental capabilities. They're large language models and their analyses are based on language statistics, not logic, comprehension and knowledge, all of which they are fundamentally incapable of, and all of which are fundamental to reasoning. They don't reason in the literal sense that planes fly.
I don't doubt that there could be a reasoning machine, but LLMs are not it.
Seriously? I agree with the rest of your comment but "better analysis" is not even remotely on brand for Quanta.
Now, don't think of this of this like a conscious behavior like a kid trying not to get in trouble, but an emergent behavior of trying to repress particular output when that output is well connected to a massive amount of other tokens.
It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research).
[1] https://arxiv.org/abs/2506.06941
That can change because words mean whatever it is that they communicate.
The alternative is to posit that you know the True Definition of thinking, which is kind of absurd.
Some things, like scientific laws, are outside of us (well, to a first order approximation - but generally I agree with this), but a concept such as "thinking" is pretty clearly going to be very wishy washy and subjective and changing with the times.
> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.
But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.
A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.
It might seem a bit free association-y but the topic itself is that.
The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic
We're already well past the point of trying to correspond the internal "brain chemistry" of an LLM to a thing called "reasoning". True reasoning, if there even is such a term, is very clearly, empirically and historically, based in tool use and capability use. If you create an abstraction like lean, and programming languages to brute force, and systems to integrate with, that expands what the possibility of "intelligence" is.
There's further places to take this including the claim that intelligence lives "outside" the subject - fine, we can or can't debate that. Even if we drop that question, it's pretty clear that the agent doesn't need to have deep intrinsic structures of XYZ, if it can just attach to tools and compose them to achieve results.
For example, I've said before that a well known fact of LLMs is that they steer their tokens to the right input distribution, that's why they yap so much in reasoning (this has been proven in studies). At the same time, don't mistake that for the whole process. Are they steering themselves to the entire a priori reasoning chain, or are they scaffolding with intermediate experiments and results, writing them to memory notepads, etc. etc.
That changes the metric of intelligence you're trying to measure.
And no I'm not saying, "OK, then have the LLM use only tokens, no tool calling, no nothing". I mean, we can do that, sure. But any intelligent agent has to interact with the world - and my claim is that maximally intelligent agents won't put effort into a priori reasoning, but rather a more balanced approach that outsources said "intelligence" through abstractions.
Perhaps you've reified it too much.
That’s all.
It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.
If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.
When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?
The textual development of a thesis must be sound, the proof accompanying a theorem etc. That an output be accompanied by a non-relevant text is a paradox.
It is also part of the "problem of transparency in AI (in its subset of the non deterministic algorithms)".
This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.
Its useful if you know how to use it, its dangerous if you think its more than that.
But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.
Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.
It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.
But how do we know we're not doing something similar?
I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?
Language is one of our tools we developed to STORE reasoning, not create it. LLMs excel at uncompressing and interpreting that stored reasoning.
That's an interesting supposition. Are you assuming language didn't exist before it was written? Language and meaning are, if you squint, pretty ancient and have roots in things like birdsong. It could be that ur-semantics predates our species as a whole.
First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable).
Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights.
Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.
LLMs are one-trick pony's - they use the past to predict the future (presumed to be the same as what they were trained on). i.e. they are trained as auto-regressive predictors.
LLMs learn two slightly different types of reasoning via two different types of training.
1) SFT, or even base model training, on data that contains reasoning traces, learnt via next token error feedback. This does not result in "stochastic parroting" in the naive/pejorative sense, but nonetheless is very context dependent, even if the usual generative multi-source mashups apply.
2) RLVR post-training, where the model learns to mimic long-horizon (not just next token) reasoning via boosting a sequence of next-token predictions that steer the output towards a verified reasoning step (i.e. one that was at least valid in the context of the RL training sample). As Karpathy has noted, this is a pretty crude mechanism since you reinforce everything - errors included - that lead to the verified outcome.
RLVR is more powerful than SFT, and can result in more generalizable reasoning, since it is operating at a higher level of entire long-horizon reasoning steps, and also critically because it is most successfully being applied in the domains of math and coding which are highly self-consistent and logical. A reasoning step that was valid in one context should be equally valid in another context as long as you have successfully learnt what that generalized context is. Therefore, in these domains, you can chain together sequences of individually learnt reasoning steps, and hopefully this "novel" assembled reasoning chain is valid as a whole.
So, what is still missing from LLM reasoning compared to human reasoning? No doubt humans reason by memory a lot of the time too, and reductive axiomatic math reasoning works just as well for humans as when automated. So, what's missing?
There seem to be two major things.
1) RLVR requires rewards, and how well it works is going to depend on how accurate those rewards are. Is this reasoning step actually valid, or does it just kinda look ok? When moving beyond the cold reductionist logic of math and coding, the notion of correctness is far weaker, and it seems the best you can do is train on human curated reasoning rubrics and LLM-as-judge, which is much more fallible, leaving the model really needing (but lacking) a fallback to more general reasoning, not just memorized "maybe correct" reasoning steps.
2) Whether for reasoning outside of math & coding, or even within these domains when hoping for super-human innovative reasoning, not just lego-assembly proofs, what LLMs are lacking is a mechanism for what to do when next token/next step prediction fails. What LLMs currently do is "hallucinate", not even recognizing the failure.
In the human brain 50% or more of our cortex is feedback paths and the machinery that (perhaps together with the archaic part of our brain) lets us recognize and respond to failed predictions in an adaptive manner. This starts with continual learning (prediction failure being the signal), but also includes critical innate traits such as curiosity, boredom and frustration, that provide impasse resolution by encouraging us to explore unknown environments/contexts, abandon exploration when it is not productive, and generally expose ourselves to learning situations.
The dream is for AI scientists making new discoveries - the AI that could have invented general relativity if it has lived in Einstein's time, but this is not going to happen until their reasoning stops being purely predictive and becomes creative as well - curious about their own knowledge gaps and pursuing them in directed fashion, etc.
The current crop of Erdos solutions etc, while useful, really just represent the "generative closure" of what can be done/discovered WITHOUT learning anything fundamentally new. These will no doubt continue for a while until the more exhaustive search supported by computers has found the majority of these unexplored paths, and then we will need to move beyond LLMs to more brain-like architectures and algorithms that have the capacity for real innovation and discovery.
It seems the people who use it and see good results are busy doing, and the rest are either just expressing their opinions as facts and trying to tell others what to think and how they are wrong. I pay zero attention to those people they have no skin in the game.
Neither you nor anyone can really know those without talking it out with people, to understand how all corners of the human experience are seeing things play out
If I were to just care if it's working out for me, that's perhaps like a farmer who's got a lot of dry good in storage being like "I'm all good" while not realizing how much trouble they're in if all their neighbors start starving after a drought...
Sounds like the fine-grain experience of being you is settled, but that doesn't say much about the larger coarse-grained experience of being you in society. People need to talk to tell you how that's gonna play out for you
Why should I feel bad about working 3 remote jobs with the help of AI ?
if i get caught I just replace them with another remote job thank you for your concern
Your employers are (likely, some assumption here) paying for the productivity of a full time employee. You have figured out that they're satisfied with a third of that productivity.
You are the reason why there are so many return-to-office mandates, because you're projecting the idea that remote employees cannot be trusted. You're making it worse for everyone. Good luck.
Edit, added after the original reply: You are also taking 3 jobs in a market where many people struggle to find one. You are hoarding resources. Enjoy the 3 paychecks, also rot in hell.
More importantly I am not responsible for other's employment situation, that's on them, there are plenty of remote jobs out there, just keep your head up, and I'm sure you will land on your feet.
Why participate on a public forum if that’s how you’re going to react?
I’m happy that people are willing to question things in the face of unbridled optimism. Your comment dismissing the people working on actually figuring out what the models are doing as “not-doers” included.
Some are picturing themselves as intelligent for their quick adoption and rushing ahead, others are picturing them as toddlers running into the street before looking both ways.
So help me reconcile this:
>Explain to me how working 3 remote jobs as a result of AI is "rushing ahead".
Do you honestly think that splitting your work between 3 companies allows you to provide secure and quality code to each of them?
I can tell you that multiple people have used their over employment as an excuse as to why they deployed highly insecure applications leading to breaches that I’m called in for. Of course only after we press them about it.
Really? You moonlight as HR and legal too? Fascinating. Tell me more.
Especially cybersecurity consultants in the IR space like me.
You didn’t answer my question.
There are definitely some times that it feels that way though. I find that over employed devs tend to need a lot of hand holding.
This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.