Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.
Was Hofstadter ever arguing that intelligence requires self-referentiality?
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.
If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know.
What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following:
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.
Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
But I suppose it doesn’t harm its reasoning!
I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.
[delayed]
Was Hofstadter ever arguing that intelligence requires self-referentiality?
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
It seems like it is the bitter lesson of the emergence of intelligence.
Kind of question-begging towards the idea that LLMs are self-referential in the same way that Hofstadter means. Even if they loop, they don’t do so strangely.
If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following:
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.
The fact that we constantly switch between the colloquial definitions of those terms and more specific, technical ones doesn’t help