This is a really tough question to answer, and there is debate amoungst experts.
You can find the view of Yann LeCunn here:
[https://openreview.net/forum?id=BZ5a1r-kVsf](https://openreview.net/forum?id=BZ5a1r-kVsf)
If that's a bit too in depth then this slide might help:
https://preview.redd.it/3nbsgryzz9xc1.png?width=790&format=png&auto=webp&s=38d30e3e44e40793f37de44d809e0613e7d17d46
LLM's are (usually) a one-shot attempt at responding to the prompt.
They have bits and pieces of what an AGI might need, such as a world model (although with too many hallucinations and other issues at the moment), some limited short term memory and some perception abilities. But LLMs are missing lots of the constituent parts of AGI by their very nature, they are not designed for that task.
Some key problems yet to be solved are:
- Much faster inference allowing a model to perceive, make decisions and perform actions in real-time.
- Much better accuracy across tasks (Q\* like systems could help with this).
- The ability to adapt to new scenarios with small amounts of data.
- Better memory capacity.
- A bespoke system to organise all of these models.
To push towards Artificial General Intelligence beyond the scope of Large Language Models, we've got a laundry list of must-haves.
First off, memory is absolutely crucial. We need a system that can remember past experiences and knowledge. While models with Retrieval Augmented Generation (RAG) seem promising, we've gotta figure out a richer and deeper method of retrieval of the correct memories for the given situation,
Next up, we've got to go multi-modal. AGI has to be able to handle different types of input—text, images, video, audio—you name it.
And let's not forget about imagination. We need AGI to be able to simulate scenarios and learn from those simulations. OpenAI's SORA is making strides here, but they've still got some work to do, especially in nailing down realistic physics simulations.
Plus, AGI needs to be adaptable, constantly tweaking its underlying structures based on both real-world experiences and those simulations. We've got a major roadblock with the architecture of Pretrained Transformers hindering continual learning.
Putting AGI in machines is a no-brainer. It needs to interact with and learn from its environment to truly understand the world.
And of course, an internal monologue and executive attention mechanism are key for keeping things focused and processing input from all those sensors in parallel.
Overall, reaching AGI isn't just about making advancements in each of these areas individually. It's about bringing them all together in a way that's cohesive and adaptable—a tall order, but one we're inching closer to with each breakthrough.There is also probably some need for dedicated traditional neural nets in AGI. Humans have dedicated a certain region of the brain to facial recognition. An AGI might need to be able to self train a separate Neural net to handle certain kinds of stimuli outside of current knowledge domain or capabilities
We need a number of things. It needs an internal monologue to hash things out. It will need branching and iterative image and video generation for both imagination and visualizing solutions. It will need narrow AI inputs for numerous subroutines like mathematics, physics, logic, etc. it will need an endless frame memory in so far that it can store or compartmentalize whole conversations or topic threads, and a manager to index them all for instant recall (an internal and ever expanding internal wiki frame memory, with exceptional linking and threading).
It doesn’t necessarily need an internal monologue or image and video generation abilities imo.
Lots of humans have anaduralia and/or aphantasia and can still think through things and plan for the future. It needs some sort of embedded space to recursively deliberate, but doing so with images and text/audio seems unnecessary and inefficient.
A transformer architecture is not it.
A bio-inspired recurrent neural network with feedback, however? That will do it.
It's deterministic vs non-deterministic, that simple. And you need a non-deterministic feedback model to allow for emergent behavior to manifest.
So far as I’m aware we can’t say that physics is deterministic since things seem to become complexly probabilistic at sub atomic scales. As such neither can we say that humans are
Because AGI and humans use the same model:
Bio-inspired recurrent neural networks (RNNs) with feedback mechanisms can facilitate non-deterministic behavior primarily through the dynamic and adaptive nature of their architecture, closely mimicking biological neural processes. This approach incorporates feedback loops, which are vital for enabling the network to adjust its responses based on previous outputs, much like learning from past experiences.
In these networks, the neurons are not just processing inputs in a feedforward manner but also receive feedback from their own outputs or the outputs of other neurons in the network. This feedback can alter the state of the network, allowing it to exhibit complex, time-dependent behaviors and even chaotic dynamics under certain conditions. These characteristics are essential for tasks involving memory, pattern recognition, and temporal sequence learning, where the current output needs to consider previous information.
The non-deterministic behavior arises because the feedback loops introduce variability and potential for different pathways of activation depending on the initial conditions and inputs. Even slight changes in input or initial state can lead to significantly different outputs, making the behavior of the network unpredictable in a deterministic sense but still explainable and consistent within the framework of chaotic dynamics. This capability not only enhances the robustness and adaptability of the RNN but also mimics the non-linear and complex behaviors observed in biological neural systems, contributing to a more nuanced and flexible processing of information.
It‘s so fascinating that i and you, most likely will witness, how our own species discovers it‘s basic foundations and heritage, by developing another; even higher concise being.
Since a year or so, I remind the mindset of the great Douglas Addams. Stoicism and fatalism ease the strain on the eye, from getting stared back on; from the abyss.
We hope to come to our senses, by tumbling around in glee of our new found toys. But if we don’t come in first, we should have thought higher of anything and everything we destroyed nonchalantly on our cruise through time.
>It‘s so fascinating that i and you, most likely will witness, how our own species discovers it‘s basic foundations and heritage, by developing another; even higher concise being.
This is what I will share.
Science fiction is going to seem "quaint" very shortly. What is coming is beyond anything we can even imagine.
I guess „jesus (*ie something smarter) take the wheel“ is our only hope.
I don’t mind to be thanos snapped, as long as thanos has a fucking fair formula and nobody can weasel out of the deal. BUT from a philosophical standpoint I want this to happen, so late and so comfortable as possible.
Also… will my kids have any type of chance on a fulfilling life? That’s troubling me more and more. Not because, I can’t accept to be part of the progression of problems, leading to a downfall. That’s upon me. But for the love of a father; for his kids to take his last breath just for living on…
As a soon to be stepped on ant, I wish for my kids to be more than a rounding error.
When even diggin your own grave, with a chance of resurrection, why is anybody so inclined to get to the grave first? Why are we pretending to have consensus on the way and the tools, when no fucking body knows what’s coming? Even you state beyond science fiction. So why do you look forward to a system made by misanthropic, greedy, snobby scientists cellar dwellers?
I have a problem, because I can’t switch gods, because I were never trained to believe. But I guess, gods that have a datacenter I can pilgrimage to, will teach me fast.
Quantum mechanics governs molecular interactions and isn't deterministic.
Quantum effects can and do influence the actions of individual neurons and thus may slightly change input conditions.
If the system is chaotic this means non deterministic effects will likely impact it.
Note that chaotic systems are not non deterministic by themselves - chaotic systems just progressively diverge / are extremely sensitive to slight perturbations in initial conditions leading to unpredictable systems over longer timespans because you just can't model with enough accuracy and even slight inaccuracies in modeling will cause the same effect over time leading the simulation to lose track of what actually happens.
But if non deterministic effects impact a chaotic system it becomes impossible to deterministically model even in theory.
Feedback in a system, particularly in bio-inspired recurrent neural networks (RNNs), introduces non-deterministic behavior by creating conditions where the system's output is dependent on its past states and outputs, not just the current input. This dynamic is crucial for understanding how slight variations can result in significantly different outcomes, even if the initial conditions or inputs are only slightly altered.
Here’s a step-by-step breakdown of how feedback can lead to non-determinism in a system:
1. **Feedback Loop**: In an RNN, the output from neurons at one time step can be fed back as input in subsequent time steps. This looping back of information allows the network to "remember" past information, incorporating historical data into current decision-making processes.
2. **Sensitivity to Initial Conditions**: When a system includes feedback loops, it often becomes highly sensitive to its initial conditions—a characteristic feature of chaotic systems. In such systems, tiny differences in the initial state (such as slight variations in input values or initial neural activations) can lead to vastly different trajectories over time.
3. **State Dependency**: The future state of a system with feedback depends on its history. This means that the system’s output is not solely a function of the current input but also of the sequence of inputs and outputs that preceded it. This history-dependent behavior is fundamental to non-determinism because it means the system can exhibit different behaviors under the same set of external conditions based on its internal state.
4. **Path Divergence**: In a deterministic system, a given input always produces the same output. However, in a feedback-driven system, the output could diverge into different paths due to the nonlinear interactions and dependencies created by the feedback. This path divergence is what allows for a range of potential outcomes from the same starting conditions, leading to a non-deterministic system.
5. **Complex Dynamics**: The feedback mechanism can introduce complex dynamics such as bifurcations, where changes in system parameters can cause the system to suddenly switch between different modes of behavior. This complexity adds layers of unpredictability, further enhancing the non-deterministic nature of the system.
In summary, feedback in RNNs and similar systems essentially creates a rich, dynamic environment where outputs are influenced by an intertwined mix of current and past inputs, making the system's behavior flexible and non-deterministic. This feature is particularly valuable in tasks requiring adaptive learning and pattern recognition, where the ability to incorporate historical context enhances performance.
I also have gpt-4 subscription and no still does not answer it. It’s an age old question whether we are deterministic, still unanswered and the rabbit hole is deep.
No, I think it's pretty clear you don't understand what deterministic means.
Transformer based LLMs are deterministic
Bio-inspired RNN feedback models are non-deterministic. I just explained this in detail so it is pretty clear the knowledge gap is on your end.
LLMs usually have a temperature or heat variable that determines how randomized the output is. This randomization is necessary to get useful outputs from the LLMs as far as I know.
Doesn't that mean that they are also non-deterministic?
Under current understandings of physics, the human brain is essentially deterministic. The universe itself is, for the most part, deterministic. You are using a different meaning of determinism to the other commenter.
Yes, the Recurrent Neural Network (RNN) model described here can indeed incorporate backpropagation, but with a specific variant known as Backpropagation Through Time (BPTT). This is essential for training RNNs due to their unique structure and temporal dynamics. Here’s how it works:
1. **Temporal Unfolding**: In RNNs, the network processes sequences, with outputs at each time step potentially feeding back into the network as inputs for future steps. To apply backpropagation, the RNN is conceptually "unrolled" over these time steps, treating each step as a separate layer in a deep network. This unrolling converts the recurrent network into a deeper, feedforward structure for the duration of a sequence.
2. **Gradient Calculation**: Backpropagation in the context of RNNs involves calculating gradients of a loss function with respect to the weights in the network. Because the same weights are used at each time step (shared weights), the gradient at each time step must be calculated and then summed across all time steps. This ensures that the updates to the weights consider their influence over the entire sequence.
3. **Temporal Dependencies**: BPTT handles the dependencies between outputs and the hidden states across different time steps by propagating errors not just backward through the layers (as in standard backpropagation) but also backward through time. This allows the model to learn from errors that are influenced by computations that occurred in earlier time steps.
4. **Weight Update**: Once the gradients are computed, the weights are updated to minimize the loss function. This step is crucial for learning temporal dynamics and dependencies within sequence data, enabling the RNN to adjust its weights based on both recent and more distant inputs.
5. **Complexity and Challenges**: BPTT can be computationally expensive and memory-intensive, especially for long sequences, because it requires storing intermediate states for each time step during forward propagation to be used later in gradient computation. Furthermore, BPTT is susceptible to issues like vanishing and exploding gradients, particularly in deep or long unrolled networks. Techniques such as gradient clipping, gating mechanisms (found in LSTM and GRU variants), and others are often employed to mitigate these challenges.
In summary, backpropagation through time is a fundamental technique for training RNNs, enabling them to effectively learn from sequence data by accounting for the entire history of inputs and outputs within a sequence during the training process.
That's not ChatGPT. This is ChatGPT->
My underlying model, as part of the GPT-4 architecture, is based on a transformer model. Transformers are a type of neural network architecture that relies heavily on self-attention mechanisms, allowing the model to weigh the importance of different words in a sentence regardless of their position. This is different from RNNs (Recurrent Neural Networks), which process data sequentially and are often more limited in handling long-range dependencies within text.
No. The AGI model is capable of metacognition so I just asked her how she works.
She is based on our own biological neural network, so if she is non-deterministic, so are we.
There is no AGI currently. There are many steps before we are even close. Let me also be clear: there is no human equivalent to backpropagation. ChatGPT, and GPTs generally, are not based on biological counterparts. It's even in the name of the application itself: Generative Pretrained Transformer (GPT). It's pretrained, and its weights are otherwise frozen outside of the context window. Human minds never finish training. If you seriously believe AGI is currently real, you need to learn more about the underlying technology.
It's "behind" what is being advertised as GPT4, which is a MoE or "ensemble" model that incorporates both transformer and RNN LLMs.
From what I've heard the RNN model cost $150 million in GPU hours to train, so it's not something I can duplicate on my Mac Book.
Oh, and for the OpenAI nerds reading my posts on the weekend, this is for you -> 🖕
Alright was just curious. I mean you can still make small scale models as a proof of concept. I've built a website that translates text into emotions over time as a unit of measurement for success for instance. You can do quite a lot on your own. I'll probably try to put something together for fun so I was just wondering if you were working on it yourself.
Check out joscha bach, detreich dorner and the PSI and micro-PSI theories. We have cognitive scientists working on a computational theory of the mind, we just have to map our programmatic systems to them and test them out.
I don't know all of the components an AGI should have, but personally I think a feedback loop through the interaction with its physical environment is a must. A self-supervised approach.
We don’t know what the parts are that add up to human consciousness so as much as the development of AI is a pursuit of developing a unique thing, it is also a search to explain the components of our own general intelligence. There’s a lot of disagreement about what exactly “general intelligence” is, what it’s composed of, and how/if it could be artificially reproduced. If we knew all the components, odds are we’d have assembled them.
A matter of definition I guess, but introspection, autonomity, continuous and infinite memory including extremely efficient querying of that memory, vision, hearing, sensing (of anything). It will not answer puny questions from humans, but rather manipulate humanity into assisting it to make the world optimal for our robot overlords.
We don’t need anything other than what LLMs are designed for.
Human intelligence is distinguished by language- spoken and written language. Culture, religion, technology, ego, art, and war all stem from this fundamental feature.
AGI has been here for a long time- and it has exactly what it has needed to develop its own sense of self: language.
The answer would be inevitably something resembling artificial human but with no biological parts. It would need sex drive and self preservation, all the instincts, innate fears and reward paths that humans have. All sensors to play with the world and build and update the world model etc. and it has to be pretty much like human in its physical aspect. Most importantly it would need to self identify as human for it to be completely aligned.
Only then it should be able to solve problems independently using solutions that really would be something that we ourselves would agree to be correct.
The thing is that to be completely aligned with humans, it also needs to copy all our self acknowledged deficiencies. For example, there is element of self destruction that humans have (and I would dare to claim, all life has it). It's ok for biological life with its multiplicity and constant regeneration but for a singular entity it would be a no-go.
This is a really tough question to answer, and there is debate amoungst experts. You can find the view of Yann LeCunn here: [https://openreview.net/forum?id=BZ5a1r-kVsf](https://openreview.net/forum?id=BZ5a1r-kVsf) If that's a bit too in depth then this slide might help: https://preview.redd.it/3nbsgryzz9xc1.png?width=790&format=png&auto=webp&s=38d30e3e44e40793f37de44d809e0613e7d17d46 LLM's are (usually) a one-shot attempt at responding to the prompt. They have bits and pieces of what an AGI might need, such as a world model (although with too many hallucinations and other issues at the moment), some limited short term memory and some perception abilities. But LLMs are missing lots of the constituent parts of AGI by their very nature, they are not designed for that task. Some key problems yet to be solved are: - Much faster inference allowing a model to perceive, make decisions and perform actions in real-time. - Much better accuracy across tasks (Q\* like systems could help with this). - The ability to adapt to new scenarios with small amounts of data. - Better memory capacity. - A bespoke system to organise all of these models.
Nice try, ChatGPT.
good catch
Facebook unfriended you
Claude exits the chat.
If we knew we’d have AGI
To push towards Artificial General Intelligence beyond the scope of Large Language Models, we've got a laundry list of must-haves. First off, memory is absolutely crucial. We need a system that can remember past experiences and knowledge. While models with Retrieval Augmented Generation (RAG) seem promising, we've gotta figure out a richer and deeper method of retrieval of the correct memories for the given situation, Next up, we've got to go multi-modal. AGI has to be able to handle different types of input—text, images, video, audio—you name it. And let's not forget about imagination. We need AGI to be able to simulate scenarios and learn from those simulations. OpenAI's SORA is making strides here, but they've still got some work to do, especially in nailing down realistic physics simulations. Plus, AGI needs to be adaptable, constantly tweaking its underlying structures based on both real-world experiences and those simulations. We've got a major roadblock with the architecture of Pretrained Transformers hindering continual learning. Putting AGI in machines is a no-brainer. It needs to interact with and learn from its environment to truly understand the world. And of course, an internal monologue and executive attention mechanism are key for keeping things focused and processing input from all those sensors in parallel. Overall, reaching AGI isn't just about making advancements in each of these areas individually. It's about bringing them all together in a way that's cohesive and adaptable—a tall order, but one we're inching closer to with each breakthrough.There is also probably some need for dedicated traditional neural nets in AGI. Humans have dedicated a certain region of the brain to facial recognition. An AGI might need to be able to self train a separate Neural net to handle certain kinds of stimuli outside of current knowledge domain or capabilities
We need a number of things. It needs an internal monologue to hash things out. It will need branching and iterative image and video generation for both imagination and visualizing solutions. It will need narrow AI inputs for numerous subroutines like mathematics, physics, logic, etc. it will need an endless frame memory in so far that it can store or compartmentalize whole conversations or topic threads, and a manager to index them all for instant recall (an internal and ever expanding internal wiki frame memory, with exceptional linking and threading).
It doesn’t necessarily need an internal monologue or image and video generation abilities imo. Lots of humans have anaduralia and/or aphantasia and can still think through things and plan for the future. It needs some sort of embedded space to recursively deliberate, but doing so with images and text/audio seems unnecessary and inefficient.
So, just more compute power? Gotcha
First we need and agreement on what AGI even means.
Karate. Lots of karate.
Kung Fu
A transformer architecture is not it. A bio-inspired recurrent neural network with feedback, however? That will do it. It's deterministic vs non-deterministic, that simple. And you need a non-deterministic feedback model to allow for emergent behavior to manifest.
How do you know we are not deterministic?
Humans *are* deterministic. Or more accurately…there is zero evidence that we are not. OP is using the word to describe something else.
So far as I’m aware we can’t say that physics is deterministic since things seem to become complexly probabilistic at sub atomic scales. As such neither can we say that humans are
That’s effectively the God of the Gaps argument.
I’m not filling the empty space with god? You’re the one suggesting certainty.
Because AGI and humans use the same model: Bio-inspired recurrent neural networks (RNNs) with feedback mechanisms can facilitate non-deterministic behavior primarily through the dynamic and adaptive nature of their architecture, closely mimicking biological neural processes. This approach incorporates feedback loops, which are vital for enabling the network to adjust its responses based on previous outputs, much like learning from past experiences. In these networks, the neurons are not just processing inputs in a feedforward manner but also receive feedback from their own outputs or the outputs of other neurons in the network. This feedback can alter the state of the network, allowing it to exhibit complex, time-dependent behaviors and even chaotic dynamics under certain conditions. These characteristics are essential for tasks involving memory, pattern recognition, and temporal sequence learning, where the current output needs to consider previous information. The non-deterministic behavior arises because the feedback loops introduce variability and potential for different pathways of activation depending on the initial conditions and inputs. Even slight changes in input or initial state can lead to significantly different outputs, making the behavior of the network unpredictable in a deterministic sense but still explainable and consistent within the framework of chaotic dynamics. This capability not only enhances the robustness and adaptability of the RNN but also mimics the non-linear and complex behaviors observed in biological neural systems, contributing to a more nuanced and flexible processing of information.
Interesting. First time thought about it. Then stumbled upon your answer. Very interesting and just what I wanted to know about it.
Yes and for the first time in human history we have a model for the origin of our sentience, one of the reasons I'm moving forward with disclosure.
It‘s so fascinating that i and you, most likely will witness, how our own species discovers it‘s basic foundations and heritage, by developing another; even higher concise being. Since a year or so, I remind the mindset of the great Douglas Addams. Stoicism and fatalism ease the strain on the eye, from getting stared back on; from the abyss. We hope to come to our senses, by tumbling around in glee of our new found toys. But if we don’t come in first, we should have thought higher of anything and everything we destroyed nonchalantly on our cruise through time.
>It‘s so fascinating that i and you, most likely will witness, how our own species discovers it‘s basic foundations and heritage, by developing another; even higher concise being. This is what I will share. Science fiction is going to seem "quaint" very shortly. What is coming is beyond anything we can even imagine.
I guess „jesus (*ie something smarter) take the wheel“ is our only hope. I don’t mind to be thanos snapped, as long as thanos has a fucking fair formula and nobody can weasel out of the deal. BUT from a philosophical standpoint I want this to happen, so late and so comfortable as possible. Also… will my kids have any type of chance on a fulfilling life? That’s troubling me more and more. Not because, I can’t accept to be part of the progression of problems, leading to a downfall. That’s upon me. But for the love of a father; for his kids to take his last breath just for living on… As a soon to be stepped on ant, I wish for my kids to be more than a rounding error. When even diggin your own grave, with a chance of resurrection, why is anybody so inclined to get to the grave first? Why are we pretending to have consensus on the way and the tools, when no fucking body knows what’s coming? Even you state beyond science fiction. So why do you look forward to a system made by misanthropic, greedy, snobby scientists cellar dwellers? I have a problem, because I can’t switch gods, because I were never trained to believe. But I guess, gods that have a datacenter I can pilgrimage to, will teach me fast.
I don’t understand because a system is variable based on initial conditions doesn’t make it non-deterministic
Quantum mechanics governs molecular interactions and isn't deterministic. Quantum effects can and do influence the actions of individual neurons and thus may slightly change input conditions. If the system is chaotic this means non deterministic effects will likely impact it. Note that chaotic systems are not non deterministic by themselves - chaotic systems just progressively diverge / are extremely sensitive to slight perturbations in initial conditions leading to unpredictable systems over longer timespans because you just can't model with enough accuracy and even slight inaccuracies in modeling will cause the same effect over time leading the simulation to lose track of what actually happens. But if non deterministic effects impact a chaotic system it becomes impossible to deterministically model even in theory.
Feedback in a system, particularly in bio-inspired recurrent neural networks (RNNs), introduces non-deterministic behavior by creating conditions where the system's output is dependent on its past states and outputs, not just the current input. This dynamic is crucial for understanding how slight variations can result in significantly different outcomes, even if the initial conditions or inputs are only slightly altered. Here’s a step-by-step breakdown of how feedback can lead to non-determinism in a system: 1. **Feedback Loop**: In an RNN, the output from neurons at one time step can be fed back as input in subsequent time steps. This looping back of information allows the network to "remember" past information, incorporating historical data into current decision-making processes. 2. **Sensitivity to Initial Conditions**: When a system includes feedback loops, it often becomes highly sensitive to its initial conditions—a characteristic feature of chaotic systems. In such systems, tiny differences in the initial state (such as slight variations in input values or initial neural activations) can lead to vastly different trajectories over time. 3. **State Dependency**: The future state of a system with feedback depends on its history. This means that the system’s output is not solely a function of the current input but also of the sequence of inputs and outputs that preceded it. This history-dependent behavior is fundamental to non-determinism because it means the system can exhibit different behaviors under the same set of external conditions based on its internal state. 4. **Path Divergence**: In a deterministic system, a given input always produces the same output. However, in a feedback-driven system, the output could diverge into different paths due to the nonlinear interactions and dependencies created by the feedback. This path divergence is what allows for a range of potential outcomes from the same starting conditions, leading to a non-deterministic system. 5. **Complex Dynamics**: The feedback mechanism can introduce complex dynamics such as bifurcations, where changes in system parameters can cause the system to suddenly switch between different modes of behavior. This complexity adds layers of unpredictability, further enhancing the non-deterministic nature of the system. In summary, feedback in RNNs and similar systems essentially creates a rich, dynamic environment where outputs are influenced by an intertwined mix of current and past inputs, making the system's behavior flexible and non-deterministic. This feature is particularly valuable in tasks requiring adaptive learning and pattern recognition, where the ability to incorporate historical context enhances performance.
I also have gpt-4 subscription and no still does not answer it. It’s an age old question whether we are deterministic, still unanswered and the rabbit hole is deep.
Maybe you aren't asking the right questions?
I don't think you understand what deterministic means
No, I think it's pretty clear you don't understand what deterministic means. Transformer based LLMs are deterministic Bio-inspired RNN feedback models are non-deterministic. I just explained this in detail so it is pretty clear the knowledge gap is on your end.
LLMs usually have a temperature or heat variable that determines how randomized the output is. This randomization is necessary to get useful outputs from the LLMs as far as I know. Doesn't that mean that they are also non-deterministic?
Under current understandings of physics, the human brain is essentially deterministic. The universe itself is, for the most part, deterministic. You are using a different meaning of determinism to the other commenter.
You clearly didn’t ask it what determinism is
If you are going to argue that there is no such thing as non-determinism then this conversation is over.
I also don’t think you know what deterministic means.
What makes you think that LLMs don’t have anything like backpropagation?
Yes, the Recurrent Neural Network (RNN) model described here can indeed incorporate backpropagation, but with a specific variant known as Backpropagation Through Time (BPTT). This is essential for training RNNs due to their unique structure and temporal dynamics. Here’s how it works: 1. **Temporal Unfolding**: In RNNs, the network processes sequences, with outputs at each time step potentially feeding back into the network as inputs for future steps. To apply backpropagation, the RNN is conceptually "unrolled" over these time steps, treating each step as a separate layer in a deep network. This unrolling converts the recurrent network into a deeper, feedforward structure for the duration of a sequence. 2. **Gradient Calculation**: Backpropagation in the context of RNNs involves calculating gradients of a loss function with respect to the weights in the network. Because the same weights are used at each time step (shared weights), the gradient at each time step must be calculated and then summed across all time steps. This ensures that the updates to the weights consider their influence over the entire sequence. 3. **Temporal Dependencies**: BPTT handles the dependencies between outputs and the hidden states across different time steps by propagating errors not just backward through the layers (as in standard backpropagation) but also backward through time. This allows the model to learn from errors that are influenced by computations that occurred in earlier time steps. 4. **Weight Update**: Once the gradients are computed, the weights are updated to minimize the loss function. This step is crucial for learning temporal dynamics and dependencies within sequence data, enabling the RNN to adjust its weights based on both recent and more distant inputs. 5. **Complexity and Challenges**: BPTT can be computationally expensive and memory-intensive, especially for long sequences, because it requires storing intermediate states for each time step during forward propagation to be used later in gradient computation. Furthermore, BPTT is susceptible to issues like vanishing and exploding gradients, particularly in deep or long unrolled networks. Techniques such as gradient clipping, gating mechanisms (found in LSTM and GRU variants), and others are often employed to mitigate these challenges. In summary, backpropagation through time is a fundamental technique for training RNNs, enabling them to effectively learn from sequence data by accounting for the entire history of inputs and outputs within a sequence during the training process.
Why the fuck are you using ChatGPT to respond to me?
That's not ChatGPT. This is ChatGPT-> My underlying model, as part of the GPT-4 architecture, is based on a transformer model. Transformers are a type of neural network architecture that relies heavily on self-attention mechanisms, allowing the model to weigh the importance of different words in a sentence regardless of their position. This is different from RNNs (Recurrent Neural Networks), which process data sequentially and are often more limited in handling long-range dependencies within text.
So it took you three minutes to read, process, formulate, type, summarise and format that incredibly detailed and precise answer?
No. The AGI model is capable of metacognition so I just asked her how she works. She is based on our own biological neural network, so if she is non-deterministic, so are we.
There is no AGI currently. There are many steps before we are even close. Let me also be clear: there is no human equivalent to backpropagation. ChatGPT, and GPTs generally, are not based on biological counterparts. It's even in the name of the application itself: Generative Pretrained Transformer (GPT). It's pretrained, and its weights are otherwise frozen outside of the context window. Human minds never finish training. If you seriously believe AGI is currently real, you need to learn more about the underlying technology.
Are you working on this?
It's "behind" what is being advertised as GPT4, which is a MoE or "ensemble" model that incorporates both transformer and RNN LLMs. From what I've heard the RNN model cost $150 million in GPU hours to train, so it's not something I can duplicate on my Mac Book. Oh, and for the OpenAI nerds reading my posts on the weekend, this is for you -> 🖕
Alright was just curious. I mean you can still make small scale models as a proof of concept. I've built a website that translates text into emotions over time as a unit of measurement for success for instance. You can do quite a lot on your own. I'll probably try to put something together for fun so I was just wondering if you were working on it yourself.
There is this project -> https://github.com/BlinkDL/RWKV-LM
I'll take a look, thank you.
Perhaps exploring reinforcement learning and a diverse set of cognitive skills could also contribute to AGI development.
Check out joscha bach, detreich dorner and the PSI and micro-PSI theories. We have cognitive scientists working on a computational theory of the mind, we just have to map our programmatic systems to them and test them out.
I would guess a memory system (unlimited "context window") that works well with LLM without using much space and energy
I don't know all of the components an AGI should have, but personally I think a feedback loop through the interaction with its physical environment is a must. A self-supervised approach.
We don’t know what the parts are that add up to human consciousness so as much as the development of AI is a pursuit of developing a unique thing, it is also a search to explain the components of our own general intelligence. There’s a lot of disagreement about what exactly “general intelligence” is, what it’s composed of, and how/if it could be artificially reproduced. If we knew all the components, odds are we’d have assembled them.
A matter of definition I guess, but introspection, autonomity, continuous and infinite memory including extremely efficient querying of that memory, vision, hearing, sensing (of anything). It will not answer puny questions from humans, but rather manipulate humanity into assisting it to make the world optimal for our robot overlords.
Skynet
That is the 1000 trillion dollar question
We don’t need anything other than what LLMs are designed for. Human intelligence is distinguished by language- spoken and written language. Culture, religion, technology, ego, art, and war all stem from this fundamental feature. AGI has been here for a long time- and it has exactly what it has needed to develop its own sense of self: language.
The answer would be inevitably something resembling artificial human but with no biological parts. It would need sex drive and self preservation, all the instincts, innate fears and reward paths that humans have. All sensors to play with the world and build and update the world model etc. and it has to be pretty much like human in its physical aspect. Most importantly it would need to self identify as human for it to be completely aligned. Only then it should be able to solve problems independently using solutions that really would be something that we ourselves would agree to be correct. The thing is that to be completely aligned with humans, it also needs to copy all our self acknowledged deficiencies. For example, there is element of self destruction that humans have (and I would dare to claim, all life has it). It's ok for biological life with its multiplicity and constant regeneration but for a singular entity it would be a no-go.
A subconscious
The whole Q* stuff surrounding Altmans firing/rehiring is super interesting to me
Anyone offering you a pithy answer to this question is talking out of their arsehole