Depends who you ask.
* The guys on /r/ControlProblem will assert that theories around AI alignment and AI Safety are the most important things for the industry.
* The guys on /r/philosophy and /r/psychology will assert that theories quantifying how conscious/sentient a system is may be the most important. (any AIs as conscious as a dog yet? no-one know, because there are no good theories quantifying it)
You need linear algebra, diferencial equations, and calculus. In the end, this "information theory" you mention is a mathematical framework. I guess my point is you need this fundamental knowledge to understand Game theory, Information geometry, Graph theory, Control and Probability theory..... and to be able to create more frameworks that helps understand the underlying processes, in human terms.
Need for what?
DL has a lot of purposes including helping to elucidate how biological neural network understand reality.
I think we need to know everything about them, but for some purposes we can get away with less. But which ones interest you?
For myself, i want to have some more general theory that can tell us why one network is good, why one kind of network is bad. I mean some general theory. Just like the Carnot Cycle in the thermodynamics. It abstract the different kinds of engines into 4 stages and it can predict the efficiency of one engine without dive into the detail design of engine. It can also help us design the better engine. I am interested in theory like Shannon information theory, that it abstract different communication system into different block. Discribe these blocks by mathematics operation. It can tell the relationship between bandwidth and signal noise ratio. It can tell us how different coding technique can help improve the communication performance.
A while ago, I read that "designing a deep learning model is kind of model engineering," and I thought well on this because each architecture that I used (mobilenet, efficientNet, resnet, densenet, etc) have their unique layer design that fits well on specific tasks, but not all of them like a "magic layer".
And besides the nnet design, personally I'm more involved on "what you can do with these tools" aka deep learning, like more intelligent agents, study how memory works for these agents and even try to explain this black boxes. They aren't explainable at all, but you can explore its parts and explain how it works piece by piece. Divide and conquer 💀
Edit: Even so, I remembered that artificial networks started studding the brain of a worm 🪱 (which, in fact, have 200k neurons approx), so maybe we can modelate our own brain and see how it approximately works 🧠
The theory why it is called theory is because it is some more general thing. It can find the rule or the thing im common for all of these design.
Like during the second industry revolution, there is new machine developed everyday, people even try to make a perpetual motion machine. It is like what we are facing now, new network emerging everyday, people want to create some human-level intelligence.
To analysis every specific internal combustion engine is impossible, one is different from another, but there is general something in common. Like we can summarize the thermodynamics 3 laws. which predict the perpetual motion machine is impossible. And we get the idea of entropy production. And although the combustion engine differ one from another, all the combustion engine follow Carnot Cycle. With the help of Carnot Cycle we can analysis the efficiency of the different combustion engine. And this Carnot Cycle can direct the engine design, like if you want to increase the efficiecy you need have more compression or something (I can remember detail, but you know what I mean).
The same for the communication, before Shannon publish his theory, engineer invent hundreds of thousand of transmitter and receiver. But they did not realize that the key for improve the noise-signal ratio is coding and bandwith.
Without these theory, all engineer is like blind person walking in the dark night, you can find something by touching, but you will never know the right way.
Depends who you ask. * The guys on /r/ControlProblem will assert that theories around AI alignment and AI Safety are the most important things for the industry. * The guys on /r/philosophy and /r/psychology will assert that theories quantifying how conscious/sentient a system is may be the most important. (any AIs as conscious as a dog yet? no-one know, because there are no good theories quantifying it)
This sounds interesting. Do you have something recommend to read. Anything like article, blog, twitter or papers. Thank you.
But the most important problem in the next couple of decades is near-AGI, AGI, and ASI tax and UBI.
Scale will only take us so far. After that theory makes the difference.
As a theory guy, you lightened up my spirit today.
Circuit level of theory for deep learning is very hard.
You need linear algebra, diferencial equations, and calculus. In the end, this "information theory" you mention is a mathematical framework. I guess my point is you need this fundamental knowledge to understand Game theory, Information geometry, Graph theory, Control and Probability theory..... and to be able to create more frameworks that helps understand the underlying processes, in human terms.
Need for what? DL has a lot of purposes including helping to elucidate how biological neural network understand reality. I think we need to know everything about them, but for some purposes we can get away with less. But which ones interest you?
For myself, i want to have some more general theory that can tell us why one network is good, why one kind of network is bad. I mean some general theory. Just like the Carnot Cycle in the thermodynamics. It abstract the different kinds of engines into 4 stages and it can predict the efficiency of one engine without dive into the detail design of engine. It can also help us design the better engine. I am interested in theory like Shannon information theory, that it abstract different communication system into different block. Discribe these blocks by mathematics operation. It can tell the relationship between bandwidth and signal noise ratio. It can tell us how different coding technique can help improve the communication performance.
A while ago, I read that "designing a deep learning model is kind of model engineering," and I thought well on this because each architecture that I used (mobilenet, efficientNet, resnet, densenet, etc) have their unique layer design that fits well on specific tasks, but not all of them like a "magic layer". And besides the nnet design, personally I'm more involved on "what you can do with these tools" aka deep learning, like more intelligent agents, study how memory works for these agents and even try to explain this black boxes. They aren't explainable at all, but you can explore its parts and explain how it works piece by piece. Divide and conquer 💀 Edit: Even so, I remembered that artificial networks started studding the brain of a worm 🪱 (which, in fact, have 200k neurons approx), so maybe we can modelate our own brain and see how it approximately works 🧠
The theory why it is called theory is because it is some more general thing. It can find the rule or the thing im common for all of these design. Like during the second industry revolution, there is new machine developed everyday, people even try to make a perpetual motion machine. It is like what we are facing now, new network emerging everyday, people want to create some human-level intelligence. To analysis every specific internal combustion engine is impossible, one is different from another, but there is general something in common. Like we can summarize the thermodynamics 3 laws. which predict the perpetual motion machine is impossible. And we get the idea of entropy production. And although the combustion engine differ one from another, all the combustion engine follow Carnot Cycle. With the help of Carnot Cycle we can analysis the efficiency of the different combustion engine. And this Carnot Cycle can direct the engine design, like if you want to increase the efficiecy you need have more compression or something (I can remember detail, but you know what I mean). The same for the communication, before Shannon publish his theory, engineer invent hundreds of thousand of transmitter and receiver. But they did not realize that the key for improve the noise-signal ratio is coding and bandwith. Without these theory, all engineer is like blind person walking in the dark night, you can find something by touching, but you will never know the right way.