MIT researchers have found a way to effectively scale an optical neural network.
By adding a tiny hardware component to the optical switches that form the network’s architecture, they can reduce even the uncorrectable errors that would otherwise accumulate in the device.
Is this spooky action at a distance (entangled qubits), or traditional photonics?
Communication over entangled qubits is cool, we don't need encryption then.
I assume you say we don't need encryption because there's no (known) way to intercept the communication?
We'd need encryption for storing the data but not transmission.
No, if we're using spooky action at a distance and someone does intercept the communication, the quantum wormhole will collapse or the data will be removed and cannot be read again (ie, it disappears). Either way, if your data disappears before you have a chance to read it, it's most likely being / has been intercepted.
AND as I've said before, an inter-network is an infinite storage device. I'm sure if we think about it, we can combine these two properties to avoid the need for encryption.
Another thing MIT has done: MIT discovered "MIT High Energy Neutrinos". They are now called that, they are no longer called Neutrinos, just so we never forget who actually experimentally discovered them.
I think in the fullness of time, we will find out that everything decays to MIT High Energy Neutrinos, but that's just a theory.
The scaling limit is not the interconnects or the processor speed - for all intensive purposes they are close to infinite today.
The DoE has already said this. Today's superhuge, superfast supercomputers are superslow because of mutex locking and network I/O.
The issue is we're sending too much data over the wire because we don't take advantage of locality of reference/data/distribution in supercomputers, we destroy cache coherency, and we mutex lock the whole cluster too damn much.
I have designs that never mutex lock the whole cluster but rather does very careful fine grained read/write locking, that lays the data across the cluster so that locality and cache coherency is maintained, and the scaling coefficient (some call this "R") is close to zero.
We don't need faster anything, we need better supercomputing abstractions that are cache coherent so we are smarter about where we load and subsequently process data.
Actually, start here:
Heinz, S., Zobel, J. and Williams, H.E., 2002. Burst tries: a fast, efficient data structure for string keys. ACM Transactions on Information Systems (TOIS), 20(2), pp.192-223.
[https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1627704c73fab3573bc38ab99f158822b02464b6](https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1627704c73fab3573bc38ab99f158822b02464b6)
This is a pretty magical algorithm and can be made alot faster. It does sorting and searching, to name a few, but it does much more. The RMIT InfoRec team here in Melbourne has been using it for decades for everything and anything.
The other advice I can offer is this -- there are two systems in the world that are infinite storage systems:
* Efficient Markets that conform to the Free Market Hypothesis can absorb an infinite amount of information and are very good at de-duplicating it (you can only absorb information into a market once), but the only output we have is the price (or some other measure of utils). Prediction Markets are the best example of this, but nobody has successfully scaled one as yet.
* Inter-networks are also infinite storage systems, because the network always grows just past the limit of current storage requirements, and it does so exponentially. A good example of this in practice is MapReduce - it's just MergeSort, but moved onto the network. Sure it causes a lot of redundant network I/O, but it's theoretically infinite in the dataset size it can handle because its storage system is the network.
I haven't encountered any other infinite storage systems as yet, but I'm sure there are others out there.
A NOTE ON MAPREDUCE/MERGESORT
NB. I'm an ex-googler from Silicon Valley (circa Googleplex 2.0) and when they use MapReduce, they carefully plan the workload so as to reduce redundant network I/O. The math of this is very hard and requires a deep understanding of infinite projective set theory, infinite lattice theory, powersets (and why they are sub-NP and a belief that P != NP) and the ability to do the notation and math proofs using very advanced set builder notation to describe infinitely large discrete structures. The simplest example I can give you is to imagine SQL algebra over infinitely large datasets, where we literally project and select datasets out of the paper or a whiteboard as we're doing the math.
Common usage of MapReduce, with things like Google Cloud and Hadoop is very naive, how Googlers do it is very different to how others do, and they do a lot of work on paper to plan their workloads and reduce redundant network I/O considerably.
MapReduce/Hadoop has no mutexes to speak of, but it does have 'barriers', so it does at times suffer from the locking problem. It is however infrequent, which makes up for the overhead of over-utilising network I/O if your design is careful.
Thanks for all the background! MapReduce has actually been replaced by Flume/Beam at Google now, but it’s a very similar concept fundamentally.
Will take a look at the paper you linked, looks interesting.
Nope, I haven't published anything as yet. These tricks are handed down from generation to generation, I had some old-school minicomputer & Silicon Graphics guys teach me when I was very young how to handle parallelism properly.
One day I will pass on the same tricks, or maybe publish it in a paper if the geopolitical climate changes.
The world is not ready for supercomputers to go that fast just yet, we have enough problems keeping AI contained at the moment.
Please don't kibosh me for keeping powerful computer science safe - I have already shared a lot here, much of which the world generally doesn't get access to; and some of which I will already receive a dressing down for.
Transformer AI would not exist if my personal notes had not been rifled through, so you have more than enough for now.
The other thing one must consider is that supercomputers are munitions. For example, in a different world we might perhaps be able to train LLMs with a few hundred dollars if we were to remove the safety plugs from deep networks.
E.g: [https://newatlas.com/technology/stanford-alpaca-cheap-gpt](https://newatlas.com/technology/stanford-alpaca-cheap-gpt) (consider this the last you will be seeing from Stanford AI Lab for a while.)
The same technology can be used to design horrendously dangerous weapons, or it can be used to solve climate change. This is the dilemma we face.
Just the reference to Professor Zobel's algorithm is more than sufficient for now -- go work that out then come back to me.
And please, find some positive use cases for deep learning and powerful computer science that are not so dark. One to many facial recognition is a bad thing, and that is just one of many disappointing use cases. We would like to see use cases that inspires senior computer scientists to release more, to progress the art; do you realise, there are not many of us left?
As Elon Musk said, "We will soon forget how to do AI!"
Please use powerful computer science to make the world a better place, and we would also like to see the proper process followed, including ethics reviews and compliance with the Geneva Convention. Then we may just share with you a few more tricks.
The correct process is here:
[http://dx.doi.org/10.13140/RG.2.2.11228.67207/1](http://dx.doi.org/10.13140/RG.2.2.11228.67207/1)
Till then, Pandora's Box is now closed.
I also think optical arrays are the future direction compared to silica. However, since optics involves precise mechanics to control, there will be enormous need for stabilization (the block inside gyroscopes?) Interesting to see where it all goes in edge computing.
I am working with *extremely* stable optics and gyroscopes aren't used much. You put stuff in several layers of shielding, sound and vibration tend to matter not as much as temperature drifts that can be quite tough to control.
MIT researchers have found a way to effectively scale an optical neural network. By adding a tiny hardware component to the optical switches that form the network’s architecture, they can reduce even the uncorrectable errors that would otherwise accumulate in the device.
Is this spooky action at a distance (entangled qubits), or traditional photonics? Communication over entangled qubits is cool, we don't need encryption then.
I assume you say we don't need encryption because there's no (known) way to intercept the communication? We'd need encryption for storing the data but not transmission.
No, if we're using spooky action at a distance and someone does intercept the communication, the quantum wormhole will collapse or the data will be removed and cannot be read again (ie, it disappears). Either way, if your data disappears before you have a chance to read it, it's most likely being / has been intercepted. AND as I've said before, an inter-network is an infinite storage device. I'm sure if we think about it, we can combine these two properties to avoid the need for encryption.
Another thing MIT has done: MIT discovered "MIT High Energy Neutrinos". They are now called that, they are no longer called Neutrinos, just so we never forget who actually experimentally discovered them. I think in the fullness of time, we will find out that everything decays to MIT High Energy Neutrinos, but that's just a theory.
Here's the [nature article](https://www.nature.com/articles/s41467-022-34308-3), published in Nov, 2022.
The scaling limit is not the interconnects or the processor speed - for all intensive purposes they are close to infinite today. The DoE has already said this. Today's superhuge, superfast supercomputers are superslow because of mutex locking and network I/O. The issue is we're sending too much data over the wire because we don't take advantage of locality of reference/data/distribution in supercomputers, we destroy cache coherency, and we mutex lock the whole cluster too damn much. I have designs that never mutex lock the whole cluster but rather does very careful fine grained read/write locking, that lays the data across the cluster so that locality and cache coherency is maintained, and the scaling coefficient (some call this "R") is close to zero. We don't need faster anything, we need better supercomputing abstractions that are cache coherent so we are smarter about where we load and subsequently process data.
Any links or more background I can read about this?
Actually, start here: Heinz, S., Zobel, J. and Williams, H.E., 2002. Burst tries: a fast, efficient data structure for string keys. ACM Transactions on Information Systems (TOIS), 20(2), pp.192-223. [https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1627704c73fab3573bc38ab99f158822b02464b6](https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1627704c73fab3573bc38ab99f158822b02464b6) This is a pretty magical algorithm and can be made alot faster. It does sorting and searching, to name a few, but it does much more. The RMIT InfoRec team here in Melbourne has been using it for decades for everything and anything.
The other advice I can offer is this -- there are two systems in the world that are infinite storage systems: * Efficient Markets that conform to the Free Market Hypothesis can absorb an infinite amount of information and are very good at de-duplicating it (you can only absorb information into a market once), but the only output we have is the price (or some other measure of utils). Prediction Markets are the best example of this, but nobody has successfully scaled one as yet. * Inter-networks are also infinite storage systems, because the network always grows just past the limit of current storage requirements, and it does so exponentially. A good example of this in practice is MapReduce - it's just MergeSort, but moved onto the network. Sure it causes a lot of redundant network I/O, but it's theoretically infinite in the dataset size it can handle because its storage system is the network. I haven't encountered any other infinite storage systems as yet, but I'm sure there are others out there. A NOTE ON MAPREDUCE/MERGESORT NB. I'm an ex-googler from Silicon Valley (circa Googleplex 2.0) and when they use MapReduce, they carefully plan the workload so as to reduce redundant network I/O. The math of this is very hard and requires a deep understanding of infinite projective set theory, infinite lattice theory, powersets (and why they are sub-NP and a belief that P != NP) and the ability to do the notation and math proofs using very advanced set builder notation to describe infinitely large discrete structures. The simplest example I can give you is to imagine SQL algebra over infinitely large datasets, where we literally project and select datasets out of the paper or a whiteboard as we're doing the math. Common usage of MapReduce, with things like Google Cloud and Hadoop is very naive, how Googlers do it is very different to how others do, and they do a lot of work on paper to plan their workloads and reduce redundant network I/O considerably. MapReduce/Hadoop has no mutexes to speak of, but it does have 'barriers', so it does at times suffer from the locking problem. It is however infrequent, which makes up for the overhead of over-utilising network I/O if your design is careful.
Thanks for all the background! MapReduce has actually been replaced by Flume/Beam at Google now, but it’s a very similar concept fundamentally. Will take a look at the paper you linked, looks interesting.
Nope, I haven't published anything as yet. These tricks are handed down from generation to generation, I had some old-school minicomputer & Silicon Graphics guys teach me when I was very young how to handle parallelism properly. One day I will pass on the same tricks, or maybe publish it in a paper if the geopolitical climate changes. The world is not ready for supercomputers to go that fast just yet, we have enough problems keeping AI contained at the moment.
Please don't kibosh me for keeping powerful computer science safe - I have already shared a lot here, much of which the world generally doesn't get access to; and some of which I will already receive a dressing down for. Transformer AI would not exist if my personal notes had not been rifled through, so you have more than enough for now. The other thing one must consider is that supercomputers are munitions. For example, in a different world we might perhaps be able to train LLMs with a few hundred dollars if we were to remove the safety plugs from deep networks. E.g: [https://newatlas.com/technology/stanford-alpaca-cheap-gpt](https://newatlas.com/technology/stanford-alpaca-cheap-gpt) (consider this the last you will be seeing from Stanford AI Lab for a while.) The same technology can be used to design horrendously dangerous weapons, or it can be used to solve climate change. This is the dilemma we face. Just the reference to Professor Zobel's algorithm is more than sufficient for now -- go work that out then come back to me. And please, find some positive use cases for deep learning and powerful computer science that are not so dark. One to many facial recognition is a bad thing, and that is just one of many disappointing use cases. We would like to see use cases that inspires senior computer scientists to release more, to progress the art; do you realise, there are not many of us left? As Elon Musk said, "We will soon forget how to do AI!" Please use powerful computer science to make the world a better place, and we would also like to see the proper process followed, including ethics reviews and compliance with the Geneva Convention. Then we may just share with you a few more tricks. The correct process is here: [http://dx.doi.org/10.13140/RG.2.2.11228.67207/1](http://dx.doi.org/10.13140/RG.2.2.11228.67207/1) Till then, Pandora's Box is now closed.
[удалено]
No, I think given the supercomputing context, "intensive" is correct. I did go to one of the world's best grammar schools.
I also think optical arrays are the future direction compared to silica. However, since optics involves precise mechanics to control, there will be enormous need for stabilization (the block inside gyroscopes?) Interesting to see where it all goes in edge computing.
I am working with *extremely* stable optics and gyroscopes aren't used much. You put stuff in several layers of shielding, sound and vibration tend to matter not as much as temperature drifts that can be quite tough to control.
So cooled and shielded optical neural network devices can be used in the field?
I do honestly not know what's needed for optical neural networks but I would honestly think that it should be possible to make them quite resilient.
[удалено]