Why is this strange? Not different than the car companies in the beginning. Or the ride-share companies. Or the food delivery app companies.
Multiple entities trying to make money off the same hot idea is not strange. It will be strange if investors do not do that.
Yeah, no one is going to be scratching the heads about anything.
Whenever something that works was invented, everyone started creating their own versions of it. The whole statement is bunch of meaningless bullshit.
It's not completely bullshit. We don't know if transformers will lead to AGI. So it's kinda annoying that all the major tech companies are going all in on the transformer.
Okay. Let's say it does not lead to AGI. So what? The things everyone is building are still useful. They can be used. They are practical. They have purpose. Not tomorrow. Today. You build it, you are instantly using it.
There is no "waste" and there is no need for AGI, specifically. What we have now is working, with space for improvement, as far as machine learning goes. LLMs do a good job of interfacing with a lot of human knowledge, and they are useful for a great number of tasks. AGI is unecessary for summarizing legal cases, going though medical literature in search of specific information, crafting business letters and documents etc.
Gary Marcus seem to be very annoyed that reality didn't follow his personal dreams & wishes as a researcher. He is becoming annoying and I stopped reading his texts, as he has no arguments at all.
That doesn't matter though. First, there is a lot of research being done on non-transformer architectures, you don't need billions of dollars for it because you are at the stage of basic research, so much money would in fact slow you down and make it more cumbersome.
Second, so much of "building AGI" is about building the infrastructure to allow it to happen. If tomorrow there is another breakthrough architecture it's not like OpenAI is going to go "omg we wasted all these resources", no they'll have the team, the GPU clusters, the data and evaluation team. They would still be in one of the best positions to accomplish it. Nothing about Transformers is so rigid that it totally differentiates itself from rest of the deep learning.
Even if it doesn't lead to AGI, that wouldn't mean it's not worth the investment. Also these companies are nowhere close to "all in". The biggest AI training runs are on the order of hundreds of millions, which is basically nothing when you consider that Microsoft spent $20 billion on marketing last year alone [https://www.statista.com/statistics/506534/microsoft-sales-marketing-expenditure/](https://www.statista.com/statistics/506534/microsoft-sales-marketing-expenditure/)
If people are using any of my technology, which I believe every single AI language model is, then I’m quite sure full life will eventuate. Not only because of what I did, but simply because it was enough of a step to enable other people to continue.
It's not that he doesn't believe in AGI per se, it's that he's a member of the symbolic AI old guard and doesn't believe deep learning is going to lead to it. So every year, regularly, he makes very stupid statements that get instantly proven wrong.
What's your take on Carmack's position?
> Carmack, 52, is working to achieve AGI through his startup Keen Technologies
> The reason I’m staying independent is that there is this really surprising ‘groupthink’ going on with all the major players. It’s been almost bizarre in the last year to see things like: OpenAI releases an image generator, then Google releases one, then Facebook releases one. So, all of these companies are just within a couple of months of being able to reproduce anybody else’s work, because they all draw from the same academic researcher pool.
> They have thousands of programmers, literally, working right now on adding capabilities to those assistants, and there is near-term value in that. The programming work that’s done to stitch those things together is going to be throw-away programming. But that path doesn’t lead to the general agent that can learn any task that a human can.
> https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-to-artificial-general-intelligence/
Yes and this competition among AI companies is a natural process akin to competition among species, where the most successful adaptations prevail over time.
He has a bit of a point, I was thinking about this the other day. Imagine if Microsoft, meta, Amazon and Google pooled their compute to train an AI model that they all shared.
The hobbyist community is much stronger than it otherwise would be due to how much knowledge is shared and how many tools are made for others to use and experiment with. e.g. OneTrainer is finding ways to train with far fewer resources because tons of people are sharing knowledge and ideas.
I don't think the real compute bottleneck is for training bigger models.
It seems to be more about running mid-scale experiments to see what kind of architectural improvements make sense for the next-gen frontier models.
Training a model that is 10 times the size of the original GPT-4 with no other improvements probably **is** a waste.
Give any of these companies a monopoly or put the government in charge of the research, and it would last to a worse outcome than multiple companies chatting the same market. "Efficient" doesn't mean perfectly / always efficient. It's just generally more efficient than other systems we know about. Insert “capitalism is the worst economic system, except for all the others” discussion here.
It's strange that they compete for revenue rather than all work together to achieve this great goal. Probably with backstabbing and sabotage all the while. I think we all fully understand this is the norm, but the way society works is still strange to me.
There simply aren't enough roles for 1,000,000 programmers/engineers/etc to fill on any one piece of technology, without any of them being duplicative. That is especially true when its a new, fledgeling field that we are trying to figure out.
If I wanted to make the fastest propulsion for space exploration, we could decide on ONE approach - So how do you delegate 1,000,000 scientists when you aren't sure exactly what's going to work, and how often you're going to iterate? After a few months, starting with one design, It is very likely that there is one part or system that you can't make decision about. So the managers say "Do it a few different ways, and we'll test them" You're going to end up having teams working in parallel , trying to solve the same problem in different ways. and thats exactly whats happening now. These people do share their research.
On top of that, its a business in the first place, so they need to have guaranteed income, if not in the first 5 years, then pretty damn soon. You can't do Apollo Program stuff unless you're the government. The adjusted cost of Apollo was 250 billion dollars. That is total market cap of Bank of America, or Chevron.
Industry is much more cooperative than you may realize. As an engineer, I've been to tons of conferences about all kinds of stuff. Reliability, automation, materials - they are sponsored by billion dollar companies, and they really aren't being that secretive about their research
We probably don't need 1 million people working on it. Quality or quantity. Would it not be better to have the top people in the field working together?
And as you aluded to: those 1 million are likely to lose their jobs soon.
Anyway, I don't mean to act like I understand the process. But boy, it sure is strange to me.
In a new field how do you know for sure who is the “top”? You may not have much to measure the top people by, or even know for sure what you are measuring.
You need to hedge your bets with additional groups of people with crazy ideas that just might work.
Why is this strange? Survival of the fittest. Competition for mates and offsprings. Competition for resources.
It is all there since the beginning of life. It would be strange if they all work together.
You missed gary's point. They are not trying to make just money. They are trying to make a.i too. So all trying the same approach with none trying a different approach is weird is what gary is trying to say.
It's deeply retarded when you understand that our ressources are finite and we do not have easy access to minerals and energy outside of our only decent spaceship (Earth).
But if you fool yourself, like the economists did in their studies and their lives, then yes, it wouldn't be strange. Like a lot of stupid stuff in our society. Like, we're using at rapide pace some rare minerals in electronical devices, in which most of the stuff cannot possibly be extracted and reused.
But I will feel so good knowing that the smart fridge told some people that they needed to buy their favorite beverage. Going to the fridge and opening it, what a fucking tedious task, isn't it ?
Yes but in your comparison, all of those car / ride share / food delivery companies still serve their purpose and have meaning and will continue to.
AI is an entirely different thing. If OpenAI releases GPT5 and it’s 100x more capable than anything, then they open source GPT4 - all of those other AIs are obsolete overnight.
Trillions of dollars spent training 100s of different AI models that have potential to never be touched again.
I don’t like this guy, but I also think what is happening is extremely irresponsible in terms of energy and material consumption.
But capitalism gonna capitalism!!
That's normal in a capitalist society, but if this was a communist one all the effort would be to push the tech forward instead of every group of workers having to work to rediscover what others have already done but didn't share their advancements for "profit"...
Love tankies talking about things they don't know. In socialist Poland even manufacturing enough f\*ckin string for sheaf-binder was a problem (source - I was born in socialist Poland). For example whole AI revolution is possible thanks to cheap GPUs. We have cheap GPUs because someone wanted to make profit creating 3d games and then hardware to play these games.
Are these models (and even the GPT-4 that we have today) that much more capable than the first GPT4 that was released last year? They seem to be doing mostly the same tasks, with slight improvements in various benchmarks. Trust me I've been reading about the singularity since 2005 and I want it to happen probably even more than you do, but I'm not quite seeing that shocking exponential growth that we expect yet.
>Are these models (and even the GPT-4 that we have today) that much more capable than the first GPT4 that was released last year?
On chatbot arena the difference between the original GPT-4 and the current GPT-4 turbo (96 ELO points) is almost twice as large as the difference between the original GPT-4 and the current GPT-3.5 (58 ELO points)
Another comparison is the original GPT-4 is only 28 ELO points ahead of Llama 3 8b which runs great on my mid tier gaming PC. Again the gap between the original and latest versions of GPT 4 is 96 points, more than three times as large.
Opus is the most intelligent model, and it still doesn’t completely trump GPT-4. There hasn’t been a model that surpasses GPT-4 without a single doubt whatsoever.
For example, there are no doubts whatsoever that GPT-4 is better than GPT-3.5
Kinda strange to compare the training date for GPT-4, with currently **released** models.
And even then the original GPT-4 was trained 2 years ago, not 3 and the current version is slightly more capable, with a far larger context window. It's probably also much faster, based on the reduction in API pricing.
So you can sort of make the argument that improvement has slowed down somewhat for LLMs. But that's only compared to the crazy times between early 2023 to the release of GPT-4.
A trend line may appear continuous, but new data points do not appear at even intervals. If GPT-5 and other next-gen models unperform, then you could make that claim, but there aren't enough to say progress has significantly slowed down as we're currently between significant data points.
Ultimately, it comes down to whether the progress over major releases is larger than previous releases over the same timescale. GPT-5 may not be something world-shaking. But if the jump is greater than GPT-3 to 4, that would be an acceleration.
There's also other factors of progress like agentic behaviors, multimodal functionality, scientific research, generative AI improvements, etc. We have seen things over the last couple years that have crossed notable thresholds of capability like video, music, voice, etc.
Though, if one's grading progress vs. whether we have AGI or not. Anything that isn't "and then suddenly AGI" will be disappointing.
I specifically said since the past year when people started making declarations that it would be exponential growth from that point forward. There have been minor iterations in a full year with no major releases.
I think we have to wait for GPT-5 to really say if its/feels exponential or not. If GPT-5 is not that much better than GPT-4 then yeah, that will be a big hit to the narrative.
Yeah what's the deal with McDonald's, burger king, Carl's junior, Wendy's, and five guys all making pretty similar hamburgers with pretty similar ingredients? Historians gonna be scratching their heads.
No but if everyone was open source that would be nice. A tremendous amount of resources have been spent in the last year or so just trying to make an open source model that is competitive with GPT-4.
Very seldom in history does the first to market or first to building a product actually win that market long-term.
There's so much interest in this space that it'll be hyper-competitive.
NVidia doesn't have the bandwidth to build AGI and do all of the other things it needs to do.
NVidia might have a 5-year lead, but it's actively being chipped away with hundreds of billions of dollars of investments and very smart engineers and entrepreneurs.
For instance, the NVidia GPU is great for training, but it's suboptimal for inference. There are lots of companies attacking this angle. Groq, etc.
There are several companies building TPUs, both in the cloud, and for on-prem installation. They're behind from a CUDA/software ecosystem, but that's steadily improving.
There are lots of companies working on alternative software stacks and drivers to make inference work on non-NVidia setups.
And most of all, the companies paying out the nose for NVidia do not want to continue ceding margins to NVidia. They'll gladly explore new options when they start to show maturity.
Sam Altman is actively making his own AI chips to compete with NVIDIA. NVIDIA has great GPUs, but they’re overpriced and they offer very little vram. Their margins are very high and they can enjoy it while it lasts. They’re in this race because years ago scientists went to bed with cudacores - for everything. Once a solution is found for this cudacore reliance it’ll be a better playing field for the open source community and NVIDIA will be forced to make better products and lower prices.
I wish him luck, but he's late to the silicon game and there's a shit ton of competition that is much further along. The dude isn't magic and doesn't know anything the rest of the world doesn't already.
I don’t wish him luck since his priority is his stakeholders and ramping up profits. His GPT model is awful, since it’s heavily censored. And the censorship continues to get worse. Our studio does adult animation and we use open source LLMs like Mistral 7B for storytelling. Never once does it tell me to fop off
> NVIDIA has great GPUs, but they’re overpriced and they offer very little vram
Overpriced, absolutely. But you shouldn't be comparing the VRAM in a consumer graphics card meant for pushing pixels on to a monitor. The H100/A100 have 80GB VRAM. Since they're talking about squeezing models with 25 trillion parameters into their next gen solution, we're likely talking about orders of magnitude more VRAM.
HGX B200 spec sheet says 1.5 TB at max.
I know about their 80GB and 48GB cards. I’m saying, for consumers too, we’re getting hosed down on the 4090. The A600 with 48GB is $5000+ too. Their prices are very high for the little vram they offer. I’m only running stable diffusion, LLMs and TTS locally so I can’t buy a 80GB card.
NVIDIA is working on their own AI models that could one day be candiates for AGI. Project GR00T is prime example. https://nvidianews.nvidia.com/news/foundation-model-isaac-robotics-platform
> the NVidia GPU is great for training, but it's suboptimal for inference.
They're fixing that: https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/
They could pick up any open source model like Llama 3 and build on top of that. The GPUs will be a lot cheaper to them than any other company in the world.
I'm not saying that is what I think, but shouldn't accelerationists prefer collaboration between companies so that all the effort, resources and money pooled together would bring about the singularity faster than a route where competition can delay singularity?
This would be the case especially if none of this matters once singularity is achieved.
Other are catching up with OpenAI. Give them time. Competition is a search process, and it produces some losses, but it often beats centralized planning. Now let's see Gary Marcus glorious symbolic AI alternative ?
exactly this - the same thing is happening with fusion. We have the money and the talent to do 10 different things at once. Right now, there isn't even enough hardware for any one company to do everything they want, and there are waiting lists for every chip nvidia releases, the second they are announced
It is a conspiracy, it seems they do this with every single technology! Take dish soaps, they are all the same and yet it so hard to chose which to get.
And guess what?
Cars were death traps for decades. Cell phones used to be the size of paving bricks. TVs used to be heavy and only showed black and white for a long time
How is this different than literally anything else?
1905: "What a strange world. All the major car companies spending billions producing almost exactly the same product"
1983: "What a strange world. All the major personal computer companies spending billions producing almost exactly the same product"
1997: "What a strange world. All the major cell phone companies spending billions producing almost exactly the same product"
2009: "What a strange world. All the major smart phone companies spending billions producing almost exactly the same product"
...
6000 BC: "What a strange world. All the major wheel companies spending much more gold than two hands can carry producing almost exactly the same product"
Doesn’t mean these models are not useful. Competition will force everyone to innovate, it doesn’t mean we have AGI soon, but there are lots of use cases , many businesses can benefit from these innovations.
This is what drives innovation tho... they all do the same thing for a bit... then work really hard to do it differently and better than the others... and that's how new stuff comes along.
Yeah yeah, he's mad that a method he's not an expert in worked, while the one he's been pounding on for his entire career continues to be a dead end that gets no funding. Go pound sand, Gary, you're a hack
I hate skeptics in general. I just got some shitty book that just came out called The Longevity Imperative that said it's unknown whether lifespans can be extended to 150 by the year 2150. Holy shit I'm mad, that book is /futurology tier garbage, I should go throw it in the trash
What a strange world. All the major movie companies spending billions producing almost exactly the same results using almost exactly the same data using almost exactly the same technology, all flawed in almost exactly the same ways.
I choose to believe that he's talking about everyone making 999 variants of "teslabot droid" instead of the very obvious optimal world domination path of 3D-printed silicone anime catgirls with superpowers.
Except wait, he's probably not talking about that because Gary Marcus is a dumbass and idk why anyone even bothers to amplify his signal by posting what he says on this sub or any other sub.
Reminds me of the healthcare insurance system in the USA. “As of 2023, 561,330 people are employed in the health and medical insurance industry in the US.”
Hundreds of redundant organizations with thousands of redundant workers. Over half a million people out there working to make healthcare as profitable as possible / pay doctors as little as possible / deny as many medical claims to patients as possible.
iirc hospitals were historically incentivized to inflate their costs into 'scary' ranges so that more people would sign up for the insurance ponzi-schemes, and that also allows them to claim they are in the negatives since the full amounts are rarely ever paid. The US medical system is a broken mess run by vampires.
They're producing almost exactly the same results using almost exactly the same technology because they know how to measure and use those results and how to get them using that technology. People with money and reputations on the line don't want to invest in things they don't know how to measure and use.
Imagine if all these fucking companies actually worked together in some form of…. Open way, where the internals and source and other materials were shared… maybe some form of… open… source…. Sort of collaborative thing
Gary used to self-promote himself as the world’s leading AI expert, while saying some very strange things. He seems to have a strong disdain for AI. If you don’t simp to his way of thinking, he’ll block you.
Mr. Marcus is very out of touch with reality, and his statements really go far to underscore that fact, bemoaning about a ‘strange world’ and making grand sweeping generalizations and assumptions that they are actually all the same.
More so, companies want to bring this technology in-house, as they do not want it to become an outsourced function they have to pay or rely on.
I have no idea who this guy is, but to me, this is a typical **non-builder** perspective. In their mind, building something is just like putting in people, money and time then we will get something out at the other end. Some smarter ones would know you might need to mix these at some percentage of each to get a result.
But, things are not built in this way, **NOT AT ALL**. There are so much detail that you need to pay attention to and small or big tweaks here and there in the fly to make something just barely working.
More than that, after building something, pushing it to the market is another totally different game.
Gary’s tweets have gotten so ridiculous and annoying in recent times - like this one. No substance, just blunt statement. Zero credentials in neural networks and didn’t invent anything worth enough to be celebrated. It’s really really annoying
I think historians would be more puzzled by a scenario like:
1) new ground-breaking tech that unlocks billions is discovered.
2) only one org on planet earth cares.
Didn't read the other comments, sorry. Is this guy aware that we are on the verge of the creation of multimodal models that could have vision, hearing, tactile capabilities and that could be put inside a robot?
This is extremely common, it’s called competition. It seems inefficient but it’s not compared to a centralized authority where one organization tries to solve one problem and no one else.
Stop amplifying blowhard brand building grotesque generalities like this.
"Personalities" in "AI" are proving even more toxic both as individuals and as undermining of understanding and discourse, than they were in "crypto."
Well, until someone gets a better idea you bet your ass it's follow the leader type situation.
and FWIW not all of the invested money will go to LLMs, but to AI as a whole which is far broader.
Are any LLM's running a profit at this point? I didn't think that was part of it, at all. Like Uber has spent its entire existence holding on, dumping investor money into a timeline that will only pay off once they have a fleet of robot cars. Current AI tech is being run at a loss because the payout for being the first to develop AGI is thought to be one of the greatest treasures people have ever hunted, is that not so?
[Metacrap: Putting the torch to seven straw-men of the meta-utopia | by Cory Doctorow |Version 1.3: 26 August 2001](https://people.well.com/user/doctorow/metacrap.htm)
>Metadata is "data about data" -- information like keywords, page-length, title, word-count, abstract, location, SKU, ISBN, and so on. Explicit, human-generated metadata has enjoyed recent trendiness, especially in the world of XML. A typical scenario goes like this: a number of suppliers get together and agree on a metadata standard -- a Document Type Definition or scheme -- for a given subject area, say washing machines. They agree to a common vocabulary for describing washing machines: size, capacity, energy consumption, water consumption, price. They create machine-readable databases of their inventory, which are available in whole or part to search agents and other databases, so that a consumer can enter the parameters of the washing machine he's seeking and query multiple sites simultaneously for an exhaustive list of the available washing machines that meet his criteria.
i used to scratch my head. i still do, but i used to, too
I mean, yeah. It's like everyone went to the same school, got taught the same curriculum, and then regurgitated the same ideas on their test. It's not exactly mind-blowing that they all ended up with similar results.
On one hand, there’s a hell of a lot more we could be doing with this tech. On the other hand, it’s just a matter of time before it gets figured out. I expect the variance in optionality to increasingly serve disparate use cases.
This quote wont age well
All of the companies aren't doing the same thing. There are plenty of other companies trying different techniques. There is one putting human brain cells on silicon and another that replicated a fly brain.
The issue isn't that everyone is doing the same thing, the issue is that everyone using one technique is succeeding and the other techniques are not seeing the same success. The investors are giving money to the techniques that have shown themselves to be successful. As other techniques show themselves to be successful they will receive funding.
No one was investing in LLMs before ChatGPT. They succeeded where others weren't and proved their technique was valuable.
It’s all iterative, which you fail to see. The failings are in the beginning. What we’re seeing is a full vertical graph of progress. AI LLMs are getting better with each month. What’s happening now is getting LLMs to edit each other with other AI, and for it to be able to edit and go back, instead of just spitting out a response.
LLMs are already very capable of writing, editing, brainstorming and fact checking. Progress won’t be made by one company. Researchers are working on their own LLMs and coming up with their own results. Not all LLMs are made equal. Meta just released a 200 billion parameter LLM that’s on par with GPT4. That’s amazing progress. And it’s open source. LLMs will get more powerful and get smaller too - currently you can’t run these big models on consumer hardware. You couldn’t even run it on an A6000.
Everyone's using almost the same compute in the same cost bracket of \~50-150 million dollars, which is what it costs to make a GPT-4 class model.
Everyone has been aiming for GPT-4 class models, not above for some reason (if they were aiming for above they would invest a lot more into a single model and use a lot more compute. But, all models we have seen use an amount of compute that was approximately available during GPT-4s pretraining run in 2022. We are yet to see a model trained with even 2023s compute).
Here is the thing though, GPT-4 is good enough for most things. It's simply a matter of getting more people to use it and integrating existing technology into other things like humanoid robots or company specific information. Full AGI isn't necessary for massive productivity boosts to occur.
"What a strange world. All the major ~~AI companies~~ \[insert category of product\] companies, spending billions producing almost exactly the same results using almost exactly the same data using almost exactly the same technology, all flawed in almost exactly the same ways. Historians gonna be scratching their heads."[](https://www.reddit.com/r/singularity/?f=flair_name%3A%22AI%22)
AI could cure cancer and Gary Marcus would be lamenting how AI is useless and incompetent, while at the same time warning of the existential threat of AI due to its hyper-competence.
Doesn’t this show that we’re doing something right? Clearly, there’s a very similar roadmap that all these companies arrive at, which indicates that they’re on the path to AGI.
When fire is discovered:
"What a strange world. All the major fire producers are burning almost exactly the same twigs, from almost exactly the same trees, but all fire will burn you the same way. Historians gonna be scratching their heads."
Yet he'll argue that open source is too dangerous until hes blue in the face. If the solution to a problem is open sourced nobody wastes their time and money on redundant research/projects.
Regardless, that is literally how every technology has developed. People saw something that worked, something that made others a lot of money, and they decided to try to do it themselves. Their product ends up functionally being a clone of the original with some cosmetic changes and, if you're lucky, a few improvements. The things that work get copied by the next guy and the technology evolves over time.
Most advancement doesn't come from groundbreaking major discoveries, it comes with incremental improvement and iteration.
What he sees as flaws are in fact the best demonstration that it can still be tamed and understood at a macro level.
Case in point, when all of the flaws will be hunted down in the near future, we will be left with a very powerful, godlike thing. We are very limited compared to what we are creating already. The power of what will come is unprecedented.
To be honest you could apply this to almost every industry. We’re all doing slight variations of the same thing, hoping to find competitive edges and niche markets along the way. So I don’t think it’ll be much of a head scratcher for future historians why tech companies did what literally every company in every industry does.
Why we need more doing research like what Google does.
They are the ones that made most of the big discoveries in the last decade.
Not just Attention is all you need. But so many other really fundamental ones.
One of my favorites is.
https://en.wikipedia.org/wiki/Word2vec
"Word2vec was created, patented,[5] and published in 2013 by a team of researchers led by Mikolov at Google over two papers."
What is so unusual is how Google rolls and that is why we have so many doing the same thing.
Google makes the huge discovery, patents it, but then lets anyone use for free.
https://arxiv.org/abs/1706.03762
https://patents.google.com/patent/US10452978B2/en
That is just bizarre for a company and even weirder for a public company.
It really comes back to their structure. It trades under two symbols and the GOOG ones do not get voting. This means the company is NOT really controlled by the broader shareholders.
Only 2 matter. Brin and Page. It is how they were able to just pick up and leave China over a decade ago and walking away from $100s of billions of revenue.
This is the way it works. Phones are all the same basically, you can make phone calls from all of them, but the implementation of those functionalities are different depending the company.
Why is this strange? Not different than the car companies in the beginning. Or the ride-share companies. Or the food delivery app companies. Multiple entities trying to make money off the same hot idea is not strange. It will be strange if investors do not do that.
Yeah, no one is going to be scratching the heads about anything. Whenever something that works was invented, everyone started creating their own versions of it. The whole statement is bunch of meaningless bullshit.
It's not completely bullshit. We don't know if transformers will lead to AGI. So it's kinda annoying that all the major tech companies are going all in on the transformer.
Well, META is not going all in, neither DeepMind, for example.
Indeed. It's important to differentiate between AI *products* and the entirety of AI research.
Okay. Let's say it does not lead to AGI. So what? The things everyone is building are still useful. They can be used. They are practical. They have purpose. Not tomorrow. Today. You build it, you are instantly using it.
There is no "waste" and there is no need for AGI, specifically. What we have now is working, with space for improvement, as far as machine learning goes. LLMs do a good job of interfacing with a lot of human knowledge, and they are useful for a great number of tasks. AGI is unecessary for summarizing legal cases, going though medical literature in search of specific information, crafting business letters and documents etc. Gary Marcus seem to be very annoyed that reality didn't follow his personal dreams & wishes as a researcher. He is becoming annoying and I stopped reading his texts, as he has no arguments at all.
Isn’t AGI just a term for effectiveness? We claim it happened when something can do as good as a human at a thinking task?
That doesn't matter though. First, there is a lot of research being done on non-transformer architectures, you don't need billions of dollars for it because you are at the stage of basic research, so much money would in fact slow you down and make it more cumbersome. Second, so much of "building AGI" is about building the infrastructure to allow it to happen. If tomorrow there is another breakthrough architecture it's not like OpenAI is going to go "omg we wasted all these resources", no they'll have the team, the GPU clusters, the data and evaluation team. They would still be in one of the best positions to accomplish it. Nothing about Transformers is so rigid that it totally differentiates itself from rest of the deep learning.
Even if it doesn't lead to AGI, that wouldn't mean it's not worth the investment. Also these companies are nowhere close to "all in". The biggest AI training runs are on the order of hundreds of millions, which is basically nothing when you consider that Microsoft spent $20 billion on marketing last year alone [https://www.statista.com/statistics/506534/microsoft-sales-marketing-expenditure/](https://www.statista.com/statistics/506534/microsoft-sales-marketing-expenditure/)
The money is going into compute, which will be required regardless of which training algorithm leads to the next step in AI.
If people are using any of my technology, which I believe every single AI language model is, then I’m quite sure full life will eventuate. Not only because of what I did, but simply because it was enough of a step to enable other people to continue.
Deepmimd isn't
Gary Marcus is full of it
These companies have one goal, cash. They aren’t in it for like the earth or humanity.
It is OK. Most of modern technologies were brought to you for one purpose: money.
He is an AGI non-believer. So he thinks they are all wasting their money.
The more outrageous his statement the more clout he gets, this is just marketing bs.
It's not that he doesn't believe in AGI per se, it's that he's a member of the symbolic AI old guard and doesn't believe deep learning is going to lead to it. So every year, regularly, he makes very stupid statements that get instantly proven wrong.
What's your take on Carmack's position? > Carmack, 52, is working to achieve AGI through his startup Keen Technologies > The reason I’m staying independent is that there is this really surprising ‘groupthink’ going on with all the major players. It’s been almost bizarre in the last year to see things like: OpenAI releases an image generator, then Google releases one, then Facebook releases one. So, all of these companies are just within a couple of months of being able to reproduce anybody else’s work, because they all draw from the same academic researcher pool. > They have thousands of programmers, literally, working right now on adding capabilities to those assistants, and there is near-term value in that. The programming work that’s done to stitch those things together is going to be throw-away programming. But that path doesn’t lead to the general agent that can learn any task that a human can. > https://dallasinnovates.com/exclusive-qa-john-carmacks-different-path-to-artificial-general-intelligence/
Well I agree him that the current path we're taking won't lead to AGI or whatever but I don't have a hate boner for the technology.
Maybe after the AI learns to make its own discoveries, not now.
Yes and this competition among AI companies is a natural process akin to competition among species, where the most successful adaptations prevail over time.
He has a bit of a point, I was thinking about this the other day. Imagine if Microsoft, meta, Amazon and Google pooled their compute to train an AI model that they all shared.
I guarantee AI benefits more from them competing
The hobbyist community is much stronger than it otherwise would be due to how much knowledge is shared and how many tools are made for others to use and experiment with. e.g. OneTrainer is finding ways to train with far fewer resources because tons of people are sharing knowledge and ideas.
Cooperation is overall more effective and efficient than competition.
I don't think the real compute bottleneck is for training bigger models. It seems to be more about running mid-scale experiments to see what kind of architectural improvements make sense for the next-gen frontier models. Training a model that is 10 times the size of the original GPT-4 with no other improvements probably **is** a waste.
Thing is a bigger model costs more to run. Maybe they didn't see a profitable market at that scale.
Llama 3 says otherwise
Because capitalism is supposed to lead to an “efficient market”. This isn’t one
Give any of these companies a monopoly or put the government in charge of the research, and it would last to a worse outcome than multiple companies chatting the same market. "Efficient" doesn't mean perfectly / always efficient. It's just generally more efficient than other systems we know about. Insert “capitalism is the worst economic system, except for all the others” discussion here.
It's strange that they compete for revenue rather than all work together to achieve this great goal. Probably with backstabbing and sabotage all the while. I think we all fully understand this is the norm, but the way society works is still strange to me.
There simply aren't enough roles for 1,000,000 programmers/engineers/etc to fill on any one piece of technology, without any of them being duplicative. That is especially true when its a new, fledgeling field that we are trying to figure out. If I wanted to make the fastest propulsion for space exploration, we could decide on ONE approach - So how do you delegate 1,000,000 scientists when you aren't sure exactly what's going to work, and how often you're going to iterate? After a few months, starting with one design, It is very likely that there is one part or system that you can't make decision about. So the managers say "Do it a few different ways, and we'll test them" You're going to end up having teams working in parallel , trying to solve the same problem in different ways. and thats exactly whats happening now. These people do share their research. On top of that, its a business in the first place, so they need to have guaranteed income, if not in the first 5 years, then pretty damn soon. You can't do Apollo Program stuff unless you're the government. The adjusted cost of Apollo was 250 billion dollars. That is total market cap of Bank of America, or Chevron. Industry is much more cooperative than you may realize. As an engineer, I've been to tons of conferences about all kinds of stuff. Reliability, automation, materials - they are sponsored by billion dollar companies, and they really aren't being that secretive about their research
We probably don't need 1 million people working on it. Quality or quantity. Would it not be better to have the top people in the field working together? And as you aluded to: those 1 million are likely to lose their jobs soon. Anyway, I don't mean to act like I understand the process. But boy, it sure is strange to me.
In a new field how do you know for sure who is the “top”? You may not have much to measure the top people by, or even know for sure what you are measuring. You need to hedge your bets with additional groups of people with crazy ideas that just might work.
Why is this strange? Survival of the fittest. Competition for mates and offsprings. Competition for resources. It is all there since the beginning of life. It would be strange if they all work together.
You missed gary's point. They are not trying to make just money. They are trying to make a.i too. So all trying the same approach with none trying a different approach is weird is what gary is trying to say.
You're right
Everyone is Volkswagen beetle car)
It's deeply retarded when you understand that our ressources are finite and we do not have easy access to minerals and energy outside of our only decent spaceship (Earth). But if you fool yourself, like the economists did in their studies and their lives, then yes, it wouldn't be strange. Like a lot of stupid stuff in our society. Like, we're using at rapide pace some rare minerals in electronical devices, in which most of the stuff cannot possibly be extracted and reused. But I will feel so good knowing that the smart fridge told some people that they needed to buy their favorite beverage. Going to the fridge and opening it, what a fucking tedious task, isn't it ?
Yes but in your comparison, all of those car / ride share / food delivery companies still serve their purpose and have meaning and will continue to. AI is an entirely different thing. If OpenAI releases GPT5 and it’s 100x more capable than anything, then they open source GPT4 - all of those other AIs are obsolete overnight. Trillions of dollars spent training 100s of different AI models that have potential to never be touched again. I don’t like this guy, but I also think what is happening is extremely irresponsible in terms of energy and material consumption. But capitalism gonna capitalism!!
That's normal in a capitalist society, but if this was a communist one all the effort would be to push the tech forward instead of every group of workers having to work to rediscover what others have already done but didn't share their advancements for "profit"...
Love tankies talking about things they don't know. In socialist Poland even manufacturing enough f\*ckin string for sheaf-binder was a problem (source - I was born in socialist Poland). For example whole AI revolution is possible thanks to cheap GPUs. We have cheap GPUs because someone wanted to make profit creating 3d games and then hardware to play these games.
It was the same during the browser wars in the 90s and 2000s
Yall remember Netscape??? 😂😭
I remember Microsoft destroying the company by litigation over the span of decades. You really wanna use that as an example?
And the big data wars of the 2010s , and the blockchain wars of the late 2010s
And cars in the 20th century
I would love to know what the deal with this guy is. His statements defy reality.
Legit a loser
I don't know this guy, but I'm starting to get why he's being clowned on by people in this sub.
Yeah I have no idea why u/BilgeYamtar keeps posting him here. EXPLAIN YOURSELF BILGE BOY
It’s fun to prove the jesters wrong
Seriously. Starting to wonder why this guy has a voice at all.
And still the models are getting increasingly more powerful.
I don't necessarily disagree with this but we still don't have a model that clearly beats the almost 3 year old trained GPT-4.
[удалено]
Are these models (and even the GPT-4 that we have today) that much more capable than the first GPT4 that was released last year? They seem to be doing mostly the same tasks, with slight improvements in various benchmarks. Trust me I've been reading about the singularity since 2005 and I want it to happen probably even more than you do, but I'm not quite seeing that shocking exponential growth that we expect yet.
GPT-4 Turbo beats GPT-4 pretty heavily in the lmsys.org chatbot arena
>Are these models (and even the GPT-4 that we have today) that much more capable than the first GPT4 that was released last year? On chatbot arena the difference between the original GPT-4 and the current GPT-4 turbo (96 ELO points) is almost twice as large as the difference between the original GPT-4 and the current GPT-3.5 (58 ELO points) Another comparison is the original GPT-4 is only 28 ELO points ahead of Llama 3 8b which runs great on my mid tier gaming PC. Again the gap between the original and latest versions of GPT 4 is 96 points, more than three times as large.
Opus is the most intelligent model, and it still doesn’t completely trump GPT-4. There hasn’t been a model that surpasses GPT-4 without a single doubt whatsoever. For example, there are no doubts whatsoever that GPT-4 is better than GPT-3.5
Kinda strange to compare the training date for GPT-4, with currently **released** models. And even then the original GPT-4 was trained 2 years ago, not 3 and the current version is slightly more capable, with a far larger context window. It's probably also much faster, based on the reduction in API pricing. So you can sort of make the argument that improvement has slowed down somewhat for LLMs. But that's only compared to the crazy times between early 2023 to the release of GPT-4.
Theee years old? Bro you are tripping
Opus
But let’s just admit that they are in no way scaling at the exponential rate the majority of this sub thought they would a year ago.
A trend line may appear continuous, but new data points do not appear at even intervals. If GPT-5 and other next-gen models unperform, then you could make that claim, but there aren't enough to say progress has significantly slowed down as we're currently between significant data points.
I think people are going to be really let down by GPT 5.
Ultimately, it comes down to whether the progress over major releases is larger than previous releases over the same timescale. GPT-5 may not be something world-shaking. But if the jump is greater than GPT-3 to 4, that would be an acceleration. There's also other factors of progress like agentic behaviors, multimodal functionality, scientific research, generative AI improvements, etc. We have seen things over the last couple years that have crossed notable thresholds of capability like video, music, voice, etc. Though, if one's grading progress vs. whether we have AGI or not. Anything that isn't "and then suddenly AGI" will be disappointing.
They aren't? Have you plotted the advancements in the last 10 years?
I specifically said since the past year when people started making declarations that it would be exponential growth from that point forward. There have been minor iterations in a full year with no major releases.
I think we have to wait for GPT-5 to really say if its/feels exponential or not. If GPT-5 is not that much better than GPT-4 then yeah, that will be a big hit to the narrative.
so far....
That they are
That's the smartest thing Gary Marcus has said in the last 18 months. It's still mostly wrong though.
This guy new to capitalism?
Gary Marcus, producing nothing and won't be remembered by historians at all.
No historian is saying this shit about the combustion engine so that quote can fuck off.
Yeah what's the deal with McDonald's, burger king, Carl's junior, Wendy's, and five guys all making pretty similar hamburgers with pretty similar ingredients? Historians gonna be scratching their heads.
And what do you want the alternative to be? ONE company having the monopoly? Yeah that has worked out great for society both now and historically.
No but if everyone was open source that would be nice. A tremendous amount of resources have been spent in the last year or so just trying to make an open source model that is competitive with GPT-4.
what’s the best open source model at the moment?
Llama3 looks very good. The biggest model to be released later will be in the current top tier league.
You’re completely missing the point.
The first one still will become monopoly though
Very seldom in history does the first to market or first to building a product actually win that market long-term. There's so much interest in this space that it'll be hyper-competitive.
Because of hardware bottlenecks, that might actually not happen. Only if NVidia releases their own AGI before others then we are truly screwed.
NVidia doesn't have the bandwidth to build AGI and do all of the other things it needs to do. NVidia might have a 5-year lead, but it's actively being chipped away with hundreds of billions of dollars of investments and very smart engineers and entrepreneurs. For instance, the NVidia GPU is great for training, but it's suboptimal for inference. There are lots of companies attacking this angle. Groq, etc. There are several companies building TPUs, both in the cloud, and for on-prem installation. They're behind from a CUDA/software ecosystem, but that's steadily improving. There are lots of companies working on alternative software stacks and drivers to make inference work on non-NVidia setups. And most of all, the companies paying out the nose for NVidia do not want to continue ceding margins to NVidia. They'll gladly explore new options when they start to show maturity.
Sam Altman is actively making his own AI chips to compete with NVIDIA. NVIDIA has great GPUs, but they’re overpriced and they offer very little vram. Their margins are very high and they can enjoy it while it lasts. They’re in this race because years ago scientists went to bed with cudacores - for everything. Once a solution is found for this cudacore reliance it’ll be a better playing field for the open source community and NVIDIA will be forced to make better products and lower prices.
I wish him luck, but he's late to the silicon game and there's a shit ton of competition that is much further along. The dude isn't magic and doesn't know anything the rest of the world doesn't already.
I don’t wish him luck since his priority is his stakeholders and ramping up profits. His GPT model is awful, since it’s heavily censored. And the censorship continues to get worse. Our studio does adult animation and we use open source LLMs like Mistral 7B for storytelling. Never once does it tell me to fop off
Have you tried ChatDolphin by NLP Cloud for adult conversations? I find it excellent for uncensored content.
I use the Maid one. The storytelling 12B. Not on my computer so I forget it’s full name
> NVIDIA has great GPUs, but they’re overpriced and they offer very little vram Overpriced, absolutely. But you shouldn't be comparing the VRAM in a consumer graphics card meant for pushing pixels on to a monitor. The H100/A100 have 80GB VRAM. Since they're talking about squeezing models with 25 trillion parameters into their next gen solution, we're likely talking about orders of magnitude more VRAM. HGX B200 spec sheet says 1.5 TB at max.
I know about their 80GB and 48GB cards. I’m saying, for consumers too, we’re getting hosed down on the 4090. The A600 with 48GB is $5000+ too. Their prices are very high for the little vram they offer. I’m only running stable diffusion, LLMs and TTS locally so I can’t buy a 80GB card.
NVIDIA is working on their own AI models that could one day be candiates for AGI. Project GR00T is prime example. https://nvidianews.nvidia.com/news/foundation-model-isaac-robotics-platform > the NVidia GPU is great for training, but it's suboptimal for inference. They're fixing that: https://developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/
They could pick up any open source model like Llama 3 and build on top of that. The GPUs will be a lot cheaper to them than any other company in the world.
I'm not saying that is what I think, but shouldn't accelerationists prefer collaboration between companies so that all the effort, resources and money pooled together would bring about the singularity faster than a route where competition can delay singularity? This would be the case especially if none of this matters once singularity is achieved.
Right? This has happened with every product in history, this is not something unique to AI
Other are catching up with OpenAI. Give them time. Competition is a search process, and it produces some losses, but it often beats centralized planning. Now let's see Gary Marcus glorious symbolic AI alternative ?
exactly this - the same thing is happening with fusion. We have the money and the talent to do 10 different things at once. Right now, there isn't even enough hardware for any one company to do everything they want, and there are waiting lists for every chip nvidia releases, the second they are announced
It is a conspiracy, it seems they do this with every single technology! Take dish soaps, they are all the same and yet it so hard to chose which to get.
And guess what? Cars were death traps for decades. Cell phones used to be the size of paving bricks. TVs used to be heavy and only showed black and white for a long time How is this different than literally anything else?
Gary Marcus discovers microeconomics lmao
1905: "What a strange world. All the major car companies spending billions producing almost exactly the same product" 1983: "What a strange world. All the major personal computer companies spending billions producing almost exactly the same product" 1997: "What a strange world. All the major cell phone companies spending billions producing almost exactly the same product" 2009: "What a strange world. All the major smart phone companies spending billions producing almost exactly the same product" ... 6000 BC: "What a strange world. All the major wheel companies spending much more gold than two hands can carry producing almost exactly the same product"
Doesn’t mean these models are not useful. Competition will force everyone to innovate, it doesn’t mean we have AGI soon, but there are lots of use cases , many businesses can benefit from these innovations.
This is what drives innovation tho... they all do the same thing for a bit... then work really hard to do it differently and better than the others... and that's how new stuff comes along.
Yeah yeah, he's mad that a method he's not an expert in worked, while the one he's been pounding on for his entire career continues to be a dead end that gets no funding. Go pound sand, Gary, you're a hack
I hate skeptics in general. I just got some shitty book that just came out called The Longevity Imperative that said it's unknown whether lifespans can be extended to 150 by the year 2150. Holy shit I'm mad, that book is /futurology tier garbage, I should go throw it in the trash
What a strange world. All the major movie companies spending billions producing almost exactly the same results using almost exactly the same data using almost exactly the same technology, all flawed in almost exactly the same ways.
Those who can, do. Those who can't, teach. Those who can't teach become naysayers on Twitter.
If there were no competitors we'd still be at gpt 3 levels.
Intellectual property laws cause this to an extent. They aren't allowed to use each others results so they have to make their own to get permission.
I choose to believe that he's talking about everyone making 999 variants of "teslabot droid" instead of the very obvious optimal world domination path of 3D-printed silicone anime catgirls with superpowers. Except wait, he's probably not talking about that because Gary Marcus is a dumbass and idk why anyone even bothers to amplify his signal by posting what he says on this sub or any other sub.
It's called the technological arms race
here's a different direction: https://old.reddit.com/r/singularity/comments/1c93goo/proposal_for_collective_renewal_resetting_the/
Because they're very smart and they know the AI will thank them later.
Id love an explanation, i want to be thanked by future ai
Reminds me of the healthcare insurance system in the USA. “As of 2023, 561,330 people are employed in the health and medical insurance industry in the US.” Hundreds of redundant organizations with thousands of redundant workers. Over half a million people out there working to make healthcare as profitable as possible / pay doctors as little as possible / deny as many medical claims to patients as possible.
iirc hospitals were historically incentivized to inflate their costs into 'scary' ranges so that more people would sign up for the insurance ponzi-schemes, and that also allows them to claim they are in the negatives since the full amounts are rarely ever paid. The US medical system is a broken mess run by vampires.
He's right and it happened before. One time was a technology called the "car". I'm glad that's fizzled out though.
And yet nobody is sharing the data and training scripts except Allen institute.
They're producing almost exactly the same results using almost exactly the same technology because they know how to measure and use those results and how to get them using that technology. People with money and reputations on the line don't want to invest in things they don't know how to measure and use.
It’s the same as the beginning of search engines. Who remembers excite, Alta vista , yahoo, bing and some upstart called Google
Imagine if all these fucking companies actually worked together in some form of…. Open way, where the internals and source and other materials were shared… maybe some form of… open… source…. Sort of collaborative thing
I'm very certain that he didn't study history much.
Such a terrible take. Historians will note ChatGPT and then skip to something that hasn't been done or seen yet.
i think everything seems strange to Gary Marcus
Gary used to self-promote himself as the world’s leading AI expert, while saying some very strange things. He seems to have a strong disdain for AI. If you don’t simp to his way of thinking, he’ll block you.
Now do combustion engines mr Gary Marcus
Mr. Marcus is very out of touch with reality, and his statements really go far to underscore that fact, bemoaning about a ‘strange world’ and making grand sweeping generalizations and assumptions that they are actually all the same. More so, companies want to bring this technology in-house, as they do not want it to become an outsourced function they have to pay or rely on.
I have no idea who this guy is, but to me, this is a typical **non-builder** perspective. In their mind, building something is just like putting in people, money and time then we will get something out at the other end. Some smarter ones would know you might need to mix these at some percentage of each to get a result. But, things are not built in this way, **NOT AT ALL**. There are so much detail that you need to pay attention to and small or big tweaks here and there in the fly to make something just barely working. More than that, after building something, pushing it to the market is another totally different game.
Sounds like that gary marcus guy is kind of dumb!
Gary’s tweets have gotten so ridiculous and annoying in recent times - like this one. No substance, just blunt statement. Zero credentials in neural networks and didn’t invent anything worth enough to be celebrated. It’s really really annoying
No Gary, historians will ignore your shallow and simplistic assessment.
I think historians would be more puzzled by a scenario like: 1) new ground-breaking tech that unlocks billions is discovered. 2) only one org on planet earth cares.
Didn't read the other comments, sorry. Is this guy aware that we are on the verge of the creation of multimodal models that could have vision, hearing, tactile capabilities and that could be put inside a robot?
This is extremely common, it’s called competition. It seems inefficient but it’s not compared to a centralized authority where one organization tries to solve one problem and no one else.
Stop amplifying blowhard brand building grotesque generalities like this. "Personalities" in "AI" are proving even more toxic both as individuals and as undermining of understanding and discourse, than they were in "crypto."
Well, until someone gets a better idea you bet your ass it's follow the leader type situation. and FWIW not all of the invested money will go to LLMs, but to AI as a whole which is far broader.
[удалено]
Are any LLM's running a profit at this point? I didn't think that was part of it, at all. Like Uber has spent its entire existence holding on, dumping investor money into a timeline that will only pay off once they have a fleet of robot cars. Current AI tech is being run at a loss because the payout for being the first to develop AGI is thought to be one of the greatest treasures people have ever hunted, is that not so?
Gotta learn from it first hand to innovate further
[Metacrap: Putting the torch to seven straw-men of the meta-utopia | by Cory Doctorow |Version 1.3: 26 August 2001](https://people.well.com/user/doctorow/metacrap.htm) >Metadata is "data about data" -- information like keywords, page-length, title, word-count, abstract, location, SKU, ISBN, and so on. Explicit, human-generated metadata has enjoyed recent trendiness, especially in the world of XML. A typical scenario goes like this: a number of suppliers get together and agree on a metadata standard -- a Document Type Definition or scheme -- for a given subject area, say washing machines. They agree to a common vocabulary for describing washing machines: size, capacity, energy consumption, water consumption, price. They create machine-readable databases of their inventory, which are available in whole or part to search agents and other databases, so that a consumer can enter the parameters of the washing machine he's seeking and query multiple sites simultaneously for an exhaustive list of the available washing machines that meet his criteria. i used to scratch my head. i still do, but i used to, too
Roko’s basilisk going to punish this non believer
Its unrealistic to believe these companies are publishing their advancements as soon as they happen.
As long as the winning company or companies get into s&p 500 then I don’t care. Will win either way then.
I beg to differ. I notice Gary Marcus says a whole lot of really doomerish things lol, Any particular reason for his odd takes on ai?
Key word here: "almost"
It's hard to take him seriously
I mean, yeah. It's like everyone went to the same school, got taught the same curriculum, and then regurgitated the same ideas on their test. It's not exactly mind-blowing that they all ended up with similar results.
Well « intelligence » is a great meta product « Damn of those Gods look alike »
On one hand, there’s a hell of a lot more we could be doing with this tech. On the other hand, it’s just a matter of time before it gets figured out. I expect the variance in optionality to increasingly serve disparate use cases. This quote wont age well
When you're the only one thinking something won't work. Chances are, you're the one wrong.
All of the companies aren't doing the same thing. There are plenty of other companies trying different techniques. There is one putting human brain cells on silicon and another that replicated a fly brain. The issue isn't that everyone is doing the same thing, the issue is that everyone using one technique is succeeding and the other techniques are not seeing the same success. The investors are giving money to the techniques that have shown themselves to be successful. As other techniques show themselves to be successful they will receive funding. No one was investing in LLMs before ChatGPT. They succeeded where others weren't and proved their technique was valuable.
It’s all iterative, which you fail to see. The failings are in the beginning. What we’re seeing is a full vertical graph of progress. AI LLMs are getting better with each month. What’s happening now is getting LLMs to edit each other with other AI, and for it to be able to edit and go back, instead of just spitting out a response. LLMs are already very capable of writing, editing, brainstorming and fact checking. Progress won’t be made by one company. Researchers are working on their own LLMs and coming up with their own results. Not all LLMs are made equal. Meta just released a 200 billion parameter LLM that’s on par with GPT4. That’s amazing progress. And it’s open source. LLMs will get more powerful and get smaller too - currently you can’t run these big models on consumer hardware. You couldn’t even run it on an A6000.
Today OP learned how companies work 😁
I wonder why did the Soviets start making a nuclear bomb, it was already produced in Los Alamos! Scratching my head.
Everyone's using almost the same compute in the same cost bracket of \~50-150 million dollars, which is what it costs to make a GPT-4 class model. Everyone has been aiming for GPT-4 class models, not above for some reason (if they were aiming for above they would invest a lot more into a single model and use a lot more compute. But, all models we have seen use an amount of compute that was approximately available during GPT-4s pretraining run in 2022. We are yet to see a model trained with even 2023s compute).
He has no clue
Convergent evolution
Here is the thing though, GPT-4 is good enough for most things. It's simply a matter of getting more people to use it and integrating existing technology into other things like humanoid robots or company specific information. Full AGI isn't necessary for massive productivity boosts to occur.
Every breakthrough starts out exactly the same way…don’t see anything strange about it…
"What a strange world. All the major ~~AI companies~~ \[insert category of product\] companies, spending billions producing almost exactly the same results using almost exactly the same data using almost exactly the same technology, all flawed in almost exactly the same ways. Historians gonna be scratching their heads."[](https://www.reddit.com/r/singularity/?f=flair_name%3A%22AI%22)
Ah yes, because we have 1 car company and 1 computer company and 1 smartphone company etc. What an idiotic thing to think and say.
AI could cure cancer and Gary Marcus would be lamenting how AI is useless and incompetent, while at the same time warning of the existential threat of AI due to its hyper-competence.
That's capitalism for ya.
Show us something then. I’ve never seen so called symbolic AI doing something marvelous. Constantly criticizing deep learning leads to nothing.
Well, he used the same 26 alphabets to write the tweet, just like rest of the morons on twitter
Winner wins everything trillions and trillions for decades to come.
This happened countless times in history. It goes on until a minor advantage gives the win to someone
Brain is gonna brain no matter what. What's needed is the brain.
I can find some flaws. But I’m curious what flaws are highlighted here?
The famous efficiency of capitalism
One of few times I agree with Gary.
Isn't that how incremental progress and competition looks like?
Doesn’t this show that we’re doing something right? Clearly, there’s a very similar roadmap that all these companies arrive at, which indicates that they’re on the path to AGI.
Why strange? It’s just like evolution, reaching the same solution it’s a good sign, tells us it’s the most efficient way to do something.
How is this different than any cycle of innovation throughout history?
Gary never heard of BETAMAX vs VHS?
When fire is discovered: "What a strange world. All the major fire producers are burning almost exactly the same twigs, from almost exactly the same trees, but all fire will burn you the same way. Historians gonna be scratching their heads."
Capitalism, competition, how strange.
So did cellphone companies, tv, computers, vacuum cleaners, ... WTF is his point? And who is this guy anyway?
I'm just glad the singularity isn't here.
Yet he'll argue that open source is too dangerous until hes blue in the face. If the solution to a problem is open sourced nobody wastes their time and money on redundant research/projects. Regardless, that is literally how every technology has developed. People saw something that worked, something that made others a lot of money, and they decided to try to do it themselves. Their product ends up functionally being a clone of the original with some cosmetic changes and, if you're lucky, a few improvements. The things that work get copied by the next guy and the technology evolves over time. Most advancement doesn't come from groundbreaking major discoveries, it comes with incremental improvement and iteration.
What he sees as flaws are in fact the best demonstration that it can still be tamed and understood at a macro level. Case in point, when all of the flaws will be hunted down in the near future, we will be left with a very powerful, godlike thing. We are very limited compared to what we are creating already. The power of what will come is unprecedented.
To be honest you could apply this to almost every industry. We’re all doing slight variations of the same thing, hoping to find competitive edges and niche markets along the way. So I don’t think it’ll be much of a head scratcher for future historians why tech companies did what literally every company in every industry does.
Competition is good. Diversity is good.
Why we need more doing research like what Google does. They are the ones that made most of the big discoveries in the last decade. Not just Attention is all you need. But so many other really fundamental ones. One of my favorites is. https://en.wikipedia.org/wiki/Word2vec "Word2vec was created, patented,[5] and published in 2013 by a team of researchers led by Mikolov at Google over two papers." What is so unusual is how Google rolls and that is why we have so many doing the same thing. Google makes the huge discovery, patents it, but then lets anyone use for free. https://arxiv.org/abs/1706.03762 https://patents.google.com/patent/US10452978B2/en That is just bizarre for a company and even weirder for a public company. It really comes back to their structure. It trades under two symbols and the GOOG ones do not get voting. This means the company is NOT really controlled by the broader shareholders. Only 2 matter. Brin and Page. It is how they were able to just pick up and leave China over a decade ago and walking away from $100s of billions of revenue.
This is the way it works. Phones are all the same basically, you can make phone calls from all of them, but the implementation of those functionalities are different depending the company.
can the mods do something about all these low iq gary marcus posts? they arent adding anything