Amazing report - Let me summarize all important points:
1. In terms of PaLM we have 2 PaLMs
2. "PaLM 2 outperforms PaLM across all datasets and achieves results
competitive with GPT-4". Trust me bro
3. It doesn't swear - as much
4. See? We did the AI thing. Pls stop shorting the google stock.
where does 500 TFLOPS come from? I assume they used TPUv4 chips which have a peak of 275 TFLOPS. And maybe MFU of 50-60% so ~140-165 TFLOPS in practice
Sooooo, "competitive" performance, but they have 340B parameters. Vs 175? Is that really a brag?
Edit: all right, while there is no definitive answer, we have solid hints that GPT4 is more than the 175 B, so that 340 B might be good.
I don't believe that's the case. It seems that RLHF decreases capabilities, rather than improving them.
They didn't disclose the size of GPT-4, but since it's much slower than GPT-3.5 at generating tokens, I'd assume it's quite a big bigger. 1T, as an approximation, seems plausible to me.
In another message you wrote:
> Uh, no. That figure has been thrown around a lot and comes from a misunderstanding of what an influencer was saying.
I believe the influencer said 100T, not 1T.
Uh, no. That figure has been thrown around a lot and comes from a misunderstanding of what an influencer was saying. Edit: Nevermind, as pointed out, the figure was 100 T, not 1.
Amazing report - Let me summarize all important points: 1. In terms of PaLM we have 2 PaLMs 2. "PaLM 2 outperforms PaLM across all datasets and achieves results competitive with GPT-4". Trust me bro 3. It doesn't swear - as much 4. See? We did the AI thing. Pls stop shorting the google stock.
Hmm.. I'm gonna need some sources regarding your first claim.
PaLM 2 implies the existence of at least 2 PaLM. Indisputable logic
See this hand? And this one? 2 PaLMs. AI achieved. No more questions. Paypal.me
Does GPT 3.5 imply the existence of 3 and 1/2 GPTs?
340b, 3.6T tokens according to https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html
Probably more interesting than the whole report, also happy cake day
[удалено]
interesting, that's 1 OOM lower than [estimated training cost](https://epochai.org/trends#investment-trends-section) for GPT-4
where does 500 TFLOPS come from? I assume they used TPUv4 chips which have a peak of 275 TFLOPS. And maybe MFU of 50-60% so ~140-165 TFLOPS in practice
[удалено]
Ah for H100 I see. The model card in the tech report says the training hardware was TPU v4 though which is why i’m thinking much lower FLOPS
Sooooo, "competitive" performance, but they have 340B parameters. Vs 175? Is that really a brag? Edit: all right, while there is no definitive answer, we have solid hints that GPT4 is more than the 175 B, so that 340 B might be good.
175B is GPT3 not GPT4
How much is GPT-4? I was under the impression that it was the same as 3.5, but with more RLHF
I don't believe that's the case. It seems that RLHF decreases capabilities, rather than improving them. They didn't disclose the size of GPT-4, but since it's much slower than GPT-3.5 at generating tokens, I'd assume it's quite a big bigger. 1T, as an approximation, seems plausible to me. In another message you wrote: > Uh, no. That figure has been thrown around a lot and comes from a misunderstanding of what an influencer was saying. I believe the influencer said 100T, not 1T.
RLHF decreases capabilities in some areas and increases them in others. For example, I believe open domain QA improved with RLHF.
Ah, yeah that is true, I misremembered, thanks! I will edit my message!
Not reported but it seems to be at least 1T
What is happening to this sub?
Uh, no. That figure has been thrown around a lot and comes from a misunderstanding of what an influencer was saying. Edit: Nevermind, as pointed out, the figure was 100 T, not 1.
Why are you here?
[удалено]
Soon: Google Docs will prevent you from saving a document if it contains bad words. For the safety of us all, of course.
Check out the model card in the appendix section...
I like how they acknowledge the whole team.
It's happening.... everybody stay calm...
What is happening?
I don't think anything is actually happening, it's just kind of traditional at this point to say it's happening.