128
GPT-4's details are leaked. (threadreaderapp.com)
submitted 1 year ago by manitcor@lemmy.intai.tech to c/chatgpt@lemmy.world
26 comments fedilink hide all child comments

cross-posted from: https://lemmy.intai.tech/post/72919

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts - Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

Related Article: https://lemmy.intai.tech/post/72922

you are viewing a single comment's thread
view the rest of the comments
[-] manitcor@lemmy.intai.tech 3 points 1 year ago* (last edited 1 year ago)

its just a bit of a rant based on this, source article is still here and kicking. https://lemmy.intai.tech/post/72922

also check out the recent Gerorge Hotz, Freidman podcast.

its been kind of an open secret for weeks now, people are digging to try and prove it out.

[-] shotgun_crab@lemmy.world 1 points 1 year ago* (last edited 1 year ago)

Oh I see. Thanks, it's interesting stuff

this post was submitted on 11 Jul 2023
128 points (100.0% liked)

ChatGPT

8912 readers
1 users here now

Unofficial ChatGPT community to discuss anything ChatGPT

founded 1 year ago
MODERATORS
marcar@lemmy.world