793
submitted 6 days ago* (last edited 5 days ago) by not_IO to c/microblogmemes@lemmy.world
you are viewing a single comment's thread
view the rest of the comments
[-] skisnow@lemmy.ca 40 points 6 days ago

I'm constantly mystified at the huge gap between all these "new model obliterates all benchmarks/passes the bar exam/writes PhD thesis" stories and my actual experience with said model.

[-] aBundleOfFerrets@sh.itjust.works 17 points 5 days ago

The real truth is just that standardized testing fucking sucks and always has

[-] CheeseNoodle@lemmy.world 15 points 5 days ago

Likely those new models are varients trained specifically on the exact material needed to perform those tasks, essentially passing the bar exam as if it were open book.

[-] Tomassci@sh.itjust.works 6 points 5 days ago

Reminds me of a video that starts with the fact you can't convince image generating AI to draw a wine glass filled to the brim. AI is great at replicating the patterns that it has seen and been trained on, like full wine glasses, but it doesn't actually know why it works or how it works. It doesn't know the things we humans know intuitively, like "filled to the brim means more liquid than full". It knows the what but doesn't get the why.

The same could apply to testing. AI knows how you solve test pages, but wouldn't be that exact if you were to try adapting it into real life.

this post was submitted on 23 Jul 2025
793 points (100.0% liked)

Microblog Memes

8707 readers
2131 users here now

A place to share screenshots of Microblog posts, whether from Mastodon, tumblr, ~~Twitter~~ X, KBin, Threads or elsewhere.

Created as an evolution of White People Twitter and other tweet-capture subreddits.

Rules:

  1. Please put at least one word relevant to the post in the post title.
  2. Be nice.
  3. No advertising, brand promotion or guerilla marketing.
  4. Posters are encouraged to link to the toot or tweet etc in the description of posts.

Related communities:

founded 2 years ago
MODERATORS