Thread on @AnthropicAI's cool new paper on how large models are both predictable (scaling laws) and surprising (capability jumps).
1. That there’s a capability jump in 3-digit addition for GPT3 (left) is unsurprising. Good challenge to better predict when such jump will occur.

2. The MMLU capability jump (center) is very different b/c it’s many diverse knowledge questions with no simple algorithm like addition.
This jump is surprising and I’d like to understand better why it happens at all.
3. Program Synthesis jump (right) feels like it should be in between 1 and 2. Less diversity than 2 and we can also imagine models grokking certain concepts in programming leading to a jump.
I’d love to see more work on this topic of predictability and surprise and how they relate to forecasting alignment/risk.
Related work:
1. @gwern's list of capability jumps and classic article on scaling
https://t.co/C8qljJa13A
https://t.co/er791mhbjP
2. @JacobSteinhardt's insightful blog series. https://t.co/0ckLgOgBiV
3. Lukas Finnveden's post on GPT-n extrapolation /scaling on different task shttps://www.lesswrong.com/posts/k2SNji3jXaLGhBeYP/extrapolating-gpt-n-performance

More from All

ChatGPT is a phenomenal AI Tool.

But don't limit yourself to just ChatGPT.

Here're 8 AI-powered tools you should try in 2023:

1. KaiberAI

@KaiberAI helps you generate beautiful videos in minutes.

Transform your ideas into the visual stories of your dreams with this Amazing Tool.

New features:
1. Upload your custom music
2. Prompt Templates
3. Camera Movements:

Check here

https://t.co/ivnDRf628L


2. @tldview TLDV

Best ChatGPT Alternative for meetings.

Make your meetings 10X more productive with this amazing tool.

Try it now:

https://t.co/vOy3sS4QfJ


3. ComposeAI

Use ComposeAI for generating any text using AI.

It’s will help you write better content in seconds.

Try it here:

https://t.co/ksj5aop5ZI


4. Browser AI

Use this AI tool to extract and monitor data from any website.

Train a robot in 2 minutes to do your work.

No coding required.

https://t.co/nNiawtUMyO

You May Also Like