Introduction
- We have accurate scaling curves for CE loss (over > 7 orders of magnitude!)
- But not for performance on downstream tasks
These emergent abilities can’t be predicted by extrapolating.
Aim of paper = show examples of this across the literature (not investigate when/why)
Emergent Abilities
Definition
Definition of emergence:
Emergence is when quantitative changes in a system result in qualitative changes in behavior
In the context of ML:
- Performance is near-random
- Until critical threshold, after which performance rapidly increases
- Also known as a “phase transition”
Measuring scaling
3 key axes:
- FLOPs
- Params
- Data
1 & 2 linked (except with MoE), so can’t differentiate here. For 3 many papers just use a fixed number of tokens at all scales, so again hard to compare. Hence main focus will be on flops (implies params proportionate)
(can also try and correlate CE/perplexity with emergent ability scaling)
Emergent Scale
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fabdcc30b-5c38-4da3-8ccc-03bcef2c3b67%2FScreenshot_2022-07-01_at_12.04.15.png%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Credential%3DAKIAT73L2G45EIPT3X45%252F20221016%252Fus-west-2%252Fs3%252Faws4_request%26X-Amz-Date%3D20221016T192357Z%26X-Amz-Expires%3D86400%26X-Amz-Signature%3D4bc0520d7f980a87ae03555aa785c747c23ad0d930f03f11b38d2added592794%26X-Amz-SignedHeaders%3Dhost%26x-id%3DGetObject?table=block&id=6e2e2033-3534-4dba-aaac-3b5878da3050&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F4b8cb940-dd36-4a37-afe5-bfe401c4a907%2FScreenshot_2022-07-01_at_12.10.16.png%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Credential%3DAKIAT73L2G45EIPT3X45%252F20221016%252Fus-west-2%252Fs3%252Faws4_request%26X-Amz-Date%3D20221016T192357Z%26X-Amz-Expires%3D86400%26X-Amz-Signature%3D8438a9e19237ae7df795e19d4c8ab8d38a9f05ef8bc4f68a8d7cfa05e37d55cd%26X-Amz-SignedHeaders%3Dhost%26x-id%3DGetObject?table=block&id=e17e1aa9-04d8-4349-b209-9fed6029a5ed&cache=v2)
Few-shot:
- MMLU (multi-task language understanding) result suggests “the ability to solve knowledge-based questions spanning a large collection of topics might require scaling up past this threshold”
- WIC is the only task requiring huge PaLM-scale models. Suggests differentiating between contexts may require massive scale
Augmented prompting: (shows the point at which prompting method surpasses few-shot - see fig 3)
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F64dd366f-4c50-4765-b90c-dc1db5a26854%2FScreenshot_2022-07-01_at_12.24.00.png%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Credential%3DAKIAT73L2G45EIPT3X45%252F20221016%252Fus-west-2%252Fs3%252Faws4_request%26X-Amz-Date%3D20221016T192357Z%26X-Amz-Expires%3D86400%26X-Amz-Signature%3Dfdb891297a76209cb5a39a18d4a23a0888c6f19185ad9c137e7bfb50d678dcd5%26X-Amz-SignedHeaders%3Dhost%26x-id%3DGetObject?table=block&id=93ecba1c-a8d7-4d94-bb96-aca2714cc44a&cache=v2)
Discussion
Explaining emergence
- Notion of emergence may be exaggerated by evaluation metrics. E.g. if exact string match is required then incremental progress may not be visible. Similar logic for multi-step problems.
- However, this is only partial explanation, as even on classification tasks emergence is seen.
- Should be noted though that CE improves even when accuracy is in the random stage, so important to note that models are “getting better” in some sense, even before emergence.
- We can think of downstream metrics as “masking” log-likelihood improvements, until a certain scale.
- However this does not explain why/how some downstream metrics have this effect.
![notion image](https://www.notion.so/image/https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fsecure.notion-static.com%2Fb799c813-e73e-4685-8c89-300e1b139216%2FScreenshot_2022-07-01_at_13.45.06.png%3FX-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Credential%3DAKIAT73L2G45EIPT3X45%252F20221016%252Fus-west-2%252Fs3%252Faws4_request%26X-Amz-Date%3D20221016T192357Z%26X-Amz-Expires%3D86400%26X-Amz-Signature%3Dab22c891d5b80c56791835fe5d4482a98be3f45425af6f94bfb713127603ed3e%26X-Amz-SignedHeaders%3Dhost%26x-id%3DGetObject?table=block&id=d032811c-1b8e-4bf0-905f-59f87a906786&cache=v2)
Â