Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay,...
IP属地：新疆
-
缩放律与模型架构：归纳偏置如何影响缩放？
-
UL2：统一语言学习范式
UL2: Unifying Language Learning Paradigms https://arxiv.org/abs/2205.05131v3 Yi Tay, Mo...
-
用0.1%的额外计算超越缩放定律
Transcending Scaling Laws with 0.1% Extra Compute https://arxiv.org/abs/2210.11399 Yi T...
-
大语言模型的涌现能力
Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 Jason Wei,...
-
预训练指南：测量数据年龄、领域覆盖率、质量和毒性的影响
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Covera...
-
自回归生成建模的缩放律
Scaling Laws for Autoregressive Generative Modeling Oct 2020 https://arxiv.org/abs/2010...
-
神经语言模型的缩放律
Scaling Laws for Neural Language Models Jan 2020 https://arxiv.org/abs/2001.08361 Jared...
-
DoReMi：优化数据混合加速语言模型预训练
预训练数据域（如维基百科、书籍、网络文本）的混合比例极大地影响了语言模型（LM）的性能。在本文中，我们提出了具有Minimax优化的域重新加权（DoReMi），它首先在域上使...
-
LoRA微调LoRA: Low-Rank Adaptation of Large Language Models Jun 2021 Edward J. Hu*, Yelong Shen*...
-
LIMA: Less Is More for Alignment
May 2023 https://arxiv.org/abs/2305.11206 [Meta AI, Carnegie Mellon University, Univers...