Studying the Effects of Task Diversity Using Small Mathematical Worlds
The study introduced two mathematical task ecosystems, IntWorld (integers) and PermWorld (permutations), to examine how increasing task diversity improves a model's internal representations. Results show that more diverse tasks lead to richer representations, and transfer learning is most effective when pretraining includes directly related tasks.