Transferrable Surrogates in Expressive Neural Architecture Search Spaces
cs.LG, cs.AI
Submitted: 2025-04-17
Updated: 2026-10-30
Comments: Accepted at AutoML 25, Project page at: https://shiwenqin.github.io/TransferrableSurrogate/
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of…
Terminology
Abstract
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.
Sources
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
- The Llama 3 Herd of Models
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Graph Neural Architecture Search with GPT-4
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- Qwen2.5 Technical Report
- GPT-NAS: Evolutionary Neural Architecture Search with the Generative Pre-Trained Model
Related papers
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis
- (Mis)generalization of Helpful-Only Fine-tuning