Semantic Synthesis over Brute-Force Search: LLM-Guided Transformation Design for Efficient Learning
cs.CV, cs.AI
Submitted: 2026-11-07
Updated: 2026-11-07
Code: https://github.com/ABrain-One/NN-GPT
License: http://creativecommons.org/licenses/by/4.0/
The gist: Data transformation pipelines shape the training distribution in computer vision, but designing them is a computationally expensive problem.
Abstract
Data transformation pipelines shape the training distribution in computer vision, but designing them is a computationally expensive problem. We investigate whether large language models can enable efficient discovery by synthesizing executable transformation programs outside predefined augmentation spaces. We compare nine strategies under a unified training protocol, including standard augmentation policies, brute-force search, direct code generation with two LLMs, and performance-guided generation using LLMs fine-tuned on accuracy-ordered transformation pairs. When the best candidates are retrained for 20 epochs, LLM-generated transformation programs lead on every dataset, reaching 0.8877 accuracy on CIFAR-10, 0.5825 on CIFAR-100, and 0.9588 on SVHN, surpassing brute-force search over 6,000 candidates. The best transform on CIFAR-10 and SVHN comes from a fine-tuned generator using 270 candidates, while the best on CIFAR-100 comes from an off-the-shelf generator using only 10. This needs 22x fewer candidate evaluations in the generation step and 600x fewer for off-the-shelf generations, relative to brute-force search. Beyond transformation discovery, we analyze the reliability of the reduced-budget evaluation commonly used for augmentation search. Correlating one-epoch selection scores with 20-epoch outcomes reveals strong, weak, and absent predictive relationships depending on the dataset and architecture, with correlations ranging from 0.756 to near zero. A positive control indicates that range restriction alone is unlikely to explain the low correlations. Our results indicate that semantic synthesis with code LLMs provides an efficient alternative to search-based transformation discovery and highlight the importance of validating evaluation fidelity when optimizing data transformations.
Sources
- AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance?
- Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability
- Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs
- Synthetic Data from Diffusion Models Improves ImageNet Classification
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Evaluating Large Language Models Trained on Code
- Selection via Proxy: Efficient Data Selection for Deep Learning
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Is synthetic data from generative models ready for image recognition?
- Qwen2.5-Coder Technical Report
- AceCoder: Utilizing Existing Code to Enhance Code Generation
- Eureka: Human-Level Reward Design via Coding Large Language Models
- Automated Image Data Preprocessing with Deep Reinforcement Learning
- Dataset Meta-Learning from Kernel Ridge-Regression
- AlphaEvolve: A coding agent for scientific and algorithmic discovery
- Training language models to follow instructions with human feedback
- The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
- Data Distillation: A Survey
- A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
- The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs