AuraLuxMuse: Adaptive Fusion Modeling for Aesthetic Stage Lighting Design with Music and Expert Guidance
cs.MM, cs.AI
Submitted: 2026-10-08
Updated: 2026-10-08
Comments: Accepted to appear in SIGGRAPH Asia 2026 Conference Papers
License: http://creativecommons.org/licenses/by/4.0/
The gist: We present AuraLuxMuse, a novel system for automated aesthetic stage lighting design that integrates expert knowledge, representation learning, and preference-adaptive modeling.
Abstract
We present AuraLuxMuse, a novel system for automated aesthetic stage lighting design that integrates expert knowledge, representation learning, and preference-adaptive modeling. Lighting design in live performance settings requires the seamless translation of musical features into dynamic lighting behaviors. However, traditional workflows remain time-consuming, labor-intensive, and difficult to transfer. AuraLuxMuse encodes music and professional cue sequences into a shared retrieval space, estimates cue-event density, and retargets selected fixture commands to the destination stage. It assists pre-production authoring by returning editable cues rather than replacing the designer with an unconstrained generator. At the heart of AuraLuxMuse are two key modules: Lighting-Aligned Music Pretraining (LAMP), which performs contrastive learning between audio and lighting cues for alignment, and Preference-Adaptive Mixture of Experts (PAMoE), which conditions preference-aware cue retrieval and adaptation on designers' intent through a gated ensemble of style-specific expert networks. To support training and evaluation, we introduce Musilux, the first dataset of paired musical audio and professional lighting cue sequences under diverse performance scenarios. We evaluate AuraLuxMuse across both virtual simulation environments and professional-grade laboratories. Experimental results, including objective and subjective evaluation, demonstrate that AuraLuxMuse retrieves and adapts stage-lighting cues that are visually cohesive, semantically meaningful, and artistically expressive, showing its potential for AI-assisted aesthetic stage design.
Sources
- Speech Enhancement via Deep Spectrum Image Translation Network
- Unveiling Visual Biases in Audio-Visual Localization Benchmarks
- WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Learning to Predict Indoor Illumination from a Single Image
- AST: Audio Spectrogram Transformer
- SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding
- AudioCLIP: Extending CLIP to Image, Text and Audio
- Deep Residual Learning for Image Recognition
- CNN Architectures for Large-Scale Audio Classification
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Style Mixture of Experts for Expressive Text-To-Speech Synthesis
- Adam: A Method for Stochastic Optimization
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- SpecTNT: a Time-Frequency Transformer for Music Audio
- Contrastive Audio-Language Learning for Music
- MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generation
- Learning Transferable Visual Models From Natural Language Supervision
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Related papers
- ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
- Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
- Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection
- A Rate-Distortion-Classification Approach for Lossy Image Compression