WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development
WAFFLE is a new fine-tuning strategy for multi-modal models to generate functional HTML code from UI designs. It uses structure-aware attention to help models understand HTML hierarchy and contrastive learning to align visual images with text codes. This approach significantly improves model performance on benchmarks like WebSight-Test, leading to substantial gains in code accuracy and visual similarity.