MELLON - Multimodal Enhanced LLM for Online Navigation

MELLON - Multimodal Enhanced LLM for Online Navigation

Ruiyu Li, Haoyang Cai, Zhitong Guo, Tong Hu

arXiv:2608.09121v1 [cs.AI] 10 Aug 2026

Abstract

Web navigation agents are capable of addressing various types of tasks on different websites. Current baselines on Web Navigation are either unimodal, or lack strong reasoning abilities given multimodal inputs. Focusing on the WebShop benchmark, a real-world website simulation, we explore the alignment of text and images and multimodal reasoning and planning abilities to enhance the performance of web navigation agents. We propose three innovative multimodal enhancements: Multimodal Enhanced LLM for Online Navigation (MELLON), VQAgent, and Multimodal Ranker. MELLON demonstrates a significant improvement in task completion accuracy, with a 9.26% increase after just one epoch of training. Our findings suggest the necessity of further exploration into multimodal approaches, with a focus on more extensive training and alignment strategies to enhance web navigation agents' effectiveness.

1. Introduction

Web Navigation Agents address real website tasks autonomously through the interpretation of natural language instructions. As the integration of web-based applications and services has grown increasingly ubiquitous in our daily lives, encompassing activities as diverse as online shopping, route planning, and participation in online forums, these agents have become highly valuable, as they can greatly improve efficiency in various online tasks, enhancing user experiences and productivity.

When humans encounter a webpage, they not only focus on text information, but also visual aspects. Visual information could be a key factor in making a decision, especially in tasks like shopping on a e-commerce website. Therefore, we argue that by incorporating text information with visual information in web navigation task, we can achieve a better performance in this task, because text information may have a lack of information that images may complement.

In this paper, we introduce a multimodal understanding of the web navigation task. Additionally, we introduce three innovative multimodal methods: (1) Multimodal Enhanced LLM for Online Navigation (MELLON), a multimodal web agent characterized by using better encoders, enhanced text and image alignment, robust planning capabilities, and formidable multimodal reasoning abilities; (2) VQAgent, which frames the WebShop task as a multi-hop VQA task; (3) Multimodal Ranker, which encourages agent to not just interact with the top few items in the page, but takes more time in making the decision.

We evaluated MELLON on the WebShop benchmark. WebShop is a website environment that simulates a real-world e-commerce website. It is naturally multimodal as it contains instructions, descriptions and images, and thus is an appropriate benchmark for our purpose.

2. Related Work

2.1. Web Navigation Benchmarks

The Mind2Web dataset presents a collection of real-world tasks and their corresponding trajectories on various websites commonly encountered in real-life scenarios. However, it is essential to note that this dataset offers these tasks in the context of webpage snapshots, which differ from the dynamic and continuously evolving nature of real web environments. In contrast, WebShop poses a more formidable challenge, primarily because the actions performed by agents within the environment can have a direct impact on the outcome of the inference process. This dynamic nature of WebShop introduces a layer of complexity absent in static webpage snapshots, making it a more demanding testing ground for web agents.

WebArena is another benchmark for web navigation task. WebArena is a fully functional web environment spanning four distinct domains: e-commerce, social forum discussions, collaborative software development, and content management. However, tasks for WebArena predominantly involve unimodal reasoning. On the contrary, WebShop is an inherently multimodal benchmark that includes visual and text information of each product, and is better suited to our purpose.

2.2. Unimodal Web Navigation Agent

ReAct is an algorithm that is developed to solve language reasoning and decision making tasks. It prompts LLMs to generate reasoning trace and actions pertaining to the task in an interleaved manner. It allows model to dynamic reasoning to create, maintain, and adjust high-level plans for acting while interacting with external sources like Wikipedia to include more information in the process of reasoning. The goal is achieved by including the action space in the language space. A reasoning trace is also know as CoT (chain of thoughts) reasoning. ReAct has been tested on our benchmark, WebShop, and achieved an score of 66.6 and a success rate of 40.0. ReAct is a unimodal web navigation agent, as it utilizes merely the text information on the webpages.

AgentBench was first developed as a systematic benchmark to evaluate the ability of LLMs as agents. It also includes a method to generate prompt for LLMs to generate a CoT reasoning. Their work is based on ReAct, since ReAct is a pioneer in combining CoT reasoning with actions specific to the task. Like ReAct, AgentBench is also a unimodal web navigation agent, as it solely focuses on prompting.

2.3. Multimodal Web Navigation Agent

There are limited number of existing works on web navigation agent that involves multimodal machine learning, and one of them is proposed by WebShop. WebShop adds an attention fusion layer on top of action representations and the bimodal representations (visual embeddings of the items learned from ResNet and the textual embeddings learned from Transformers), to generate actions.

WebGUM is another multimodal web navigation agent, which preprocesses visual imagery with a vision transformer and then merging its output embedding with tokenized HTML. This combined feature is processed by a T5 encoder-decoder transformer to predict the agent's optimal next step. This approach effectively incorporates visual signals. However, it might still face challenges in utilizing the hierarchical structure of webpages to discern the semantic relationships between its elements, particularly regarding the pairings between images and text. Furthermore, in the application within the WebShop environment, the model underwent a significant alteration. The image encoder component was removed, resulting in the exclusive use of text encoders for processing tasks in this context.

To our knowledge, our paper is the first to (1) develop a multimodal web navigation agent that incorporates LLMs, despite the difficulties in engineering the LLM into dynamic web environments; (2) align visual and textual representation via a projection layer into the LLM, which is also efficient for training; (3) enable strong reasoning and planning abilities in a multimodal setting via re-designing ReAct prompts with visual representations.

3. Proposed Approach

3.1. MELLON - Multimodal Enhanced LLM for Online Navigation

The challenge with the WebShop task, unlike typical multimodal reasoning tasks, is its nature as a Markov decision process where only predefined actions are valid, and any deviation can lead to task failure; and some instruction contains qualities not present in the textual description. This motivates us to incorporate LLMs and better pre-trained visual encoders for better reasoning techniques. However, since LLM has open-ended generation, it's particularly challenging due to the difficulty in mapping generated actions to a specific action space.

Moreover, for the current LLM-based baselines for the WebShop task, they are all unimodal, while the data analysis we performed suggests that there is information represented in the image that is not present in the texts. While the accessibility tree contains essential textual content of a webpage, the agent lacks visual perception by only inputting an accessibility tree into LLM. Visual input usually contains a wealth of information necessary for many web tasks, especially for WebShop, as we can indicate from the images some qualitative aspects of the product that are not present in the textual descriptions. So we propose to integrate visual information with the textual state representations to offer the agent a broader context from both text and image modalities and thus a more precise task completion.

We developed MELLON (Multimodal Enhanced LLM for Online Navigation), a web agent with better encoders, better alignment, strong planning abilities and strong multimodal reasoning abilities. Our innovations are three-fold: (1) we generated aligned visual and textual embeddings with better visual and language encoders, that was not done by SOTA web navigation agents before; (2) we innovated an end-to-end learning objective of our model as a cross-entropy generation loss and collected data for training; we solved huge amount of engineering challenges posted by the Webshop environments and processed its datasets to create trajectories for our experiments (3) we designed multimodal prompts for LLM. We initially wanted to use ReAct prompts, but ended up re-designing the prompt to incorporate projected visual input and to fit input embeddings and our model into memory.

3.1.1. Better Encoders

For the model architecture, we propose to use pre-trained visual encoders to generate image encoding of web screenshots, inspired by (Huang et al., 2023). More specifically, we can build a ViT backbond coupled with a pre-trained Q-Former. Unlike traditional convolutional neural networks, ViT relies on a transformer architecture. It divides an image into patches, treats them as tokens, and processes them through self-attention mechanisms, enabling it to capture spatial relationships and features within the image. This approach has demonstrated remarkable performance on various vision tasks. The Q-Former consists of two transformer submodules that share the same self-attention layers. A first image transformer performs image-text matching and a second text transformer generates text based on visual information.

3.1.2. Text and Image Alignment and Efficient End-to-End Training

Since pre-trained LLMs cannot directly understand the output of a visual encoder, we developed a single projection layer to align the visual encoder with the LLM inspired by (Su et al., 2023) and (Zhu et al., 2023). The projection layer takes in the output from Q-Former and outputs a prompt with our designed format for the WebShop tasks, which is aligned with the observation space and other instruction texts in the prompt.

Initially, the projection layer was trained on pairs of images and text to achieve multimodal alignment, thereby endowing the model with visual perception capabilities. However, subsequent experimentation revealed a significant drawback: the model exhibited a diminished reasoning ability, evident in its failure to align with prompts and frequent generation of invalid actions.

To address these challenges, we developed a training approach. Specifically, we trained the model on pairs consisting of multimodal inputs and successful actions, employing a generative loss function. Additionally, we integrated the WebShop completion score into the loss function as a contributing factor. This dual-focused training strategy aims to enhance both the model's reasoning capabilities and its alignment with the intended actions. We freeze LLM and ViT and only trained the projection layer and Q-former, leading to efficient training and computation saving.

Given a ground truth action text sequence y = y1, y2,..., yN and the model output action text sequence ŷ = ŷ1, ŷ2,..., ŷN, the new learning objective L can be defined as a cross-entropy loss:

L(y, ŷ) = -Σ i=1 N yi log(ŷi)

where N is the length of the action text sequence, yi is the true probability distribution of the ith action, and ŷi is the predicted probability distribution of the ith action by the model.

3.1.3. Extended ReAct Prompts

To enable better reasoning abilities for the LLM, we designed an aligned multimodal prompt, based on text-only ReAct prompt. In reality, to fit the prompt in our 24 GB GPU memory to run experiments, we had to significantly shorten the length of our multimodal prompts and put numerous efforts into prompt engineering.

is where we insert our aligned visual representation into the prompts. We augment the agent's action space to A' = A ∩ L, where L is the space of language. Each action a't ∈ A' is composed of useful information by reasoning over the current context ct. The context will also be updated with a't following ct+1 = (ct, a't) for future reasoning and acting. Based on this, the designed prompts contains in-context example with a human trajectory of actions, thoughts, and environment observations to solve the WebShop task. Since there could be a large number of action choices, thoughts only need to appear sparsely in the most relevant positions of a trajectory and the occurence of actions and thoughts is decided by the LLM itself. We formulate ReAct prompts with (1) actions to search, choose product, choose options, and buy, combined with (2) reasoning to determine what to explore, when to buy, and what products options are relevant to the instruction.

3.2. VQAgent

Visual Question Answering (VQA) is a multimodal task requiring the integration of visual and textual understanding, involving both visual context comprehension and textual question interpretation. WebShop, similar to VQA, demands a thorough understanding of images, textual content, and instructions. Recognizing this parallel, we adopted WebShop as a downstream task of VQA, structuring it as a multiple-choice VQA task where the answers are actions to be taken. This approach is innovative in the context of WebShop, utilizing a multimodal model based on VQA principles.

For this purpose, we selected the state-of-the-art BEiT3 model, known for its versatility in language, vision, and vision-language tasks, and its capability in both unimodal and multimodal settings. BEiT3 processes image patch embeddings and word embeddings through a multi-head self-attention layer, addressing visual and textual information separately before fusing them in a vision-language model. This fused encoding is then used for generating answers, adapting BEiT3's architecture for the specific needs of WebShop tasks.

Nevertheless, our model works differently from the original BEiT3 model. BEiT3 for the VQA task is inherently a generation task, while our task is rather a classification task. We are provided with all possible actions, and we want to know which action to take. Furthermore, we are provided an image, a description of the product, and an instruction, but BEiT3 does not support multiple inputs of the same modality. Therefore, we need to add novelty to the model for better incorporate the task. We name our model VQAgent.

Given a dataset D' = (o, A(o), a* of M' = 9558 samples from the training human trajectories, we train our model using imitation learning, which is a learning technique that tries to mimic human action in a task. We encode observation o, which include the image and detail of the product we are trying to decide whether to buy, as well as the instruction, into embeddings using a 24-layer BEiT3 model. We parameterize this model as θ. We do the same for each possible action. Then we pool our embeddings, and pass the observation embeddings and embeddings of each action into a bi-attention layer. Then, we mean pooled the results for all actions into a single vector, and use a linear layer to obtain score for executing each action. πθ(ao, A(o)) refers to the softmax distribution over action scores, S(o, a). Therefore, we define the loss function for our task as:

Lchoose = E o,A(o),a*∼D' [-log πθ(a* o, A(o))]

where our policy, πθ is formulated as:

πθ(a o,A(o)) ∼ exp(W T average[cross-attn(pool(BEiT3(o; θ)), pool(BEiT3(a; θ)))]).

3.3. Multimodal Ranker

From the analysis of the WebShop baseline, it was observed that the existing baseline models exhibit a limited exploratory behavior, predominantly focusing on the initial option presented by the search module, rather than investigating a broader range of potential choices. To address this, we are in the process of integrating…

Improvements to AI systems based on this paper:

1. Multimodal Action Alignment via Projection Layer: Implement a trainable projection layer between a frozen vision encoder (e.g., ViT + Q-Former) and a frozen LLM, trained end-to-end with cross-entropy loss on successful action sequences. This enables the AI to map visual features directly into the LLM's token embedding space, allowing it to reason over both text and images simultaneously without fine-tuning the LLM. The improved system can navigate dynamic web environments (e.g., e-commerce) by interpreting product images and text together, leading to a 9.26% accuracy boost in task completion after one epoch.

2. ReAct Prompt Redesign for Multimodal Context: Extend the ReAct prompting framework to include an `` token where aligned visual embeddings are inserted, alongside interleaved reasoning traces and actions. The AI system can then perform chain-of-thought reasoning over both modalities, deciding when to search, select products, choose options, and purchase—while dynamically updating its context with each action. This enables more robust planning in partially observable environments, reducing invalid actions and improving decision quality.

3. Parameter-Efficient Multimodal Training: Freeze the LLM and vision transformer, training only the projection layer and Q-Former with a generative loss that incorporates task completion scores. This reduces computational cost (trainable on a single 24GB GPU) while still improving multimodal reasoning. The improved AI can be rapidly adapted to new web navigation tasks with minimal data and compute, making it practical for real-world deployment.

4. Multimodal Ranker with Learnable Fusion: Integrate a ranking module that combines BERTScore (text similarity) and CLIP score (image-text similarity) via a learnable weight, trained with MSE loss against task-specific reward scores. This allows the AI to systematically evaluate all candidate items before acting, rather than focusing only on top search results. The improved system can better balance visual and textual cues for relevance assessment, though the paper notes this requires careful alignment with task-specific scoring criteria to avoid performance degradation.

5. VQA-Style Action Selection (VQAgent): Frame web navigation as a multi-hop visual question answering task, using a bi-attention layer to fuse observation embeddings (image + text + instruction) with action embeddings, then selecting the best action via softmax over scores. The improved AI can handle multiple inputs of the same modality (e.g., multiple product images) and perform classification-based action selection, which is more stable than open-ended generation for constrained action spaces.

6. Error-Aware Generative Action Constraint: Implement a post-processing mechanism to detect and correct invalid or empty outputs from the LLM (e.g., CodeLlama), by constraining the action space to valid predefined actions and re-prompting when outputs are malformed. The improved system reduces hallucination and ensures every step produces a valid action, increasing reliability in real-world web tasks.

7. Modality Informativeness Analysis: Before applying multimodal fusion, assess the relative information content of visual vs. textual modalities for the specific task. The improved AI can dynamically weight modalities based on their informativeness (e.g., in WebShop, text is more informative than images), preventing performance drops seen when visual features dominate or add noise. This leads to more robust multimodal agents across different domains.

8. Continuous Trajectory Learning: Collect and train on continuous interaction trajectories (not just static snapshots) to capture dynamic state transitions and long-term dependencies. The improved AI can learn from richer sequential data, improving generalization and planning over multiple steps, especially in environments where actions affect future states (e.g., WebShop's POMDP).

Web navigation agents are capable of addressing various types of tasks on different websites. Current baselines on web navigation are either unimodal or lack strong reasoning abilities given multimodal inputs. Focusing on the WebShop benchmark, a real-world website simulation, we explore the alignment of text and images, as well as multimodal reasoning and planning abilities, to enhance the performance of web navigation agents. We propose three innovative multimodal enhancements: Multimodal Enhanced LLM for Online Navigation (MELLON), VQAgent, and Multimodal Ranker. MELLON demonstrates a significant improvement in task completion accuracy, with a 9.26% increase after just one epoch of training. Our findings suggest the necessity of further exploration into multimodal approaches, with a focus on more extensive training and alignment strategies to enhance the effectiveness of web navigation agents.