AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search

summary

The gist

Text-based person anomaly search has emerged as a critical task aiming to retrieve individuals with abnormal behaviors via natural language descriptions The first framework that harnesses LMMs for…

In short

AnomalyLMM introduces a framework that uses Large Multi-modal Models (LMMs) for text-based person anomaly search by employing a coarse-to-fine retrieval pipeline. It reframes the search as a masked cloze reasoning task, using training-free adaptation techniques like masked cross-modal prompting to guide LMMs to focus on subtle visual cues and improve retrieval accuracy.

Key concepts

Coarse-to-Fine Retrieval Pipeline
This method first uses a standard text-to-image model to quickly narrow down a large set of possible images based on the initial query. Then, it refines this smaller list by using the LMM's reasoning abilities to perform a more detailed search, moving from broad candidates to highly specific matches.
Masked Cloze Reasoning Task
This is a technique where key words or attributes in the original text query are replaced with placeholders. The LMM then has to fill these blanks using visual evidence from candidate images. This forces the model to reason about how the image content relates to the missing textual details.
Training-Free Adaptation Cookbook
This is a set of techniques that allow LMMs to perform well on new tasks without needing extra training or fine-tuning data. It uses methods like masked cross-modal prompting and behavioral saliency prediction to help the model focus on subtle anomaly clues in zero-shot scenarios.
Semantic Comparison and Re-ranking
Instead of just checking for exact word matches, this step uses an LLM to compare the original query with image completions semantically. It then re-ranks the results based on how well the substituted words match in meaning, ensuring a more robust and accurate final ranking.

Terminology used across episodes

This episode discusses

The paper

AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search · Read on arXiv

Hao Ju, Hu Zhang, Zhedong Zheng

Faculty of Science and Technology and Institute of Collaborative Innovation, University of Macau · CSIRO Data61

Text-based person anomaly search requires fine-grained alignment between language queries and subtle visual cues, a challenge amplified by the long-tailed nature of anomalous behaviors. While Large Multimodal Models (LMMs) offer powerful visual understanding, their generative pre-training is fundamentally misaligned with the discriminative needs of this retrieval task. Adapting them without costly fine-tuning remains a significant hurdle. We introduce AnomalyLMM, a training-free, coarse-to-fine framework that effectively repurposes generative LMMs for zero-shot anomaly retrieval. Our method first uses a general retrieval model to produce an initial ranking. To refine this list, we introduce a novel cloze-based re-ranking mechanism with three steps. The first step Cloze Generation converts the text query into a ``fill-in-the-blank'' prompt. Next, Cloze Completion compels the LMM to focus on specific visual regions and generate a description of the potential anomaly. The final step Comparison & Re-ranking calculates semantic alignment between the LMM's generated completion and the original query, which serves as a powerful re-ranking score. Experiments on the PAB and UCC datasets show that AnomalyLMM surpasses the competitive baseline by +1.93 % and +1.79 % Recall@1 accuracy on two datasets. Crucially, our method provides highly interpretable visual-textual alignments without any task-specific training, a vital feature for real-world deployment. The code will be made publicly available.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search".

Title and authors: Tom: So we’re talking about the paper called "AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search." Jane, what’s the basic setup here?

Jane: Well, Tom, it tackles the problem of finding people who are acting abnormally when you describe their behavior in plain language. It points out that standard person search doesn't work well because it struggles with two things: getting a really fine alignment between what you say and what the image actually shows, and dealing with those rare cases where we only have very few examples to look at.

Lu: Exactly. The core idea is that Large Multi-modal Models, or LMMs, are great at understanding both text and images together, but they seem to have this gap between their general knowledge—the generative stuff—and the specific skill needed for precise retrieval of anomalies.

Meng: So, if I'm running this in production, what’s the immediate problem we’re solving? Is it just getting better recall on a test set?

Tom: It aims to solve that domain gap by using a new pipeline. Instead of just feeding the text query directly into the LMM, they propose a coarse-to-fine retrieval process where you narrow down the possibilities first and then refine the search using the LMM's reasoning ability.

Jane: That sounds like they’re trying to use that generative knowledge in a more targeted way. How do they actually structure this pipeline?

Lu: They start with a pre-trained text-to-image retrieval model to create an initial list of candidate images based on your natural language query, which is the coarse step. Then, instead of stopping there, they reformulate the search as a masked cloze style reasoning task to do that finer alignment.

Meng: Masked cloze? So you take the text and hide key parts and ask the LMM to fill them in using what it sees in an image? Does that mean you need a perfect match for those missing words?

Tom: Not exactly, Meng. They use a training-free adaptation cookbook featuring masked cross-modal prompting, behavioral saliency prediction, and knowledge-aware re-ranking to help the LMM focus on those subtle anomaly cues even with sparse visual samples.

Jane: That’s interesting—training-free means we don't need tons of new data to adapt it for these specific tasks. What kind of prompts are they using to get that initial focused attention?

Lu: They use the cookbook to replace key verbs and color attributes in the original query with structured placeholders, like "and" or "is," creating a masked query. Then they feed candidate images into the LMM, instructing it to fill those placeholders based on what’s visually present.

Meng: So you’re basically letting the image provide the evidence to complete your text prompt? That sounds risky if the visual evidence is ambiguous.

Title and authors: Tom: It’s not risky because they have a second step afterward. After that, they compare the original query with all the image-specific completions using another LLM to do a semantic comparison and then re-rank everything based on how many placeholders matched semantically.

Jane: That semantic comparison step sounds really smart for making sure it's not just matching keywords like "orange" when the context is wrong. It’s about the meaning, right?

Lu: Precisely. The re-ranking isn't just about counting matches; they use a weighted combination of the initial list score and this new semantic re-ranking list score to get a final ranking.

Meng: How does that scoring formula actually work? Is it just adding them up, or is there some decay involved?

Tom: They use a score fusion strategy where you combine the initial retrieval list with the semantically obtained reranking list using an exponential decay function to get that final score. It’s a weighted combination of those two lists.

Jane: So, to summarize this paper, AnomalyLMM is proposing a unified pipeline that turns the anomaly search into a masked completion and re-ranking process, leveraging LMM reasoning without any extra training.

Lu: Yeah, and their main contributions are that coarse-to-fine pipeline that uses generative knowledge with discriminative detection, and this training-free cookbook to highlight cues. They also benchmarked it rigorously on the PAB dataset, which is the only public one for this task, showing a zero point ninety-six percent Recall at one accuracy improvement over baselines.

Meng: A zero point nine six percent recall improvement sounds promising if it’s truly training-free and works on real data. But what about when you look at the limitations? What doesn't this framework handle well?

Tom: The authors state clearly that the performance of AnomalyLMM still depends heavily on the capabilities of the underlying LMMs they use, so it inherits any limitations those models have regarding fine-grained visual-language understanding.

Jane: That makes sense. So, while it’s a solid improvement over what was out there, it isn't a complete fix for every possible visual subtlety that could trip up an LMM.

Lu: Future work they suggest includes integrating dynamic anomaly memory banks and exploring few-shot adaptation to handle rare behaviors better. They also think the framework could be extended to dynamic multi-camera environments for real-time detection in cities, and coupling it with decision support systems for proactive responses.

Meng: Real-time in urban infrastructure sounds complex to deploy at scale, but if you can get that kind of accuracy on edge devices, that’s valuable. What about privacy concerns?

Lalam: The paper mentions they plan to investigate federated learning paradigms specifically to address privacy concerns when deploying this at a city scale.

Tom: So, we’ve covered the pipeline, the training method, and where it stands against other work on this topic using the AnomalyLMM framework. We're going to take a quick break before we talk about what these results actually mean for how we use AI in public safety systems.

The paper's summary: Tom: So, we've talked about how AnomalyLMM uses a coarse-to-fine pipeline to find abnormal people using Large Multi-modal Models, and now we need to talk about what the authors actually claim in their summary.

Jane: Right. So basically, the big takeaway is that they aren't just trying to make a general image generator better; they are specifically building a system for retrieval that can handle those tricky, subtle human anomalies when you describe them in text.

Lu: They introduce this "coarse-to-fine" pipeline which moves past just feeding the text query directly into the LMM. Instead, they narrow down the search space first with a standard model, and then they use the LMM’s reasoning power to refine that list progressively.

Meng: So, if I'm looking at this from a deployment standpoint, what does that coarse step actually do for us in practice? Does it save compute?

Tom: It saves compute by filtering out millions of irrelevant images before the more expensive reasoning steps kick in. It's like sorting through a massive pile of photos to find the relevant ones before you start reading every caption closely.

Jane: And then they use this "masked cloze style reasoning task" to do that fine-grained alignment, which means they prompt the LMM to fill in missing details based on the visual evidence it sees. It’s like asking the AI, "I see a person balancing something, what word should I put there?"

Lu: Exactly. That’s where they leverage the LMM's generative knowledge to anchor itself to what is visually present, even if the text description is vague. They use a training-free cookbook for this—masked cross-modal prompting and behavioral saliency prediction—to get zero-shot focus on those subtle cues.

Meng: Training-free sounds good for development speed, but how sure are they that this works well when the anomaly is really rare? We deal with things that don't look like anything we've seen before.

Tom: That’s where their re-ranking comes in. They use a semantic comparison between the original text query and the LMM’s visual completion to figure out if they are actually describing the same thing, not just using similar words. Then they rank everything based on how many of those semantically matched placeholders there are.

Jane: So, it's not just looking for keywords; it's checking if the *meaning* matches across the text and the picture. That semantic comparison step is crucial because it makes the system robust to different ways you might phrase an anomaly description.

Lu: And they combine that initial list score with this new re-ranking list score using a weighted combination, which means they get a final ranking that balances both stages of retrieval. It’s not just one method, it’s two methods working together.

Tom: The results on the PAB dataset show this approach beats the baseline by zero point nine six percent in recall at one accuracy level. That's a concrete number we can look at, which is what I need to see.

Meng: A zero point nine six percent improvement is respectable if it holds up outside of that specific benchmark set. What about the limitations they admit?

Jane: They are quite honest: the performance still depends on how good the underlying Large Multi-modal Models are at understanding both language and vision together, so it inherits any weaknesses those models have in that fine-grained understanding.

Lu: Their future work includes things like adding dynamic anomaly memory banks to remember past behaviors, and exploring few-shot adaptation for those rare behaviors they mentioned. They also see this framework being extended to multi-camera environments for real-time detection across urban infrastructure.

Tom: So, the big picture is that while this AnomalyLMM framework gives us a better way to use LMMs for retrieval, we still have to trust the base model's visual understanding and plan for more memory and adaptation techniques. Next up, we're going to look at how this kind of focused retrieval fits into the broader picture of AI in public safety systems.

The paper's improvements: Tom: We just covered how AnomalyLMM works by looking at its pipeline, and now we need to talk about what improvements the authors are suggesting for this research direction.

Jane: So, what’s the next step in making this system better? What fixes are they proposing for where it falls short?

Lu: They're focusing on making the LMM operate more like a retrieval tool rather than just a creative one. The idea is to integrate that coarse-to-fine retrieval pipeline even deeper, so the LMM focuses more on finding specific anomalies instead of just generating general images.

Meng: So they want to stop it from just being a fancy image generator and start making it good at discriminating between normal and abnormal people? That would be useful for production because we need precision.

Tom: Exactly. They’re suggesting that the way they build the training signal, using that masked cross-modal prompting, should be made even more targeted to help zero-shot focus on those subtle anomaly cues even when you don't have a lot of examples.

Jane: That sounds like they want to refine the training cookbook so it’s less about general prompting and more about specifically highlighting what makes an image an anomaly.

Lu: And they propose making that comparison and re-ranking step even better by shifting from surface-level matching to a true semantic comparison, which means matching concepts like similar verbs or colors, not just exact words.

Meng: Semantic comparison is key if we want it to work across different ways people describe the same strange behavior. That robustness against phrasing changes is something I can actually think about when we try to deploy this in a real-world scenario.

Tom: And they aren't stopping there either; they’re looking at making things dynamic, suggesting integrating dynamic anomaly memory banks so the system can remember patterns over time instead of just looking at one frame.

Jane: So it moves from a single snapshot analysis to something that understands context and history, which is important for long-term monitoring. It means the AI wouldn't forget what it saw yesterday when looking at today's image.

Lu: And they also think this framework could be extended to dynamic multi-camera environments, meaning you could have the system constantly checking several feeds at once using natural language queries. That opens up possibilities for real-time anomaly detection across a whole city infrastructure.

Meng: Real-time across multiple cameras is ambitious because of the computational load, but if they can keep it efficient on edge devices, that’s a huge win for deployment cost and latency.

Tom: And finally, they are looking into federated learning paradigms to tackle privacy issues when deploying this at a city scale where you have sensitive data. That addresses the ethical side of using this kind of surveillance technology.

Jane: So the improvements focus on making it smarter—more focused, more context-aware, and more capable of handling real-world deployment challenges like memory and privacy. This paper is showing us the direction for taking these LMMs from cool experiments to practical tools.

Conclusion: Tom: So we're wrapping up this look at AnomalyLMM: Bridging Generative Knowledge and Discriminative Retrieval for Text-Based Person Anomaly Search, by summarizing its implications for AI in public safety systems.

Jane: Basically, they show us a way to use Large Multi-modal Models not just to create pictures, but to actually find specific human anomalies when we describe them in text.

Lu: The paper proves that you can use a training-free approach with this coarse-to-fine pipeline and semantic re-ranking to get a measurable improvement in recall, specifically zero point nine six percent at one accuracy level on the PAB dataset.

Meng: That number is solid, but we have to remember that it's on their benchmark data. The real question for me is how much this system actually breaks when it hits production traffic where the anomaly descriptions are messy and unstructured.

Lalam: I’m confident in my assessment, Tom and Jane. Based on the literature I’ve seen, this method is very sure that it provides a more robust way to link text descriptions to visual evidence than what we have now.

Tom: It does sound promising for building more precise systems, but we still have to deal with the limitations they mentioned regarding how much the underlying LMM's visual understanding limits the final performance.

Jane: Right, so it’s a step forward, not a complete solution for every possible visual subtlety in real life. But the idea of using semantic comparison instead of just keyword matching is really interesting for how we design these systems going forward.

Lu: It shifts the focus from simple pattern matching to actual meaning alignment between what you read and what you see. That's a significant change in how we think about multimodal retrieval tasks.

Meng: For me, the future work they mentioned, like integrating dynamic anomaly memory banks for long-term behavior tracking, is where I’ll be looking next if I wanted to deploy this kind of thing at scale and keep it from getting stale.

Tom: Exactly. So while AnomalyLMM gives us a concrete method and a number to compare against, the next phase is making it more adaptive and robust for messy real-world data streams.

Jane: That leads perfectly into our next topic, where we're going to look at some papers that deal with how these models handle long conversations and context windows in role-playing agents.

More episodes

← Home