Meme Template Identification in the Wild: Comparing Methods for Semi-Open-Set Recognition

arXiv:2408.08126 · cs.CY · Submitted 2024-08-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Meme Template Identification in the Wild".

Paper summary: Tom: So we're looking at this paper today called "Meme Template Identification in the Wild: Comparing Methods for Semi-Open-Set Recognition". Basically, they're tackling how to spot meme templates on social media.

Jane: That sounds like something every internet user has wondered about: how do you know when an image is just a random picture and when it’s actually part of a recurring meme format?

Tom: Exactly. The core idea here is that most existing methods are trained on clean, controlled sets of templates, but those sets don't work well in the messy world of actual social media feeds.

Lu: They formalize this as a semi-open-set recognition problem because you have to do two things: correctly classify the templates you already know, and also correctly reject everything else that isn't a template or is just random content.

Meng: That sounds complicated for production. So what's the big picture they're trying to show?

Jane: They are pointing out that in real life, only about twenty-one percent of images actually use a known template, but fifty-eight percent are just template-free content, which is way more common than you might think.

Tom: The paper sets up three research questions to test different approaches: performance in a controlled setting, performance in the real world where things are mixed up, and how efficient these methods are when you try to use them at scale.

Lu: They compare a variety of supervised and unsupervised methods across those three settings.

Meng: Which part is most important for someone actually trying to deploy this stuff? Is it getting a high score in the perfect lab test or handling the noise?

Jane: That's what they're testing with research question number two, which looks at how robust these models are when they see a mix of template-based memes, template-free content, and just regular images.

Tom: And research question number three focuses on efficiency and scalability. They want to see the trade-offs between getting good results and how much time or computing power it takes to run everything on massive datasets.

Lalam: I see they’re using a specific dataset called ImgFlip, which has a known list of templates, for the controlled setting, right?

Jane: That's right. ImgFlip is used for research question one, which is the performance in a controlled setting where you have all those known templates.

Tom: So for that first test, they are looking at how well supervised and unsupervised methods perform when classifying known meme templates on that clean dataset.

Paper summary: Lu: They use two different evaluation settings. Setting one is the controlled set with one thousand seven hundred four known templates to see classification performance in a closed-set environment.

Meng: What about the real world? That's where things get tricky because it’s not just clean data anymore, right?

Jane: It is. Research question two assesses model robustness on a manually labeled set of one thousand images sampled from Reddit, X, and Facebook to see if they can both classify memes and reject non-memes.

Tom: They also look at how the methods handle those things—they score not just correct template classification but also correct rejection of non-template based content.

Lalam: And for the efficiency part, they are looking at practical tradeoffs across different identification methods when they're applied to large scale datasets.

Meng: I'm curious about what kind of algorithms they are actually testing there. Are we talking about running a few models against everything, or something more streamlined?

Tom: They compare things like the fused SigLIP2 and DINOv2 embedding that combines language and visual structure, against a retrieval-augmented LLM pipeline that prunes the template inventory down to a smaller set before verification.

Lu: That retrieval-augmented approach is interesting because it tries to narrow down the search space early in the process.

Jane: And they also have some established literature baselines they are comparing against, which gives us a good idea of what's already out there.

Tom: The paper introduces a fused SigLIP2 and DINOv2 clustering that combines two different embedding techniques for their unsupervised methods.

Lu: It’s designed to combine the pooled SigLIP2 output and the DINOv2 CLS token at two hundred twenty-four pixels, which are then combined by weighted concatenation.

Meng: So what's the payoff here compared to just using one embedding technique alone? Is that combination worth the complexity?

Jane: The results show that representation choice is really decisive in performance because semantic embeddings substantially outperform perceptual hashing, which is more vulnerable to real-world cropping and recombination.

Tom: That’s a big point. They found that the fused SigLIP2 and DINOv2 clustering achieved a Matthews Correlation Coefficient of zero point nine four zero in the controlled setting without even using template labels during discovery, which is close to their strongest supervised methods <ref:2408.08126#pg1>.

Lu: That is significant because it shows you can find good representation without needing those expensive ground truth templates for the initial discovery phase.

Paper summary: Meng: But then they move to research question two, and the best closed-set Matthews Correlation Coefficient drops to zero point six eight four, achieved by GPT-five point six Luna. So how big of a drop are we talking about when we move from perfect lab conditions to real social media data?

Jane: That drop is pretty noticeable, but the paper notes that this difference is mainly driven by the decision between template-based and template-free content rather than just the naming of the templates themselves.

Tom: It sounds like over-prediction dominates their error budget in those real-world scenarios, meaning most of their false positives are actually just template-free memes instead of completely random stuff.

Meng: So when you're deploying this, you have to be really careful about setting your threshold because the noise is heavily skewed towards things that look like templates but aren't.

Jane: They also measured a weighted F1 score of zero point four nine five for the overall model performance on real-world data, which shows they aren't perfect at distinguishing everything from each other in that mixed environment.

Lu: The authors acknowledge their limitations right there: they are focused only on visual templates, excluding video and text driven formats, and there’s a gap between static training datasets and the constantly evolving live feeds.

Tom: They also used a one thousand image annotated sample for research question two, which limits how much subgroup analysis you can do with that specific part of the data.

Meng: So what does this mean for someone who just wants to use this to flag something quickly? Is Luna the answer for modest volume settings?

Jane: The paper suggests Luna is suitable where the decision between template based versus template free has to be reliable, especially if you're flagging pipelines that have costly false positives.

Tom: Overall, "Meme Template Identification in the Wild: Comparing Methods for Semi-Open-Set Recognition" provides a comprehensive framework for comparing vision only identification methods across controlled, real world, and efficiency settings.

Lu: It also provides an extensive dataset of one point one million social media images that they soft labeled with the template and meme nonmeme predictions from all the methods they evaluated.

Meng: That dataset sounds valuable if it helps us build better systems for deployment because it gives us something to test against before we even start building our own stuff.

Jane: So, if you're looking at how to move beyond just looking at individual memes and start understanding the patterns of templates across a whole corpus, this paper gives a solid baseline for computational memetics.

Conclusion: Tom: So we've been talking about how researchers are trying to figure out if an image is a meme template or just random noise on social media, and this paper, "Meme Template Identification in the Wild," puts a lot of different methods head-to-head.

Jane: Exactly. It’s looking at how those identification methods actually do when you take them out of the controlled lab and put them on real internet feeds. The authors are comparing several techniques to see which one holds up best under messy, real-world conditions where things are all mixed up and not perfectly labeled.

Lu: They frame this as a semi-open-set recognition problem, meaning the models have to know both the templates they were trained on and reject everything else that isn't a template. It’s not just one task, it’s a classification plus a rejection task.

Meng: From my side, what I find interesting is that they aren't just testing if you can find the right pattern; they are testing how fast you can run the system when you have millions of images to check. Efficiency matters for deployment.

Tom: Right, so what they show is that representation choice makes a huge difference. The way they combine different image features—like combining semantic language embeddings with visual structure embeddings—is actually where the best results come from, even without having perfect labels for every single template.

Jane: And the numbers show a gap between what happens in a perfect test set and what happens when you look at actual social media data. The performance drops when they move from that clean ImgFlip dataset to real Reddit or Facebook images.

Lu: They found that some methods, like this fused SigLIP2 and DINOv2 approach, can find good patterns without using template labels during the initial discovery phase, which is pretty smart because labeling everything takes forever.

Meng: But the trade-off is clear: you get better discovery power by using more complex embeddings, but you have to deal with slower processing times when you try to scale it up.

Tom: So, the authors are presenting a framework that lets us compare these different ways of identifying memes—supervised versus unsupervised—across three distinct settings: controlled testing, real-world testing, and speed.

Jane: It really helps for anyone trying to build something practical because it shows you exactly what kind of accuracy you can expect when dealing with the noise and variety found online.

Lu: The dataset they created alongside the methods is also a big contribution. Having one point one million images soft-labeled with all these predictions gives people a lot to work with for following up on this research.

Tom: It’s definitely a solid baseline for anyone trying to tackle computational memetics, showing that combining different AI techniques can give you much more reliable results in the wild than relying on just one method.

Jane: And that brings us to what they suggest next: looking beyond just static images to see how these template ideas evolve as trends change in real-time.

Department of Math and Computer Science, Augustana College · Department of Stochastics, Budapest University of Technology and Economics · University of Colorado Boulder

cs.CY

Submitted: 2024-08-15

Updated: 2026-11-18

Comments: 18 pages, 4 figures

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

The gist: Image-with-text memes are a dominant form of online communication, and much of their spread happens through meme templates which users adapt with new text or imagery The paper formalizes meme…

Key concepts

Semi-Open-Set Recognition
This is a machine learning problem where the system must recognize known patterns (templates) while simultaneously rejecting any new or unseen patterns that do not match those known templates. It's harder than simple classification because there are infinitely many possible 'new' memes, requiring robust methods to handle novel content.
Fused SigLIP2+DINOv2 Embedding
This is a powerful visual representation technique that combines two different deep learning models. It merges the semantic understanding from SigLIP2 with the structural information from DINOv2. This fusion creates a richer feature that captures both what an image *means* (semantics) and how it is *visually structured*, leading to superior template identification.
Template-Free Content
This refers to images found in real-world social media that do not use any recognizable meme template. The paper highlights that distinguishing these 'template-free' images from actual memes is a key challenge, as this distinction heavily influences the overall performance metrics of the identification models.
Retrieval-Augmented LLM Pipeline
This method uses a large language model (like GPT-5.6 Luna) to first narrow down potential meme templates by searching a precomputed inventory. This pruning step reduces the search space before verification, making the process more efficient and focused on highly relevant candidates.

Terminology

Summary

Image-with-text memes are a dominant form of online communication, and much of their spread happens through meme templates which users adapt with new text or imagery The paper formalizes meme template identification as a semiopen-set recognition problem, requiring methods to both classify known templates and reject template-free memes and non-meme content In real-world social media sources dedicated to meme sharing, only 21% of images use a known template, while 58% are template-free and 21% are not memes at all The main contributions of this paper are threefold: first, it formalizes meme template identification as a semi-open-set recognition problem rather than a closedset classification task and provides the first comprehensive framework for comparing vision-only identification methods across a controlled setting (RQ1), a real-world heterogeneous setting (RQ2), and efficiency/scalability (RQ3) Second, it proposes and evaluates two novel entrants alongside established literature baselines: a fused SigLIP2+DINOv2 embedding that combines language-aligned semantic similarity with layout-sensitive visual structure, and a retrieval-augmented LLM pipeline that prunes the template inventory to a small candidate set before verification Third, it provides an extensive dataset of 1. p. 1

Where the paper says it

First and foremost, we formalize meme template identification as a semi-open-set recognition problem rather than a closedset classification task, and provide the first comprehensive framework for comparing vision-only identification methods across a controlled setting (RQ1), a real-world heterogeneous setting (RQ2), and efficiency/scalability (RQ3). Second, within this framework, we additionally propose and evaluate two novel entrants alongside established literature baselines: a fused SigLIP2 and DINOv2 embedding that combines language-aligned semantic similarity with layoutsensitive visual structure, and a retrieval-augmented LLM pipeline that prunes the template inventory to a small candidate set before verification, avoiding the need to present the full label space to the model at inference time.

1 million social media images, soft-labeled with the template and meme/non-meme predictions produced by the methods evaluated in this paper, along with code to support reproducibility and follow-up work

Pruning redundant reasoning tokens

The study addresses three research questions (RQs): RQ1: Performance in a Controlled Setting (meme template classification) How effectively do different supervised and unsupervised methods perform at classifying known meme templates when evaluated on a large, clean dataset composed solely of template-based memes (i.e., ImgFlip)? p. 0

Where the paper says it

We formalize meme template identification as a semiopen-set recognition problem, requiring methods to both classify known templates and reject template-free memes and non-meme content. We introduce an evaluation framework spanning a controlled setting (1,704 ImgFlip templates) and a heterogeneous real-world social media sample, comparing supervised CNN- and distance-based methods, unsupervised density-based clustering, a novel fused SigLIP2+DINOv2 representation, and a retrieval-augmented LLM pipeline.

RQ2: Performance in a Real-World Setting (meme template identification) How robust are these models in a realworld scenario when confronted with a heterogeneous mix of template-based, template-free, and non-meme images? p. 1
Where the paper says it

, ImgFlip)? RQ2: Performance in a Real-World Setting (meme template identification) How robust are these models in a realworld scenario when confronted with a heterogeneous mix of template-based, template-free, and non-meme images?

Specifically, how well do they distinguish known templates from the broader category of “template-free” content? p. 1
Where the paper says it

RQ2: Performance in a Real-World Setting (meme template identification) How robust are these models in a realworld scenario when confronted with a heterogeneous mix of template-based, template-free, and non-meme images? Specifically, how well do they distinguish known templates from the broader category of “template-free” content?

RQ3: Efficiency and Scalability What are the practical tradeoffs between prediction performance, time efficiency, and scalability across the various identification methods when applied to large-scale datasets? p. 1
Where the paper says it

Specifically, how well do they distinguish known templates from the broader category of “template-free” content? RQ3: Efficiency and Scalability What are the practical tradeoffs between prediction performance, time efficiency, and scalability across the various identification methods when applied to large-scale datasets?

The methodology combines a structured, labeled source (ImgFlip) with unstructured, unlabeled social media data (Reddit, Facebook, X), and implements trains and evaluates a wide array of supervised and unsupervised methods against an evaluation framework that scores not only correct template classification but also correct rejection of non-template-based content

How the training signal is built

The researchers utilize two distinct evaluation settings: Evaluation Setting 1 evaluates models on the ImgFlip dataset with 1,704 known templates to study classification performance in a controlled, closed-set environment Evaluation Setting 2 assesses model robustness on a manually labeled set of 1,000 images sampled from Reddit, X, and Facebook to require models to both classify template-based memes and reject non-template-based content For supervised approaches in the controlled setting, models are fit on the 80% portion of ImgFlip, with hyperparameters selected on a stratified 10% validation subset, and the held-out 20% was never seen during training or selection Unsupervised methods like HDBSCAN are applied to CLIP and pHash embeddings, first reduced with UMAP to preserve global structure, then clustered with HDBSCAN for its noise handling The fused SigLIP2+DINOv2 clustering combines the pooled SigLIP2 output and the DINOv2 CLS token at 224px, which are then combined by weighted concatenation

Where the gains come from

The results indicate that representation choice is decisive in performance, as semantic embeddings substantially outperform perceptual hashing, which is vulnerable to real-world cropping and recombination Specifically, fused SigLIP2+DINOv2 clustering achieved a MCC of 0.940 in the controlled setting without using template labels during discovery, close to the strongest supervised methods In the real-world setting (RQ2), the best closed-set MCC falls to 0. p. 7

Where the paper says it

Our fused SigLIP2+DINOv2 pipeline reached MCC 0. 940 without using template labels during discovery, close to the strongest supervised methods, while producing substantially fewer, less fragmented clusters than UMAP+HDBSCAN (7,328 vs.

684, achieved by GPT-5. p. 8
Where the paper says it

974, Table 2) falls to 0. 684, achieved by GPT-5.

p. 8
Where the paper says it

974, Table 2) falls to 0. 684, achieved by GPT-5.

6 Luna, which also has the highest binary precision The paper notes that representation matters more than algorithm family because semantic embeddings achieve True Template-Based F1 scores between 0. p. 9
Where the paper says it

Representation matters more than algorithm family: semantic embeddings (our fused SigLIP2+DINOv2 clustering, Luna, rNN over DenseNet features) substantially outperform perceptual hashing, which is vulnerable to real-world cropping and recombination. The precision/recall behavior reflects threshold calibration rather than pipeline family: threshold-based hashing is most conservative (recall 0.

789 and 0.862, substantially outperforming perceptual hashing methods (0. p. 9
Where the paper says it

684), driven almost entirely by the template-based versus templatefree decision rather than template naming. Representation matters more than algorithm family: semantic embeddings (our fused SigLIP2+DINOv2 clustering, Luna, rNN over DenseNet features) substantially outperform perceptual hashing, which is vulnerable to real-world cropping and recombination.

283 to 0.504)

Cost of a two-stage gate

The efficiency analysis shows that the supervised models, the retrieval methods over precomputed representations, and all four clustering pipelines complete in under three hours per seed The two matching-based methods instead compare every query against a reference set of individual images, requiring 20.6 hours per seed for Sparse Matching and 41. p. 8

Where the paper says it

The two matching-based methods instead compare every query against a reference set of individual images, requiring 20. 6 hours per seed for Sparse Matching and 41.

1 hours for rNN GPT-5. p. 8
Where the paper says it

6 hours per seed for Sparse Matching and 41. 1 hours for rNN

6 Luna occupies a third regime bounded by API throughput rather than corpus size, processing the ImgFlip test split at just 12,201 images per hour The fused SigLIP2+DINOv2 pipeline sustains approximately 418,000 images per hour while achieving the strongest unsupervised accuracy

Model performance on real-world data

The models show poor overall performance (mean weighted F1 = 0.495) and, in the binary task of separating template-based memes from everything else, moderate recall (mean = 0. p. 8

Where the paper says it

495, max = 0. 799) and, in the binary task of separating template-based memes from everything else, moderate recall (mean = 0.

743, max = 0. p. 8
Where the paper says it

799) and, in the binary task of separating template-based memes from everything else, moderate recall (mean = 0. 743, max = 0.

981) at consistently low precision The best closed-set MCC (0. p. 8
Where the paper says it

981) at consistently low precision. The best closed-set MCC (0.

974, Table 2) falls to 0. p. 8
Where the paper says it

The best closed-set MCC (0. 974, Table 2) falls to 0.

684, achieved by GPT-5.6 Luna The drop relative to RQ1 is driven mainly by the template-based versus non-template-based decision rather than by template naming itself Over-prediction dominates the error budget, as false positives consist overwhelmingly of templatefree memes rather than non-memes The precision/recall behavior reflects threshold calibration rather than pipeline family, with Luna escaping this trade-off entirely

Limitations and future directions

Limitations include a focus on visual templates (excluding video and text-driven formats), a discrepancy between static datasets and evolving live feeds, and a 1,000-image annotated sample that limits subgroup analysis Future work should incorporate text via NLP or VLMs to address “Memetic Trends” and rapidly emerging templates (Hazman, McKeever, and Griffith 2025a) and use targeted degradation augmentation to bridge the domain gap between training sets and social media The paper concludes that Luna suits modest-volume settings where the templatebased versus template-free decision must be reliable, such as flagging pipelines with costly false positives The framework and released datasets offer a solid baseline for computational memetics

Data Collection details

ImgFlip provided the core training dataset with reliable ground-truth template labels, while Reddit, X, and Facebook provided real-world, unlabeled data serving as a discovery corpus for unsupervised methods A dedicated “NonMeme” dataset was curated following hard-negative design principles established in Related Work to effectively distinguish template-based memes from other image types The ImgFlip system of generating memes from specific templates provided the necessary data integrity for the study because KYM’s data lacked reliably accurate template-to-instance pairing

Feature Engineering details

Low-level descriptors include RGB and Grayscale Histograms for color/brightness distributions and Local Binary Patterns (LBP) for textural patterns Content-based fingerprints build on Zannettou et al. (2018) and Courtois and Frissen (2023), including Perceptual Hashing (pHash) for holistic image fingerprinting and ORB for local keypoints Deep semantic embeddings include CLIP image-encoder embeddings, a fused dual-backbone representation concatenating l2-normalized SigLIP2 and DINOv2 embeddings under mixing weight α

Model comparison details

The models compared include Multinomial Logistic Regression (MLR) trained on low-level features, Radius Nearest Neighbors (rNN) using pHash, and various fine-tuned CNN architectures such as ResNet-18, EfficientNetV2, and DenseNet Unsupervised approaches include HDBSCAN applied to CLIP and pHash embeddings reduced with UMAP to preserve global structure The LLM-Based Approach uses GPT-5.6 Luna, employing a retrieval step that narrows the candidate set using a weighted combination of six low-level descriptors

Evaluation metrics

Performance is measured using Matthews Correlation Coefficient (MCC), Cohen’s Kappa (κ), and the weighted F1 score, which are well-suited for imbalanced multiclass problems For RQ2, binary metrics (precision, recall, and F1 of the templatebased class) are used to evaluate each model’s ability to distinguish template-based memes from anything else The paper reports inter-annotator agreement for the 1,000-image social media sample in Evaluation Setting 2, with Fleiss’ κ = 0.693

ImgFlip provided the core training dataset with reliable ground-truth template labels, while Reddit, X, and Facebook provided real-world, unlabeled data serving as a discovery corpus for unsupervised methods The NonMeme dataset was curated following hard-negative design principles established in Related Work to effectively distinguish template-based memes from other image types The ImgFlip system of generating memes from specific templates provided the necessary data integrity for the study because KYM’s data lacked reliably accurate template-to-instance pairing

Performance is measured using Matthews Correlation Coefficient (MCC), Cohen’s Kappa (κ), and the weighted F1 score, which are well-suited for imbalanced multiclass problems For RQ2, binary metrics (precision, recall, and F1 of the templatebased class) are used to evaluate each model’s ability to distinguish template-based memes from anything else The paper reports inter-annotator agreement for the 1,000-image social media sample in Evaluation Setting 2, with Fleiss’ κ = 0.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Semi-Open-Set Recognition Framework: Implement a framework that formally identifies meme template identification as a semi-open-set recognition problem, requiring methods to both classify known templates and reject template-free memes and non-meme content.

  2. Representation Superiority for Discrimination: Prioritize the use of fused SigLIP2+DINOv2 or GPT5.6 Luna retrieval augmentation when distinguishing between template-based memes and template-free content, as these semantic embeddings substantially outperform perceptual hashing, which is vulnerable to real-world cropping and recombination.

  3. Robust Rejection Criterion: Adapt models designed for closed-set classification to support a common protocol for template-free/non-meme rejection by using distance-based methods like Radius Nearest Neighbors (rNN) or unsupervised clustering where a cluster is named by majority vote over its labeled members, and clusters without labeled members are rejected.

  4. LLM Verification Pipeline: Employ a retrieval-augmented LLM pipeline for semi-open-set identification, utilizing a two-stage process where GPT5.6 Luna performs an initial classification (V1) followed by a verification stage (V2) that "checks every V1 prediction in a separate model call: the verifier sees the query side by side with the reference image of the selected template and decides whether the two show the same underlying template, answering yes or no."

  5. Cost-Effective Self-Hosted Default: Use the fused SigLIP2+DINOv2 pipeline as a strong self-hosted custom default for high accuracy, as it is statistically indistinguishable from rNN over DenseNet features on the overall metrics while incurring a lower per-image cost than API calls.

Abstract

Image-with-text memes are a dominant form of online communication, and much of their spread happens through meme templates which are recurring visual formats that users adapt with new text or imagery. Most prior work scores memes individually for engagement or harmful content, an approach that is structurally blind to templates. Templates might amplify these patterns and enable coordinated harassment, which becomes visible only when memes are grouped by shared template. We formalize meme template identification as a semi-open-set recognition problem, requiring methods to both classify known templates and reject template-free memes and non-meme content. We introduce an evaluation framework spanning a controlled setting (1,704 ImgFlip templates) and a heterogeneous real-world social media sample, comparing supervised CNN- and distance-based methods, unsupervised density-based clustering, a novel fused SigLIP2+DINOv2 representation, and a retrieval-augmented LLM pipeline. In real-world social media sources dedicated to meme sharing, we find that only 21% of images use a known template, while 58% are template-free and 21% are not memes at all; identification performance also drops sharply from the controlled setting (best MCC 0.974 to 0.684). The loss comes mainly from deciding whether an image is a template-based meme at all, rather than identifying the template. Our dataset of 1.1M soft-labeled social media images, together with code for reproducibility, is available upon request.

Sources

Related papers