Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Black-Box Optimization of Mixed Binary-Continuous Variables".
Jane: Here's the gist of the paper. Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training The DFS merging problem is formally characterized as a mixed binary-continuous black-box optimization problem identifying three fundamental challenges Empirical evaluation shows that all tested merging methods fail to surpass the stronger source model on official benchmarks The structured DFS approach achieves a 49.5% reduction in the effective search space Results reveal that all tested merging methods fail to surpass the stronger source model on official benchmarks.
Tom: Problem Formulation and Characterization The DFS merging problem is defined as minimizing a function subject to mixed variable constraints The variables involved are the binary layer selection vector z in N and continuous scaling weights x in R N Three fundamental challenges characterize this problem: (1) mixed binary-continuous variable types, (2) high-dimensional search spaces with up to 291 parameter groups, and (3) conditional variable dependencies corresponding to Type-I interaction.
Jane: Proposition 1 demonstrates that DFS merging with binary layer selection exhibits Type-I interaction.
Tom: Search Space Reduction and Interaction The structured DFS approach explicitly models the dependency by sampling z ∼ Bernoulli(p) N and optimizing x only for active groups This structured approach achieves a 49.5% reduction in the effective search space (mean 146.96 active layers out of 291 total).
Title and authors: Tom: So today we're looking at this paper called "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging." It’s about how we can combine two large language models without retraining them from scratch.
Jane: That sounds like a really practical problem, Tom. What exactly is this DFS merging thing they’re talking about? Is it just averaging the weights?
Lu: It's actually more than that. They look at selective activation of parameter groups across model layers instead of averaging everything uniformly <ref:2605.12326#pg2>. It allows for finer-grained knowledge transfer because you only pick certain parts of the model to merge together.
Meng: Selective activation sounds flexible, but how much flexibility are we actually talking about? If it's just picking a few parameter groups, does that mean we can really skip a lot of the learning? What happens when things break at scale on real traffic?
Tom: Well, the paper formally characterizes this whole process as a mixed binary-continuous black-box optimization problem <ref:2605.12326#pg1>. They found three main hurdles: first, you have those mixed binary and continuous variables together, second, the search space is massive with up to two hundred ninety-one parameter groups, and third, there are these conditional dependencies where the continuous parts depend on which binary groups you picked <ref:2605.12326#pg1>.
Jane: So it’s not just a simple average; it's this complex optimization problem where you have to decide which layers to use, and then how much to blend them? That sounds like a lot of variables for the system to handle.
Lu: Exactly. And they point out that this mixed nature leads directly into Type-I interaction <ref:2605.12326#pg3>. This means the effective continuous parts of the problem actually depend on your binary choices, which complicates things a lot <ref:2605.12326#pg3>.
Meng: So if that dependency is strong, it makes standard optimization tools pretty unreliable, right? What tools are they testing against this complexity?
Tom: They tested several methods and found that none of the existing merging approaches managed to beat the stronger source model on official benchmarks <ref:2605.12326#pg1>. This suggests the underlying optimization structure itself is where the difficulty lies, not just a simple tuning issue.
Jane: That’s kind of a sobering result for model merging, Tom. If all these methods fail to surpass the source model on things like GSM8K or MMLU, what does that imply about how we should be approaching this?
Lu: It implies we need a fundamentally different approach to optimization algorithms because the standard ones aren't set up right for this specific mixed variable problem <ref:2605.12326#pg3>. They are suggesting that the theoretical tool for this is IGBD, which they say is designed specifically to handle Type-I interaction through coupled bilevel optimization <ref:2605.12326#pg3>.
Title and authors: Meng: IGBD sounds complicated. How does that translate to something we can actually deploy without needing a super powerful research team just for the setup? What’s the real cost of this theoretical solution?
Tom: The structured approach they tested, by sampling those binary selections using a Bernoulli distribution, managed to reduce the effective search space by about forty-nine point five percent <ref:2605.12326#pg2>. That means instead of looking at all two hundred ninety-one groups every time, you only focus on about one hundred forty-seven active layers on average <ref:2605.12326#pg3>.
Jane: So they managed to cut down the search space significantly just by structuring how they sample the binary choices? That’s a big win for making the problem tractable.
Lu: It also showed that the probability you pick for those binary selections affects things in a specific way, because when you increase that probability, the variance actually decreases <ref:2605.12326#pg3>. That’s exactly what Type-I interaction predicts—the behavior is conditional on your sampling decision.
Meng: If the variance drops as you sample more aggressively, that means we might want to be very selective about which layers we activate during the merge process <ref:2605.12326#pg3>. Does this mean we should prioritize stability over exploring every possible combination?
Tom: That’s a good question, Meng. The paper suggests that for future work, you need multi-objective optimization techniques to balance task specific knowledge against general knowledge when merging Improv two <ref:2605.12326#pg1>. It’s not just about finding *a* model; it's about finding the right trade-off.
Jane: And they also highlighted a really important point about starting with the right source model, which is something we saw in their evaluation Improv three <ref:2605.12326#pg1>. They noted that using a stronger base model, like Model B over Model A, makes a big difference because the weak base model scores very poorly on GSM8K at only six point six seven percent compared to Model B’s thirty-six point seven seven percent Improv three.
Lu: That supports the idea that starting with a superior foundation is critical for these merging operations to succeed, which ties back into the bigger picture of model selection Improv three <ref:2605.12326#pg1>.
Meng: So it seems like if we want this to work in production, we need two things: either a mathematically rigorous framework like IGBD, or at least a very smart way to select our starting models and sample our layer activations Improv one four. What about the gap between testing and real-world use?
Tom: There’s also this issue with what they tested versus what people actually see in the real world Improv two <ref:2605.12326#pg1>. The CMA-ES approach showed relatively stronger performance on MMLU than on GSM8K during their proxy evaluations, but that doesn't always translate to official benchmarks Improv two <ref:2605.12326#pg1>.
Title and authors: Jane: So even when an algorithm performs well on a test set, like MMLU, it might not perform as well where it really matters for math reasoning tasks like GSM8K Improv two <ref:2605.12326#pg1>. That gap between proxy performance and official benchmark results is something we need to keep watching.
Lu: And that's why the next step should probably involve developing those multi-objective merging formulations we discussed, so we can actually balance the different types of knowledge during the merge process Improv two <ref:2605.12326#pg1>.
Meng: For me, I’m focused on what actually breaks when you try to deploy this. If we use these complex binary selections, how do we keep the pipeline stable? We need something that handles that scaling issue efficiently.
Tom: The paper points toward applying IGBD to DFS merging as the natural next step because it directly tackles that Type-I interaction structure Improv one <ref:2605.12326#pg1>. It’s the theoretically appropriate algorithm for what they’ve defined as a core challenge <ref:2605.12326#pg3>.
Jane: So, if we distill this down for someone listening who doesn't read papers, it means that model merging isn't just about mixing things up; it’s about managing a very specific type of interaction where the choices you make at one level affect the quality of everything below it <ref:2605.12326#pg3>.
Lu: Right. It’s about understanding that dependency structure, formalizing it, and then using optimization tools that are built to handle that exact kind of coupling <ref:2605.12326#pg3>.
Meng: If we had to deploy this today, I’d probably start by picking the strongest base model available just to avoid losing crucial instruction-following knowledge during the merge Improv three <ref:2605.12326#pg1>.
Tom: That makes sense for stability. So, to wrap up this discussion on "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging," we see that while merging offers a way to combine model capabilities, the complexity comes from the mixed variable types and the conditional dependencies <ref:2605.12326#pg1>.
Jane: The paper shows that standard methods struggle because they don't account for how those binary choices actually change the continuous optimization space <ref:2605.12326#pg3>. It gives us a direction toward using bilevel optimization frameworks like IGBD to solve this specific interaction problem <ref:2605.12326#pg3>.
Lu: And for the future, we need to look at how we can balance general knowledge versus task-specific knowledge in those merging steps through multi-objective methods Improv two <ref:2605.12326#pg1>. That seems like where the next big theoretical push needs to go.
Meng: From an engineering standpoint, I’d want to see a practical way to sample those layers that keeps the training pipeline stable at scale, maybe something that ties into how variance decreases with probability Improv four.
Title and authors: Tom: So we're looking at formal characterization, showing the search space reduction of forty-nine point five percent for structured sampling, and pointing toward IGBD as the right theoretical tool for this problem <ref:2605.12326#pg3>. It’s a lot to take in about how tricky these interactions are.
Jane: It really shows that when we look at model merging, we have to be super precise about what variables are discrete and what variables are continuous because the interaction between them is the real bottleneck <ref:2605.12326#pg1>.
Lu: Yeah, and remembering that starting with a stronger base model makes a huge difference for performance on things like math reasoning Improv three, it’s important context to keep in mind when we think about implementation <ref:2605.12326#pg1>.
Meng: It's not just about the math; it's about making sure the deployed model actually performs well on what users care about, which is where our practical testing needs to be more rigorous than just some proxy evaluations Improv two <ref:2605.12326#pg1>.
Tom: So, in summary, the DFS merging problem is a mixed binary-continuous black-box optimization challenge characterized by three main difficulties: the variable types, the high dimensionality of parameter groups up to two hundred ninety-one and those conditional dependencies between them <ref:2605.12326#pg1>.
Jane: And while current methods aren't beating the source model on official benchmarks yet, they’ve given us a clear path forward by suggesting IGBD as the right algorithm for handling that Type-I interaction <ref:2605.12326#pg3>.
Lu: We also see evidence that selective sampling based on variance helps reduce the search space to about one hundred forty-seven active layers out of two hundred ninety-one total, which is a solid reduction <ref:2605.12326#pg3>.
Meng: And remember, for real world deployment, we need to focus on selecting a stronger base model first because that seems essential for preserving the instruction following weights during the merge Improv three <ref:2605.12326#pg1>.
Tom: That’s the practical advice. So, applying IGBD or using structured sampling with variance-based probabilities is the next logical step in tackling this DFS merging problem <ref:2605.12326#pg3>.
Jane: It really shows that understanding the structure of interaction, and then using tools designed for that structure, is key to making these merging techniques more effective Improv one <ref:2605.12326#pg1>.
Lu: And we should also keep pushing the idea of multi-objective optimization to balance different knowledge types during the process Improv two <ref:2605.12326#pg1>. That seems like a major avenue for future research.
Meng: For now, I'm focused on how we can make sure our deployment pipeline handles those binary layer selections robustly without introducing too much instability Improv four.
Tom: So that’s our take on "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging." It’s a deep dive into the mechanics of model combination and what it takes to solve the underlying optimization puzzles.
The paper's summary: Tom: So we’ve been looking at the mechanics of model merging, and now we're going to talk about what this paper is actually saying about that whole process <ref:2605.12326#pg1>.
Jane: Basically, they’re characterizing it as a mixed binary-continuous black-box optimization problem. That just means they have these variables—some are choices you make, like which model layers to use, and some are numbers you tune continuously—and the whole thing is a messy optimization challenge <ref:2605.12326#pg1>.
Lu: The core idea here is that this problem has three major hurdles. First, those mixed variable types themselves. Second, the search space is huge, with up to two hundred ninety-one different parameter groups you can pick from. And third, there’s this conditional dependency where the continuous variables actually depend on which binary choices you made <ref:2605.12326#pg1>.
Meng: That makes sense for a researcher, but what does that look like for someone trying to actually ship a model? It sounds incredibly difficult to navigate those dependencies.
Tom: Exactly. And they show that this interaction is called Type-I interaction <ref:2605.12326#pg3>. That means the continuous parts of the problem actually change based on your binary choices, which messes up standard optimization tools like CMA-ES <ref:2605.12326#pg1>.
Jane: So, what’s the big takeaway for us? They tested several merging methods and none of them managed to beat a strong source model on official benchmarks like GSM8K or MMLU <ref:2605.12326#pg1>. That’s a pretty telling result.
Lu: It shows that the current approaches just aren't set up correctly for this specific kind of interaction. They suggest that the theoretically right tool for this is IGBD, which is a bilevel optimization framework designed specifically to handle Type-I interaction <ref:2605.12326#pg3>.
Tom: IGBD sounds like heavy math, but the structured approach they tested did something interesting. By sampling the binary choices with a Bernoulli distribution, they managed to cut down the effective search space by nearly fifty percent <ref:2605.12326#pg2>. That’s about fourteen hundred layers active on average out of two hundred ninety-one total <ref:2605.12326#pg3>.
Jane: So, it gives us a concrete way to make the search space manageable by being selective about which parts of the model we look at first. But they also point out that using a stronger base model from the start makes a huge difference in performance on tasks like math reasoning <ref:2605.12326#pg3>.
Lu: Yeah, and they noted that variance in performance actually decreases as you increase the probability you select those binary layers <ref:2605.12326#pg3>. That’s exactly what Type-I interaction predicts—the behavior is conditional on your sampling decision.
Meng: So, for deployment, it sounds like we need to be very smart about how we structure our merging process and prioritize starting with a better foundation, because the standard tools just don't work for this <ref:2605.12326#pg1>.
Tom: That’s the practical angle. The paper isn't just about finding a better merge; it’s about figuring out why the current search methods fail and pointing us toward a mathematically sound framework like IGBD <ref:2605.12326#pg3>.
Jane: It really shows that when we think about model merging, we have to be extremely precise about separating our binary choices from our continuous tuning parameters because that interaction is the actual bottleneck <ref:2605.12326#pg1>.
Lu: And for future work, they suggest we need multi-objective optimization techniques to balance general knowledge against task specific knowledge during the merge steps <ref:2605.12326#pg3>. That’s where the next big theoretical push should probably go.
The paper's improvements: Tom: So we just talked about how hard it is to optimize these model merges because of those mixed variables and Type-I interaction <ref:2605.12326#pg1>. Now, the authors propose some ways to fix that difficulty.
Jane: They suggest using a structured DFS approach where you sample the binary layer choices using a Bernoulli distribution <ref:2605.12326#pg2>. This helps reduce the search space significantly, cutting it down by about fifty percent <ref:2605.12326#pg3>.
Lu: That sampling strategy is good because they showed that the performance variance actually goes down as you increase the probability of selection for those binary layers <ref:2605.12326#pg3>. It’s a way to control the interaction directly.
Meng: Controlling it by adjusting a probability sounds manageable, but what about scaling this up? If we have millions of parameters, is sampling still feasible for production pipelines?
Tom: That's the million-dollar question, Meng. The paper notes that they need multi-objective optimization techniques to balance general knowledge against task specific knowledge during the merge <ref:2605.12326#pg1>. That’s a necessary step for real-world use.
Jane: And they also stressed the importance of starting with a stronger base model, Model B over Model A, because the weaker base model scores very poorly on math tasks like GSM8K <ref:2605.12326#pg3>. That’s a practical tip for anyone trying to improve their merge quality right now.
Lu: They also mentioned that applying IGBD, which is that bilevel optimization framework, would be the most theoretically appropriate way to tackle this Type-I interaction problem <ref:2605.12326#pg3>. It’s the next logical algorithmic step.
Tom: So, it’s a mix of smart sampling for tractability and heavy math frameworks like IGBD for theoretical correctness <ref:2605.12326#pg3>. It shows that we need to be thinking about the structure of the interaction itself, not just throwing more computing power at it.
Jane: What this means for us is that instead of just trying every possible merge recipe, we need a method that understands how those choices affect the outcome <ref:2605.12326#pg3>.
Meng: I still have my reservations about the complexity of implementing something like IGBD in a production environment, but it’s clear that ignoring that interaction structure is just going to lead to more failed merges at scale <ref:2605.12326#pg1>.
Lu: Exactly. We need the theory behind the optimization, and then we need engineering solutions that respect those theoretical constraints <ref:2605.12326#pg3>.
Conclusion: Tom: So we’ve gone through the mechanics of optimizing model merges and those three major hurdles they found—mixed variables, high dimensionality, and conditional dependencies <ref:2605.12326#pg1>.
Jane: And they showed that standard methods just can't handle that Type-I interaction, which means the continuous optimization space actually changes based on your binary choices <ref:2605.12326#pg3>.
Lu: So the main suggestion is to use a bilevel optimization framework called IGBD, which is designed specifically to handle that type of dependency structure <ref:2605.12326#pg3>.
Meng: That sounds like a lot of theoretical heavy lifting, but what does it actually mean for us in terms of deployment stability?
Tom: The structured approach they tested showed we can reduce the search space by about fifty percent just by being selective with layer sampling <ref:2605.12326#pg2>. That means less time spent searching, which is good for engineering.
Jane: It also highlighted that you absolutely need to start with a stronger base model, like Model B over Model A, because the weaker starting point really hurts performance on things like math reasoning <ref:2605.12326#pg3>.
Lu: And they pointed out that variance in performance actually goes down as you increase the probability you select those binary layers <ref:2605.12326#pg3>. That’s a direct way to manage the interaction using sampling decisions.
Meng: I still have my reservations about implementing a framework like IGBD when we have to worry about running these things on real traffic where things might break <ref:2605.12326#pg1>.
Tom: The paper does flag that the structured approach is just one way to handle it, but it points toward multi-objective optimization as a necessary future direction <ref:2605.12326#pg1>.
Jane: So we need to balance task specific knowledge against general knowledge during the merge process for better results <ref:2605.12326#pg1>.
Lu: And they also noted that the gap between what they tested and what people actually see in real-world evaluations is still a big issue, especially when using proxy tasks instead of official benchmarks <ref:2605.12326#pg3>.
Meng: So the next step has to be more rigorous testing on actual usage patterns, not just on development subsets <ref:2605.12326#pg3>.
Tom: So to wrap up, this paper on "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging" shows us that the complexity comes from the interaction structure, and IGBD is the theoretical path forward <ref:2605.12326#pg3>.
Jane: It really shows that when we look at model merging, we have to be super precise about separating our discrete choices from our continuous tuning parameters because that interaction is the real bottleneck <ref:2605.12326#pg1>.
Lu: And remember that starting with a stronger base model makes a huge difference for performance on math reasoning <ref:2605.12326#pg3>.
Meng: For me, I’m focused on how we can make sure our deployment pipeline handles those binary layer selections robustly without introducing too much instability <ref:2605.12326#pg4>.
Tom: So that’s our take on "Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging." It’s a deep dive into the mechanics of model combination and what it takes to solve the underlying optimization puzzles.
Department of Computer Science and Engineering, BRAC University, Dhaka, Bangladesh · Institute of Systems and Information Engineering / Center for Artificial Intelligence Research, University of Tsukuba
cs.NE
Submitted: 2026-05-12
Updated: 2026-11-12
Code: https://github.com/Mdniloykhan/dfs-merging
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 65/100
The gist: Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training The DFS merging problem is formally…
Key concepts
- DFS Merging Problem
- This is the core challenge of combining two LLMs. It involves finding the best combination of which layers to keep (binary choice) and how much to scale them (continuous weights). The difficulty arises from these mixed variable types and complex dependencies between them.
- Type-I Interaction
- This describes a specific dependency in the optimization problem where the relationship between binary layer selection and continuous scaling weights changes based on which layers are selected. It means the optimal continuous settings depend on which discrete layers are chosen, making standard optimization difficult.
- Structured DFS Approach
- This method tackles the complexity by only optimizing continuous weights for a subset of active (selected) layers. By sampling layer selection probabilities based on performance variance, it significantly cuts down the search space from 291 total groups to about 147 active ones.
- IGBD Framework
- Information-Geometric Bilevel Decomposition is a theoretical optimization framework specifically designed to handle problems with Type-I interaction. Applying this framework could provide a theoretically sound method for solving the complex model merging problem.
Terminology
Summary
Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training The DFS merging problem is formally characterized as a mixed binary-continuous black-box optimization problem identifying three fundamental challenges Empirical evaluation shows that all tested merging methods fail to surpass the stronger source model on official benchmarks The structured DFS approach achieves a 49. p. 0 jp
October 2026
Abstract
Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training. Among existing approaches, Data
Flow Space (DFS) merging operates by selectively activating parameter groups across model layers,
offering greater flexibility than naive parameter averaging. 5% reduction
in effective search space. Results reveal that all tested merging methods fail to surpass the stronger
source model on official benchmarks, with TIES (13.Where the paper says it
Where the paper says it
Problem Formulation and Characterization
The DFS merging problem is defined as minimizing a function subject to mixed variable constraints The variables involved are the binary layer selection vector z in N and continuous scaling weights x in R N Three fundamental challenges characterize this problem: (1) mixed binary-continuous variable types, (2) high-dimensional search spaces with up to 291 parameter groups, and (3) conditional variable dependencies corresponding to Type-I interaction. Proposition 1 demonstrates that DFS merging with binary layer selection exhibits Type-I interaction. p. 2 Proposition 1. DFS merging with binary layer selection exhibits Type-I interaction.Where the paper says it
Search Space Reduction and Interaction
The structured DFS approach explicitly models the dependency by sampling z ∼ Bernoulli(p) N and optimizing x only for active groups This structured approach achieves a 49.5% reduction in the effective search space (mean 146. p. 1 Search space reduction: Our structured DFS approach achieves a 49. 5% reduction in the effective search space (mean 146. 5% reduction in the effective search space (mean 146. 96 active layers out of 291 total). 5% reduction confirmed. Right:
selection probability vs active layers showing theoretical and measured agreement.Where the paper says it
Where the paper says it
Where the paper says it
Empirical Performance and Failure Analysis
The comprehensive evaluation on official GSM8K and MMLU benchmarks reveals that all tested merging methods fail to surpass Model B on official GSM8K TIES (13.72%) and DELLA (4. p. 0 Results reveal that all tested merging methods fail to surpass the stronger
source model on official benchmarks, with TIES (13. 72%) and DELLA (4. 72%) and DELLA (4. 12%) performing catastrophically on GSM8K. 67%), demonstrating
that standard parameter space methods fail when merging base and instruction-tuned models. These
findings directly validate the difficulty of the Type-I interaction structure we identify, and motivate
the application of algorithms specifically designed for this challenge — such as the concurrent IGBD
work of Ong et al. 3 Cross-Benchmark Analysis
Figure 3 reveals an important pattern: methods optimized or designed for general parameter averaging (PS Merging, Unstructured DFS) maintain performance across both benchmarks, while methods
involving complex manipulation (TIES, DELLA) fail on both. Our CMA-ES approach shows relatively
stronger performance on MMLU, reflecting its proxy optimization on GSM8K questions.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Implications for Future Research
The findings point clearly toward IGBD [2] as the theoretically appropriate algorithm for this problem Ong et al. [2] independently developed IGBD—a bilevel optimization framework designed specifically for Type-I interaction. p. 2 Concurrent work by Ong et al. [2] proposes IGBD, a bilevel informationgeometric framework that directly handles Type-I interaction through coupled bilevel optimization. Methods performing similarly on both benchmarks cluster near the diagonal. PS Merging and Unstructured DFS maintain competitive performance
on both benchmarks, while CMA-ES shows stronger relative performance on MMLU than GSM8K,
reflecting proxy optimization on GSM8K questions. Comprehensive empirical analysis: We evaluate seven merging methods on official GSM8K
(1,319 questions) and MMLU (14,042 questions, 57 subjects), providing the most honest largescale analysis of evolutionary model merging to date. Critical failure analysis: We identify why standard parameter space methods fail catastrophically when merging base and instruction-tuned models — a finding with direct implications for
practitioners.Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Conclusions
Limitations include the single model family restriction and the weak base model performance on GSM8K. The proxy evaluation gap explains why CMA-ES performs well on development subsets but not on official benchmarks. p. 3 24B parameters)
4. 2 Benchmarks
Official Benchmarks:
• GSM8K [8]: 1,319 test questions, chain-of-thought prompting
• MMLU [9]: 14,042 questions across 57 subjects
Development Subsets (ablation only):
• GSM8K subset: 97 questions for ablation studies
• MMLU subset: 100 questions across 10 subjects for ablation studies
4. Table 2: Official MMLU Results (14,042 questions, 57 subjects, 3 runs)
Method Mean Accuracy Std vs Model B
Model B (Mistral-7B-Instruct) 51. 10% — Baseline
PS Merging 50. We
demonstrate that DFS merging exhibits exactly this structure, establishing it as a concrete real-world
application domain for these methods. 3 Large Language Model Evaluation
GSM8K [8] contains 1,319 test questions requiring multi-step math reasoning.Where the paper says it
Where the paper says it
Where the paper says it
REFERENCES
[1] Y. Akimoto, X. p. 0 niloy26643@gmail. com
Youhei Akimoto
Institute of Systems and Information Engineering / Center for Artificial Intelligence Research
University of Tsukuba, Tsukuba, Japan
akimoto@cs. 2 Mixed-Variable Black-Box Optimization
Akimoto et al. [1] formally characterized Type-I interaction in mixed categorical-continuous optimization — where effective continuous dimensions depend on categorical variables.Where the paper says it
Where the paper says it
[2] M. Ong, S. Shirakawa, and Y. Akimoto, “Mixed-categorical black-box optimization via information-geometric bilevel decomposition,” arXiv:2606. p. 2 2 Mixed-Variable Black-Box Optimization
Akimoto et al. [1] formally characterized Type-I interaction in mixed categorical-continuous optimization — where effective continuous dimensions depend on categorical variables.Where the paper says it
[3] T. Akiba, M. Shing, Y. Tang, Q. Sun, and D. Ha, “Evolutionary optimization of model merging recipes,” arXiv:2403. p. 0 Keywords: model merging, evolutionary optimization, black-box optimization, large language models, CMA-ES, Data Flow Space, Type-I interaction
1 Introduction
The rapid proliferation of large language models (LLMs) has created significant demand for efficient
methods to combine their complementary capabilities. Model merging — combining parameters from
1
arXiv:2605.Where the paper says it
[4] K. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “TIES-Merging: Resolving interference when merging models,” in NeurIPS, 2023
[5] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” arXiv:2311. p. 0 Keywords: model merging, evolutionary optimization, black-box optimization, large language models, CMA-ES, Data Flow Space, Type-I interaction
1 Introduction
The rapid proliferation of large language models (LLMs) has created significant demand for efficient
methods to combine their complementary capabilities. Model merging — combining parameters from
1
arXiv:2605.Where the paper says it
[6] P. Deep et al., “DELLA-Merging: Reducing interference in model merging through magnitudebased sampling,” arXiv:2406. p. 1 DELLA [6] extends DARE with magnitude-based sampling. Evolutionary approaches to model merging were pioneered by Akiba et al.Where the paper says it
[7] R. Hamano, S. Saito, M. Nomura, K. Uchida, and S. Shirakawa, “CatCMA: Stochastic optimization for mixed-category problems,” in Proc. p. 2 Hamano et al. [7] proposed CatCMA, extending
CMA-ES to handle mixed categorical-continuous variables through joint optimization of Gaussian and
categorical distributions.Where the paper says it
[8] K. Cobbe et al., “Training verifiers to solve math word problems,” arXiv:2110. p. 0 Keywords: model merging, evolutionary optimization, black-box optimization, large language models, CMA-ES, Data Flow Space, Type-I interaction
1 Introduction
The rapid proliferation of large language models (LLMs) has created significant demand for efficient
methods to combine their complementary capabilities. Model merging — combining parameters from
1
arXiv:2605.Where the paper says it
[9] D. Hendrycks et al., “Measuring massive multitask language understanding,” in ICLR, 2021
[10] M. Wortsman et al.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Concurrent IGBD implementation: Apply Ong et al.'s IGBD framework to DFS merging to directly handle Type-I interaction, which is currently a fundamental challenge for standard optimization algorithms like CMA-ES. This would allow for the design of theoretically correct optimization methods instead of relying on heuristic search spaces.
-
Multi-objective merging formulations: Develop multi-objective optimization techniques to balance task specific knowledge and general knowledge during model merging, addressing the limitation that CMA-ES optimized on a proxy subset
does not generalize to MMLU.
-
Stronger base model selection: Prioritize selecting a stronger base model (Model B over Model A) as the initial source, as the paper finds that
The weak base model: Mistral-7B-v0.1 scores only 6.67% on GSM8K — far below Model B (36.77%).
This ensures that merging operations do not destroy essential instruction following weights. -
Sparse binary selection strategy: Implement a sampling probability for the binary layer selection vector z that decreases monotonically with performance variance, as shown in Ablation 1 where
performance variance decreases monotonically with selection probability — exactly what Type-I interaction predicts.
Abstract
Model merging has emerged as a cost-effective paradigm for combining capabilities of multiple large language models (LLMs) without additional training. Among existing approaches, Data Flow Space (DFS) merging operates by selectively activating parameter groups across model layers, offering greater flexibility than naive parameter averaging. However, the optimization challenges underlying DFS merging remain poorly understood. In this paper, we formally characterize the DFS merging problem as a mixed binary-continuous black-box optimization problem, identifying three fundamental challenges: (1) mixed binary-continuous variable types, (2) high-dimensional search spaces with up to 291 parameter groups, and (3) conditional variable dependencies corresponding to Type-I interaction as characterized by Akimoto et al. [1]. We empirically evaluate seven merging methods on Mistral-7B models across official GSM8K (1,319 questions) and MMLU (14,042 questions, 57 subjects) benchmarks. Our structured approach achieves a 49.5% reduction in effective search space. Results reveal that all tested merging methods fail to surpass the stronger source model on official benchmarks, with TIES (13.72%) and DELLA (4.12%) performing catastrophically on GSM8K. Notably, DELLA scores below even the base model (6.67%), demonstrating that standard parameter space methods fail when merging base and instruction-tuned models. These findings directly validate the difficulty of the Type-I interaction structure we identify, and motivate the application of algorithms specifically designed for this challenge such as the concurrent IGBD work of Ong et al. [2].
Sources
- Mixed-Categorical Black-Box Optimization via Information-Geometric Bilevel Decomposition
- Evolutionary Optimization of Model Merging Recipes
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
- DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
- Training Verifiers to Solve Math Word Problems
- The CMA Evolution Strategy: A Tutorial
Related papers
- Evolutionary Ensemble of Agents
- EvoDR: Evolving Dispatching Rules via Large Language Model for Dynamic Flexible Assembly Flow Shop Scheduling
- RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics
- Encoding and Decoding Temporal Signals with Spiking Bandpass Wavelets
- Design and Analysis of Temporal Representations in Neural Networks
- Evolutionary Computation: A Paradigm of Natural Generative AI and Creative Synthesis