ThinkSum: Probabilistic reasoning over sets using large language models

arXiv:2210.01293 · cs.CL · Submitted 2022-10-04 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "ThinkSum: Probabilistic reasoning over sets using large language models".

Title and authors: Tom: So, we're looking at this paper called ThinkSum, and what it’s doing is trying to fix a problem where large language models struggle with things that need multiple steps of logic or reasoning over several facts at once.

Jane: Right, so the title itself says "Probabilistic reasoning over sets using large language models," and that sounds like it's moving away from just asking the model one long question at a time.

Lu: Exactly. The core idea here is to structure the LLM's thinking into two distinct phases: first you get associations, and then you aggregate those associations probabilistically to get the final answer.

Meng: So, what does this actually mean for how we build these things? Is it just a fancy way of writing a prompt that works better?

Tom: No, it's different because the thinking part—the "Think" stage—is done by querying an LLM in parallel over several small phrases you extract from the prompt or maybe some other model call.

Jane: That sounds like breaking down a big problem into smaller pieces that you can check all at once before you put them back together.

Lu: Precisely, and the second part, the "Sum" stage, is where you take all those individual results from the first stage and combine them using some probabilistic inference techniques to make the final prediction.

Meng: I’m wondering what kind of things they're calling these inference techniques. Are they just simple voting mechanisms? Because when I try to run stuff at scale, I always worry about the overhead of running too many parallel calls.

Tom: They use several methods, including majority or minority votes and ratio of likelihoods for aggregation, but they also do product aggregation, which is multiplying the LLM likelihoods for all the strings in a set.

Jane: That sounds like you’re taking all those different potential paths the model suggests and mathematically combining their probabilities to find the most likely final outcome.

Lu: And they've even looked at more advanced structures, like using latent variable models in that Sum stage to assume an underlying structure for those items and facts.

Meng: Latent variables sound ambitious when you think about deployment. How does that help us figure out what actually runs on a production system?

Tom: It helps by assuming there's a hidden cluster, so you select the facts or items that belong to the majority class, which can simplify the decision process.

Jane: So instead of checking every single possible combination, you’re leaning into an assumed underlying structure to guide your selection.

Meng: That makes sense if we think about reducing the search space when we have a million possible inputs. But what about prompt design sensitivity? Isn't that a big issue with current methods?

Tom: That's where ThinkSum really shines because the reasoning step is performed outside of the LLM’s direct generation process, which makes it less sensitive to things like accidental distractions in the prompt.

Title and authors: Jane: So if you accidentally put a weird word in your prompt that confuses the model, it doesn't derail this entire set-based inference structure as much?

Lu: It reduces vulnerability because they engineer that slow thinking process to make parallel calls for intermediate info before combining them, which isn't just letting the LLM do all the heavy lifting in one go.

Meng: So it’s more robust to prompt design, but what about the cost? I need to know if this scales reasonably when we have a whole set of objects or facts to reason over.

Tom: The computation cost scales with how many parallel calls you make, which depends on the number of Think prompts you use for each task and how many objects you need likelihood computations for at the Sum stage.

Jane: So it’s not free scaling; it costs more as the problem gets bigger in terms of objects. But they say it's generally faster than some chain-of-thought methods because those are sequential generation problems, whereas ThinkSum does parallel calls.

Lu: And for tasks like logical deduction, they translated the problems into sets of mathematical inequalities and then used product aggregation followed by mixture aggregation to determine the correct ordering.

Meng: That sounds like a solid way to handle complex constraints without having to force the LLM into generating a perfect step-by-step narrative. But what about those tasks where it’s not working well?

Tom: They tested it across thirteen difficult tasks from the BIG-bench suite, and they showed improvements over the state of the art using GPT family models on thirteen specific difficult ones.

Jane: And they found that for a task like logical deduction with five objects, ThinkSum got an accuracy of seventy-seven percent, which is better than what average humans get for that specific type of problem.

Meng: Seventy-seven percent is good, but I need to know how it holds up when we move to real traffic or complex video reasoning problems where the input isn't just text sets.

Tom: They also noted that in the ODD ONE OUT task, they maintained near-average human accuracy for models ranging from GPT two small to the largest GPT three model.

Jane: So it’s pretty consistent across different model sizes for those specific benchmark tasks, which is encouraging when you're trying to deploy something.

Lu: The difference between ThinkSum and traditional chain-of-thought methods, like the ones from Wei et al., is that the reasoning isn't reduced to a generation problem for the LLM anymore.

Meng: So you’re not just chaining text outputs together; you’re doing something different with those intermediate results before you hit the final answer.

Tom: Yeah, thinkSum performs classical probabilistic reasoning externally, which is what allows it to reduce vulnerability to prompt features like spurious patterns that might confuse the LLM.

Jane: That external reasoning step means the structure of your set-based approach is more important than making the LLM write a perfect little narrative for you.

Lu: And they’ve shown this can be combined with latent variable models to extract structured knowledge from the LLMs, which is really interesting for understanding what’s actually happening inside those large models.

Title and authors: Meng: That sounds like a way to pull out hidden knowledge without having to retrain the whole model just for that one set of facts. What about practical deployment costs again?

Tom: The cost scales with the number of parallel calls, which you determine based on how many Think prompts you use and how many objects you need likelihood computations for at the Sum stage.

Jane: So if you have a really massive set, that computational load will grow significantly because of those parallel calls. It’s not a cheap operation for very large inputs.

Lu: The authors also pointed out a limitation: as the number of objects or facts you're reasoning over increases, the computation cost will also rise.

Meng: That’s the caveat I was worried about; scaling up means scaling up the compute demand for this inference structure.

Tom: There’s another point, though, that when you are solving a brand new task, it requires human effort to select specific operations in each step for both Think and Sum.

Jane: So even if the framework is good at reasoning over sets, you still have to figure out the right sequence of operations for a novel problem.

Lu: Future work they suggest could focus on automating that prompt engineering procedure so you don't have to manually select every operation every time.

Meng: It sounds like they’re focused on making the process more automated or at least less manual for the user who has to set it up.

Tom: So, in short, ThinkSum is a two-stage approach that uses parallel LLM queries for association and then probabilistic aggregation for reasoning over sets.

Jane: It’s a way to use LLMs to handle complex logical deductions by treating the problem as a set of facts instead of just one long sequence.

Lu: It offers more interpretability because you can trace the results back through those explicit retrieval steps, which is useful when we want to understand *why* an AI made a certain choice.

Meng: For us in production, if this means less sensitivity to prompt formatting errors and more structured output, that could be a win for stability.

Tom: It really does sound like they're proposing a way to get more reliable reasoning from existing large models by controlling the inference step outside of the model itself.

Jane: So we’ve seen how this two-stage paradigm works and what it means for tackling problems that require reasoning over multiple objects or facts.

Lu: The paper "ThinkSum: Probabilistic reasoning over sets using large language models" proposes this structured approach to make LLMs more reliable when dealing with complex logical deductions.

Meng: It sounds like a promising direction, especially with the focus on probabilistic inference being done externally from the LLM's core generation.

Tom: We’ll keep an eye on how they extend this paradigm to newer models and how they handle those scaling costs as we move towards even larger architectures.

The paper's summary: Tom: So, basically, ThinkSum is this two-stage approach where you use an LLM to quickly grab associations from a set of phrases, and then you take all those results and combine them using probability math to get the final answer.

Jane: It’s trying to fix a problem where models struggle with reasoning over multiple facts or objects at the same time, which is what this paper is all about.

Tom: Right. They show that instead of just having the LLM generate one long chain of thought, you can do this structured thinking process externally and then perform the inference outside of the LLM itself.

Jane: And they found that because you're doing that probabilistic math outside of the model, it makes the whole system much more robust to how you write your prompt—it’s less sensitive to accidental distractions.

Tom: Exactly. They tested this on thirteen difficult tasks from a benchmark suite called BIG-bench, and they actually got better results than the state of the art using GPT family models on thirteen specific tough challenges.

Jane: For instance, in a task that involves logical deduction with five objects, ThinkSum achieved seventy-seven percent accuracy which beats the average human performance for that type of problem.

Tom: That’s a concrete number. And they found that this method outperformed traditional chain-of-thought prompting on all those tasks except for one specific variant, KNOWN UNKNOWNS with the davinci model.

Jane: So what does it mean practically? It suggests that we can get more reliable reasoning from existing large models just by adding a smart controller layer that handles the set reasoning and probabilistic inference.

Tom: The cost analysis shows that the computation scales with how many parallel calls you make, which depends on how many Think prompts you use and how many objects you need to run likelihood computations for at the Sum stage.

Jane: So it’s not free scaling; if you have a really big set of facts, the computation load goes up because of those parallel calls. But they also mentioned that when you're starting a brand new task, you still need human effort to select the right operations for both stages.

Tom: That’s the limitation they flagged. The authors say future work needs to focus on automating that process of selecting the right operations so it’s less manual for users.

Jane: Still, this whole structure—using parallel retrieval followed by structured probabilistic aggregation—gives us a way to see what's happening inside these models more clearly because you can trace the results back through those explicit steps.

Tom: It’s about treating the LLM like a retrieval and aggregation engine rather than just a pure generation machine for complex tasks.

Jane: And that structure, combined with latent variable models, lets them extract structured knowledge from the AI that they couldn't get before.

Tom: So, ThinkSum is proposing this two-stage paradigm—Think for associations, Sum for inference—to enhance reasoning over sets. But where does it leave us when we think about deploying this kind of structured thinking on a massive scale?

The paper's improvements: Tom: So, we're looking at how they actually make ThinkSum better than just using a standard chain of thought prompt to solve these set problems.

Jane: Right, so they aren't just relying on the LLM to write a long narrative anymore; they’ve introduced a two-stage process that does retrieval first, and then probabilistic inference second.

Tom: Exactly. The first stage is "Think," where the LLM queries several small phrases in parallel—maybe things like "Order inversion" or generating a list of facts about cats—to build up a set of candidate associations.

Jane: And then in the second stage, the "Sum," they take all those different results and use specific math techniques like mixture aggregation or product aggregation to pick the most likely final answer.

Tom: They also introduced this latent variable model approach for the Sum stage, which lets them assume there’s an underlying hidden structure in those items and facts, essentially modeling them as coming from a single cluster.

Jane: That part is interesting because it means they can select facts that belong to the majority class if that structure makes sense for the problem.

Tom: It helps explain their decisions because they aren't just guessing; they are selecting based on that assumed underlying structure of what’s possible.

Jane: So, what does this mean for us? It means we can use existing large models to handle complex logical deductions over many facts in a structured way, and the results are more interpretable because you can trace back through those explicit retrieval steps.

Tom: It makes the system less sensitive to how you format your prompt because the heavy lifting of probabilistic inference is happening outside of the LLM’s main generation step.

Jane: That robustness is key when we are trying to deploy these things in real applications where prompt formatting can easily get messy or accidentally confusing.

Tom: And they showed this method performing better than older chain-of-thought methods on thirteen different hard tasks from BIG-bench, often even with smaller model variants like GPT two small.

Jane: So, if you’re a developer trying to build a system that needs to reason over multiple items, this shows you that breaking the problem down into parallel retrieval and structured inference is a much more promising way forward than just one big prompt.

Tom: The main point is that we can get more reliable reasoning from models of existing scale when we use a controller that allows for this kind of probabilistic inference to happen externally.

Jane: It’s about controlling the reasoning process by separating the search and the final decision-making into two distinct, measurable steps.

Tom: And while they pointed out that the cost scales with those parallel calls and you still need human effort to set up those specific operations for a new task, it’s a solid framework for getting structured output.

Jane: It’s definitely a way to enhance the capability of current models without needing a massive retraining effort just to get better at multi-object reasoning.

Tom: So, the idea is that this two-stage paradigm—Think for retrieval and Sum for inference—is a more robust way to tackle complex set problems. But we still have those scaling costs and manual setup hurdles to deal with before it’s ready for mass deployment.

Conclusion: Tom: So, to wrap up, ThinkSum is a two-stage method that uses parallel LLM calls for association and then uses probabilistic inference for aggregation to reason over sets of objects or facts.

Jane: Basically, it shows we can use existing large language models effectively by giving them a smart controller that handles the set reasoning in a structured way instead of just letting them generate one long answer.

Tom: It’s about making the model less sensitive to prompt design because the probabilistic math happens externally, which gives us more interpretable predictions.

Jane: And they achieved improvements on thirteen hard tasks from BIG-bench, proving that this set-based thinking actually beats some of the state of the art using GPT family models.

Tom: The main implication is that we can get better logical reasoning out of existing AI by controlling the inference step outside of the model itself.

Jane: It suggests a direction for building more robust systems, where structure and explicit retrieval steps are prioritized over just hoping the LLM writes a perfect narrative.

Tom: We still have those scaling costs to worry about though, because the computation scales with how many parallel calls you make for each task.

Jane: And there's that necessary human effort involved in setting up those specific operations for every new task you want to solve.

Tom: So, it’s a promising approach for enhancing reasoning capabilities with models of existing scale, but we gotta keep an eye on those scaling costs and setup overhead when we move this into production.

Jane: It definitely points toward needing these kinds of structured controllers if we want AI to handle complex logic reliably in the real world.

Lu: From my side, I see the latent variable model idea as really interesting because it gives us a way to assume an underlying structure for those facts, which could change how we think about knowledge representation in these models.

Meng: For me, at the production level, the key is that you have to be able to predict exactly how many parallel calls you’re going to make so you can manage the compute budget and avoid running into unexpected bottlenecks.

Lalam: I'm sure this works well because my internal literature shows that when we decompose a complex reasoning task into these retrieval and aggregation steps, the confidence in the final prediction is much higher than if I just tried to do it all in one pass.

Batu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa Jojic

Stanford University · Université de Montréal · Ohio State University · Microsoft Research

cs.CL

Submitted: 2022-10-04

Updated: 2026-11-18

Comments: ACL 2023

DOI: 10.18653/v1/2023.acl-long.68

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the…

Key concepts

Think Stage
The first stage where an LLM is queried in parallel over a set of phrases derived from the prompt or auxiliary calls. This step creates associations between text spans and sets of strings, such as generating examples or performing string manipulations like 'Order inversion', to build a pool of potential knowledge for later inference.
Sum Stage
The second stage where the results from the Think stage are aggregated using various probabilistic inference techniques like majority voting or product aggregation. This step takes the LLM outputs and combines them mathematically to produce the final, reasoned prediction.
Probabilistic Inference
The mathematical process used in the Sum stage to aggregate multiple LLM outputs into a single answer. Techniques like mixture aggregation or product aggregation are employed to find the most likely outcome from a set of possibilities, effectively performing reasoning outside of direct LLM generation.
Latent Variable Models
A technique used in the Sum stage where items and facts are modeled as being generated from an underlying, unobserved structure or 'latent class'. This allows the system to select facts that belong to the majority class, helping to explain decisions through this structured knowledge.

Terminology

Summary

Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the provided context (few-shot in-context learning) However, recent studies show that even the more advanced LLMs fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions We propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner In the first stage (Think – retrieval of associations), a LLM is queried in parallel over a set of phrases extracted from the prompt or an auxiliary model call In the second stage (Sum – probabilistic inference or reasoning), the results of these queries are aggregated to make the final prediction We demonstrate the possibilities and advantages of ThinkSum on the BIG-bench suite of LLM evaluation tasks, achieving improvements over the state of the art using GPT-family models on thirteen difficult tasks, often with far smaller model variants Our results suggest that because the probabilistic inference in ThinkSum is performed outside of calls to the LLM, ThinkSum is less sensitive to prompt design, yields more interpretable predictions, and can be flexibly combined with latent variable models to extract structured knowledge from LLMs Overall, our proposed paradigm represents a promising approach for enhancing the reasoning capabilities of LLMs. p. 0

Where the paper says it

com Abstract Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the provided context (few-shot in-context learning). However, recent studies show that even the more advanced LLMs fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions.

ThinkSum Paradigm and Mechanism

The ThinkSum paradigm is a two-stage process where the first stage, Think, involves querying an LLM in parallel over a set of phrases extracted from the prompt or an auxiliary model call The second stage, Sum, aggregates the results of these queries to produce the final prediction through probabilistic inference. The Think step involves creating an association of text spans with sets of strings, which may involve generation from a language model or solely a scoring mechanism resulting in a matrix of probabilities Examples of the Think method include elementary string manipulations such as Order inversion and Premise erasure, as well as example generation, list extension, and fact generation.

Inference Aggregation Techniques

The Sum stage employs various techniques to aggregate the results obtained from the Think step, including several elementary inference methods These include Majority/minority vote (argmin/argmax), Ratio of likelihoods, and Mixture (average) aggregation. Product aggregation is also utilized, such as in the LOGICAL DEDUCTION task where it involves multiplying the LLM likelihoods of all strings in a set. p. 6

Where the paper says it

The term inside the sum is a case of Product aggregation: the LLM likelihoods of all strings in the set are multiplied. In summary, our solution to this task involves composition of two Think operations – a Translation into a set of equations and then Substitution of numbers in place of item names – and two Sum operations – a Product aggregation followed by a Mixture aggregation.

Performance Across Diverse Tasks

ThinkSum has been demonstrated on thirteen difficult tasks from BIG-bench, achieving improvements over the state of the art using GPT-family models on thirteen difficult tasks, often with far smaller model variants For instance, in the LOGICAL DEDUCTION task with 500 problems where N = 5 objects, ThinkSum yields an accuracy of 77%, besting the average human performance Furthermore, in the ODD ONE OUT task, near-average-human accuracy is maintained for all model sizes from GPT-2 Small to the largest GPT-3 model.

Comparison with Existing Methods

An important difference between ThinkSum and existing chain-of-thought-like prompt engineering methods is that our reasoning step is not reduced to a generation problem for the LLM, but is performed as a probabilistic inference external to the LLM This external probabilistic inference reduces vulnerability to features of the prompt, such as accidental distraction of the LLM by spurious patterns ThinkSum outperforms chain-of-thought prompting on all tasks with all variants except KNOWN UNKNOWNS with the davinci variant.

Conclusion and Future Directions

The main claim of this paper is that more is possible with LLMs of existing scale when they are used in concert with a wise controller that allows for probabilistic inference ThinkSum is more robust to prompt design, yields more interpretable predictions, and can be combined with many probabilistic inference approaches to tackle a diverse set of tasks. The computation cost for ThinkSum scales with the number of parallel calls to the LLM, which is determined for each task by the number of Think prompts used and the number of objects for which likelihood computations are required at the Sum stage. p. 9

Where the paper says it

Firstly, as the number of objects or facts that are reasoned over increases, the computation cost will also rise. However, increasing the number of objects will also make the task harder, and direct prompting may cease to work at all (as we indeed observe in BIG-bench results, such as LOGICAL DEDUCTION with more than five objects), while ThinkSum offers a generalizable methodology, as the atomic Think operations do not increase in complexity as the number of objects grows.

The authors suggest that a general twostage paradigm with an external inference step aims to facilitate adaptation to new tasks with minimal modifications to the Think and Sum steps. p. 9
Where the paper says it

We have described ThinkSum as a general twostage paradigm, with an external inference step. This generality aims to facilitate the adaptation of ThinkSum to new tasks, with minimal modifications to the Think and Sum steps.

Acknowledgments and Limitations

The authors thank Alexandros Graikos, Sudha Rao, and Alessandro Sordoni for valuable discussions A limitation noted is that as the number of objects or facts that are reasoned over increases, the computation cost will also rise Additionally, when solving a new task, it is necessary to expend human effort to select specific operations in each step.

Ethics and Impact Statement

Control of LLMs’ reasoning in the way we have proposed can potentially mitigate such bias due both to the decomposition of the argumentation process into interpretable fact-retrieval steps and to the averaging effect of smoothing out spurious triggers when aggregating many hypotheses and reasoning chains.

References

Yoshua Bengio. 2017. The consciousness prior. p. 9

Where the paper says it

References Yoshua Bengio. The consciousness prior.

p. 9
Where the paper says it

References Yoshua Bengio. The consciousness prior.

p. 9
Where the paper says it

References Yoshua Bengio. The consciousness prior.

arXiv preprint arXiv:1709. p. 9
Where the paper says it

The consciousness prior. arXiv preprint arXiv:1709.

p. 9
Where the paper says it

The consciousness prior. arXiv preprint arXiv:1709.

p. 9
Where the paper says it

The consciousness prior. arXiv preprint arXiv:1709.

08568 Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger Hyung Won Chung et al. p. 9
Where the paper says it

arXiv preprint arXiv:1709. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

2022. Scaling instruction-finetuned language models Antonia Creswell et al. p. 9
Where the paper says it

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.

p. 9
Where the paper says it

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.

p. 9
Where the paper says it

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.

2022. Selection-inference: Exploiting large language models for interpretable logical reasoning A P. p. 9
Where the paper says it

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning.

p. 9
Where the paper says it

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning.

p. 9
Where the paper says it

Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning.

Dempster et al. p. 5
Where the paper says it

70 Table 2: Different alternatives of probabilistic reasoning with ThinkSum for solving ODD ONE OUT: list of words, latent variable model, minority voting. (EM; Dempster et al.

p. 5
Where the paper says it

70 Table 2: Different alternatives of probabilistic reasoning with ThinkSum for solving ODD ONE OUT: list of words, latent variable model, minority voting. (EM; Dempster et al.

p. 5
Where the paper says it

70 Table 2: Different alternatives of probabilistic reasoning with ThinkSum for solving ODD ONE OUT: list of words, latent variable model, minority voting. (EM; Dempster et al.

1977. Maximum likelihood from incomplete data via the EM algorithm David Dohan et al. p. 9
Where the paper says it

Laird, and D. Maximum likelihood from incomplete data via the EM algorithm.

p. 9
Where the paper says it

Laird, and D. Maximum likelihood from incomplete data via the EM algorithm.

p. 9
Where the paper says it

Laird, and D. Maximum likelihood from incomplete data via the EM algorithm.

2022. Language model cascades

Additional Tasks and Details

The paper includes case studies on several BIG-bench tasks, such as Conceptual combinations, Odd one out, Logical deduction, and Phrase relatedness Detailed experimental parameters include maximum generation length of 100 for example and list generation tasks The experiments are performed using GPT-3 variants and GPT-2 models. The paper also details a latent variable model approach to select facts that characterize the majority class for ODD ONE OUT

Inference Cost Analysis

The inference cost for ThinkSum scales with the number of parallel calls to the LLM, which is determined for each task by the number of Think prompts used and the number of objects for which likelihood computations are required at the Sum stage In some cases, ThinkSum is faster than chain-of-thought prompting due to its ability to perform parallel calls to the LLM The inference cost increase is marginal for tasks where the number of Think prompts is not typically high and the prompts are short.

Inference as Reasoning

The work generalizes such approaches to perform arbitrary probabilistic inference outside of the LLM The authors model items i ∈ I and facts f ∈ F as being generated from a latent class c ∈ 0, 1 using an expectation-maximization (EM) algorithm to derive P(i c) and P(f c) This inference procedure can be seen as a Sum (inference) operation and can be applicable in other tasks of similar structure.

Examples of Inference Procedures

The paper details specific inference procedures for different tasks, such as the translation into a set of inequalities and then substitution of numbers in place of item names This involves composition of two Think operations – a Translation into a set of equations and then Substitution of numbers in place of item names – and two Sum operations – a Product aggregation followed by a Mixture aggregation.

Results Summary

For the 500 LOGICAL DEDUCTION problems with N = 5 objects, ThinkSum yields an accuracy of 77% (see Table 1), besting the average human performance The final result for INVENTED WORDS was Borger folpts are pet toys

Table Summaries

Table 3 shows the comparison of ThinkSum with algorithms that append auxiliary knowledge as an oracle ‘reasoning chain’ Table 4 compares ThinkSum with chain-of-thought prompting approaches Table D.1 lists examples for the studied BIG-bench tasks The results show that ThinkSum outperforms auxiliary knowledge and chain of thought on LOGICAL DEDUCTION. p. 7

Where the paper says it

Above we have compared ThinkSum with approaches that add extracted (from an auxiliary LM call), not generated (within the LM’s linear workspace) token sequences after the prompt, for the ODD ONE OUT, PHRASE RELATEDNESS, and LOGICAL DEDUCTION tasks (see Table 3). With suitable examples, it may be possible for a CoT approach to replace the Think phase, by learning from demonstrations to generate the appropriate knowledge, and parts of the Sum phase, although inference over parallel evaluations of the LLM is no longer possible.

The results show that ThinkSum outperforms CoT prompting on all tasks with all variants except KNOWN UNKNOWNS with the davinci variant

Figure Analysis

Figure 3 shows details for LOGICAL DEDUCTION, including a demonstration for the Think prompt, and an example LLM output Figure 4 shows how performance varies with the size of the LLM used for GPT-3, auxiliary knowledge and ThinkSum on ODD ONE OUT Figure D.1 shows the probabilities of different (in)equalities according to GPT-3 text-davinci-002 The results show that ThinkSum dramatically improves over much larger largest zero- or few-shot models with or without auxiliary knowledge. p. 5

Where the paper says it

2 (left) shows how performance varies with the size of the LLM used for GPT-3, auxiliary knowledge and ThinkSum on ODD ONE OUT. Even with GPT-2 Small, ThinkSum dramatically improves over much larger largest zero- or few-shot models with or without auxiliary knowledge.

The results show that ThinkSum is 23% faster for PHRASE RELATEDNESS compared to chain-of-thought approaches with 5 facts generated using InstructGPT

Final Conclusion

The two-step paradigm allows operating over sets instead of manipulating the prompt itself, preventing sensitivity to prompt format during the probabilistic inference in ThinkSum As a result, ThinkSum is more robust to prompt design, yields more interpretable predictions, and can be combined with many probabilistic inference approaches to tackle a diverse set of tasks. The paper concludes that more is possible with LLMs of existing scale when they are used in concert with a wise controller that allows for probabilistic inference. p. 2

Where the paper says it

As is the case with humans, there is always the next frontier of problems where a trained model with remarkable ‘intuition’ needs to be slowed down. The main claim of this paper is that more is possible with LLMs of existing scale when they are used in concert with a wise controller that allows for probabilistic inference.

Limitations and Extensions

The authors acknowledge that as the number of objects or facts that are reasoned over increases, the computation cost will also rise A limitation is that when solving a new task, it is necessary to expend human effort to select specific operations in each step Future work could focus on automating the prompt engineering procedure for overcoming this limitation. The authors suggest that applying ThinkSum to more recent instruction-tuned models such as Flan-PaLM, text-davinci-003, ChatGPT, and GPT-4 may be beneficial The paper foresees no direct or immediate societal impacts arising from this work. p. 9

Where the paper says it

, 2022), text-davinci-003, ChatGPT, and GPT-4, which seem more capable of robustly performing Think steps. Ethics and impact statement We foresee no direct or immediate societal impacts arising from this work.

Yoshua Bengio. 2017. The consciousness prior. arXiv preprint arXiv:1709.08568 Tom Brown et al. p. 9

Where the paper says it

arXiv preprint arXiv:1709. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

p. 9
Where the paper says it

arXiv preprint arXiv:1709. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

2020. Language models are few-shot learners Hyung Won Chung et al. p. 9
Where the paper says it

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.

p. 9
Where the paper says it

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models.

2023. Scaling instruction-finetuned language models Antonia Creswell et al. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning A P. Dempster et al. 1977. Maximum likelihood from incomplete data via the EM algorithm David Dohan et al. 2022. Language model cascades

Figure 3 shows details for LOGICAL DEDUCTION, including a demonstration for the Think prompt, and an example LLM output Figure 4 shows how performance varies with the size of the LLM used for GPT-3, auxiliary knowledge and ThinkSum on ODD ONE OUT Figure D.1 shows the probabilities of different (in)equalities according to GPT-3 text-davinci-002 The results show that ThinkSum dramatically improves over much larger largest zero- or few-shot models with or without auxiliary knowledge The results show that ThinkSum is 23% faster for PHRASE RELATEDNESS compared to chain-of-thought approaches with 5 facts generated using InstructGPT. p. 5

Where the paper says it

2 (left) shows how performance varies with the size of the LLM used for GPT-3, auxiliary knowledge and ThinkSum on ODD ONE OUT. Even with GPT-2 Small, ThinkSum dramatically improves over much larger largest zero- or few-shot models with or without auxiliary knowledge.

The two-step paradigm allows operating over sets instead of manipulating the prompt itself, preventing sensitivity to prompt format during the probabilistic inference in ThinkSum As a result, ThinkSum is more robust to prompt design, yields more interpretable predictions, and can be combined with many probabilistic inference approaches to tackle a diverse set of tasks The paper concludes that more is possible with LLMs of existing scale when they are used in concert with a wise controller that allows for probabilistic inference.

The authors acknowledge that as the number of objects or facts that are reasoned over increases, the computation cost will also rise A limitation is that when solving a new task, it is necessary to expend human effort to select specific operations in each step Future work could focus on automating the prompt engineering procedure for overcoming this limitation

Yoshua Bengio. 2017. The consciousness prior. arXiv preprint arXiv:1709.08568 Tom Brown et al. 2020. Language models are few-shot learners Hyung Won Chung et al. 2023. Scaling instruction-finetuned language models Antonia Creswell et al. 2023. Selection-inference: Exploiting large language models for interpretable logical reasoning A P. Dempster et al. 1977. Maximum likelihood from incomplete data via the EM algorithm David Dohan et al. 2022.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Probabilistic reasoning over sets using ThinkSum allows models to reason over multiple objects or facts by performing a two-stage process: Think (retrieval of associations via auxiliary model calls) and Sum (probabilistic inference). This structure makes the system less sensitive to prompt design, yields more interpretable predictions, which enhances robustness compared to direct prompting.

  2. The Think stage can be implemented using elementary string manipulations such as Order inversion or Premise erasure, and list generation prompts like “List facts about cats,” enabling the model to create sets of examples for later inference. This allows the system to perform fast thinking / association / knowledge retrieval step: creating an association of text spans with sets of strings.

  3. The Sum stage utilizes various probabilistic inference methods, including Mixture aggregation and Product aggregation, which aggregate generated strings or probabilities to produce the final answer. For tasks like Conceptual combinations, this involves forming a mixture model over substitutions and normalizing likelihoods to select the most plausible sentence.

  4. Latent variable models can be incorporated into the Sum stage to assume an underlying structure in items and facts, modeling them as being generated from a latent class such that the facts and items belonging to a cluster can be freely interchanged. This helps explain decisions by selecting facts that characterize the majority class.

  5. The system can perform structured inference for logical deduction tasks by translating problems into sets of mathematical inequalities and then performing Product aggregation followed by a Mixture aggregation over all possible mappings to determine the correct ordering.

Abstract

Large language models (LLMs) have a substantial capacity for high-level analogical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the provided context (few-shot in-context learning). However, recent studies show that even the more advanced LLMs fail in scenarios that require reasoning over multiple objects or facts and making sequences of logical deductions. We propose a two-stage probabilistic inference paradigm, ThinkSum, which reasons over sets of objects or facts in a structured manner. In the first stage (Think - retrieval of associations), a LLM is queried in parallel over a set of phrases extracted from the prompt or an auxiliary model call. In the second stage (Sum - probabilistic inference or reasoning), the results of these queries are aggregated to make the final prediction. We demonstrate the possibilities and advantages of ThinkSum on the BIG-bench suite of LLM evaluation tasks, achieving improvements over the state of the art using GPT-family models on thirteen difficult tasks, often with far smaller model variants. We also compare and contrast ThinkSum with other proposed modifications to direct prompting of LLMs, such as variants of chain-of-thought prompting. Our results suggest that because the probabilistic inference in ThinkSum is performed outside of calls to the LLM, ThinkSum is less sensitive to prompt design, yields more interpretable predictions, and can be flexibly combined with latent variable models to extract structured knowledge from LLMs. Overall, our proposed paradigm represents a promising approach for enhancing the reasoning capabilities of LLMs.

Sources

Related papers