How and why does CoT work?
Chain of thought is one of the most powerful prompting techniques nowadays. However, the internal mechanics and intuition behind it are still a grey area.
Several well-proven methods exist for discovering the internals of LLMs(including prompting techniques). Fictional ontology - to ensure that the model can not use world knowledge acquired during pre-training. Circuits - finding subgraphs in the model responsible for completing a task. Activation patching -comparing outputs of the model when one output is original and the second is corrupted. Knockout - prune nodes in the computational graph to find circuits. A prerequisite for such research is an understanding of Transformer architecture(attention heads) and an inference pipeline. The first step is to do a task decomposition of a prompting technique in question.
CoT can be split into 3 sub-tasks: decision-making(what reasoning to follow), copying(what information needs to be copied from the input to the output) and induction(finding new relations). The 'How to think step-by-step' paper reviews these sub-tasks in detail. The first conclusion is that LLM uses the question context, generated context and few-shot examples to generate an answer, which means that the model benefits from the dynamic nature of CoT. At the same time, different sub-tasks use different parts of the context. Based on Llama-2 experiments, there is a functional shift, where before the 16-th layer, the model relies more on pre-trained information and, after this - more on the input context.
If the previous paper focuses on the internal mechanics of LLM, the 'Understanding Chain-of-Though' paper gives us a formal language to describe CoT reasoning. This work assumes that each new step in CoT adds meaningful and relevant information towards the correct final output. The paper calls this the information gain. A separate LLM (the supervisor model) was trained to calculate the information gain. So, we send each step from CoT to the supervisor model, which returns a score. If the score is positive, we have added new insights, and the step is correct.
The information gain can be used for evaluation, but it also shows why CoT works. The first paper shows that asking a model to generate reasoning steps(CoT prompting) helps the model generate better results. The second paper shows that each reasoning step adds information to the current answer.
References
https://arxiv.org/abs/2402.18312 - How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning
https://arxiv.org/abs/2411.11984 - Understanding Chain-of-Thought in LLMs through Information Theory
https://arxiv.org/abs/2201.11903 - Chain-of-Thought Prompting Elicits Reasoning in Large Language Models