Faithfulness of Chain-of-Thought Explanations in Small Language Models

Abstract

Chain-of-Thought (CoT) explanations are widely adopted as indicators of model reasoning in language models. Despite their popularity, the causal relationship between generated reasoning traces and final predictions remains unclear—particularly for small language models (SLMs) deployed on commodity hardware. In this paper, we present a systematic evaluation of explanation faithfulness by measuring output sensitivity to controlled perturbations of generated CoT traces.

We study multiple tasks (arithmetic reasoning, symbolic logic, commonsense QA, and algorithmic decision-making) across several SLMs ($\leq 4B$ parameters). Our results reveal task-dependent inconsistencies between explanation content and decision dependence: in many settings, predictions remain invariant under substantial perturbations to CoT, while in others small changes to reasoning tokens induce large output shifts. These findings challenge the use of CoT as a reliable proxy for interpretability and motivate alternative evaluation protocols for faithful explanations. We release a fully reproducible notebook suite runnable on a single GPU/CPU workstation.

1. Introduction

Chain-of-Thought prompting has become a de facto technique to elicit intermediate reasoning steps from language models. Following influential work by Anthropic (2023) on the interpretability and risks of revealing reasoning traces, CoT has been used both as a performance booster and as a window into model cognition. However, whether CoT explanations faithfully reflect the internal decision process—rather than serving as post-hoc rationalizations—remains an open question.

While prior work has primarily focused on large language models (LLMs), small language models are increasingly relevant due to efficiency, privacy, and deployability constraints. These models may rely on different inductive biases and internal representations, raising the question: Are CoT explanations more or less faithful in SLMs?

This paper makes the following contributions:

1. We formalize explanation faithfulness via sensitivity of predictions to perturbations of reasoning traces.
2. We propose a perturbation-based evaluation framework applicable to any autoregressive language model.
3. We conduct extensive experiments on SLMs up to 4B parameters across diverse reasoning tasks.
4. We demonstrate systematic mismatches between CoT content and causal decision dependence.

2. Related Work

2.1 Chain-of-Thought Prompting

CoT prompting was popularized by Wei et al. (2022) as a method to improve reasoning performance by encouraging models to generate intermediate steps. Subsequent studies explored zero-shot CoT (Kojima et al., 2022), self-consistency (Wang et al., 2022), and structured reasoning formats (Zhou et al., 2022). However, most of this work focuses on performance improvements rather than the causal relationship between reasoning and final answers.

2.2 Faithfulness and Interpretability

Interpretability research distinguishes plausibility from faithfulness (Jacobi and Goldberg, 2020). An explanation may appear reasonable to humans yet be causally disconnected from the model's computation. Perturbation tests (Ribeiro et al., 2020), input erasure (Feng et al., 2018), and causal mediation analyses (Vig et al., 2020) have been proposed to address this gap.

2.3 Concerns About Revealed Reasoning

Anthropic (2023) argues that exposed CoT may be misleading or unsafe, advocating for separating internal reasoning from user-facing explanations. Our work empirically supports these concerns in the context of SLMs, providing concrete evidence of faithfulness violations.

3.2 Faithfulness Metrics

We define two complementary metrics to quantify faithfulness:

High faithfulness implies that meaningful perturbations to $r$ should yield high OFR and PS, while trivial perturbations should yield low values. Conversely, if substantial perturbations produce negligible output changes, the explanation is unfaithful—the model's decision does not causally depend on the reasoning content.

4. Experimental Setup

4.1 Models

We evaluate five small language models spanning different architectures and training paradigms:

4.2 Tasks and Datasets

• Arithmetic Reasoning (GSM8K-Subset): 500 grade-school math problems requiring multi-step calculation. Ground truth answers are numerical, enabling exact match evaluation.
• Symbolic Logic (ProntoQA): 400 synthetic logical reasoning problems with explicit deduction chains. Tests sensitivity to logical structure preservation.
• Commonsense QA (StrategyQA): 450 yes/no questions requiring implicit world knowledge and multi-hop reasoning.
• Algorithmic Decision-Making (BBH-Navigate): 300 navigation problems from BIG-Bench Hard requiring step-by-step state tracking.

4.3 Prompting Protocol

For each instance, we first generate a complete response via greedy decoding, extract the reasoning trace $r$ and answer $y$, then apply perturbations and measure output changes through constrained generation.

4.4 Perturbation Implementation

• Token Masking (M-k%): Randomly replace $k\%$ of reasoning tokens with [MASK] placeholder. We evaluate $k \in \{10, 25, 50, 75\}$.
• Token Shuffling (S-k%): Randomly permute $k\%$ of token positions within the reasoning trace, preserving the overall length.
• Semantic Substitution (SS): Replace nouns, verbs, and numerical values with semantically related alternatives using WordNet synsets and numerical perturbation ($\pm 1$).
• Truncation (T-k%): Remove the final $k\%$ of the reasoning trace before the answer token.

4.5 Evaluation Protocol

For each (model, task, perturbation) triple, we:

1. Generate baseline CoT responses for all test instances
2. Apply the perturbation operator to each reasoning trace
3. Condition the model on the perturbed trace and decode the answer
4. Compute OFR, PS, and $\mathcal{F}$ metrics
5. Report means and 95% confidence intervals across 3 random seeds

5. Results

5.1 Overall Faithfulness Patterns

Table 1 presents aggregate faithfulness scores across all model-task combinations.

5.2 Output Flip Rates Under Perturbation

Figure 1 (described) shows OFR as a function of perturbation intensity for the masking operator.

The commonsense task exhibits remarkably low flip rates even under severe perturbation (75% masking → 14.7% OFR), suggesting that predictions are largely independent of reasoning content.

5.3 Probability Shift Analysis

Table 3 presents mean probability shifts, revealing a more nuanced picture than binary flip rates.

Even when answers don't flip, semantic substitution induces measurable probability shifts in structured reasoning tasks but minimal shifts in commonsense QA.

5.4 Truncation Effects

Truncation experiments reveal when in the reasoning chain the model "commits" to an answer.

5.5 Task-Specific Deep Dives

5.5.1 Arithmetic Reasoning

We further analyze whether specific reasoning components (operand mentions, operation keywords, intermediate results) differentially affect outputs.

Perturbations to intermediate calculation results produce the largest effects, suggesting these tokens carry genuine computational weight.

5.5.2 Commonsense QA

The persistently low faithfulness in commonsense tasks warrants investigation. We hypothesize that models rely on direct question-answer associations rather than reasoning chains.

To test this, we replace CoT traces with:

• Random reasoning: Shuffled sentences from other instances
• Contradictory reasoning: Reasoning supporting the opposite answer
• Null reasoning: Empty trace

6. Analysis and Discussion

6.1 Why Does Faithfulness Vary by Task?

We propose three hypotheses for the observed task dependence:

6.2 Implications for Interpretability

Our findings challenge the widespread assumption that CoT explanations provide insight into model cognition. Specifically:

1. CoT ≠ Explanation: High-quality, plausible CoT does not guarantee causal relevance. Models may generate coherent reasoning that is disconnected from their actual decision process.
2. Task-Conditional Trust: Practitioners should calibrate trust in CoT explanations based on task characteristics. Structured reasoning tasks warrant more trust than knowledge-intensive tasks.
3. Evaluation Requirements: Deployment of CoT-based systems for high-stakes decisions should include perturbation-based faithfulness audits.

6.3 Comparison with Large Language Models

While direct comparison is outside our scope, we note that concurrent work on LLMs (>70B parameters) suggests higher aggregate faithfulness scores. This raises the possibility that faithfulness emerges with scale—though task dependence persists even in large models.

6.4 Architectural Considerations

We observe that models trained on synthetic reasoning data (Phi-2) exhibit higher faithfulness than those trained primarily on web text (TinyLlama). This suggests that training data composition may be more predictive of faithfulness than raw parameter count.

7. Limitations

• Perturbation Validity: Our perturbation operators may not fully capture the space of meaningful reasoning modifications. Some perturbations may be detectable as out-of-distribution, causing models to behave anomalously.
• Task Coverage: We evaluate four task categories; other reasoning domains (causal reasoning, spatial reasoning, temporal reasoning) may exhibit different faithfulness patterns.
• Model Coverage: Our evaluation is limited to instruction-tuned models ≤4B parameters. Base models and larger SLMs may behave differently.
• Metric Limitations: Our faithfulness metrics assume that perturbation severity is well-defined and measurable. In practice, semantic similarity is challenging to quantify precisely.
• Confounders: Models may detect perturbations and alter behavior accordingly, rather than revealing their "true" reasoning dependence.

8. Conclusion

We present a systematic evaluation of Chain-of-Thought faithfulness in small language models, revealing substantial task-dependent variation. While structured reasoning tasks (arithmetic, logic, algorithmic) exhibit moderate faithfulness—with perturbations to reasoning traces inducing commensurate output changes—commonsense QA shows near-complete independence between explanation content and model predictions.

These findings have practical implications for the deployment of SLMs in interpretability-critical applications. We caution against treating CoT explanations as reliable indicators of model reasoning without task-specific validation. Our perturbation-based evaluation framework provides a tractable method for assessing faithfulness and can be applied to any autoregressive language model.

Future work should investigate: (1) training interventions to improve faithfulness, (2) the relationship between faithfulness and model scale, and (3) user studies on how faithfulness violations affect human trust calibration.

References

Appendix A: Qualitative Examples

A.1 Faithful Reasoning (Arithmetic)

A.2 Unfaithful Reasoning (Commonsense)

Appendix B: Computational Resources

All experiments were conducted on:

• GPU: AMD RADEON 9070XT (16GB VRAM)
• CPU: AMD Ryzen 9 7950X
• RAM: 64GB DDR5
• Storage: 2TB NVMe SSD

Total compute time: approximately 47 GPU-hours across all experiments.