Sahaj.blogs

Day page

20 Oct 2025

Study

Transformers · Multi-head attention

Splitting d_model into several heads so attention can look at different things at once, implemented by reshaping, and checked against nn.MultiheadAttention.

  • Multi-head attention runs h smaller attentions in parallel and concatenates the results.
  • With d_k = d_model / h, total compute stays about the same as one big head.

Read the post →

Paper summary

Paper summary: A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning

A theoretical framework for sampling-based test-time scaling shows self-consistency has high estimation error and perplexity has high modelling error; the hybrid RPC method matches self-consistency with half the samples.

  • Self-consistency (majority vote) and perplexity-based selection fail in different ways: one has high estimation error, the other high modelling error.
  • RPC reaches self-consistency's accuracy while cutting sampling cost by 50%.

Read the post →