Sahaj.blogs

Day page

25 Nov 2025

Study

Transformers · GPT

GPT as a decoder-only Transformer trained to predict the next token; generating with GPT-2, measuring perplexity, and seeing why causal masking makes this possible.

  • A decoder-only model is trained on one simple objective: predict the next token given everything before it.
  • Perplexity is the exponential of the average next-token loss; lower means the text looks more expected to the model.

Read the post →

Paper summary

Paper summary: General Agentic Memory Via Deep Research

General Agentic Memory keeps a light memory plus the full history in a page-store, then uses a "researcher" agent to build the right context at request time, like just-in-time compilation for memory.

  • Pre-summarising memory in advance loses information; GAM keeps everything and does the "deep research" when a request arrives.
  • Two parts: a Memorizer that keeps a light index and the full page-store, and a Researcher that retrieves and integrates at runtime.

Read the post →