Transformers · GPT

GPT models are decoder-only Transformers: masked self-attention and feed-forward layers, no encoder and no cross-attention. The training objective is next-token prediction:

$\mathcal{L} = -\sum_t \log p(x_t \mid x_{<t})$

The causal mask is what makes this efficient. One forward pass over a sequence gives predictions for every position at once, each seeing only its left context.

Perplexity = exp(average loss): roughly, how many choices the model is “confused between” per token. Normal text should get a low perplexity and shuffled text a high one.

I used GPT-2 small (124M parameters), which OpenAI released openly in 2019, through Hugging Face. Greedy decoding keeps the output deterministic.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
gpt = AutoModelForCausalLM.from_pretrained("gpt2").eval()
print("parameters:", round(sum(p.numel() for p in gpt.parameters()) / 1e6, 1), "M")

prompt = "Machine learning is"
ids = tok(prompt, return_tensors="pt").input_ids
with torch.no_grad():
    out = gpt.generate(ids, max_new_tokens=20, do_sample=False, pad_token_id=tok.eos_token_id)
print(repr(tok.decode(out[0])))

def perplexity(text):
    enc = tok(text, return_tensors="pt")
    with torch.no_grad():
        loss = gpt(**enc, labels=enc.input_ids).loss
    return torch.exp(loss).item()

normal = "The students went to the library to study for their exams."
shuffled = "exams library the study went for to their The students to."
print(f"perplexity normal: {perplexity(normal):.1f} | shuffled: {perplexity(shuffled):.1f}")

with torch.no_grad():
    logits = gpt(ids).logits[0, -1]
top = logits.softmax(-1).topk(5)
print("next-token candidates:", [(tok.decode(i), round(p.item(), 3)) for p, i in zip(top.values, top.indices)])

Output:

parameters: 124.4 M
"Machine learning is a very powerful tool for learning about the world around us. It's a tool that can help us"
perplexity normal: 19.8 | shuffled: 2939.9
next-token candidates: [(' a', 0.138), (' the', 0.055), (' not', 0.042), (' an', 0.036), (' one', 0.017)]