Predicting the Next Character
bigram counts → learned embeddings → NLL.
The least mystical version of language modeling: a counting table vs. a tiny neural model with learned embeddings. Watch the neural model overfit a tiny corpus, then deliberately destroy its embeddings and see what fails.
The Concept
Start with the least mystical version of language generation possible: a bigram counting table. For every pair of adjacent characters in the training corpus, increment a counter. To generate text, look at the current character, read the matching row, turn counts into probabilities, sample.
Then upgrade: replace the explicit count table with a tiny MLP that uses learned character embeddings and a fixed context window of block_size = 3 previous characters. The neural model can share statistical strength across similar characters; the counting table cannot.
Why It Matters
Once you build both, "embedding" stops sounding abstract. It becomes a concrete row of numbers indexed by token ID, learned so that distinct inputs land at distinct internal identities. Temperature stops being a mysterious slider and becomes division of logits before softmax. Cross-entropy stops being a category in PyTorch and becomes the natural penalty for "how surprised should the model be by the actual next character?"
And — the most important lesson of this project — baseline models are not toys. They are sanity rails. The first time you train a neural character LM on a tiny corpus, you may find the bigram baseline ties or beats it on held-out data because the neural model overfit. That experience permanently calibrates your trust in training loss alone.
The Build
build.py$python projects/02_predicting-the-next-character/build.py --tiny
"""
Project 2: Predicting the Next Character — complete working build.
Two character-level language models on a tiny names corpus:
1. A bigram counting model (no learning — just a count table normalized to probabilities).
2. A neural MLP with learned character embeddings and a fixed context window.
Both produce samples and a negative log-likelihood (NLL) score on held-out data.
Run:
python build.py --tiny # 500 training steps, <30s on CPU
python build.py --full # 10000 steps, ~3 min on CPU
"""
from __future__ import annotations
import argparse
import math
from pathlib import Path
import torch
import torch.nn.functional as F
# A small built-in names corpus (no network, no file fetch).
# 80 short names; enough for the bigram to make sense and the neural model
# to clearly outperform it.
DEFAULT_NAMES = [
"emma",
"olivia",
"ava",
"isabella",
"sophia",
"charlotte",
"mia",
"amelia",
"harper",
"evelyn",
"abigail",
"emily",
"elizabeth",
"mila",
"ella",
"avery",
"sofia",
"camila",
"aria",
"scarlett",
"victoria",
"madison",
"luna",
"grace",
"chloe",
"penelope",
"layla",
"riley",
"zoey",
"nora",
"lily",
"eleanor",
"hannah",
"lillian",
"addison",
"aubrey",
"ellie",
"stella",
"natalie",
"zoe",
"leah",
"hazel",
"violet",
"aurora",
"savannah",
"audrey",
"brooklyn",
"bella",
"claire",
"skylar",
"lucy",
"paisley",
"everly",
"anna",
"caroline",
"nova",
"genesis",
"emilia",
"kennedy",
"samantha",
"maya",
"willow",
"kinsley",
"naomi",
"aaliyah",
"elena",
"sarah",
"ariana",
"allison",
"gabriella",
"alice",
"madelyn",
"cora",
"ruby",
"eva",
"serenity",
"autumn",
"adeline",
"hailey",
"gianna",
]
def build_vocab(words: list[str]) -> tuple[list[str], dict[str, int], dict[int, str]]:
"""Vocabulary is all unique characters in the corpus, plus a special `.` token."""
chars = sorted(set("".join(words)))
chars = ["."] + chars # `.` as start/end token
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
return chars, stoi, itos
def build_bigram_counts(words: list[str], stoi: dict[str, int]) -> torch.Tensor:
"""Count transitions. Rows = current char, cols = next char. Returns int32 (V, V)."""
V = len(stoi)
N = torch.zeros((V, V), dtype=torch.int32)
for word in words:
chs = ["."] + list(word) + ["."]
for ch1, ch2 in zip(chs, chs[1:]):
i, j = stoi[ch1], stoi[ch2]
N[i, j] += 1
return N
def bigram_probs(N: torch.Tensor, smoothing: int = 1) -> torch.Tensor:
"""Convert counts to probabilities; add `+smoothing` to every cell to avoid zeros."""
P = (N + smoothing).float()
P = P / P.sum(dim=1, keepdim=True)
return P
def bigram_nll(P: torch.Tensor, words: list[str], stoi: dict[str, int]) -> float:
"""Negative log-likelihood per character on `words`. Lower is better."""
log_likelihood = 0.0
n = 0
for word in words:
chs = ["."] + list(word) + ["."]
for ch1, ch2 in zip(chs, chs[1:]):
i, j = stoi[ch1], stoi[ch2]
log_likelihood += math.log(P[i, j].item())
n += 1
return -log_likelihood / n
def bigram_sample(
P: torch.Tensor, itos: dict[int, str], n_samples: int, seed: int = 0
) -> list[str]:
g = torch.Generator().manual_seed(seed)
samples: list[str] = []
start = 0 # `.` is index 0
for _ in range(n_samples):
ix = start
out: list[str] = []
while True:
ix = int(torch.multinomial(P[ix], 1, replacement=True, generator=g).item())
ch = itos[ix]
if ch == ".":
break
out.append(ch)
if len(out) > 30: # safety bound
break
samples.append("".join(out))
return samples
def build_neural_training_data(
words: list[str], stoi: dict[str, int], block_size: int
) -> tuple[torch.Tensor, torch.Tensor]:
"""For each word, create (context, target) pairs with block_size context."""
X: list[list[int]] = []
Y: list[int] = []
for word in words:
context = [stoi["."]] * block_size
for ch in list(word) + ["."]:
ix = stoi[ch]
X.append(context.copy())
Y.append(ix)
context = context[1:] + [ix]
return torch.tensor(X, dtype=torch.long), torch.tensor(Y, dtype=torch.long)
class NeuralCharLM:
"""A minimal MLP language model with learned character embeddings.
Parameters: C (embedding table), W1/b1 (hidden), W2/b2 (output).
No PyTorch nn.Module — we keep tensors explicit so the reader sees every weight.
"""
def __init__(
self,
vocab_size: int,
block_size: int,
embed_dim: int,
hidden_size: int,
seed: int = 0,
) -> None:
g = torch.Generator().manual_seed(seed)
self.V = vocab_size
self.block_size = block_size
self.C = torch.randn((vocab_size, embed_dim), generator=g).requires_grad_(True)
self.W1 = (
torch.randn((block_size * embed_dim, hidden_size), generator=g)
* (1.0 / math.sqrt(block_size * embed_dim))
).requires_grad_(True)
self.b1 = torch.zeros(hidden_size, requires_grad=True)
self.W2 = (
torch.randn((hidden_size, vocab_size), generator=g) * (1.0 / math.sqrt(hidden_size))
).requires_grad_(True)
self.b2 = torch.zeros(vocab_size, requires_grad=True)
def parameters(self) -> list[torch.Tensor]:
return [self.C, self.W1, self.b1, self.W2, self.b2]
def forward(self, X: torch.Tensor) -> torch.Tensor:
emb = self.C[X] # (B, block_size, embed_dim)
x = emb.view(emb.shape[0], -1)
h = torch.tanh(x @ self.W1 + self.b1)
logits = h @ self.W2 + self.b2
return logits
def loss(self, X: torch.Tensor, Y: torch.Tensor) -> torch.Tensor:
return F.cross_entropy(self.forward(X), Y)
@torch.no_grad()
def sample(
self,
itos: dict[int, str],
stoi: dict[str, int],
n_samples: int,
temperature: float = 1.0,
seed: int = 0,
) -> list[str]:
g = torch.Generator().manual_seed(seed)
samples: list[str] = []
for _ in range(n_samples):
context = [stoi["."]] * self.block_size
out: list[str] = []
while True:
x = torch.tensor([context], dtype=torch.long)
logits = self.forward(x)
probs = torch.softmax(logits / max(temperature, 1e-6), dim=1)
ix = int(torch.multinomial(probs, 1, generator=g).item())
ch = itos[ix]
if ch == ".":
break
out.append(ch)
context = context[1:] + [ix]
if len(out) > 30:
break
samples.append("".join(out))
return samples
def train_neural(
model: NeuralCharLM,
X_tr: torch.Tensor,
Y_tr: torch.Tensor,
X_val: torch.Tensor,
Y_val: torch.Tensor,
steps: int,
batch_size: int,
lr: float,
eval_every: int,
seed: int = 0,
) -> tuple[list[float], list[float]]:
g = torch.Generator().manual_seed(seed)
train_history: list[float] = []
val_history: list[float] = []
n = X_tr.shape[0]
for step in range(steps):
idx = torch.randint(0, n, (batch_size,), generator=g)
Xb, Yb = X_tr[idx], Y_tr[idx]
loss = model.loss(Xb, Yb)
for p in model.parameters():
if p.grad is not None:
p.grad = None
loss.backward()
for p in model.parameters():
if p.grad is None:
continue # parameter was frozen (e.g. via requires_grad=False)
with torch.no_grad():
p -= lr * p.grad # type: ignore[operator]
if step % eval_every == 0 or step == steps - 1:
train_history.append(float(loss.item()))
with torch.no_grad():
val_history.append(float(model.loss(X_val, Y_val).item()))
return train_history, val_history
def write_loss_curve_png(
train_hist: list[float], val_hist: list[float], steps_per_point: int, path: Path
) -> None:
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
except ImportError:
return
xs = [i * steps_per_point for i in range(len(train_hist))]
fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(xs, train_hist, label="train")
ax.plot(xs, val_hist, label="val")
ax.set_xlabel("training step")
ax.set_ylabel("cross-entropy loss")
ax.set_title("Project 2: neural char-LM training")
ax.legend()
ax.grid(True, alpha=0.3)
fig.tight_layout()
fig.savefig(path, dpi=120)
plt.close(fig)
def main() -> int:
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("--tiny", action="store_true")
parser.add_argument("--full", action="store_true")
parser.add_argument("--steps", type=int, default=None)
parser.add_argument("--block-size", type=int, default=3)
parser.add_argument("--embed-dim", type=int, default=8)
parser.add_argument("--hidden-size", type=int, default=64)
parser.add_argument("--batch-size", type=int, default=32)
parser.add_argument("--lr", type=float, default=0.1)
parser.add_argument("--seed", type=int, default=0)
parser.add_argument(
"--output-dir",
type=Path,
default=Path(__file__).parent / "outputs",
)
args = parser.parse_args()
if args.steps is None:
args.steps = 10_000 if args.full else 500
args.output_dir.mkdir(parents=True, exist_ok=True)
torch.manual_seed(args.seed)
words = DEFAULT_NAMES
chars, stoi, itos = build_vocab(words)
V = len(chars)
print(f"Corpus: {len(words)} names, vocab size {V}")
print(f"Vocab: {chars}")
# === Bigram counting model ===
N = build_bigram_counts(words, stoi)
P = bigram_probs(N, smoothing=1)
bigram_loss = bigram_nll(P, words, stoi)
print(f"\nBigram NLL (full corpus): {bigram_loss:.4f}")
bigram_samples = bigram_sample(P, itos, n_samples=10, seed=args.seed)
print("Bigram samples:", bigram_samples)
# === Neural model with embeddings ===
X, Y = build_neural_training_data(words, stoi, args.block_size)
n_total = X.shape[0]
n_train = int(n_total * 0.85)
X_tr, Y_tr = X[:n_train], Y[:n_train]
X_val, Y_val = X[n_train:], Y[n_train:]
print(f"\nNeural training pairs: {n_train} train / {n_total - n_train} val")
model = NeuralCharLM(
vocab_size=V,
block_size=args.block_size,
embed_dim=args.embed_dim,
hidden_size=args.hidden_size,
seed=args.seed,
)
n_params = sum(p.numel() for p in model.parameters())
print(f"Neural model parameters: {n_params}")
eval_every = max(1, args.steps // 20)
train_hist, val_hist = train_neural(
model,
X_tr,
Y_tr,
X_val,
Y_val,
steps=args.steps,
batch_size=args.batch_size,
lr=args.lr,
eval_every=eval_every,
seed=args.seed,
)
print(
f"\nNeural after {args.steps} steps: "
f"train_loss={train_hist[-1]:.4f} val_loss={val_hist[-1]:.4f}"
)
# Temperature sweep — generate samples at several temperatures.
print("\nNeural samples at varied temperatures:")
temp_samples: dict[float, list[str]] = {}
for temp in [0.5, 1.0, 1.5]:
s = model.sample(itos, stoi, n_samples=8, temperature=temp, seed=args.seed)
temp_samples[temp] = s
print(f" T={temp}: {s}")
# === Outputs ===
write_loss_curve_png(
train_hist, val_hist, steps_per_point=eval_every, path=args.output_dir / "loss_curve.png"
)
log = args.output_dir / "run_log.txt"
lines = [
"# Project 2 run log",
f"corpus_size : {len(words)} names",
f"vocab_size : {V}",
f"block_size : {args.block_size}",
f"embed_dim : {args.embed_dim}",
f"hidden_size : {args.hidden_size}",
f"steps : {args.steps}",
"",
"# bigram model (no learning, just counts + smoothing)",
f"bigram_NLL : {bigram_loss:.4f}",
f"bigram_samples : {bigram_samples}",
"",
"# neural model",
f"n_parameters : {n_params}",
f"train_loss_final : {train_hist[-1]:.4f}",
f"val_loss_final : {val_hist[-1]:.4f}",
"",
"# temperature sweep",
]
for temp, samples in temp_samples.items():
lines.append(f"T={temp}: {samples}")
lines += [
"",
"# comparison",
f" bigram NLL: {bigram_loss:.4f}",
f" neural val: {val_hist[-1]:.4f}",
f" delta: {bigram_loss - val_hist[-1]:+.4f}",
]
log.write_text("\n".join(lines) + "\n", encoding="utf-8")
print(f"\nOutputs written to {args.output_dir.resolve()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
The Steps
5 reference filesEach step is the code for one piece, in isolation — read them as the book introduces them. They assemble into build.py above.
Step 02Build The Vocabularystep_02_build-the-vocabulary.py
"""
Project 2: Step 2 — Build the vocabulary
Pedagogical reference: this file shows the code for this step in isolation.
For the full assembled, runnable build, use build.py in this same folder.
"""
chars = sorted(list(set(text)))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
chars = ["."] + sorted(list(set(text.replace("\n", ""))))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}
Step 03Bigram Countsstep_03_bigram-counts.py
"""
Project 2: Step 3 — Bigram counts
Pedagogical reference: this file shows the code for this step in isolation.
For the full assembled, runnable build, use build.py in this same folder.
"""
import torch
N = torch.zeros((V, V), dtype=torch.int32)
for word in words:
chs = ["."] + list(word) + ["."]
for ch1, ch2 in zip(chs, chs[1:]):
i = stoi[ch1]
j = stoi[ch2]
N[i, j] += 1
Step 04Turn Counts Into Probabilitiesstep_04_turn-counts-into-probabilities.py
"""
Project 2: Step 4 — Turn counts into probabilities
Pedagogical reference: this file shows the code for this step in isolation.
For the full assembled, runnable build, use build.py in this same folder.
"""
P = N.float()
P = P / P.sum(dim=1, keepdim=True)
P = (N + 1).float()
P = P / P.sum(dim=1, keepdim=True)
Step 05Sample From The Bigram Modelstep_05_sample-from-the-bigram-model.py
"""
Project 2: Step 5 — Sample from the bigram model
Pedagogical reference: this file shows the code for this step in isolation.
For the full assembled, runnable build, use build.py in this same folder.
"""
g = torch.Generator().manual_seed(42)
ix = stoi["."]
out = []
while True:
probs = P[ix]
ix = torch.multinomial(probs, num_samples=1, replacement=True, generator=g).item()
ch = itos[ix]
if ch == ".":
break
out.append(ch)
print("".join(out))
Step 06Measure Negative Log Likelihood For The Bigram Modelstep_06_measure-negative-log-likelihood-for-the-bigram-model.py
"""
Project 2: Step 6 — Measure negative log-likelihood for the bigram model
Pedagogical reference: this file shows the code for this step in isolation.
For the full assembled, runnable build, use build.py in this same folder.
"""
log_likelihood = 0.0
n = 0
for word in words:
chs = ["."] + list(word) + ["."]
for ch1, ch2 in zip(chs, chs[1:]):
i = stoi[ch1]
j = stoi[ch2]
prob = P[i, j]
log_likelihood += torch.log(prob)
n += 1
nll = -log_likelihood / n
print(nll.item())
Break It
break_it.pyWe collapse every row of the embedding table C to be the same vector, then freeze C so gradients cannot re-introduce row distinctions. The lookup C[X] now returns identical vectors for every character: the model can no longer condition on which character it sees.
mode final train final val
--------------------------------------------------
baseline 1.8098 2.5044
collapsed embeds 2.7993 2.7817
max row-to-row difference in broken model's C after training: 0.000000
(rows stayed identical — model genuinely could not condition on character)
The broken model's loss plateaus around 2.78. For comparison: uniform-random would be log(24) ≈ 3.18, so the model still learns the global character frequency distribution (some characters are more common at end-of-sequence than mid-sequence; that signal survives without distinguishing inputs). But it cannot do better — there is no usable information about what character it just saw.
Lesson: an embedding's job is to route distinct token IDs to distinct vectors. That is the entire mechanism. "Embedding represents meaning" is a one-liner that hides the work. The work is the lookup. Break the routing and the rest of the network has nothing to do.
(A subtle gotcha: if you collapse C without freezing it, gradients flowing back through the embedding lookup will re-introduce row distinctions within a few hundred steps. The break_it.py here freezes C so the sabotage is observable.)
$python projects/02_predicting-the-next-character/break_it.py --tiny
"""
Project 2: BREAK IT — destroy the embedding table's ability to distinguish characters.
We force every row of the embedding table to be the same vector. The forward
pass still runs (shapes are fine), but every character lookup returns the same
vector, so the model can no longer condition on which character it sees.
Expected: the broken model converges to a "global character frequency" predictor
and its loss plateaus far above the baseline.
Run:
python break_it.py --tiny
"""
from __future__ import annotations
import argparse
from pathlib import Path
import torch
from build import (
DEFAULT_NAMES,
NeuralCharLM,
build_neural_training_data,
build_vocab,
train_neural,
)
def collapse_embeddings(model: NeuralCharLM) -> None:
"""
Force every embedding row to equal row 0 AND freeze C so the sabotage sticks.
Without freezing, gradients flowing back through C re-introduce row
distinctions within a few hundred steps. Freezing keeps the collapse
intact so we can see the model's loss plateau without the embedding signal.
"""
with torch.no_grad():
model.C[:] = model.C[0]
model.C.requires_grad_(False)
def main() -> int:
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("--tiny", action="store_true")
parser.add_argument("--full", action="store_true")
parser.add_argument("--steps", type=int, default=None)
parser.add_argument("--block-size", type=int, default=3)
parser.add_argument("--embed-dim", type=int, default=8)
parser.add_argument("--hidden-size", type=int, default=64)
parser.add_argument("--batch-size", type=int, default=32)
parser.add_argument("--lr", type=float, default=0.1)
parser.add_argument("--seed", type=int, default=0)
parser.add_argument(
"--output-dir",
type=Path,
default=Path(__file__).parent / "outputs",
)
args = parser.parse_args()
if args.steps is None:
args.steps = 10_000 if args.full else 500
args.output_dir.mkdir(parents=True, exist_ok=True)
torch.manual_seed(args.seed)
words = DEFAULT_NAMES
_chars, stoi, _itos = build_vocab(words)
V = len(stoi)
X, Y = build_neural_training_data(words, stoi, args.block_size)
n_train = int(X.shape[0] * 0.85)
X_tr, Y_tr = X[:n_train], Y[:n_train]
X_val, Y_val = X[n_train:], Y[n_train:]
eval_every = max(1, args.steps // 10)
# === Baseline ===
torch.manual_seed(args.seed)
baseline = NeuralCharLM(
vocab_size=V,
block_size=args.block_size,
embed_dim=args.embed_dim,
hidden_size=args.hidden_size,
seed=args.seed,
)
base_train, base_val = train_neural(
baseline,
X_tr,
Y_tr,
X_val,
Y_val,
steps=args.steps,
batch_size=args.batch_size,
lr=args.lr,
eval_every=eval_every,
seed=args.seed,
)
# === Broken: collapse all embedding rows to the same vector ===
torch.manual_seed(args.seed)
broken = NeuralCharLM(
vocab_size=V,
block_size=args.block_size,
embed_dim=args.embed_dim,
hidden_size=args.hidden_size,
seed=args.seed,
)
collapse_embeddings(broken)
broken_train, broken_val = train_neural(
broken,
X_tr,
Y_tr,
X_val,
Y_val,
steps=args.steps,
batch_size=args.batch_size,
lr=args.lr,
eval_every=eval_every,
seed=args.seed,
)
print(f"{'mode':20s} {'final train':>14s} {'final val':>12s}")
print(f"{'-' * 50}")
print(f"{'baseline':20s} {base_train[-1]:>14.4f} {base_val[-1]:>12.4f}")
print(f"{'collapsed embeds':20s} {broken_train[-1]:>14.4f} {broken_val[-1]:>12.4f}")
# Check whether the embedding rows stayed identical during training.
# If gradient flow doesn't preserve the collapse, broken model recovers.
rows_max_diff = (broken.C - broken.C[0]).abs().max().item()
print(f"\nmax row-to-row difference in broken model's C after training: {rows_max_diff:.6f}")
if rows_max_diff > 0.01:
print("(rows drifted — gradients re-introduced distinction; sabotage was overcome)")
else:
print("(rows stayed identical — model genuinely could not condition on character)")
log = args.output_dir / "break_it_log.txt"
log.write_text(
"# Project 2 BREAK IT — collapse all embedding rows to the same vector\n"
f"baseline: train={base_train[-1]:.4f} val={base_val[-1]:.4f}\n"
f"collapsed C: train={broken_train[-1]:.4f} val={broken_val[-1]:.4f}\n"
f"max row diff: {rows_max_diff:.6f}\n"
"\n"
"Lesson: an embedding's job is to route distinct token IDs to distinct\n"
"vectors. If every row of C is the same, the lookup C[X] returns identical\n"
"vectors for every character — the network sees one input regardless of\n"
"actual characters. The flattened context becomes a constant, the hidden\n"
"layer can only encode 'how long until end-of-sequence', and the model\n"
"regresses to a global character-frequency predictor.\n"
"\n"
"Note: if you let training continue, gradients flowing back through C\n"
"will gradually re-introduce row distinctions. The 'collapse + retrain'\n"
"demo above intentionally collapses BEFORE training so we can watch loss\n"
"plateau higher than baseline.\n",
encoding="utf-8",
)
print(f"\nLog written to {log.resolve()}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Outputs
break_it_log.txt
# Project 2 BREAK IT — collapse all embedding rows to the same vector
baseline: train=1.8098 val=2.5044
collapsed C: train=2.7993 val=2.7817
max row diff: 0.000000
Lesson: an embedding's job is to route distinct token IDs to distinct
vectors. If every row of C is the same, the lookup C[X] returns identical
vectors for every character — the network sees one input regardless of
actual characters. The flattened context becomes a constant, the hidden
layer can only encode 'how long until end-of-sequence', and the model
regresses to a global character-frequency predictor.
Note: if you let training continue, gradients flowing back through C
will gradually re-introduce row distinctions. The 'collapse + retrain'
demo above intentionally collapses BEFORE training so we can watch loss
plateau higher than baseline.
run_log.txt
# Project 2 run log
corpus_size : 80 names
vocab_size : 24
block_size : 3
embed_dim : 8
hidden_size : 64
steps : 500
# bigram model (no learning, just counts + smoothing)
bigram_NLL : 2.3337
bigram_samples : ['szkbubf', 'a', 'llvilrtvdhi', 'a', 'nera', 'chsotepruyfdia', 'vmnoctpdvtapcsckiomctzs', 'mallia', 'gipdwkcwimz', 'evgta']
# neural model
n_parameters : 3352
train_loss_final : 1.8098
val_loss_final : 2.5044
# temperature sweep
T=0.5: ['aanisley', 'alia', 'ara', 'daile', 'aaria', 'hanna', 'aula', 'aalia']
T=1.0: ['sik', 'aha', 'amlovil', 'avdh', 'bella', 'aha', 'sete', 'ruylara']
T=1.5: ['siklah', 'baoliviortvdh', 'belnereprhsetepruyleyi', 'vmnoctldvta', 'ciariomctrsemmdlia', 'onpdika', 'smo', 'emm']
# comparison
bigram NLL: 2.3337
neural val: 2.5044
delta: -0.1707Read it in the book
This page ships the code. The full derivation — why the chain rule is mechanical not magical, what the computational graph buys you, the long-form reasoning — lives in Chapter 2 of Under the Hood.
Get the book →