atlas-7b-sft

RUN 4F21C0 · 8 x H100 · STEP 41,600 OF 42,000 · EPOCH 3.0

Val loss rising since 39k 02:41:07

Loss

train and held-out validation, log scale
4.03.02.21.6
010.5k21k31.5k42k

Train Validation The gap opening after step 39k is the reason the run stops at 42k.

Validation loss

best 1.471 at 38.5k

1.488

Train loss
1.452
Perplexity
4.43
Grad norm
0.61
Tokens seen
1.38 T

Learning rate

warmup 2k, cosine to 3e-5
Current
3.4e-5
Peak
3.0e-4
Optimiser
AdamW

Confusion matrix

eval set, row-normalised
neujoyangfeasadsurdistru
neutralneutral predicted neutral: 80 percentneutral predicted joy: 1 percentneutral predicted anger: 0 percentneutral predicted fear: 9 percentneutral predicted sad: 9 percentneutral predicted surprise: 1 percentneutral predicted disgust: 5 percentneutral predicted trust: 5 percent
joyjoy predicted neutral: 11 percentjoy predicted joy: 91 percentjoy predicted anger: 10 percentjoy predicted fear: 3 percentjoy predicted sad: 2 percentjoy predicted surprise: 4 percentjoy predicted disgust: 5 percentjoy predicted trust: 4 percent
angeranger predicted neutral: 0 percentanger predicted joy: 3 percentanger predicted anger: 87 percentanger predicted fear: 5 percentanger predicted sad: 6 percentanger predicted surprise: 2 percentanger predicted disgust: 2 percentanger predicted trust: 5 percent
fearfear predicted neutral: 2 percentfear predicted joy: 0 percentfear predicted anger: 8 percentfear predicted fear: 94 percentfear predicted sad: 19 percentfear predicted surprise: 1 percentfear predicted disgust: 8 percentfear predicted trust: 8 percent
sadsad predicted neutral: 8 percentsad predicted joy: 5 percentsad predicted anger: 7 percentsad predicted fear: 14 percentsad predicted sad: 95 percentsad predicted surprise: 14 percentsad predicted disgust: 6 percentsad predicted trust: 7 percent
surprisesurprise predicted neutral: 6 percentsurprise predicted joy: 2 percentsurprise predicted anger: 1 percentsurprise predicted fear: 5 percentsurprise predicted sad: 1 percentsurprise predicted surprise: 84 percentsurprise predicted disgust: 14 percentsurprise predicted trust: 1 percent
disgustdisgust predicted neutral: 2 percentdisgust predicted joy: 1 percentdisgust predicted anger: 2 percentdisgust predicted fear: 5 percentdisgust predicted sad: 4 percentdisgust predicted surprise: 18 percentdisgust predicted disgust: 95 percentdisgust predicted trust: 2 percent
trusttrust predicted neutral: 7 percenttrust predicted joy: 7 percenttrust predicted anger: 8 percenttrust predicted fear: 7 percenttrust predicted sad: 9 percenttrust predicted surprise: 6 percenttrust predicted disgust: 9 percenttrust predicted trust: 82 percent

Rows are the true class, columns the prediction, and each row sums to one. Anger against fear is the only confusion above nine percent.

Devices

utilisation, memory, temperature
cuda:0 98% 72 GB70°C
cuda:1 92% 84 GB67°C
cuda:2 89% 83 GB71°C
cuda:3 89% 82 GB64°C
cuda:4 97% 87 GB65°C
cuda:5 91% 72 GB68°C
cuda:6 98% 90 GB62°C
cuda:7 95% 85 GB62°C

cuda:3 runs two degrees hotter than the rest and holds the step time for the whole group, because the all-reduce waits for the slowest rank.

Throughput

tokens per second, last 80 steps
Mean
40,180 tok/s
Step time
1.94 s
Checkpoint every
23 steps
MFU
48.2%

Benchmarks

against the previous checkpoint
Benchmark scores and change against the previous checkpoint
SuiteScore Delta
MMLU62.4 +1.8
HellaSwag79.1 +0.6
GSM8K44.7 +3.2
HumanEval31.2 +2.1
TruthfulQA48.9 -0.4

TruthfulQA moving down while the others move up is the trade this fine-tune makes. It is a decision, not a regression to chase.