Skip to content

profile: measure the chunked_cross_entropy memory spike and find its source (#2190) [2/3] #2311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
OnePunchMonk wants to merge 1 commit into
base: main
Choose a base branch
from
Open

profile: measure the chunked_cross_entropy memory spike and find its source (#2190) [2/3] #2311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/profiling/budget_formula_sweep.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
[
{
"vocab_size": 8000,
"auto_chunk_size": 524,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 264242176,
"budget_bytes": 33554432
},
{
"vocab_size": 16000,
"auto_chunk_size": 262,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 524427264,
"budget_bytes": 33554432
},
{
"vocab_size": 32000,
"auto_chunk_size": 131,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 1048862720,
"budget_bytes": 33554432
},
{
"vocab_size": 50304,
"auto_chunk_size": 83,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 1652100608,
"budget_bytes": 33554432
},
{
"vocab_size": 100000,
"auto_chunk_size": 41,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 3314145792,
"budget_bytes": 33554432
},
{
"vocab_size": 152064,
"auto_chunk_size": 27,
"single_chunk_peak_bytes": 50332672,
"full_run_peak_extra_bytes": 5037360128,
"budget_bytes": 33554432
}
]
Binary file added docs/profiling/budget_formula_sweep.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
22 changes: 22 additions & 0 deletions docs/profiling/compile_vs_eager.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"chunk_0": {
"eager": {
"peak_bytes": 2097185792,
"elapsed_ms": 160.83151245117188
},
"compiled": {
"peak_bytes": 1048643072,
"elapsed_ms": 6.78707218170166
}
},
"chunk_128": {
"eager": {
"peak_bytes": 1585480704,
"elapsed_ms": 222.7342987060547
},
"compiled": {
"peak_bytes": 1065419264,
"elapsed_ms": 7.819295883178711
}
}
}
Binary file added docs/profiling/compile_vs_eager.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 10 additions & 0 deletions docs/profiling/end_to_end_step.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"model_name": "pythia-14m",
"vocab_size": 50304,
"seq_length": 512,
"batch_size": 8,
"peak_unchunked_bytes": 3580535808,
"peak_chunked_bytes": 2764874752,
"table_unchunked": "------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg CPU Mem Self CPU Mem CUDA Mem Self CUDA Mem # of Calls \n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n aten::mm 4.59% 33.924ms 8.62% 63.641ms 1.248ms 60.906ms 49.38% 79.954ms 1.568ms 0 B 0 B 909.19 MB 909.19 MB 51 \n aten::_log_softmax 3.41% 25.159ms 6.34% 46.822ms 46.822ms 12.868ms 10.43% 25.736ms 25.736ms 0 B 0 B 786.00 MB 786.00 MB 1 \n aten::nll_loss_backward 0.37% 2.706ms 0.47% 3.476ms 3.476ms 25.948us 0.02% 3.587ms 3.587ms 0 B 0 B 786.00 MB 786.00 MB 1 \n aten::_log_softmax_backward_data 0.01% 71.293us 0.08% 626.965us 626.965us 13.043ms 10.57% 13.043ms 13.043ms 0 B 0 B 786.00 MB 786.00 MB 1 \n aten::empty 0.30% 2.194ms 0.91% 6.718ms 32.933us 0.000us 0.00% 0.000us 0.000us 96 B 96 B 236.15 MB 236.15 MB 204 \n aten::addmm 0.95% 7.013ms 8.23% 60.752ms 2.531ms 4.617ms 3.74% 5.130ms 213.756us 0 B 0 B 116.12 MB 116.12 MB 24 \n aten::cat 4.85% 35.799ms 5.96% 44.042ms 1.468ms 857.023us 0.69% 870.941us 29.031us 0 B 0 B 66.00 MB 66.00 MB 30 \n aten::gelu 4.76% 35.142ms 8.35% 61.671ms 10.279ms 433.023us 0.35% 505.492us 84.249us 0 B 0 B 48.00 MB 48.00 MB 6 \n aten::gelu_backward 0.02% 159.754us 0.06% 442.076us 73.679us 622.145us 0.50% 622.145us 103.691us 0 B 0 B 48.00 MB 48.00 MB 6 \n aten::add 2.06% 15.207ms 3.68% 27.144ms 904.804us 499.731us 0.41% 535.918us 17.864us 0 B 0 B 42.00 MB 42.00 MB 30 \n aten::mul 1.44% 10.601ms 2.53% 18.685ms 346.027us 624.287us 0.51% 633.821us 11.737us 0 B 0 B 36.00 MB 36.00 MB 54 \n aten::neg 1.94% 14.311ms 3.27% 24.137ms 1.006ms 200.932us 0.16% 212.802us 8.867us 0 B 0 B 6.00 MB 6.00 MB 24 \n aten::resize_ 0.01% 95.246us 0.39% 2.910ms 970.154us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 2.03 MB 2.03 MB 3 \n aten::sum 2.51% 18.552ms 5.71% 42.181ms 1.406ms 1.448ms 1.17% 1.448ms 48.273us 0 B 0 B 411.00 KB 411.00 KB 30 \n aten::nll_loss_forward 1.01% 7.480ms 1.76% 12.986ms 12.986ms 31.963us 0.03% 63.926us 63.926us 0 B 0 B 1.00 KB 1.00 KB 1 \n aten::empty_strided 0.00% 24.476us 0.00% 24.476us 24.476us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 512 B 512 B 1 \n aten::slice 0.09% 632.750us 0.11% 793.038us 5.432us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 0 B 0 B 146 \n aten::as_strided 0.08% 574.081us 0.08% 574.081us 1.348us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 0 B 0 B 426 \n aten::unsqueeze 0.00% 17.726us 0.00% 19.395us 9.697us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 0 B 0 B 2 \n aten::embedding 0.01% 90.507us 6.01% 44.382ms 44.382ms 0.000us 0.00% 39.802us 39.802us 0 B 0 B 2.00 MB 0 B 1 \n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \nSelf CPU time total: 738.597ms\nSelf CUDA time total: 123.343ms\n",
"table_chunked": "------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg CPU Mem Self CPU Mem CUDA Mem Self CUDA Mem # of Calls \n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \n aten::mm 0.70% 1.291ms 1.57% 2.902ms 56.905us 65.031ms 45.73% 65.031ms 1.275ms 0 B 0 B 901.06 MB 901.06 MB 51 \n aten::cat 0.36% 658.672us 1.06% 1.963ms 61.359us 7.856ms 5.52% 7.861ms 245.643us 0 B 0 B 852.02 MB 852.02 MB 32 \n aten::_log_softmax 0.46% 855.081us 3.68% 6.815ms 212.978us 16.445ms 11.56% 16.445ms 513.894us 0 B 0 B 786.00 MB 786.00 MB 32 \n aten::_log_softmax_backward_data 0.21% 381.745us 0.50% 928.148us 29.005us 15.293ms 10.75% 15.293ms 477.894us 0 B 0 B 786.00 MB 786.00 MB 32 \n aten::nll_loss_backward 0.28% 510.827us 2.04% 3.773ms 117.904us 151.103us 0.11% 4.188ms 130.874us 0 B 0 B 786.00 MB 785.98 MB 32 \n aten::empty 0.76% 1.400ms 1.05% 1.944ms 8.239us 0.000us 0.00% 0.000us 0.000us 96 B 96 B 236.16 MB 236.16 MB 236 \n aten::addmm 0.52% 966.759us 1.22% 2.264ms 94.354us 4.957ms 3.49% 4.957ms 206.534us 0 B 0 B 108.00 MB 108.00 MB 24 \n aten::gelu 0.07% 133.055us 0.34% 626.038us 104.340us 485.558us 0.34% 485.558us 80.926us 0 B 0 B 48.00 MB 48.00 MB 6 \n aten::gelu_backward 0.05% 87.548us 0.08% 143.945us 23.991us 620.754us 0.44% 620.754us 103.459us 0 B 0 B 48.00 MB 48.00 MB 6 \n aten::add 0.29% 538.806us 1.30% 2.401ms 80.036us 522.004us 0.37% 522.004us 17.400us 0 B 0 B 42.00 MB 42.00 MB 30 \n aten::mul 0.45% 824.568us 0.83% 1.543ms 28.579us 644.691us 0.45% 644.691us 11.939us 0 B 0 B 36.00 MB 36.00 MB 54 \n aten::neg 0.21% 388.406us 0.33% 613.127us 25.547us 204.636us 0.14% 204.636us 8.527us 0 B 0 B 6.00 MB 6.00 MB 24 \n aten::resize_ 0.01% 19.952us 0.01% 19.952us 9.976us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 2.03 MB 2.03 MB 2 \n aten::sum 0.40% 746.197us 0.72% 1.340ms 41.885us 1.483ms 1.04% 1.510ms 47.181us 0 B 0 B 412.00 KB 380.00 KB 32 \n aten::empty_strided 0.02% 37.059us 0.02% 37.059us 12.353us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 33.00 KB 33.00 KB 3 \n aten::nll_loss_forward 0.33% 615.660us 0.78% 1.449ms 45.292us 158.556us 0.11% 266.937us 8.342us 0 B 0 B 32.00 KB 32.00 KB 32 \n aten::ne 3.16% 5.845ms 15.84% 29.324ms 29.324ms 4.448us 0.00% 8.896us 8.896us 0 B 0 B 4.00 KB 4.00 KB 1 \n aten::div 6.30% 11.668ms 10.70% 19.794ms 9.897ms 9.825us 0.01% 14.977us 7.488us 0 B 0 B 1.00 KB 1.00 KB 2 \n aten::maximum 5.99% 11.084ms 12.36% 22.866ms 22.866ms 3.808us 0.00% 7.616us 7.616us 0 B 0 B 512 B 512 B 1 \n aten::slice 0.36% 663.813us 0.48% 893.968us 3.694us 0.000us 0.00% 0.000us 0.000us 0 B 0 B 0 B 0 B 242 \n------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ \nSelf CPU time total: 185.071ms\nSelf CUDA time total: 142.206ms\n"
}
Binary file added docs/profiling/end_to_end_step.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading