working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
[2024-07-09_21:16:03]: Failed to import xformers.
[2024-07-09_21:16:03]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm.
Overriding: out_dir = /app/suno/checkpoints
Overriding: data_dir = /app/suno/data/dpo/13b_2h_v3
Overriding: train_filename = data_tr.bin
Overriding: train_metas_filename = meta_tr.jsonl
Overriding: train_info_filename = info_tr.json
Overriding: val_filename = data_val.bin
Overriding: val_metas_filename = meta_val.jsonl
Overriding: val_info_filename = info_val.json
Overriding: learning_rate = 5e-07
Overriding: min_lr = 1e-08
Overriding: do_ipo = True
Overriding: dpo_beta = 5.0
Overriding: semantic_codebook_weight = 4.0
Overriding: last_codebook_weight = 1.0
Overriding: warmup_iters = 100
Overriding: max_iters = 2000
Overriding: grad_clip = 0.1
Overriding: eval_interval = 2000
Overriding: eval_iters = 25
Overriding: step_save_iters = 2000
Overriding: block_size = 8832
Overriding: t_text = 2560
Overriding: t_memmap = 6016
Overriding: t_audio = 6272
Overriding: use_rotary_pos_emb = True
Overriding: rope_theta = 500000
Overriding: use_qk_norm = True
Overriding: activation_f = silu
Overriding: embed_scale_factor = 10.0
Overriding: global_every_n_layers = 1
Overriding: n_layer = 40
Overriding: n_head = 40
Overriding: d_head = 128
Overriding: n_kv_head = 2
Overriding: attention_type = tao
Overriding: gradient_accumulation_steps = 1
Overriding: batch_size = 4
Overriding: fsdp = True
Overriding: sharding_strategy = full_shard
Overriding: grad_checkpointing = True
Overriding: preload_checkpoint = /app/suno/data/dpo/models/model_13b_2h.pt
Overriding: model_cache_loss_name = 13b_2h_dpo
Overriding: preload_strict = False
Overriding: local_cache_dir = /mnt/localdisk/tmp_tony
Overriding: wandb_log = True
Overriding: wandb_project = chirp-v4-dpo
Overriding: wandb_run_name = dpo_13b_2head_ft_v3_bt4_reweight
[2024-07-09_21:16:04]: ddp init, rank 8, local_rank 0
[2024-07-09_21:16:05]: ddp init, rank 16, local_rank 0
[2024-07-09_21:16:05]: ddp init, rank 0, local_rank 0
[2024-07-09_21:16:05]: ddp init, rank 12, local_rank 4
[2024-07-09_21:16:05]: ddp init, rank 9, local_rank 1
[2024-07-09_21:16:05]: ddp init, rank 2, local_rank 2
[2024-07-09_21:16:05]: ddp init, rank 11, local_rank 3
[2024-07-09_21:16:05]: ddp init, rank 15, local_rank 7
[2024-07-09_21:16:05]: ddp init, rank 13, local_rank 5
[2024-07-09_21:16:05]: ddp init, rank 14, local_rank 6
[2024-07-09_21:16:05]: ddp init, rank 10, local_rank 2
[2024-07-09_21:16:05]: ddp init, rank 7, local_rank 7
[2024-07-09_21:16:05]: ddp init, rank 6, local_rank 6
[2024-07-09_21:16:05]: ddp init, rank 1, local_rank 1
[2024-07-09_21:16:05]: ddp init, rank 5, local_rank 5
[2024-07-09_21:16:05]: ddp init, rank 4, local_rank 4
[2024-07-09_21:16:05]: ddp init, rank 3, local_rank 3
NCCL version 2.20.5+cuda12.4
[2024-07-09_21:16:05]: ddp init, rank 18, local_rank 2
[2024-07-09_21:16:05]: ddp init, rank 22, local_rank 6
[2024-07-09_21:16:05]: ddp init, rank 23, local_rank 7
[2024-07-09_21:16:05]: ddp init, rank 21, local_rank 5
[2024-07-09_21:16:05]: ddp init, rank 17, local_rank 1
[2024-07-09_21:16:05]: ddp init, rank 19, local_rank 3
[2024-07-09_21:16:05]: ddp init, rank 20, local_rank 4
[2024-07-09_21:16:16]: ddp init: world size 24 ddp_rank 0.
[2024-07-09_21:16:16]: loss discounts for codebooks: [0.25  0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062
 0.062]
[2024-07-09_21:16:29]: Total world size 24
[2024-07-09_21:16:29]: logging checkpoint here: /app/suno/checkpoints/2024-07-09_21-16-29
[2024-07-09_21:16:29]: loading data...
[2024-07-09_21:16:29]: indexed 100.0% of data
[2024-07-09_21:16:29]: 582 lines of data_val.bin loaded.
[2024-07-09_21:16:30]: indexed 100.0% of data
[2024-07-09_21:16:30]: 57,456 lines of data_tr.bin loaded.
[2024-07-09_21:16:30]: train data weights:
 50.0% perference_0
 50.0% perference_1
[2024-07-09_21:16:30]: done loading data
[2024-07-09_21:16:30]: GPU capacity: NVIDIA H100 80GB HBM3 (0) with 79.33GiB memory
[2024-07-09_21:16:30]: Initializing train model from scratch
[2024-07-09_21:18:26]: number of parameters: 11154M
[2024-07-09_21:18:26]: finish init train model
[2024-07-09_21:18:26]: not compiling model.
[2024-07-09_21:18:26]: start loading state dict
[2024-07-09_21:18:26]: verifying model args...
[2024-07-09_21:18:26]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-07-09_21:18:29]: warning: checkpoint missing config keys: {'global_every_n_layers'}
[2024-07-09_21:18:29]: warning: checkpoint config key mismatch on global_every_n_layers
[2024-07-09_21:18:29]: warning: checkpoint config key mismatch on attention_sliding_window_size
[2024-07-09_21:18:29]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-07-09_21:18:29]: copying checkpoint file to local cachedir...
[2024-07-09_21:20:39]: loading model state_dict on gpu 0
[2024-07-09_21:20:42]: loading model state_dict on gpu 1
[2024-07-09_21:20:46]: loading model state_dict on gpu 2
[2024-07-09_21:20:49]: loading model state_dict on gpu 3
[2024-07-09_21:20:52]: loading model state_dict on gpu 4
[2024-07-09_21:20:55]: loading model state_dict on gpu 5
[2024-07-09_21:20:58]: loading model state_dict on gpu 6
[2024-07-09_21:21:01]: loading model state_dict on gpu 7
[2024-07-09_21:21:04]: finish loading state dict
[2024-07-09_21:21:04]: wrapping model in FSDP ....
[2024-07-09_21:21:15]: GPU memory usage for model: 7.90GiB(9.96%)
[2024-07-09_21:21:15]: applying fsdp activation checkpointing...
[2024-07-09_21:21:15]: num decayed parameter tensors: 387, with 470,798,960 parameters
[2024-07-09_21:21:15]: num non-decayed parameter tensors: 84, with 204,800 parameters
[2024-07-09_21:21:15]: using fused Optimizer: False
[2024-07-09_21:21:15]: model setup done
[2024-07-09_21:21:15]: Validate random number: 0.9918216287004739
[2024-07-09_21:21:15]: Evaluating
[2024-07-09_21:21:15]: Start the ref model loss eval loop.
[2024-07-09_21:21:15]: Pre-compute cache loss
[2024-07-09_21:21:15]: Start evaluating loss for slipt train
[2024-07-09_21:21:15]: estimated total number of iterations 300,size of the data 57456,size of batch 8,ddp_rank is 0
[2024-07-09_21:21:22]: iter 0/300: step_time 7020.2ms, throughput 10k tok/s/node,
[2024-07-09_21:22:57]: iter 25/300: step_time 3826.2ms, throughput 18k tok/s/node,
[2024-07-09_21:24:33]: iter 50/300: step_time 3836.9ms, throughput 18k tok/s/node,
[2024-07-09_21:26:09]: iter 75/300: step_time 3840.1ms, throughput 18k tok/s/node,
[2024-07-09_21:27:45]: iter 100/300: step_time 3846.1ms, throughput 18k tok/s/node,
[2024-07-09_21:29:21]: iter 125/300: step_time 3843.8ms, throughput 18k tok/s/node,
[2024-07-09_21:30:57]: iter 150/300: step_time 3835.7ms, throughput 18k tok/s/node,
[2024-07-09_21:32:33]: iter 175/300: step_time 3834.2ms, throughput 18k tok/s/node,
[2024-07-09_21:34:09]: iter 200/300: step_time 3836.6ms, throughput 18k tok/s/node,
[2024-07-09_21:35:44]: iter 225/300: step_time 3827.2ms, throughput 18k tok/s/node,
[2024-07-09_21:37:20]: iter 250/300: step_time 3834.3ms, throughput 18k tok/s/node,
[2024-07-09_21:38:56]: iter 275/300: step_time 3833.0ms, throughput 18k tok/s/node,
[2024-07-09_21:40:28]: Start evaluating loss for slipt val
[2024-07-09_21:40:28]: estimated total number of iterations 4,size of the data 582,size of batch 8,ddp_rank is 0
[2024-07-09_21:40:32]: iter 0/4: step_time 3647.7ms, throughput 19k tok/s/node,
[2024-07-09_21:40:44]: 24
[2024-07-09_21:40:45]: Saving pre-computed cache loss: /app/suno/data/dpo/13b_2h_v3/13b_2h_dpo_cached_loss.json
[2024-07-09_21:40:45]: Check if loaded correctly: 
 train 57456 vs 57456
 val 582 vs 582
[2024-07-09_21:40:45]: Validate random number: 0.9762858123034472
[2024-07-09_21:40:45]: training...
[2024-07-09_21:42:17]: loss estimation took 92.2 seconds. (100.0% of loop)
[2024-07-09_21:42:17]: step 0: train loss 3.6343, val loss 3.7009
[2024-07-09_21:42:26]: iter 0: avg_loss 0.010, step_time 101527.0ms, mfu 0.0%, throughput 0k tok/s, total time 102s, memory 59.64GiB(75.18%)
[2024-07-09_21:45:42]: iter 25: avg_loss 0.010, step_time 7811.0ms, mfu 128.5%, throughput 109k tok/s, total time 297s, memory 60.57GiB(76.35%)
[2024-07-09_21:48:57]: iter 50: avg_loss 0.010, step_time 7808.4ms, mfu 128.5%, throughput 109k tok/s, total time 492s, memory 60.57GiB(76.35%)
[2024-07-09_21:52:12]: iter 75: avg_loss 0.010, step_time 7806.4ms, mfu 128.6%, throughput 109k tok/s, total time 688s, memory 60.57GiB(76.35%)
[2024-07-09_21:55:28]: iter 100: avg_loss 0.010, step_time 7808.2ms, mfu 128.5%, throughput 109k tok/s, total time 883s, memory 60.57GiB(76.35%)
[2024-07-09_21:58:43]: iter 125: avg_loss 0.010, step_time 7823.0ms, mfu 128.3%, throughput 108k tok/s, total time 1078s, memory 60.57GiB(76.35%)
[2024-07-09_22:01:58]: iter 150: avg_loss 0.010, step_time 7826.3ms, mfu 128.2%, throughput 108k tok/s, total time 1273s, memory 60.57GiB(76.35%)
[2024-07-09_22:05:14]: iter 175: avg_loss 0.010, step_time 7811.2ms, mfu 128.5%, throughput 109k tok/s, total time 1469s, memory 60.57GiB(76.35%)
[2024-07-09_22:08:29]: iter 200: avg_loss 0.010, step_time 7841.4ms, mfu 128.0%, throughput 108k tok/s, total time 1664s, memory 60.57GiB(76.35%)
[2024-07-09_22:11:44]: iter 225: avg_loss 0.011, step_time 7814.3ms, mfu 128.4%, throughput 109k tok/s, total time 1860s, memory 60.57GiB(76.35%)
[2024-07-09_22:15:00]: iter 250: avg_loss 0.009, step_time 7805.9ms, mfu 128.6%, throughput 109k tok/s, total time 2055s, memory 60.57GiB(76.35%)
[2024-07-09_22:18:15]: iter 275: avg_loss 0.010, step_time 7802.9ms, mfu 128.6%, throughput 109k tok/s, total time 2250s, memory 60.57GiB(76.35%)
[2024-07-09_22:21:30]: iter 300: avg_loss 0.009, step_time 7803.6ms, mfu 128.6%, throughput 109k tok/s, total time 2445s, memory 60.57GiB(76.35%)
[2024-07-09_22:24:46]: iter 325: avg_loss 0.010, step_time 7808.7ms, mfu 128.5%, throughput 109k tok/s, total time 2641s, memory 60.57GiB(76.35%)
[2024-07-09_22:28:01]: iter 350: avg_loss 0.009, step_time 7781.4ms, mfu 129.0%, throughput 109k tok/s, total time 2836s, memory 61.48GiB(77.50%)
[2024-07-09_22:31:16]: iter 375: avg_loss 0.010, step_time 7805.5ms, mfu 128.6%, throughput 109k tok/s, total time 3031s, memory 61.48GiB(77.50%)
[2024-07-09_22:34:31]: iter 400: avg_loss 0.010, step_time 7809.9ms, mfu 128.5%, throughput 109k tok/s, total time 3227s, memory 61.48GiB(77.50%)
[2024-07-09_22:37:47]: iter 425: avg_loss 0.009, step_time 7814.5ms, mfu 128.4%, throughput 109k tok/s, total time 3422s, memory 61.48GiB(77.50%)
[2024-07-09_22:41:02]: iter 450: avg_loss 0.010, step_time 7814.3ms, mfu 128.4%, throughput 109k tok/s, total time 3617s, memory 61.48GiB(77.50%)
[2024-07-09_22:44:17]: iter 475: avg_loss 0.009, step_time 7800.7ms, mfu 128.6%, throughput 109k tok/s, total time 3812s, memory 61.48GiB(77.50%)
[2024-07-09_22:47:32]: iter 500: avg_loss 0.009, step_time 7807.0ms, mfu 128.5%, throughput 109k tok/s, total time 4007s, memory 61.48GiB(77.50%)
[2024-07-09_22:50:47]: iter 525: avg_loss 0.010, step_time 7824.4ms, mfu 128.3%, throughput 108k tok/s, total time 4203s, memory 61.48GiB(77.50%)
[2024-07-09_22:54:03]: iter 550: avg_loss 0.009, step_time 7816.6ms, mfu 128.4%, throughput 108k tok/s, total time 4398s, memory 61.48GiB(77.50%)
[2024-07-09_22:57:18]: iter 575: avg_loss 0.009, step_time 7809.7ms, mfu 128.5%, throughput 109k tok/s, total time 4593s, memory 61.48GiB(77.50%)
[2024-07-09_23:00:33]: iter 600: avg_loss 0.009, step_time 7804.0ms, mfu 128.6%, throughput 109k tok/s, total time 4788s, memory 61.48GiB(77.50%)
[2024-07-09_23:03:49]: iter 625: avg_loss 0.010, step_time 7808.8ms, mfu 128.5%, throughput 109k tok/s, total time 4984s, memory 61.48GiB(77.50%)
[2024-07-09_23:07:04]: iter 650: avg_loss 0.010, step_time 7797.4ms, mfu 128.7%, throughput 109k tok/s, total time 5179s, memory 61.48GiB(77.50%)
[2024-07-09_23:10:19]: iter 675: avg_loss 0.010, step_time 7806.3ms, mfu 128.6%, throughput 109k tok/s, total time 5374s, memory 61.48GiB(77.50%)
[2024-07-09_23:13:34]: iter 700: avg_loss 0.010, step_time 7805.5ms, mfu 128.6%, throughput 109k tok/s, total time 5570s, memory 61.48GiB(77.50%)
[2024-07-09_23:16:49]: iter 725: avg_loss 0.009, step_time 7799.1ms, mfu 128.7%, throughput 109k tok/s, total time 5765s, memory 61.48GiB(77.50%)
[2024-07-09_23:20:05]: iter 750: avg_loss 0.009, step_time 7809.7ms, mfu 128.5%, throughput 109k tok/s, total time 5960s, memory 61.48GiB(77.50%)
[2024-07-09_23:23:20]: iter 775: avg_loss 0.009, step_time 7804.4ms, mfu 128.6%, throughput 109k tok/s, total time 6155s, memory 61.48GiB(77.50%)
[2024-07-09_23:26:35]: iter 800: avg_loss 0.009, step_time 7806.1ms, mfu 128.6%, throughput 109k tok/s, total time 6350s, memory 61.48GiB(77.50%)
[2024-07-09_23:29:50]: iter 825: avg_loss 0.009, step_time 7801.8ms, mfu 128.6%, throughput 109k tok/s, total time 6546s, memory 61.48GiB(77.50%)
[2024-07-09_23:33:06]: iter 850: avg_loss 0.009, step_time 7807.1ms, mfu 128.5%, throughput 109k tok/s, total time 6741s, memory 61.48GiB(77.50%)
[2024-07-09_23:36:21]: iter 875: avg_loss 0.009, step_time 7806.1ms, mfu 128.6%, throughput 109k tok/s, total time 6936s, memory 61.48GiB(77.50%)
[2024-07-09_23:39:36]: iter 900: avg_loss 0.009, step_time 7796.2ms, mfu 128.7%, throughput 109k tok/s, total time 7131s, memory 61.48GiB(77.50%)
[2024-07-09_23:42:51]: iter 925: avg_loss 0.009, step_time 7816.6ms, mfu 128.4%, throughput 108k tok/s, total time 7327s, memory 61.48GiB(77.50%)
[2024-07-09_23:46:07]: iter 950: avg_loss 0.009, step_time 7806.3ms, mfu 128.6%, throughput 109k tok/s, total time 7522s, memory 61.48GiB(77.50%)
[2024-07-09_23:49:22]: iter 975: avg_loss 0.009, step_time 7803.6ms, mfu 128.6%, throughput 109k tok/s, total time 7717s, memory 61.48GiB(77.50%)
[2024-07-09_23:52:37]: iter 1000: avg_loss 0.008, step_time 7802.7ms, mfu 128.6%, throughput 109k tok/s, total time 7912s, memory 61.48GiB(77.50%)
[2024-07-09_23:55:52]: iter 1025: avg_loss 0.009, step_time 7816.4ms, mfu 128.4%, throughput 108k tok/s, total time 8107s, memory 61.48GiB(77.50%)
[2024-07-09_23:59:07]: iter 1050: avg_loss 0.009, step_time 7799.3ms, mfu 128.7%, throughput 109k tok/s, total time 8302s, memory 61.48GiB(77.50%)
[2024-07-10_00:02:22]: iter 1075: avg_loss 0.009, step_time 7804.0ms, mfu 128.6%, throughput 109k tok/s, total time 8498s, memory 61.48GiB(77.50%)
[2024-07-10_00:05:38]: iter 1100: avg_loss 0.009, step_time 7811.8ms, mfu 128.5%, throughput 109k tok/s, total time 8693s, memory 61.48GiB(77.50%)
[2024-07-10_00:08:53]: iter 1125: avg_loss 0.008, step_time 7786.7ms, mfu 128.9%, throughput 109k tok/s, total time 8888s, memory 61.48GiB(77.50%)
[2024-07-10_00:12:08]: iter 1150: avg_loss 0.009, step_time 7809.2ms, mfu 128.5%, throughput 109k tok/s, total time 9083s, memory 61.48GiB(77.50%)
[2024-07-10_00:15:23]: iter 1175: avg_loss 0.009, step_time 7808.9ms, mfu 128.5%, throughput 109k tok/s, total time 9278s, memory 61.48GiB(77.50%)
[2024-07-10_00:18:38]: iter 1200: avg_loss 0.009, step_time 7816.9ms, mfu 128.4%, throughput 108k tok/s, total time 9474s, memory 61.48GiB(77.50%)
[2024-07-10_00:21:54]: iter 1225: avg_loss 0.009, step_time 7797.9ms, mfu 128.7%, throughput 109k tok/s, total time 9669s, memory 61.48GiB(77.50%)
[2024-07-10_00:25:09]: iter 1250: avg_loss 0.009, step_time 7797.0ms, mfu 128.7%, throughput 109k tok/s, total time 9864s, memory 61.48GiB(77.50%)
[2024-07-10_00:28:24]: iter 1275: avg_loss 0.009, step_time 7815.0ms, mfu 128.4%, throughput 108k tok/s, total time 10059s, memory 61.48GiB(77.50%)
[2024-07-10_00:31:39]: iter 1300: avg_loss 0.009, step_time 7808.8ms, mfu 128.5%, throughput 109k tok/s, total time 10255s, memory 61.48GiB(77.50%)
[2024-07-10_00:34:54]: iter 1325: avg_loss 0.010, step_time 7794.5ms, mfu 128.7%, throughput 109k tok/s, total time 10450s, memory 61.48GiB(77.50%)
[2024-07-10_00:38:10]: iter 1350: avg_loss 0.008, step_time 7815.8ms, mfu 128.4%, throughput 108k tok/s, total time 10645s, memory 61.48GiB(77.50%)
[2024-07-10_00:41:25]: iter 1375: avg_loss 0.009, step_time 7820.1ms, mfu 128.3%, throughput 108k tok/s, total time 10840s, memory 61.48GiB(77.50%)
[2024-07-10_00:44:40]: iter 1400: avg_loss 0.009, step_time 7799.2ms, mfu 128.7%, throughput 109k tok/s, total time 11035s, memory 61.48GiB(77.50%)
[2024-07-10_00:47:55]: iter 1425: avg_loss 0.008, step_time 7797.6ms, mfu 128.7%, throughput 109k tok/s, total time 11230s, memory 61.48GiB(77.50%)
[2024-07-10_00:51:10]: iter 1450: avg_loss 0.008, step_time 7818.9ms, mfu 128.3%, throughput 108k tok/s, total time 11425s, memory 61.48GiB(77.50%)
[2024-07-10_00:54:25]: iter 1475: avg_loss 0.009, step_time 7805.1ms, mfu 128.6%, throughput 109k tok/s, total time 11620s, memory 61.48GiB(77.50%)
[2024-07-10_00:57:40]: iter 1500: avg_loss 0.009, step_time 7799.7ms, mfu 128.7%, throughput 109k tok/s, total time 11816s, memory 61.48GiB(77.50%)
[2024-07-10_01:00:56]: iter 1525: avg_loss 0.008, step_time 7805.2ms, mfu 128.6%, throughput 109k tok/s, total time 12011s, memory 61.48GiB(77.50%)
[2024-07-10_01:04:11]: iter 1550: avg_loss 0.009, step_time 7802.2ms, mfu 128.6%, throughput 109k tok/s, total time 12206s, memory 61.48GiB(77.50%)
[2024-07-10_01:07:26]: iter 1575: avg_loss 0.009, step_time 7804.0ms, mfu 128.6%, throughput 109k tok/s, total time 12402s, memory 61.48GiB(77.50%)
[2024-07-10_01:10:41]: iter 1600: avg_loss 0.008, step_time 7804.3ms, mfu 128.6%, throughput 109k tok/s, total time 12597s, memory 61.48GiB(77.50%)
[2024-07-10_01:13:57]: iter 1625: avg_loss 0.008, step_time 7813.3ms, mfu 128.4%, throughput 109k tok/s, total time 12792s, memory 61.48GiB(77.50%)
[2024-07-10_01:17:12]: iter 1650: avg_loss 0.008, step_time 7789.5ms, mfu 128.8%, throughput 109k tok/s, total time 12987s, memory 61.48GiB(77.50%)
[2024-07-10_01:20:27]: iter 1675: avg_loss 0.009, step_time 7810.4ms, mfu 128.5%, throughput 109k tok/s, total time 13182s, memory 61.48GiB(77.50%)
[2024-07-10_01:23:42]: iter 1700: avg_loss 0.009, step_time 7804.4ms, mfu 128.6%, throughput 109k tok/s, total time 13377s, memory 61.48GiB(77.50%)
[2024-07-10_01:26:57]: iter 1725: avg_loss 0.008, step_time 7812.3ms, mfu 128.5%, throughput 109k tok/s, total time 13572s, memory 61.48GiB(77.50%)
[2024-07-10_01:30:12]: iter 1750: avg_loss 0.008, step_time 7805.0ms, mfu 128.6%, throughput 109k tok/s, total time 13767s, memory 61.48GiB(77.50%)
[2024-07-10_01:33:27]: iter 1775: avg_loss 0.009, step_time 7805.6ms, mfu 128.6%, throughput 109k tok/s, total time 13962s, memory 61.48GiB(77.50%)
[2024-07-10_01:36:42]: iter 1800: avg_loss 0.008, step_time 7812.8ms, mfu 128.4%, throughput 109k tok/s, total time 14157s, memory 61.48GiB(77.50%)
[2024-07-10_01:39:58]: iter 1825: avg_loss 0.009, step_time 7809.4ms, mfu 128.5%, throughput 109k tok/s, total time 14353s, memory 61.48GiB(77.50%)
[2024-07-10_01:43:13]: iter 1850: avg_loss 0.009, step_time 7801.5ms, mfu 128.6%, throughput 109k tok/s, total time 14548s, memory 61.48GiB(77.50%)
[2024-07-10_01:46:28]: iter 1875: avg_loss 0.010, step_time 7801.8ms, mfu 128.6%, throughput 109k tok/s, total time 14743s, memory 61.48GiB(77.50%)
[2024-07-10_01:49:43]: iter 1900: avg_loss 0.008, step_time 7800.4ms, mfu 128.7%, throughput 109k tok/s, total time 14938s, memory 61.48GiB(77.50%)
[2024-07-10_01:52:58]: iter 1925: avg_loss 0.009, step_time 7801.4ms, mfu 128.6%, throughput 109k tok/s, total time 15133s, memory 61.48GiB(77.50%)
[2024-07-10_01:56:13]: iter 1950: avg_loss 0.009, step_time 7801.5ms, mfu 128.6%, throughput 109k tok/s, total time 15328s, memory 61.48GiB(77.50%)
[2024-07-10_01:59:28]: iter 1975: avg_loss 0.008, step_time 7799.5ms, mfu 128.7%, throughput 109k tok/s, total time 15523s, memory 61.48GiB(77.50%)
[2024-07-10_02:04:00]: loss estimation took 92.2 seconds. (0.6% of loop)
[2024-07-10_02:04:00]: step 1999: train loss 3.6437, val loss 3.6603
[2024-07-10_02:05:30]: saving checkpoint to /app/suno/checkpoints/2024-07-09_21-16-29
[2024-07-10_02:06:13]: saving took 133.5 seconds. (0.9% of loop)
[2024-07-10_02:06:31]: iter 1999: avg_loss 0.009, step_time 242951.5ms, mfu 4.1%, throughput 3k tok/s, total time 15946s, memory 63.31GiB(79.81%)
[2024-07-10_02:06:31]: done.