working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT [2024-06-24_09:11:51]: Failed to import xformers. [2024-06-24_09:11:51]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_extend_mix Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 5.0 Overriding: semantic_codebook_weight = 4.0 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 100 Overriding: max_iters = 700 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: batch_size = 4 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-05-27_02-51-46/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-13b-dpo-extend Overriding: wandb_run_name = dpo_13b_extend_mix_v0 [2024-06-24_09:11:53]: ddp init, rank 40, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 48, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 8, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 56, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 38, local_rank 6 [2024-06-24_09:11:53]: ddp init, rank 0, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 33, local_rank 1 [2024-06-24_09:11:53]: ddp init, rank 34, local_rank 2 [2024-06-24_09:11:53]: ddp init, rank 36, local_rank 4 [2024-06-24_09:11:53]: ddp init, rank 32, local_rank 0 [2024-06-24_09:11:53]: ddp init, rank 6, local_rank 6 [2024-06-24_09:11:53]: ddp init, rank 1, local_rank 1 [2024-06-24_09:11:53]: ddp init, rank 5, local_rank 5 [2024-06-24_09:11:53]: ddp init, rank 2, local_rank 2 [2024-06-24_09:11:53]: ddp init, rank 7, local_rank 7 [2024-06-24_09:11:53]: ddp init, rank 4, local_rank 4 [2024-06-24_09:11:53]: ddp init, rank 3, local_rank 3 [2024-06-24_09:11:53]: ddp init, rank 39, local_rank 7 [2024-06-24_09:11:53]: ddp init, rank 37, local_rank 5 NCCL version 2.20.5+cuda12.4 [2024-06-24_09:11:53]: ddp init, rank 42, local_rank 2 [2024-06-24_09:11:53]: ddp init, rank 35, local_rank 3 [2024-06-24_09:11:53]: ddp init, rank 41, local_rank 1 [2024-06-24_09:11:53]: ddp init, rank 46, local_rank 6 [2024-06-24_09:11:53]: ddp init, rank 49, local_rank 1 [2024-06-24_09:11:53]: ddp init, rank 55, local_rank 7 [2024-06-24_09:11:53]: ddp init, rank 43, local_rank 3 [2024-06-24_09:11:53]: ddp init, rank 44, local_rank 4 [2024-06-24_09:11:53]: ddp init, rank 47, local_rank 7 [2024-06-24_09:11:53]: ddp init, rank 45, local_rank 5 [2024-06-24_09:11:53]: ddp init, rank 51, local_rank 3 [2024-06-24_09:11:53]: ddp init, rank 54, local_rank 6 [2024-06-24_09:11:53]: ddp init, rank 52, local_rank 4 [2024-06-24_09:11:53]: ddp init, rank 50, local_rank 2 [2024-06-24_09:11:53]: ddp init, rank 53, local_rank 5 [2024-06-24_09:11:54]: ddp init, rank 63, local_rank 7 [2024-06-24_09:11:54]: ddp init, rank 13, local_rank 5 [2024-06-24_09:11:54]: ddp init, rank 61, local_rank 5 [2024-06-24_09:11:54]: ddp init, rank 12, local_rank 4 [2024-06-24_09:11:54]: ddp init, rank 15, local_rank 7 [2024-06-24_09:11:54]: ddp init, rank 14, local_rank 6 [2024-06-24_09:11:54]: ddp init, rank 58, local_rank 2 [2024-06-24_09:11:54]: ddp init, rank 11, local_rank 3 [2024-06-24_09:11:54]: ddp init, rank 10, local_rank 2 [2024-06-24_09:11:54]: ddp init, rank 59, local_rank 3 [2024-06-24_09:11:54]: ddp init, rank 9, local_rank 1 [2024-06-24_09:11:54]: ddp init, rank 62, local_rank 6 [2024-06-24_09:11:54]: ddp init, rank 57, local_rank 1 [2024-06-24_09:11:54]: ddp init, rank 60, local_rank 4 [2024-06-24_09:11:54]: ddp init, rank 16, local_rank 0 [2024-06-24_09:11:54]: ddp init, rank 24, local_rank 0 [2024-06-24_09:11:54]: ddp init, rank 22, local_rank 6 [2024-06-24_09:11:54]: ddp init, rank 19, local_rank 3 [2024-06-24_09:11:54]: ddp init, rank 21, local_rank 5 [2024-06-24_09:11:54]: ddp init, rank 18, local_rank 2 [2024-06-24_09:11:54]: ddp init, rank 23, local_rank 7 [2024-06-24_09:11:54]: ddp init, rank 17, local_rank 1 [2024-06-24_09:11:54]: ddp init, rank 20, local_rank 4 [2024-06-24_09:11:54]: ddp init, rank 27, local_rank 3 [2024-06-24_09:11:54]: ddp init, rank 25, local_rank 1 [2024-06-24_09:11:54]: ddp init, rank 31, local_rank 7 [2024-06-24_09:11:54]: ddp init, rank 29, local_rank 5 [2024-06-24_09:11:54]: ddp init, rank 26, local_rank 2 [2024-06-24_09:11:54]: ddp init, rank 30, local_rank 6 [2024-06-24_09:11:54]: ddp init, rank 28, local_rank 4 [2024-06-24_09:12:06]: ddp init: world size 64 ddp_rank 0. [2024-06-24_09:12:06]: loss discounts for codebooks: [0.308 0.077 0.073 0.07 0.066 0.063 0.059 0.056 0.052 0.049 0.045 0.042 0.038] [2024-06-24_09:12:17]: logging checkpoint here: /app/suno/checkpoints/2024-06-24_09-12-17 [2024-06-24_09:12:17]: loading data... [2024-06-24_09:12:17]: indexed 100.0% of data [2024-06-24_09:12:17]: 1,972 lines of data_val.bin loaded. [2024-06-24_09:12:20]: indexed 100.0% of data [2024-06-24_09:12:20]: 195,148 lines of data_tr.bin loaded. [2024-06-24_09:12:20]: train data weights: 50.0% perference_0 50.0% perference_1 [2024-06-24_09:12:20]: done loading data [2024-06-24_09:12:20]: Initializing train model from scratch [2024-06-24_09:13:57]: number of parameters: 11363M [2024-06-24_09:13:57]: finish init train model [2024-06-24_09:13:57]: not compiling model. [2024-06-24_09:13:57]: start loading state dict [2024-06-24_09:13:57]: verifying model args... [2024-06-24_09:13:57]: careful, using approximation for checkpoint loading. could be wrong in principle [2024-06-24_09:13:57]: careful, using approximation for checkpoint loading. could be wrong in principle [2024-06-24_09:13:57]: loading model state_dict on gpu 0 [2024-06-24_09:14:00]: loading model state_dict on gpu 1 [2024-06-24_09:14:03]: loading model state_dict on gpu 2 [2024-06-24_09:14:06]: loading model state_dict on gpu 3 [2024-06-24_09:14:09]: loading model state_dict on gpu 4 [2024-06-24_09:14:12]: loading model state_dict on gpu 5 [2024-06-24_09:14:15]: loading model state_dict on gpu 6 [2024-06-24_09:14:18]: loading model state_dict on gpu 7 [2024-06-24_09:14:21]: finish loading state dict [2024-06-24_09:14:21]: wrapping model in FSDP .... [2024-06-24_09:14:31]: applying fsdp activation checkpointing... [2024-06-24_09:14:31]: num decayed parameter tensors: 267, with 179,698,240 parameters [2024-06-24_09:14:31]: num non-decayed parameter tensors: 84, with 204,800 parameters [2024-06-24_09:14:31]: using fused Optimizer: False [2024-06-24_09:14:31]: model setup done [2024-06-24_09:14:31]: Validate random number: 0.9918216287004739 [2024-06-24_09:14:31]: Evaluating [2024-06-24_09:14:31]: Start the ref model loss eval loop. [2024-06-24_09:14:31]: Pre-compute cache loss [2024-06-24_09:14:31]: Start evaluating loss for slipt train [2024-06-24_09:14:31]: estimated total number of iterations 3050,size of the data 195148,size of batch 8,ddp_rank is 0 [2024-06-24_09:14:36]: iter 0/3050: step_time 5047.8ms, throughput 14k tok/s/node, [2024-06-24_09:16:10]: iter 25/3050: step_time 3758.3ms, throughput 19k tok/s/node, [2024-06-24_09:17:44]: iter 50/3050: step_time 3741.3ms, throughput 19k tok/s/node, [2024-06-24_09:19:18]: iter 75/3050: step_time 3756.5ms, throughput 19k tok/s/node, [2024-06-24_09:20:52]: iter 100/3050: step_time 3757.8ms, throughput 19k tok/s/node, [2024-06-24_09:22:26]: iter 125/3050: step_time 3753.8ms, throughput 19k tok/s/node, [2024-06-24_09:23:59]: iter 150/3050: step_time 3750.2ms, throughput 19k tok/s/node, [2024-06-24_09:25:33]: iter 175/3050: step_time 3756.6ms, throughput 19k tok/s/node, [2024-06-24_09:27:07]: iter 200/3050: step_time 3750.6ms, throughput 19k tok/s/node, [2024-06-24_09:28:41]: iter 225/3050: step_time 3754.1ms, throughput 19k tok/s/node, [2024-06-24_09:30:15]: iter 250/3050: step_time 3755.2ms, throughput 19k tok/s/node, [2024-06-24_09:31:49]: iter 275/3050: step_time 3757.6ms, throughput 19k tok/s/node, [2024-06-24_09:33:23]: iter 300/3050: step_time 3757.9ms, throughput 19k tok/s/node, [2024-06-24_09:34:57]: iter 325/3050: step_time 3764.1ms, throughput 18k tok/s/node, [2024-06-24_09:36:31]: iter 350/3050: step_time 3752.4ms, throughput 19k tok/s/node, [2024-06-24_09:38:05]: iter 375/3050: step_time 3759.0ms, throughput 19k tok/s/node, [2024-06-24_09:39:39]: iter 400/3050: step_time 3752.8ms, throughput 19k tok/s/node, [2024-06-24_09:41:13]: iter 425/3050: step_time 3756.5ms, throughput 19k tok/s/node, [2024-06-24_09:42:47]: iter 450/3050: step_time 3759.9ms, throughput 19k tok/s/node, [2024-06-24_09:44:21]: iter 475/3050: step_time 3760.7ms, throughput 19k tok/s/node, [2024-06-24_09:45:55]: iter 500/3050: step_time 3755.7ms, throughput 19k tok/s/node, [2024-06-24_09:47:29]: iter 525/3050: step_time 3755.8ms, throughput 19k tok/s/node, [2024-06-24_09:49:03]: iter 550/3050: step_time 3768.5ms, throughput 18k tok/s/node, [2024-06-24_09:50:37]: iter 575/3050: step_time 3764.3ms, throughput 18k tok/s/node, [2024-06-24_09:52:11]: iter 600/3050: step_time 3759.0ms, throughput 19k tok/s/node, [2024-06-24_09:53:45]: iter 625/3050: step_time 3767.2ms, throughput 18k tok/s/node, [2024-06-24_09:55:19]: iter 650/3050: step_time 3747.3ms, throughput 19k tok/s/node, [2024-06-24_09:56:53]: iter 675/3050: step_time 3755.4ms, throughput 19k tok/s/node, [2024-06-24_09:58:27]: iter 700/3050: step_time 3757.2ms, throughput 19k tok/s/node, [2024-06-24_10:00:01]: iter 725/3050: step_time 3753.3ms, throughput 19k tok/s/node, [2024-06-24_10:01:35]: iter 750/3050: step_time 3760.6ms, throughput 19k tok/s/node, [2024-06-24_10:03:09]: iter 775/3050: step_time 3765.5ms, throughput 18k tok/s/node, [2024-06-24_10:04:43]: iter 800/3050: step_time 3752.8ms, throughput 19k tok/s/node, [2024-06-24_10:06:17]: iter 825/3050: step_time 3761.3ms, throughput 19k tok/s/node, [2024-06-24_10:07:51]: iter 850/3050: step_time 3758.8ms, throughput 19k tok/s/node, [2024-06-24_10:09:25]: iter 875/3050: step_time 3751.8ms, throughput 19k tok/s/node, [2024-06-24_10:10:59]: iter 900/3050: step_time 3761.6ms, throughput 19k tok/s/node, [2024-06-24_10:12:33]: iter 925/3050: step_time 3757.9ms, throughput 19k tok/s/node, [2024-06-24_10:14:07]: iter 950/3050: step_time 3758.7ms, throughput 19k tok/s/node, [2024-06-24_10:15:41]: iter 975/3050: step_time 3750.7ms, throughput 19k tok/s/node, [2024-06-24_10:17:15]: iter 1000/3050: step_time 3760.3ms, throughput 19k tok/s/node, [2024-06-24_10:18:49]: iter 1025/3050: step_time 3753.4ms, throughput 19k tok/s/node, [2024-06-24_10:20:23]: iter 1050/3050: step_time 3759.6ms, throughput 19k tok/s/node, [2024-06-24_10:21:57]: iter 1075/3050: step_time 3760.1ms, throughput 19k tok/s/node, [2024-06-24_10:23:31]: iter 1100/3050: step_time 3766.3ms, throughput 18k tok/s/node, [2024-06-24_10:25:05]: iter 1125/3050: step_time 3751.4ms, throughput 19k tok/s/node, [2024-06-24_10:26:39]: iter 1150/3050: step_time 3763.2ms, throughput 19k tok/s/node, [2024-06-24_10:28:13]: iter 1175/3050: step_time 3758.0ms, throughput 19k tok/s/node, [2024-06-24_10:29:47]: iter 1200/3050: step_time 3763.4ms, throughput 19k tok/s/node, [2024-06-24_10:31:21]: iter 1225/3050: step_time 3754.8ms, throughput 19k tok/s/node, [2024-06-24_10:32:55]: iter 1250/3050: step_time 3757.5ms, throughput 19k tok/s/node, [2024-06-24_10:34:29]: iter 1275/3050: step_time 3759.8ms, throughput 19k tok/s/node, [2024-06-24_10:36:03]: iter 1300/3050: step_time 3748.0ms, throughput 19k tok/s/node, [2024-06-24_10:37:37]: iter 1325/3050: step_time 3763.7ms, throughput 19k tok/s/node, [2024-06-24_10:39:11]: iter 1350/3050: step_time 3761.9ms, throughput 19k tok/s/node, [2024-06-24_10:40:45]: iter 1375/3050: step_time 3760.5ms, throughput 19k tok/s/node, [2024-06-24_10:42:19]: iter 1400/3050: step_time 3780.7ms, throughput 18k tok/s/node, [2024-06-24_10:43:53]: iter 1425/3050: step_time 3762.6ms, throughput 19k tok/s/node, [2024-06-24_10:45:27]: iter 1450/3050: step_time 3755.6ms, throughput 19k tok/s/node, [2024-06-24_10:47:01]: iter 1475/3050: step_time 3751.6ms, throughput 19k tok/s/node, [2024-06-24_10:48:35]: iter 1500/3050: step_time 3763.2ms, throughput 19k tok/s/node, [2024-06-24_10:50:09]: iter 1525/3050: step_time 3761.5ms, throughput 19k tok/s/node, [2024-06-24_10:51:43]: iter 1550/3050: step_time 3753.8ms, throughput 19k tok/s/node, [2024-06-24_10:53:17]: iter 1575/3050: step_time 3767.7ms, throughput 18k tok/s/node, [2024-06-24_10:54:51]: iter 1600/3050: step_time 3752.0ms, throughput 19k tok/s/node, [2024-06-24_10:56:25]: iter 1625/3050: step_time 3757.6ms, throughput 19k tok/s/node, [2024-06-24_10:57:59]: iter 1650/3050: step_time 3761.6ms, throughput 19k tok/s/node, [2024-06-24_10:59:33]: iter 1675/3050: step_time 3751.3ms, throughput 19k tok/s/node, [2024-06-24_11:01:07]: iter 1700/3050: step_time 3770.5ms, throughput 18k tok/s/node, [2024-06-24_11:02:41]: iter 1725/3050: step_time 3753.6ms, throughput 19k tok/s/node, [2024-06-24_11:04:15]: iter 1750/3050: step_time 3756.4ms, throughput 19k tok/s/node, [2024-06-24_11:05:49]: iter 1775/3050: step_time 3759.7ms, throughput 19k tok/s/node, [2024-06-24_11:07:23]: iter 1800/3050: step_time 3751.8ms, throughput 19k tok/s/node, [2024-06-24_11:08:57]: iter 1825/3050: step_time 3765.7ms, throughput 18k tok/s/node, [2024-06-24_11:10:31]: iter 1850/3050: step_time 3758.3ms, throughput 19k tok/s/node, [2024-06-24_11:12:04]: iter 1875/3050: step_time 3770.6ms, throughput 18k tok/s/node, [2024-06-24_11:13:38]: iter 1900/3050: step_time 3750.3ms, throughput 19k tok/s/node, [2024-06-24_11:15:12]: iter 1925/3050: step_time 3770.9ms, throughput 18k tok/s/node, [2024-06-24_11:16:46]: iter 1950/3050: step_time 3773.6ms, throughput 18k tok/s/node, [2024-06-24_11:18:20]: iter 1975/3050: step_time 3765.2ms, throughput 18k tok/s/node, [2024-06-24_11:19:54]: iter 2000/3050: step_time 3747.0ms, throughput 19k tok/s/node, [2024-06-24_11:21:28]: iter 2025/3050: step_time 3760.1ms, throughput 19k tok/s/node, [2024-06-24_11:23:02]: iter 2050/3050: step_time 3763.7ms, throughput 19k tok/s/node, [2024-06-24_11:24:36]: iter 2075/3050: step_time 3762.9ms, throughput 19k tok/s/node, [2024-06-24_11:26:10]: iter 2100/3050: step_time 3760.0ms, throughput 19k tok/s/node, [2024-06-24_11:27:44]: iter 2125/3050: step_time 3753.4ms, throughput 19k tok/s/node, [2024-06-24_11:29:18]: iter 2150/3050: step_time 3753.5ms, throughput 19k tok/s/node, [2024-06-24_11:30:52]: iter 2175/3050: step_time 3751.5ms, throughput 19k tok/s/node, [2024-06-24_11:32:26]: iter 2200/3050: step_time 3764.1ms, throughput 18k tok/s/node, [2024-06-24_11:34:00]: iter 2225/3050: step_time 3769.3ms, throughput 18k tok/s/node, [2024-06-24_11:35:34]: iter 2250/3050: step_time 3752.0ms, throughput 19k tok/s/node, [2024-06-24_11:37:08]: iter 2275/3050: step_time 3763.8ms, throughput 19k tok/s/node, [2024-06-24_11:38:43]: iter 2300/3050: step_time 3758.7ms, throughput 19k tok/s/node, [2024-06-24_11:40:17]: iter 2325/3050: step_time 3763.8ms, throughput 19k tok/s/node, [2024-06-24_11:41:51]: iter 2350/3050: step_time 3760.2ms, throughput 19k tok/s/node, [2024-06-24_11:43:25]: iter 2375/3050: step_time 3764.2ms, throughput 18k tok/s/node, [2024-06-24_11:44:59]: iter 2400/3050: step_time 3765.7ms, throughput 18k tok/s/node, [2024-06-24_11:46:33]: iter 2425/3050: step_time 3755.9ms, throughput 19k tok/s/node, [2024-06-24_11:48:07]: iter 2450/3050: step_time 3767.5ms, throughput 18k tok/s/node, [2024-06-24_11:49:41]: iter 2475/3050: step_time 3760.2ms, throughput 19k tok/s/node, [2024-06-24_11:51:15]: iter 2500/3050: step_time 3756.8ms, throughput 19k tok/s/node, [2024-06-24_11:52:49]: iter 2525/3050: step_time 3764.0ms, throughput 18k tok/s/node, [2024-06-24_11:54:23]: iter 2550/3050: step_time 3775.6ms, throughput 18k tok/s/node, [2024-06-24_11:55:57]: iter 2575/3050: step_time 3753.1ms, throughput 19k tok/s/node, [2024-06-24_11:57:31]: iter 2600/3050: step_time 3759.9ms, throughput 19k tok/s/node, [2024-06-24_11:59:05]: iter 2625/3050: step_time 3764.0ms, throughput 18k tok/s/node, [2024-06-24_12:00:39]: iter 2650/3050: step_time 3757.0ms, throughput 19k tok/s/node, [2024-06-24_12:02:14]: iter 2675/3050: step_time 3745.6ms, throughput 19k tok/s/node, [2024-06-24_12:03:48]: iter 2700/3050: step_time 3758.7ms, throughput 19k tok/s/node, [2024-06-24_12:05:22]: iter 2725/3050: step_time 3766.7ms, throughput 18k tok/s/node, [2024-06-24_12:06:56]: iter 2750/3050: step_time 3761.3ms, throughput 19k tok/s/node, [2024-06-24_12:08:30]: iter 2775/3050: step_time 3759.5ms, throughput 19k tok/s/node, [2024-06-24_12:10:04]: iter 2800/3050: step_time 3758.9ms, throughput 19k tok/s/node, [2024-06-24_12:11:38]: iter 2825/3050: step_time 3756.8ms, throughput 19k tok/s/node, [2024-06-24_12:13:12]: iter 2850/3050: step_time 3759.1ms, throughput 19k tok/s/node, [2024-06-24_12:14:46]: iter 2875/3050: step_time 3753.8ms, throughput 19k tok/s/node, [2024-06-24_12:16:20]: iter 2900/3050: step_time 3753.2ms, throughput 19k tok/s/node, [2024-06-24_12:17:54]: iter 2925/3050: step_time 3768.2ms, throughput 18k tok/s/node, [2024-06-24_12:19:28]: iter 2950/3050: step_time 3754.2ms, throughput 19k tok/s/node, [2024-06-24_12:21:02]: iter 2975/3050: step_time 3752.3ms, throughput 19k tok/s/node, [2024-06-24_12:22:36]: iter 3000/3050: step_time 3760.4ms, throughput 19k tok/s/node, [2024-06-24_12:24:10]: iter 3025/3050: step_time 3767.3ms, throughput 18k tok/s/node, [2024-06-24_12:25:40]: Start evaluating loss for slipt val [2024-06-24_12:25:40]: estimated total number of iterations 31,size of the data 1972,size of batch 8,ddp_rank is 0 [2024-06-24_12:25:44]: iter 0/31: step_time 3571.2ms, throughput 19k tok/s/node, [2024-06-24_12:27:18]: iter 25/31: step_time 3760.5ms, throughput 19k tok/s/node, [2024-06-24_12:27:40]: 64 [2024-06-24_12:27:44]: Saving pre-computed cache loss: /app/suno/data/dpo/13b_extend_mix/13b_dpo_cached_loss.json [2024-06-24_12:27:44]: Check if loaded correctly: train 195148 vs 195148 val 1972 vs 1972 [2024-06-24_12:27:44]: Validate random number: 0.38074935973175184 [2024-06-24_12:27:44]: training... [2024-06-24_12:29:12]: loss estimation took 88.8 seconds. (100.0% of loop) [2024-06-24_12:29:12]: step 0: train loss 3.4190, val loss 3.4502 [2024-06-24_12:29:21]: iter 0: avg_loss 0.010, avg_acc 0.000, step_time 97690.6ms, mfu 0.0%, throughput 0k tok/s, total time 98s [2024-06-24_12:32:24]: iter 25: avg_loss 0.010, avg_acc 0.560, step_time 7297.7ms, mfu 137.0%, throughput 305k tok/s, total time 281s [2024-06-24_12:35:26]: iter 50: avg_loss 0.010, avg_acc 0.540, step_time 7267.0ms, mfu 137.5%, throughput 307k tok/s, total time 462s [2024-06-24_12:38:29]: iter 75: avg_loss 0.010, avg_acc 0.360, step_time 7260.0ms, mfu 137.7%, throughput 307k tok/s, total time 645s [2024-06-24_12:41:32]: iter 100: avg_loss 0.010, avg_acc 0.480, step_time 7279.3ms, mfu 137.3%, throughput 306k tok/s, total time 828s [2024-06-24_12:44:34]: iter 125: avg_loss 0.010, avg_acc 0.560, step_time 7277.4ms, mfu 137.3%, throughput 306k tok/s, total time 1011s [2024-06-24_12:47:37]: iter 150: avg_loss 0.010, avg_acc 0.480, step_time 7276.2ms, mfu 137.4%, throughput 306k tok/s, total time 1193s [2024-06-24_12:50:40]: iter 175: avg_loss 0.010, avg_acc 0.660, step_time 7274.2ms, mfu 137.4%, throughput 306k tok/s, total time 1376s [2024-06-24_12:53:42]: iter 200: avg_loss 0.009, avg_acc 0.660, step_time 7270.3ms, mfu 137.5%, throughput 306k tok/s, total time 1559s [2024-06-24_12:56:45]: iter 225: avg_loss 0.009, avg_acc 0.660, step_time 7278.3ms, mfu 137.3%, throughput 306k tok/s, total time 1741s [2024-06-24_12:59:48]: iter 250: avg_loss 0.010, avg_acc 0.600, step_time 7275.3ms, mfu 137.4%, throughput 306k tok/s, total time 1924s [2024-06-24_13:02:50]: iter 275: avg_loss 0.010, avg_acc 0.500, step_time 7289.8ms, mfu 137.1%, throughput 306k tok/s, total time 2107s [2024-06-24_13:05:52]: iter 300: avg_loss 0.009, avg_acc 0.760, step_time 7307.9ms, mfu 136.8%, throughput 305k tok/s, total time 2289s [2024-06-24_13:08:55]: iter 325: avg_loss 0.010, avg_acc 0.600, step_time 7276.0ms, mfu 137.4%, throughput 306k tok/s, total time 2472s [2024-06-24_13:11:58]: iter 350: avg_loss 0.009, avg_acc 0.600, step_time 7289.6ms, mfu 137.1%, throughput 306k tok/s, total time 2655s [2024-06-24_13:15:01]: iter 375: avg_loss 0.009, avg_acc 0.580, step_time 7570.8ms, mfu 132.0%, throughput 294k tok/s, total time 2837s [2024-06-24_13:18:03]: iter 400: avg_loss 0.010, avg_acc 0.500, step_time 7271.2ms, mfu 137.5%, throughput 306k tok/s, total time 3020s [2024-06-24_13:21:06]: iter 425: avg_loss 0.009, avg_acc 0.560, step_time 7288.4ms, mfu 137.1%, throughput 306k tok/s, total time 3202s [2024-06-24_13:24:09]: iter 450: avg_loss 0.009, avg_acc 0.580, step_time 7282.4ms, mfu 137.3%, throughput 306k tok/s, total time 3385s [2024-06-24_13:27:11]: iter 475: avg_loss 0.010, avg_acc 0.520, step_time 7278.2ms, mfu 137.3%, throughput 306k tok/s, total time 3568s [2024-06-24_13:30:14]: iter 500: avg_loss 0.009, avg_acc 0.640, step_time 7260.0ms, mfu 137.7%, throughput 307k tok/s, total time 3750s [2024-06-24_13:33:16]: iter 525: avg_loss 0.010, avg_acc 0.540, step_time 7256.4ms, mfu 137.7%, throughput 307k tok/s, total time 3933s [2024-06-24_13:36:19]: iter 550: avg_loss 0.009, avg_acc 0.680, step_time 7266.9ms, mfu 137.5%, throughput 307k tok/s, total time 4115s [2024-06-24_13:39:22]: iter 575: avg_loss 0.008, avg_acc 0.680, step_time 7270.2ms, mfu 137.5%, throughput 306k tok/s, total time 4298s [2024-06-24_13:42:24]: iter 600: avg_loss 0.009, avg_acc 0.620, step_time 7278.9ms, mfu 137.3%, throughput 306k tok/s, total time 4480s [2024-06-24_13:45:26]: iter 625: avg_loss 0.010, avg_acc 0.500, step_time 7292.9ms, mfu 137.1%, throughput 306k tok/s, total time 4663s [2024-06-24_13:48:29]: iter 650: avg_loss 0.009, avg_acc 0.580, step_time 7277.1ms, mfu 137.4%, throughput 306k tok/s, total time 4845s [2024-06-24_13:51:32]: iter 675: avg_loss 0.010, avg_acc 0.540, step_time 7276.6ms, mfu 137.4%, throughput 306k tok/s, total time 5028s [2024-06-24_13:55:49]: loss estimation took 88.9 seconds. (1.7% of loop) [2024-06-24_13:55:49]: step 699: train loss 3.4602, val loss 3.4503 [2024-06-24_13:57:17]: saving checkpoint to /app/suno/checkpoints/2024-06-24_09-12-17 [2024-06-24_13:57:59]: saving took 130.2 seconds. (2.5% of loop) [2024-06-24_13:58:10]: iter 699: avg_loss 0.010, avg_acc 0.562, step_time 230583.6ms, mfu 4.3%, throughput 10k tok/s, total time 5427s [2024-06-24_13:58:10]: done.