working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
[2024-08-04_01:34:40]: Failed to import xformers.
[2024-08-04_01:34:40]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm.
Overriding: out_dir = /app/suno/checkpoints
Overriding: data_dir = /app/suno/data/dpo/30b_t1_v12
Overriding: train_filename = data_tr.bin
Overriding: train_metas_filename = meta_tr.jsonl
Overriding: train_info_filename = info_tr.json
Overriding: val_filename = data_val.bin
Overriding: val_metas_filename = meta_val.jsonl
Overriding: val_info_filename = info_val.json
Overriding: learning_rate = 5e-07
Overriding: min_lr = 1e-08
Overriding: do_ipo = True
Overriding: dpo_beta = 5.0
Overriding: semantic_codebook_weight = 4.0
Overriding: last_codebook_weight = 0.5
Overriding: warmup_iters = 100
Overriding: max_iters = 1250
Overriding: grad_clip = 0.1
Overriding: eval_interval = 2000
Overriding: eval_iters = 25
Overriding: step_save_iters = 2000
Overriding: block_size = 8832
Overriding: t_text = 2560
Overriding: t_memmap = 6016
Overriding: t_audio = 6272
Overriding: use_rotary_pos_emb = True
Overriding: rope_theta = 500000
Overriding: use_qk_norm = True
Overriding: activation_f = silu
Overriding: embed_scale_factor = 10.0
Overriding: global_every_n_layers = 1
Overriding: n_layer = 60
Overriding: n_head = 56
Overriding: d_head = 128
Overriding: n_kv_head = 4
Overriding: attention_type = tao
Overriding: gradient_accumulation_steps = 1
Overriding: batch_size = 2
Overriding: eval_loss_batch_size = 12
Overriding: fsdp = True
Overriding: sharding_strategy = full_shard
Overriding: grad_checkpointing = True
Overriding: preload_checkpoint = /app/suno/data/dpo/models/model_30b_fix_ft2_20k.pt
Overriding: model_cache_loss_name = 30b_repro_s1_bt12
Overriding: preload_strict = False
Overriding: local_cache_dir = /mnt/localdisk/tmp_tony
Overriding: wandb_log = True
Overriding: wandb_project = chirp-v4-dpo
Overriding: wandb_run_name = dpo_30b_t1_v12
[2024-08-04_01:34:41]: ddp init, rank 16, local_rank 0
[2024-08-04_01:34:41]: ddp init, rank 24, local_rank 0
[2024-08-04_01:34:42]: ddp init, rank 8, local_rank 0
[2024-08-04_01:34:42]: ddp init, rank 0, local_rank 0
[2024-08-04_01:34:42]: ddp init, rank 10, local_rank 2
[2024-08-04_01:34:42]: ddp init, rank 12, local_rank 4
[2024-08-04_01:34:42]: ddp init, rank 5, local_rank 5
[2024-08-04_01:34:42]: ddp init, rank 28, local_rank 4
[2024-08-04_01:34:42]: ddp init, rank 27, local_rank 3
[2024-08-04_01:34:42]: ddp init, rank 21, local_rank 5
[2024-08-04_01:34:42]: ddp init, rank 4, local_rank 4
[2024-08-04_01:34:42]: ddp init, rank 3, local_rank 3
[2024-08-04_01:34:42]: ddp init, rank 23, local_rank 7
[2024-08-04_01:34:42]: ddp init, rank 7, local_rank 7
[2024-08-04_01:34:42]: ddp init, rank 18, local_rank 2
[2024-08-04_01:34:42]: ddp init, rank 22, local_rank 6
[2024-08-04_01:34:42]: ddp init, rank 13, local_rank 5
[2024-08-04_01:34:42]: ddp init, rank 11, local_rank 3
[2024-08-04_01:34:42]: ddp init, rank 15, local_rank 7
[2024-08-04_01:34:42]: ddp init, rank 19, local_rank 3
[2024-08-04_01:34:42]: ddp init, rank 17, local_rank 1
[2024-08-04_01:34:42]: ddp init, rank 1, local_rank 1
[2024-08-04_01:34:42]: ddp init, rank 6, local_rank 6
[2024-08-04_01:34:42]: ddp init, rank 2, local_rank 2
[2024-08-04_01:34:42]: ddp init, rank 20, local_rank 4
[2024-08-04_01:34:42]: ddp init, rank 14, local_rank 6
NCCL version 2.20.5+cuda12.4
[2024-08-04_01:34:42]: ddp init, rank 9, local_rank 1
[2024-08-04_01:34:42]: ddp init, rank 29, local_rank 5
[2024-08-04_01:34:42]: ddp init, rank 25, local_rank 1
[2024-08-04_01:34:42]: ddp init, rank 31, local_rank 7
[2024-08-04_01:34:42]: ddp init, rank 26, local_rank 2
[2024-08-04_01:34:42]: ddp init, rank 30, local_rank 6
[2024-08-04_01:34:54]: ddp init: world size 32 ddp_rank 0.
[2024-08-04_01:34:54]: loss discounts for codebooks: [0.308 0.077 0.073 0.07  0.066 0.063 0.059 0.056 0.052 0.049 0.045 0.042
 0.038]
[2024-08-04_01:35:06]: Total world size 32
[2024-08-04_01:35:06]: logging checkpoint here: /app/suno/checkpoints/2024-08-04_01-35-06
[2024-08-04_01:35:06]: loading data...
[2024-08-04_01:35:06]: indexed 100.0% of data
[2024-08-04_01:35:06]: 736 lines of data_val.bin loaded.
[2024-08-04_01:35:08]: indexed 100.0% of data
[2024-08-04_01:35:08]: 72,742 lines of data_tr.bin loaded.
[2024-08-04_01:35:08]: train data weights:
 50.0% perference_0
 50.0% perference_1
[2024-08-04_01:35:08]: done loading data
[2024-08-04_01:35:08]: GPU capacity: NVIDIA H100 80GB HBM3 (0) with 79.33GiB memory
[2024-08-04_01:35:08]: Initializing train model from scratch
[2024-08-04_01:41:05]: number of parameters: 32020M
[2024-08-04_01:41:05]: finish init train model
[2024-08-04_01:41:05]: not compiling model.
[2024-08-04_01:41:05]: start loading state dict
[2024-08-04_01:41:05]: verifying model args...
[2024-08-04_01:41:05]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-08-04_01:41:09]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-08-04_01:41:09]: copying checkpoint file to local cachedir...
[2024-08-04_01:45:59]: loading model state_dict on gpu 0
[2024-08-04_01:46:07]: loading model state_dict on gpu 1
[2024-08-04_01:46:16]: loading model state_dict on gpu 2
[2024-08-04_01:46:25]: loading model state_dict on gpu 3
[2024-08-04_01:46:34]: loading model state_dict on gpu 4
[2024-08-04_01:46:43]: loading model state_dict on gpu 5
[2024-08-04_01:46:52]: loading model state_dict on gpu 6
[2024-08-04_01:47:00]: loading model state_dict on gpu 7
[2024-08-04_01:47:09]: finish loading state dict
[2024-08-04_01:47:09]: wrapping model in FSDP ....
[2024-08-04_01:47:40]: GPU memory usage for model: 12.32GiB(15.53%)
[2024-08-04_01:47:40]: applying fsdp activation checkpointing...
[2024-08-04_01:47:40]: num decayed parameter tensors: 567, with 1,006,788,192 parameters
[2024-08-04_01:47:40]: num non-decayed parameter tensors: 124, with 430,080 parameters
[2024-08-04_01:47:40]: using fused Optimizer: False
[2024-08-04_01:47:40]: model setup done
[2024-08-04_01:47:40]: Validate random number: 0.9918216287004739
[2024-08-04_01:47:40]: Evaluating
[2024-08-04_01:47:40]: Start the ref model loss eval loop.
[2024-08-04_01:47:40]: Pre-compute cache loss
[2024-08-04_01:47:40]: Start evaluating loss for slipt train
[2024-08-04_01:47:40]: estimated total number of iterations 190,size of the data 72742,size of batch 12,ddp_rank is 0
[2024-08-04_01:47:58]: iter 0/190: step_time 17586.4ms, throughput 6k tok/s/node,
[2024-08-04_01:54:13]: iter 25/190: step_time 15071.3ms, throughput 7k tok/s/node,
[2024-08-04_02:00:29]: iter 50/190: step_time 15064.4ms, throughput 7k tok/s/node,
[2024-08-04_02:06:45]: iter 75/190: step_time 15007.0ms, throughput 7k tok/s/node,
[2024-08-04_02:13:01]: iter 100/190: step_time 15022.6ms, throughput 7k tok/s/node,
[2024-08-04_02:19:17]: iter 125/190: step_time 15005.2ms, throughput 7k tok/s/node,
[2024-08-04_02:25:33]: iter 150/190: step_time 15051.2ms, throughput 7k tok/s/node,
[2024-08-04_02:31:49]: iter 175/190: step_time 15009.6ms, throughput 7k tok/s/node,
[2024-08-04_02:35:20]: Start evaluating loss for slipt val
[2024-08-04_02:35:20]: estimated total number of iterations 2,size of the data 736,size of batch 12,ddp_rank is 0
[2024-08-04_02:35:34]: iter 0/2: step_time 14673.5ms, throughput 7k tok/s/node,
[2024-08-04_02:35:50]: 32
[2024-08-04_02:35:51]: Saving pre-computed cache loss: /app/suno/data/dpo/30b_t1_v12/30b_repro_s1_bt12_cached_loss.json
[2024-08-04_02:35:51]: Check if loaded correctly: 
 train 72742 vs 72742
 val 736 vs 736
[2024-08-04_02:35:51]: Validate random number: 0.9298063676094717
[2024-08-04_02:35:51]: training...
[2024-08-04_02:35:54]: Eval -- Policy positive loss: tensor([3.3458]), Policy negative loss: tensor([2.5122])
[2024-08-04_02:35:54]: Eval -- Reference positive loss: tensor([3.3458]), Reference negative loss: tensor([2.5122])
[2024-08-04_02:35:54]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:35:59]: Eval -- Policy positive loss: tensor([2.7622]), Policy negative loss: tensor([2.3310])
[2024-08-04_02:35:59]: Eval -- Reference positive loss: tensor([2.7622]), Reference negative loss: tensor([2.3310])
[2024-08-04_02:35:59]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:36:04]: Eval -- Policy positive loss: tensor([1.8245]), Policy negative loss: tensor([2.2677])
[2024-08-04_02:36:04]: Eval -- Reference positive loss: tensor([1.8245]), Reference negative loss: tensor([2.2677])
[2024-08-04_02:36:04]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-5.9605e-07]), Rejected rewards: tensor([1.1921e-06])
[2024-08-04_02:36:09]: Eval -- Policy positive loss: tensor([2.4122]), Policy negative loss: tensor([2.5681])
[2024-08-04_02:36:09]: Eval -- Reference positive loss: tensor([2.4122]), Reference negative loss: tensor([2.5681])
[2024-08-04_02:36:09]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:36:14]: Eval -- Policy positive loss: tensor([2.7374]), Policy negative loss: tensor([3.0487])
[2024-08-04_02:36:14]: Eval -- Reference positive loss: tensor([2.7374]), Reference negative loss: tensor([3.0487])
[2024-08-04_02:36:14]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:36:20]: Eval -- Policy positive loss: tensor([2.8609]), Policy negative loss: tensor([3.1650])
[2024-08-04_02:36:20]: Eval -- Reference positive loss: tensor([2.8609]), Reference negative loss: tensor([3.1650])
[2024-08-04_02:36:20]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([0.])
[2024-08-04_02:36:25]: Eval -- Policy positive loss: tensor([2.5840]), Policy negative loss: tensor([3.2727])
[2024-08-04_02:36:25]: Eval -- Reference positive loss: tensor([2.5840]), Reference negative loss: tensor([3.2727])
[2024-08-04_02:36:25]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:36:30]: Eval -- Policy positive loss: tensor([1.4428]), Policy negative loss: tensor([2.1317])
[2024-08-04_02:36:30]: Eval -- Reference positive loss: tensor([1.4428]), Reference negative loss: tensor([2.1317])
[2024-08-04_02:36:30]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-5.9605e-07]), Rejected rewards: tensor([1.1921e-06])
[2024-08-04_02:36:35]: Eval -- Policy positive loss: tensor([3.1352]), Policy negative loss: tensor([3.3717])
[2024-08-04_02:36:35]: Eval -- Reference positive loss: tensor([3.1352]), Reference negative loss: tensor([3.3717])
[2024-08-04_02:36:35]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:36:40]: Eval -- Policy positive loss: tensor([2.3278]), Policy negative loss: tensor([3.0515])
[2024-08-04_02:36:40]: Eval -- Reference positive loss: tensor([2.3278]), Reference negative loss: tensor([3.0515])
[2024-08-04_02:36:40]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:36:45]: Eval -- Policy positive loss: tensor([3.5802]), Policy negative loss: tensor([3.5324])
[2024-08-04_02:36:45]: Eval -- Reference positive loss: tensor([3.5802]), Reference negative loss: tensor([3.5324])
[2024-08-04_02:36:45]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([2.3842e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:36:50]: Eval -- Policy positive loss: tensor([2.8089]), Policy negative loss: tensor([2.5901])
[2024-08-04_02:36:50]: Eval -- Reference positive loss: tensor([2.8089]), Reference negative loss: tensor([2.5901])
[2024-08-04_02:36:50]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([0.])
[2024-08-04_02:36:56]: Eval -- Policy positive loss: tensor([3.7300]), Policy negative loss: tensor([4.1375])
[2024-08-04_02:36:56]: Eval -- Reference positive loss: tensor([3.7300]), Reference negative loss: tensor([4.1375])
[2024-08-04_02:36:56]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([2.3842e-06])
[2024-08-04_02:37:01]: Eval -- Policy positive loss: tensor([2.6683]), Policy negative loss: tensor([2.6366])
[2024-08-04_02:37:01]: Eval -- Reference positive loss: tensor([2.6683]), Reference negative loss: tensor([2.6366])
[2024-08-04_02:37:01]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:37:06]: Eval -- Policy positive loss: tensor([3.2560]), Policy negative loss: tensor([2.6781])
[2024-08-04_02:37:06]: Eval -- Reference positive loss: tensor([3.2560]), Reference negative loss: tensor([2.6781])
[2024-08-04_02:37:06]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:37:11]: Eval -- Policy positive loss: tensor([2.7291]), Policy negative loss: tensor([3.1080])
[2024-08-04_02:37:11]: Eval -- Reference positive loss: tensor([2.7291]), Reference negative loss: tensor([3.1080])
[2024-08-04_02:37:11]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([0.])
[2024-08-04_02:37:16]: Eval -- Policy positive loss: tensor([3.0644]), Policy negative loss: tensor([4.0173])
[2024-08-04_02:37:16]: Eval -- Reference positive loss: tensor([3.0644]), Reference negative loss: tensor([4.0173])
[2024-08-04_02:37:16]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([-2.3842e-06])
[2024-08-04_02:37:21]: Eval -- Policy positive loss: tensor([2.7491]), Policy negative loss: tensor([3.2212])
[2024-08-04_02:37:21]: Eval -- Reference positive loss: tensor([2.7491]), Reference negative loss: tensor([3.2212])
[2024-08-04_02:37:21]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:37:26]: Eval -- Policy positive loss: tensor([2.8212]), Policy negative loss: tensor([2.9103])
[2024-08-04_02:37:26]: Eval -- Reference positive loss: tensor([2.8212]), Reference negative loss: tensor([2.9103])
[2024-08-04_02:37:26]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:37:32]: Eval -- Policy positive loss: tensor([2.8565]), Policy negative loss: tensor([3.6911])
[2024-08-04_02:37:32]: Eval -- Reference positive loss: tensor([2.8565]), Reference negative loss: tensor([3.6911])
[2024-08-04_02:37:32]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-1.1921e-06]), Rejected rewards: tensor([1.1921e-06])
[2024-08-04_02:37:37]: Eval -- Policy positive loss: tensor([3.4659]), Policy negative loss: tensor([2.7419])
[2024-08-04_02:37:37]: Eval -- Reference positive loss: tensor([3.4659]), Reference negative loss: tensor([2.7419])
[2024-08-04_02:37:37]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([0.])
[2024-08-04_02:37:42]: Eval -- Policy positive loss: tensor([3.2303]), Policy negative loss: tensor([3.3165])
[2024-08-04_02:37:42]: Eval -- Reference positive loss: tensor([3.2303]), Reference negative loss: tensor([3.3165])
[2024-08-04_02:37:42]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([-1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:37:47]: Eval -- Policy positive loss: tensor([2.9491]), Policy negative loss: tensor([2.5980])
[2024-08-04_02:37:47]: Eval -- Reference positive loss: tensor([2.9491]), Reference negative loss: tensor([2.5980])
[2024-08-04_02:37:47]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([-1.1921e-06])
[2024-08-04_02:37:52]: Eval -- Policy positive loss: tensor([3.3898]), Policy negative loss: tensor([2.7385])
[2024-08-04_02:37:52]: Eval -- Reference positive loss: tensor([3.3898]), Reference negative loss: tensor([2.7385])
[2024-08-04_02:37:52]: Eval -- Preference loss: tensor([0.0100]), Chosen rewards: tensor([1.1921e-06]), Rejected rewards: tensor([0.])
[2024-08-04_02:37:55]: loss estimation took 123.5 seconds. (100.0% of loop)
[2024-08-04_02:37:55]: step 0: train loss 3.4652, val loss 3.4113
[2024-08-04_02:38:06]: iter 0: avg_loss 0.010, step_time 134905.8ms, mfu 0.0%, throughput 0k tok/s, total time 135s, memory 45.75GiB(57.67%)
[2024-08-04_02:42:24]: iter 25: avg_loss 0.010, step_time 10335.3ms, mfu 130.2%, throughput 55k tok/s, total time 393s, memory 49.68GiB(62.63%)
[2024-08-04_02:46:42]: iter 50: avg_loss 0.010, step_time 10315.2ms, mfu 130.5%, throughput 55k tok/s, total time 651s, memory 49.68GiB(62.63%)
[2024-08-04_02:51:01]: iter 75: avg_loss 0.010, step_time 10343.3ms, mfu 130.1%, throughput 55k tok/s, total time 909s, memory 49.68GiB(62.63%)
[2024-08-04_02:55:19]: iter 100: avg_loss 0.010, step_time 10279.3ms, mfu 130.9%, throughput 55k tok/s, total time 1167s, memory 49.68GiB(62.63%)
[2024-08-04_02:59:37]: iter 125: avg_loss 0.010, step_time 10323.5ms, mfu 130.4%, throughput 55k tok/s, total time 1426s, memory 49.68GiB(62.63%)
[2024-08-04_03:03:55]: iter 150: avg_loss 0.010, step_time 10320.6ms, mfu 130.4%, throughput 55k tok/s, total time 1684s, memory 49.68GiB(62.63%)
[2024-08-04_03:08:13]: iter 175: avg_loss 0.010, step_time 10341.6ms, mfu 130.1%, throughput 55k tok/s, total time 1942s, memory 49.68GiB(62.63%)
[2024-08-04_03:12:31]: iter 200: avg_loss 0.009, step_time 10326.2ms, mfu 130.3%, throughput 55k tok/s, total time 2200s, memory 49.68GiB(62.63%)
[2024-08-04_03:16:49]: iter 225: avg_loss 0.009, step_time 10303.1ms, mfu 130.6%, throughput 55k tok/s, total time 2458s, memory 49.68GiB(62.63%)
[2024-08-04_03:21:07]: iter 250: avg_loss 0.009, step_time 10343.5ms, mfu 130.1%, throughput 55k tok/s, total time 2716s, memory 49.68GiB(62.63%)
[2024-08-04_03:25:25]: iter 275: avg_loss 0.009, step_time 10341.5ms, mfu 130.1%, throughput 55k tok/s, total time 2974s, memory 49.68GiB(62.63%)
[2024-08-04_03:29:44]: iter 300: avg_loss 0.010, step_time 10323.2ms, mfu 130.4%, throughput 55k tok/s, total time 3232s, memory 49.68GiB(62.63%)
[2024-08-04_03:34:02]: iter 325: avg_loss 0.009, step_time 10325.9ms, mfu 130.3%, throughput 55k tok/s, total time 3491s, memory 49.68GiB(62.63%)
[2024-08-04_03:38:20]: iter 350: avg_loss 0.009, step_time 10326.8ms, mfu 130.3%, throughput 55k tok/s, total time 3749s, memory 49.68GiB(62.63%)
[2024-08-04_03:42:38]: iter 375: avg_loss 0.010, step_time 10340.0ms, mfu 130.2%, throughput 55k tok/s, total time 4007s, memory 49.68GiB(62.63%)
[2024-08-04_03:46:56]: iter 400: avg_loss 0.009, step_time 10316.0ms, mfu 130.5%, throughput 55k tok/s, total time 4265s, memory 49.68GiB(62.63%)
[2024-08-04_03:51:14]: iter 425: avg_loss 0.010, step_time 10350.3ms, mfu 130.0%, throughput 55k tok/s, total time 4523s, memory 49.68GiB(62.63%)
[2024-08-04_03:55:33]: iter 450: avg_loss 0.009, step_time 10319.3ms, mfu 130.4%, throughput 55k tok/s, total time 4781s, memory 49.68GiB(62.63%)
[2024-08-04_03:59:51]: iter 475: avg_loss 0.010, step_time 10311.2ms, mfu 130.5%, throughput 55k tok/s, total time 5039s, memory 49.68GiB(62.63%)
[2024-08-04_04:04:09]: iter 500: avg_loss 0.010, step_time 10308.4ms, mfu 130.6%, throughput 55k tok/s, total time 5297s, memory 49.68GiB(62.63%)
[2024-08-04_04:08:27]: iter 525: avg_loss 0.009, step_time 10348.0ms, mfu 130.1%, throughput 55k tok/s, total time 5555s, memory 49.68GiB(62.63%)
[2024-08-04_04:12:45]: iter 550: avg_loss 0.009, step_time 10308.9ms, mfu 130.5%, throughput 55k tok/s, total time 5813s, memory 49.68GiB(62.63%)
[2024-08-04_04:17:03]: iter 575: avg_loss 0.010, step_time 10347.1ms, mfu 130.1%, throughput 55k tok/s, total time 6072s, memory 49.68GiB(62.63%)
[2024-08-04_04:21:21]: iter 600: avg_loss 0.009, step_time 10293.0ms, mfu 130.7%, throughput 55k tok/s, total time 6330s, memory 49.68GiB(62.63%)
[2024-08-04_04:25:40]: iter 625: avg_loss 0.009, step_time 10337.9ms, mfu 130.2%, throughput 55k tok/s, total time 6588s, memory 49.68GiB(62.63%)
[2024-08-04_04:29:58]: iter 650: avg_loss 0.008, step_time 10317.1ms, mfu 130.4%, throughput 55k tok/s, total time 6846s, memory 49.68GiB(62.63%)
[2024-08-04_04:34:16]: iter 675: avg_loss 0.009, step_time 10329.3ms, mfu 130.3%, throughput 55k tok/s, total time 7104s, memory 49.68GiB(62.63%)
[2024-08-04_04:38:34]: iter 700: avg_loss 0.008, step_time 10324.3ms, mfu 130.3%, throughput 55k tok/s, total time 7362s, memory 49.68GiB(62.63%)
[2024-08-04_04:42:52]: iter 725: avg_loss 0.009, step_time 10303.3ms, mfu 130.6%, throughput 55k tok/s, total time 7620s, memory 49.68GiB(62.63%)
[2024-08-04_04:47:10]: iter 750: avg_loss 0.009, step_time 10317.2ms, mfu 130.4%, throughput 55k tok/s, total time 7878s, memory 49.68GiB(62.63%)
[2024-08-04_04:51:28]: iter 775: avg_loss 0.009, step_time 10301.3ms, mfu 130.6%, throughput 55k tok/s, total time 8136s, memory 49.68GiB(62.63%)
[2024-08-04_04:55:46]: iter 800: avg_loss 0.008, step_time 10342.9ms, mfu 130.1%, throughput 55k tok/s, total time 8394s, memory 49.68GiB(62.63%)
[2024-08-04_05:00:03]: iter 825: avg_loss 0.008, step_time 10318.6ms, mfu 130.4%, throughput 55k tok/s, total time 8652s, memory 49.68GiB(62.63%)
[2024-08-04_05:04:21]: iter 850: avg_loss 0.008, step_time 10302.6ms, mfu 130.6%, throughput 55k tok/s, total time 8910s, memory 49.68GiB(62.63%)
[2024-08-04_05:08:39]: iter 875: avg_loss 0.010, step_time 10316.1ms, mfu 130.5%, throughput 55k tok/s, total time 9168s, memory 49.68GiB(62.63%)
[2024-08-04_05:12:57]: iter 900: avg_loss 0.010, step_time 10304.8ms, mfu 130.6%, throughput 55k tok/s, total time 9426s, memory 49.68GiB(62.63%)
[2024-08-04_05:17:16]: iter 925: avg_loss 0.008, step_time 10339.0ms, mfu 130.2%, throughput 55k tok/s, total time 9685s, memory 49.68GiB(62.63%)
[2024-08-04_05:21:34]: iter 950: avg_loss 0.010, step_time 10321.9ms, mfu 130.4%, throughput 55k tok/s, total time 9943s, memory 49.68GiB(62.63%)
[2024-08-04_05:25:52]: iter 975: avg_loss 0.008, step_time 10323.3ms, mfu 130.4%, throughput 55k tok/s, total time 10201s, memory 49.68GiB(62.63%)
[2024-08-04_05:30:10]: iter 1000: avg_loss 0.009, step_time 10276.7ms, mfu 131.0%, throughput 55k tok/s, total time 10459s, memory 49.68GiB(62.63%)
[2024-08-04_05:34:28]: iter 1025: avg_loss 0.012, step_time 10292.3ms, mfu 130.8%, throughput 55k tok/s, total time 10717s, memory 49.68GiB(62.63%)
[2024-08-04_05:38:46]: iter 1050: avg_loss 0.009, step_time 10335.4ms, mfu 130.2%, throughput 55k tok/s, total time 10975s, memory 49.68GiB(62.63%)
[2024-08-04_05:43:04]: iter 1075: avg_loss 0.009, step_time 10335.1ms, mfu 130.2%, throughput 55k tok/s, total time 11233s, memory 49.68GiB(62.63%)
[2024-08-04_05:47:22]: iter 1100: avg_loss 0.009, step_time 10319.4ms, mfu 130.4%, throughput 55k tok/s, total time 11491s, memory 49.68GiB(62.63%)
[2024-08-04_05:51:40]: iter 1125: avg_loss 0.008, step_time 10336.9ms, mfu 130.2%, throughput 55k tok/s, total time 11749s, memory 49.68GiB(62.63%)
[2024-08-04_05:55:59]: iter 1150: avg_loss 0.008, step_time 10347.8ms, mfu 130.1%, throughput 55k tok/s, total time 12007s, memory 49.68GiB(62.63%)
[2024-08-04_06:00:17]: iter 1175: avg_loss 0.008, step_time 10315.4ms, mfu 130.5%, throughput 55k tok/s, total time 12266s, memory 49.68GiB(62.63%)
[2024-08-04_06:04:35]: iter 1200: avg_loss 0.008, step_time 10346.5ms, mfu 130.1%, throughput 55k tok/s, total time 12524s, memory 49.68GiB(62.63%)
[2024-08-04_06:08:54]: iter 1225: avg_loss 0.009, step_time 10323.9ms, mfu 130.4%, throughput 55k tok/s, total time 12782s, memory 49.68GiB(62.63%)
[2024-08-04_06:12:54]: Eval -- Policy positive loss: tensor([2.7437]), Policy negative loss: tensor([3.1332])
[2024-08-04_06:12:54]: Eval -- Reference positive loss: tensor([2.7428]), Reference negative loss: tensor([3.1209])
[2024-08-04_06:12:54]: Eval -- Preference loss: tensor([0.0078]), Chosen rewards: tensor([-0.0044]), Rejected rewards: tensor([-0.0615])
[2024-08-04_06:12:59]: Eval -- Policy positive loss: tensor([3.8419]), Policy negative loss: tensor([3.6855])
[2024-08-04_06:12:59]: Eval -- Reference positive loss: tensor([3.8468]), Reference negative loss: tensor([3.6831])
[2024-08-04_06:12:59]: Eval -- Preference loss: tensor([0.0086]), Chosen rewards: tensor([0.0240]), Rejected rewards: tensor([-0.0120])
[2024-08-04_06:13:04]: Eval -- Policy positive loss: tensor([3.3763]), Policy negative loss: tensor([2.9829])
[2024-08-04_06:13:04]: Eval -- Reference positive loss: tensor([3.3546]), Reference negative loss: tensor([2.9452])
[2024-08-04_06:13:04]: Eval -- Preference loss: tensor([0.0071]), Chosen rewards: tensor([-0.1084]), Rejected rewards: tensor([-0.1885])
[2024-08-04_06:13:09]: Eval -- Policy positive loss: tensor([3.3461]), Policy negative loss: tensor([2.7045])
[2024-08-04_06:13:09]: Eval -- Reference positive loss: tensor([3.3528]), Reference negative loss: tensor([2.6998])
[2024-08-04_06:13:09]: Eval -- Preference loss: tensor([0.0078]), Chosen rewards: tensor([0.0336]), Rejected rewards: tensor([-0.0235])
[2024-08-04_06:13:14]: Eval -- Policy positive loss: tensor([2.9659]), Policy negative loss: tensor([3.0675])
[2024-08-04_06:13:14]: Eval -- Reference positive loss: tensor([2.9560]), Reference negative loss: tensor([3.0626])
[2024-08-04_06:13:14]: Eval -- Preference loss: tensor([0.0110]), Chosen rewards: tensor([-0.0497]), Rejected rewards: tensor([-0.0246])
[2024-08-04_06:13:19]: Eval -- Policy positive loss: tensor([3.1975]), Policy negative loss: tensor([3.0369])
[2024-08-04_06:13:19]: Eval -- Reference positive loss: tensor([3.1975]), Reference negative loss: tensor([3.0243])
[2024-08-04_06:13:19]: Eval -- Preference loss: tensor([0.0076]), Chosen rewards: tensor([0.]), Rejected rewards: tensor([-0.0628])
[2024-08-04_06:13:25]: Eval -- Policy positive loss: tensor([3.6360]), Policy negative loss: tensor([3.6929])
[2024-08-04_06:13:25]: Eval -- Reference positive loss: tensor([3.6471]), Reference negative loss: tensor([3.6830])
[2024-08-04_06:13:25]: Eval -- Preference loss: tensor([0.0063]), Chosen rewards: tensor([0.0552]), Rejected rewards: tensor([-0.0492])
[2024-08-04_06:13:30]: Eval -- Policy positive loss: tensor([3.2452]), Policy negative loss: tensor([3.6812])
[2024-08-04_06:13:30]: Eval -- Reference positive loss: tensor([3.2474]), Reference negative loss: tensor([3.6872])
[2024-08-04_06:13:30]: Eval -- Preference loss: tensor([0.0108]), Chosen rewards: tensor([0.0109]), Rejected rewards: tensor([0.0298])
[2024-08-04_06:13:35]: Eval -- Policy positive loss: tensor([2.9896]), Policy negative loss: tensor([2.7939])
[2024-08-04_06:13:35]: Eval -- Reference positive loss: tensor([2.9773]), Reference negative loss: tensor([2.7591])
[2024-08-04_06:13:35]: Eval -- Preference loss: tensor([0.0060]), Chosen rewards: tensor([-0.0617]), Rejected rewards: tensor([-0.1740])
[2024-08-04_06:13:40]: Eval -- Policy positive loss: tensor([3.8754]), Policy negative loss: tensor([2.8886])
[2024-08-04_06:13:40]: Eval -- Reference positive loss: tensor([3.8905]), Reference negative loss: tensor([2.8691])
[2024-08-04_06:13:40]: Eval -- Preference loss: tensor([0.0043]), Chosen rewards: tensor([0.0754]), Rejected rewards: tensor([-0.0975])
[2024-08-04_06:13:45]: Eval -- Policy positive loss: tensor([3.0471]), Policy negative loss: tensor([1.9386])
[2024-08-04_06:13:45]: Eval -- Reference positive loss: tensor([3.0335]), Reference negative loss: tensor([1.8510])
[2024-08-04_06:13:45]: Eval -- Preference loss: tensor([0.0007]), Chosen rewards: tensor([-0.0683]), Rejected rewards: tensor([-0.4382])
[2024-08-04_06:13:50]: Eval -- Policy positive loss: tensor([2.7300]), Policy negative loss: tensor([2.6935])
[2024-08-04_06:13:50]: Eval -- Reference positive loss: tensor([2.6932]), Reference negative loss: tensor([2.6573])
[2024-08-04_06:13:50]: Eval -- Preference loss: tensor([0.0101]), Chosen rewards: tensor([-0.1841]), Rejected rewards: tensor([-0.1806])
[2024-08-04_06:13:55]: Eval -- Policy positive loss: tensor([2.6009]), Policy negative loss: tensor([2.4565])
[2024-08-04_06:13:55]: Eval -- Reference positive loss: tensor([2.5998]), Reference negative loss: tensor([2.4360])
[2024-08-04_06:13:55]: Eval -- Preference loss: tensor([0.0065]), Chosen rewards: tensor([-0.0055]), Rejected rewards: tensor([-0.1027])
[2024-08-04_06:14:01]: Eval -- Policy positive loss: tensor([2.9325]), Policy negative loss: tensor([3.0617])
[2024-08-04_06:14:01]: Eval -- Reference positive loss: tensor([2.9288]), Reference negative loss: tensor([3.0628])
[2024-08-04_06:14:01]: Eval -- Preference loss: tensor([0.0110]), Chosen rewards: tensor([-0.0188]), Rejected rewards: tensor([0.0055])
[2024-08-04_06:14:06]: Eval -- Policy positive loss: tensor([2.6795]), Policy negative loss: tensor([3.0108])
[2024-08-04_06:14:06]: Eval -- Reference positive loss: tensor([2.6596]), Reference negative loss: tensor([2.9922])
[2024-08-04_06:14:06]: Eval -- Preference loss: tensor([0.0103]), Chosen rewards: tensor([-0.0994]), Rejected rewards: tensor([-0.0931])
[2024-08-04_06:14:11]: Eval -- Policy positive loss: tensor([3.2307]), Policy negative loss: tensor([3.5346])
[2024-08-04_06:14:11]: Eval -- Reference positive loss: tensor([3.2303]), Reference negative loss: tensor([3.5302])
[2024-08-04_06:14:11]: Eval -- Preference loss: tensor([0.0092]), Chosen rewards: tensor([-0.0022]), Rejected rewards: tensor([-0.0219])
[2024-08-04_06:14:16]: Eval -- Policy positive loss: tensor([2.7591]), Policy negative loss: tensor([2.4819])
[2024-08-04_06:14:16]: Eval -- Reference positive loss: tensor([2.7367]), Reference negative loss: tensor([2.4370])
[2024-08-04_06:14:16]: Eval -- Preference loss: tensor([0.0060]), Chosen rewards: tensor([-0.1117]), Rejected rewards: tensor([-0.2248])
[2024-08-04_06:14:21]: Eval -- Policy positive loss: tensor([2.1257]), Policy negative loss: tensor([2.2963])
[2024-08-04_06:14:21]: Eval -- Reference positive loss: tensor([2.0878]), Reference negative loss: tensor([2.2821])
[2024-08-04_06:14:21]: Eval -- Preference loss: tensor([0.0153]), Chosen rewards: tensor([-0.1890]), Rejected rewards: tensor([-0.0710])
[2024-08-04_06:14:26]: Eval -- Policy positive loss: tensor([2.9962]), Policy negative loss: tensor([1.9723])
[2024-08-04_06:14:26]: Eval -- Reference positive loss: tensor([2.9953]), Reference negative loss: tensor([1.9519])
[2024-08-04_06:14:26]: Eval -- Preference loss: tensor([0.0065]), Chosen rewards: tensor([-0.0044]), Rejected rewards: tensor([-0.1019])
[2024-08-04_06:14:31]: Eval -- Policy positive loss: tensor([2.8890]), Policy negative loss: tensor([3.2812])
[2024-08-04_06:14:31]: Eval -- Reference positive loss: tensor([2.9016]), Reference negative loss: tensor([3.3022])
[2024-08-04_06:14:31]: Eval -- Preference loss: tensor([0.0118]), Chosen rewards: tensor([0.0626]), Rejected rewards: tensor([0.1052])
[2024-08-04_06:14:37]: Eval -- Policy positive loss: tensor([3.4397]), Policy negative loss: tensor([3.3392])
[2024-08-04_06:14:37]: Eval -- Reference positive loss: tensor([3.4481]), Reference negative loss: tensor([3.3397])
[2024-08-04_06:14:37]: Eval -- Preference loss: tensor([0.0085]), Chosen rewards: tensor([0.0421]), Rejected rewards: tensor([0.0025])
[2024-08-04_06:14:42]: Eval -- Policy positive loss: tensor([3.2472]), Policy negative loss: tensor([3.0667])
[2024-08-04_06:14:42]: Eval -- Reference positive loss: tensor([3.2060]), Reference negative loss: tensor([3.0412])
[2024-08-04_06:14:42]: Eval -- Preference loss: tensor([0.0134]), Chosen rewards: tensor([-0.2060]), Rejected rewards: tensor([-0.1273])
[2024-08-04_06:14:47]: Eval -- Policy positive loss: tensor([3.6684]), Policy negative loss: tensor([3.5970])
[2024-08-04_06:14:47]: Eval -- Reference positive loss: tensor([3.6720]), Reference negative loss: tensor([3.5939])
[2024-08-04_06:14:47]: Eval -- Preference loss: tensor([0.0087]), Chosen rewards: tensor([0.0180]), Rejected rewards: tensor([-0.0158])
[2024-08-04_06:14:52]: Eval -- Policy positive loss: tensor([3.4208]), Policy negative loss: tensor([3.6236])
[2024-08-04_06:14:52]: Eval -- Reference positive loss: tensor([3.4198]), Reference negative loss: tensor([3.6182])
[2024-08-04_06:14:52]: Eval -- Preference loss: tensor([0.0091]), Chosen rewards: tensor([-0.0049]), Rejected rewards: tensor([-0.0268])
[2024-08-04_06:14:55]: loss estimation took 123.5 seconds. (0.9% of loop)
[2024-08-04_06:14:55]: step 1249: train loss 3.5409, val loss 3.4765
[2024-08-04_06:19:10]: saving checkpoint to /app/suno/checkpoints/2024-08-04_01-35-06
[2024-08-04_06:21:19]: saving took 384.2 seconds. (3.0% of loop)
[2024-08-04_06:21:50]: iter 1249: avg_loss 0.009, step_time 539250.6ms, mfu 2.5%, throughput 1k tok/s, total time 13559s, memory 51.06GiB(64.37%)
[2024-08-04_06:21:50]: done.