working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
[2024-09-26_03:56:56]: Failed to import xformers.
[2024-09-26_03:56:56]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm.
Overriding: out_dir = /app/suno/checkpoints
Overriding: data_dir = /app/suno/data/dpo/13b_s8_v9
Overriding: train_filename = data_tr.bin
Overriding: train_metas_filename = meta_tr.jsonl
Overriding: train_info_filename = info_tr.json
Overriding: val_filename = data_val.bin
Overriding: val_metas_filename = meta_val.jsonl
Overriding: val_info_filename = info_val.json
Overriding: learning_rate = 5e-07
Overriding: min_lr = 1e-08
Overriding: do_ipo = True
Overriding: dpo_beta = 20.0
Overriding: semantic_codebook_weight = 4.0
Overriding: last_codebook_weight = 0.5
Overriding: warmup_iters = 100
Overriding: max_iters = 1600
Overriding: grad_clip = 0.1
Overriding: eval_interval = 2000
Overriding: eval_iters = 25
Overriding: step_save_iters = 2000
Overriding: block_size = 8704
Overriding: t_text = 2560
Overriding: t_memmap = 6016
Overriding: t_audio = 6144
Overriding: use_rotary_pos_emb = True
Overriding: rope_theta = 500000
Overriding: use_qk_norm = False
Overriding: activation_f = gelu
Overriding: n_layer = 40
Overriding: n_head = 40
Overriding: d_head = 128
Overriding: n_kv_head = 8
Overriding: attention_type = tao
Overriding: gradient_accumulation_steps = 1
Overriding: eval_loss_batch_size = 16
Overriding: batch_size = 4
Overriding: fsdp = True
Overriding: sharding_strategy = full_shard
Overriding: grad_checkpointing = True
Overriding: preload_checkpoint = /app/suno/checkpoints/2024-06-17_01-25-46/last_ckpt_infer.pt
Overriding: model_cache_loss_name = 13b_dpo_s8_v3
Overriding: preload_strict = False
Overriding: local_cache_dir = /mnt/localdisk/tmp_tony
Overriding: wandb_log = True
Overriding: wandb_project = chirp-30b-dpo-dev
Overriding: wandb_run_name = dpo_13b_s8_v9_beta20
[2024-09-26_03:56:58]: ddp init, rank 16, local_rank 0
[2024-09-26_03:56:58]: ddp init, rank 0, local_rank 0
[2024-09-26_03:56:58]: ddp init, rank 24, local_rank 0
[2024-09-26_03:56:58]: ddp init, rank 1, local_rank 1
NCCL version 2.20.5+cuda12.4
[2024-09-26_03:56:58]: ddp init, rank 7, local_rank 7
[2024-09-26_03:56:58]: ddp init, rank 5, local_rank 5
[2024-09-26_03:56:58]: ddp init, rank 3, local_rank 3
[2024-09-26_03:56:58]: ddp init, rank 6, local_rank 6
[2024-09-26_03:56:58]: ddp init, rank 4, local_rank 4
[2024-09-26_03:56:58]: ddp init, rank 2, local_rank 2
[2024-09-26_03:56:59]: ddp init, rank 18, local_rank 2
[2024-09-26_03:56:59]: ddp init, rank 22, local_rank 6
[2024-09-26_03:56:59]: ddp init, rank 25, local_rank 1
[2024-09-26_03:56:59]: ddp init, rank 17, local_rank 1
[2024-09-26_03:56:59]: ddp init, rank 23, local_rank 7
[2024-09-26_03:56:59]: ddp init, rank 20, local_rank 4
[2024-09-26_03:56:59]: ddp init, rank 26, local_rank 2
[2024-09-26_03:56:59]: ddp init, rank 31, local_rank 7
[2024-09-26_03:56:59]: ddp init, rank 30, local_rank 6
[2024-09-26_03:56:59]: ddp init, rank 27, local_rank 3
[2024-09-26_03:56:59]: ddp init, rank 19, local_rank 3
[2024-09-26_03:56:59]: ddp init, rank 21, local_rank 5
[2024-09-26_03:56:59]: ddp init, rank 28, local_rank 4
[2024-09-26_03:56:59]: ddp init, rank 29, local_rank 5
[2024-09-26_03:56:59]: ddp init, rank 8, local_rank 0
[2024-09-26_03:56:59]: ddp init, rank 9, local_rank 1
[2024-09-26_03:56:59]: ddp init, rank 10, local_rank 2
[2024-09-26_03:56:59]: ddp init, rank 11, local_rank 3
[2024-09-26_03:56:59]: ddp init, rank 15, local_rank 7
[2024-09-26_03:56:59]: ddp init, rank 14, local_rank 6
[2024-09-26_03:56:59]: ddp init, rank 12, local_rank 4
[2024-09-26_03:56:59]: ddp init, rank 13, local_rank 5
[2024-09-26_03:57:11]: ddp init: world size 32 ddp_rank 0.
[2024-09-26_03:57:11]: loss discounts for codebooks: [0.308 0.077 0.073 0.07  0.066 0.063 0.059 0.056 0.052 0.049 0.045 0.042
 0.038]
[2024-09-26_03:57:23]: Total world size 32
[2024-09-26_03:57:23]: logging checkpoint here: /app/suno/checkpoints/2024-09-26_03-57-23
[2024-09-26_03:57:23]: loading data...
[2024-09-26_03:57:23]: indexed 100.0% of data
[2024-09-26_03:57:23]: 2,126 lines of data_val.bin loaded.
[2024-09-26_03:57:25]: indexed 100.0% of data
[2024-09-26_03:57:25]: 210,472 lines of data_tr.bin loaded.
[2024-09-26_03:57:25]: train data weights:
 50.0% perference_0
 50.0% perference_1
[2024-09-26_03:57:25]: done loading data
[2024-09-26_03:57:27]: GPU capacity: NVIDIA H100 80GB HBM3 (0) with 79.33GiB memory
[2024-09-26_03:57:27]: Initializing train model from scratch
[2024-09-26_03:59:38]: number of parameters: 11363M
[2024-09-26_03:59:38]: finish init train model
[2024-09-26_03:59:38]: not compiling model.
[2024-09-26_03:59:38]: start loading state dict
[2024-09-26_03:59:38]: verifying model args...
[2024-09-26_03:59:38]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-09-26_03:59:38]: warning: checkpoint missing config keys: {'global_every_n_layers'}
[2024-09-26_03:59:38]: warning: checkpoint config key mismatch on global_every_n_layers
[2024-09-26_03:59:38]: warning: checkpoint config key mismatch on attention_sliding_window_size
[2024-09-26_03:59:39]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-09-26_03:59:40]: loading model state_dict on gpu 0
[2024-09-26_03:59:45]: loading model state_dict on gpu 1
[2024-09-26_03:59:48]: loading model state_dict on gpu 2
[2024-09-26_03:59:51]: loading model state_dict on gpu 3
[2024-09-26_03:59:54]: loading model state_dict on gpu 4
[2024-09-26_03:59:57]: loading model state_dict on gpu 5
[2024-09-26_04:00:01]: loading model state_dict on gpu 6
[2024-09-26_04:00:04]: loading model state_dict on gpu 7
[2024-09-26_04:00:07]: finish loading state dict
[2024-09-26_04:00:07]: wrapping model in FSDP ....
[2024-09-26_04:00:17]: GPU memory usage for model: 7.40GiB(9.33%)
[2024-09-26_04:00:17]: applying fsdp activation checkpointing...
[2024-09-26_04:00:17]: num decayed parameter tensors: 267, with 359,601,280 parameters
[2024-09-26_04:00:17]: num non-decayed parameter tensors: 84, with 204,800 parameters
[2024-09-26_04:00:17]: using fused Optimizer: False
[2024-09-26_04:00:17]: model setup done
[2024-09-26_04:00:17]: Validate random number: 0.9918216287004739
[2024-09-26_04:00:17]: Evaluating
[2024-09-26_04:00:17]: Start the ref model loss eval loop.
[2024-09-26_04:00:17]: Loading pre-computed cache loss: /app/suno/data/dpo/13b_s8_v9/13b_dpo_s8_v3_cached_loss.json
[2024-09-26_04:00:18]: Check if loaded correctly: 
 train 210472 vs 210472
 val 2126 vs 2126
[2024-09-26_04:00:18]: Validate random number: 0.3328835026423017
[2024-09-26_04:00:18]: training...
[2024-09-26_04:00:21]: Eval -- [162236, 162237, 204484, 204485] Policy positive loss: tensor([2.3264, 3.1864]), torch.float32, Policy negative loss: tensor([2.7230, 3.1710]), torch.float32
[2024-09-26_04:00:21]: Eval -- Reference positive loss: tensor([2.3264, 3.1864]), torch.float32, Reference negative loss: tensor([2.7230, 3.1710]), torch.float32
[2024-09-26_04:00:21]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0.0000e+00, 4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:25]: Eval -- [104894, 104895, 164368, 164369] Policy positive loss: tensor([2.3515, 2.3021]), torch.float32, Policy negative loss: tensor([2.2936, 2.2295]), torch.float32
[2024-09-26_04:00:25]: Eval -- Reference positive loss: tensor([2.3515, 2.3021]), torch.float32, Reference negative loss: tensor([2.2936, 2.2295]), torch.float32
[2024-09-26_04:00:25]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06, 0.0000e+00]), torch.float32, Rejected rewards: tensor([-4.7684e-06,  0.0000e+00]), torch.float32
[2024-09-26_04:00:29]: Eval -- [138138, 138139, 109366, 109367] Policy positive loss: tensor([3.7297, 2.9040]), torch.float32, Policy negative loss: tensor([3.4927, 3.1234]), torch.float32
[2024-09-26_04:00:29]: Eval -- Reference positive loss: tensor([3.7297, 2.9040]), torch.float32, Reference negative loss: tensor([3.4927, 3.1234]), torch.float32
[2024-09-26_04:00:29]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-9.5367e-06,  0.0000e+00]), torch.float32, Rejected rewards: tensor([4.7684e-06, 4.7684e-06]), torch.float32
[2024-09-26_04:00:32]: Eval -- [92878, 92879, 31016, 31017] Policy positive loss: tensor([3.0490, 3.1334]), torch.float32, Policy negative loss: tensor([3.2520, 2.2107]), torch.float32
[2024-09-26_04:00:32]: Eval -- Reference positive loss: tensor([3.0490, 3.1334]), torch.float32, Reference negative loss: tensor([3.2520, 2.2107]), torch.float32
[2024-09-26_04:00:32]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0.0000e+00, 4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:36]: Eval -- [194602, 194603, 164572, 164573] Policy positive loss: tensor([2.3598, 2.7844]), torch.float32, Policy negative loss: tensor([2.4589, 3.0788]), torch.float32
[2024-09-26_04:00:36]: Eval -- Reference positive loss: tensor([2.3598, 2.7844]), torch.float32, Reference negative loss: tensor([2.4589, 3.0788]), torch.float32
[2024-09-26_04:00:36]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06,  4.7684e-06]), torch.float32, Rejected rewards: tensor([9.5367e-06, 0.0000e+00]), torch.float32
[2024-09-26_04:00:39]: Eval -- [180492, 180493, 67332, 67333] Policy positive loss: tensor([2.7999, 3.3879]), torch.float32, Policy negative loss: tensor([3.4162, 3.3379]), torch.float32
[2024-09-26_04:00:39]: Eval -- Reference positive loss: tensor([2.7999, 3.3879]), torch.float32, Reference negative loss: tensor([3.4162, 3.3379]), torch.float32
[2024-09-26_04:00:39]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0.0000e+00, 4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:43]: Eval -- [53120, 53121, 82390, 82391] Policy positive loss: tensor([3.4861, 3.1828]), torch.float32, Policy negative loss: tensor([3.5225, 2.9638]), torch.float32
[2024-09-26_04:00:43]: Eval -- Reference positive loss: tensor([3.4861, 3.1828]), torch.float32, Reference negative loss: tensor([3.5225, 2.9638]), torch.float32
[2024-09-26_04:00:43]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([ 9.5367e-06, -4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:47]: Eval -- [155468, 155469, 69614, 69615] Policy positive loss: tensor([3.2391, 2.1652]), torch.float32, Policy negative loss: tensor([3.2805, 2.3731]), torch.float32
[2024-09-26_04:00:47]: Eval -- Reference positive loss: tensor([3.2391, 2.1652]), torch.float32, Reference negative loss: tensor([3.2805, 2.3731]), torch.float32
[2024-09-26_04:00:47]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:50]: Eval -- [158472, 158473, 36880, 36881] Policy positive loss: tensor([3.1923, 2.4832]), torch.float32, Policy negative loss: tensor([3.1849, 2.7050]), torch.float32
[2024-09-26_04:00:50]: Eval -- Reference positive loss: tensor([3.1923, 2.4832]), torch.float32, Reference negative loss: tensor([3.1849, 2.7050]), torch.float32
[2024-09-26_04:00:50]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([-4.7684e-06,  0.0000e+00]), torch.float32
[2024-09-26_04:00:54]: Eval -- [54868, 54869, 76904, 76905] Policy positive loss: tensor([2.8785, 3.2058]), torch.float32, Policy negative loss: tensor([2.8966, 3.2566]), torch.float32
[2024-09-26_04:00:54]: Eval -- Reference positive loss: tensor([2.8785, 3.2058]), torch.float32, Reference negative loss: tensor([2.8966, 3.2566]), torch.float32
[2024-09-26_04:00:54]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:00:58]: Eval -- [49160, 49161, 199234, 199235] Policy positive loss: tensor([1.8407, 2.8994]), torch.float32, Policy negative loss: tensor([1.6852, 2.8509]), torch.float32
[2024-09-26_04:00:58]: Eval -- Reference positive loss: tensor([1.8407, 2.8994]), torch.float32, Reference negative loss: tensor([1.6852, 2.8509]), torch.float32
[2024-09-26_04:00:58]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([ 0.0000e+00, -4.7684e-06]), torch.float32, Rejected rewards: tensor([-2.3842e-06,  0.0000e+00]), torch.float32
[2024-09-26_04:01:01]: Eval -- [2908, 2909, 20462, 20463] Policy positive loss: tensor([3.2928, 3.2318]), torch.float32, Policy negative loss: tensor([3.3282, 3.2196]), torch.float32
[2024-09-26_04:01:01]: Eval -- Reference positive loss: tensor([3.2928, 3.2318]), torch.float32, Reference negative loss: tensor([3.3282, 3.2196]), torch.float32
[2024-09-26_04:01:01]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06,  0.0000e+00]), torch.float32, Rejected rewards: tensor([4.7684e-06, 0.0000e+00]), torch.float32
[2024-09-26_04:01:05]: Eval -- [660, 661, 916, 917] Policy positive loss: tensor([3.3802, 3.0163]), torch.float32, Policy negative loss: tensor([3.1782, 2.8424]), torch.float32
[2024-09-26_04:01:05]: Eval -- Reference positive loss: tensor([3.3802, 3.0163]), torch.float32, Reference negative loss: tensor([3.1782, 2.8424]), torch.float32
[2024-09-26_04:01:05]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:09]: Eval -- [316, 317, 1320, 1321] Policy positive loss: tensor([2.2713, 3.3021]), torch.float32, Policy negative loss: tensor([2.7124, 2.7714]), torch.float32
[2024-09-26_04:01:09]: Eval -- Reference positive loss: tensor([2.2713, 3.3021]), torch.float32, Reference negative loss: tensor([2.7124, 2.7714]), torch.float32
[2024-09-26_04:01:09]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([ 4.7684e-06, -4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:12]: Eval -- [340, 341, 1612, 1613] Policy positive loss: tensor([3.1959, 2.3842]), torch.float32, Policy negative loss: tensor([2.5581, 2.1346]), torch.float32
[2024-09-26_04:01:12]: Eval -- Reference positive loss: tensor([3.1959, 2.3842]), torch.float32, Reference negative loss: tensor([2.5581, 2.1346]), torch.float32
[2024-09-26_04:01:12]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0.0000e+00, 4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:16]: Eval -- [1786, 1787, 1258, 1259] Policy positive loss: tensor([2.7652, 2.1261]), torch.float32, Policy negative loss: tensor([2.5555, 2.3988]), torch.float32
[2024-09-26_04:01:16]: Eval -- Reference positive loss: tensor([2.7652, 2.1261]), torch.float32, Reference negative loss: tensor([2.5555, 2.3988]), torch.float32
[2024-09-26_04:01:16]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:20]: Eval -- [1304, 1305, 2080, 2081] Policy positive loss: tensor([3.1019, 2.8089]), torch.float32, Policy negative loss: tensor([3.1368, 3.0564]), torch.float32
[2024-09-26_04:01:20]: Eval -- Reference positive loss: tensor([3.1019, 2.8089]), torch.float32, Reference negative loss: tensor([3.1368, 3.0564]), torch.float32
[2024-09-26_04:01:20]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([ 0.0000e+00, -4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:23]: Eval -- [1656, 1657, 28, 29] Policy positive loss: tensor([2.5479, 2.5212]), torch.float32, Policy negative loss: tensor([2.7497, 3.3167]), torch.float32
[2024-09-26_04:01:23]: Eval -- Reference positive loss: tensor([2.5479, 2.5212]), torch.float32, Reference negative loss: tensor([2.7497, 3.3167]), torch.float32
[2024-09-26_04:01:23]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06, 4.7684e-06]), torch.float32, Rejected rewards: tensor([ 4.7684e-06, -4.7684e-06]), torch.float32
[2024-09-26_04:01:27]: Eval -- [1054, 1055, 2060, 2061] Policy positive loss: tensor([3.6326, 2.8285]), torch.float32, Policy negative loss: tensor([3.3763, 3.0630]), torch.float32
[2024-09-26_04:01:27]: Eval -- Reference positive loss: tensor([3.6326, 2.8285]), torch.float32, Reference negative loss: tensor([3.3763, 3.0630]), torch.float32
[2024-09-26_04:01:27]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0., 0.]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:31]: Eval -- [324, 325, 2100, 2101] Policy positive loss: tensor([3.3639, 2.9423]), torch.float32, Policy negative loss: tensor([3.4583, 3.2288]), torch.float32
[2024-09-26_04:01:31]: Eval -- Reference positive loss: tensor([3.3639, 2.9423]), torch.float32, Reference negative loss: tensor([3.4583, 3.2288]), torch.float32
[2024-09-26_04:01:31]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06, 0.0000e+00]), torch.float32, Rejected rewards: tensor([ 4.7684e-06, -4.7684e-06]), torch.float32
[2024-09-26_04:01:34]: Eval -- [940, 941, 296, 297] Policy positive loss: tensor([3.0391, 2.9232]), torch.float32, Policy negative loss: tensor([3.2560, 2.7353]), torch.float32
[2024-09-26_04:01:34]: Eval -- Reference positive loss: tensor([3.0391, 2.9232]), torch.float32, Reference negative loss: tensor([3.2560, 2.7353]), torch.float32
[2024-09-26_04:01:34]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06, -4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:38]: Eval -- [1242, 1243, 1092, 1093] Policy positive loss: tensor([3.3077, 2.6127]), torch.float32, Policy negative loss: tensor([3.3029, 2.6732]), torch.float32
[2024-09-26_04:01:38]: Eval -- Reference positive loss: tensor([3.3077, 2.6127]), torch.float32, Reference negative loss: tensor([3.3029, 2.6732]), torch.float32
[2024-09-26_04:01:38]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([0.0000e+00, 4.7684e-06]), torch.float32, Rejected rewards: tensor([4.7684e-06, 4.7684e-06]), torch.float32
[2024-09-26_04:01:42]: Eval -- [164, 165, 718, 719] Policy positive loss: tensor([2.7371, 3.2401]), torch.float32, Policy negative loss: tensor([3.1363, 3.0167]), torch.float32
[2024-09-26_04:01:42]: Eval -- Reference positive loss: tensor([2.7371, 3.2401]), torch.float32, Reference negative loss: tensor([3.1363, 3.0167]), torch.float32
[2024-09-26_04:01:42]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06, -4.7684e-06]), torch.float32, Rejected rewards: tensor([0., 0.]), torch.float32
[2024-09-26_04:01:45]: Eval -- [1908, 1909, 1728, 1729] Policy positive loss: tensor([2.7776, 2.5543]), torch.float32, Policy negative loss: tensor([3.2888, 2.3572]), torch.float32
[2024-09-26_04:01:45]: Eval -- Reference positive loss: tensor([2.7776, 2.5543]), torch.float32, Reference negative loss: tensor([3.2888, 2.3572]), torch.float32
[2024-09-26_04:01:45]: Eval -- Preference loss: tensor([0.0006, 0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06,  0.0000e+00]), torch.float32, Rejected rewards: tensor([-4.7684e-06,  0.0000e+00]), torch.float32
[2024-09-26_04:01:47]: loss estimation took 88.6 seconds. (100.0% of loop)
[2024-09-26_04:01:47]: step 0: train loss 3.6504, val loss 3.6866
[2024-09-26_04:01:55]: iter 0: avg_loss 0.001, step_time 96159.6ms, mfu 0.0%, throughput 0k tok/s, total time 96s, memory 40.03GiB(50.46%)
[2024-09-26_04:04:54]: iter 25: avg_loss 0.001, step_time 7148.9ms, mfu 139.8%, throughput 156k tok/s, total time 275s, memory 43.34GiB(54.63%)
[2024-09-26_04:07:53]: iter 50: avg_loss 0.001, step_time 7159.1ms, mfu 139.6%, throughput 156k tok/s, total time 454s, memory 43.34GiB(54.63%)
[2024-09-26_04:10:51]: iter 75: avg_loss 0.001, step_time 7152.6ms, mfu 139.7%, throughput 156k tok/s, total time 633s, memory 43.34GiB(54.63%)
[2024-09-26_04:13:50]: iter 100: avg_loss 0.001, step_time 7146.3ms, mfu 139.9%, throughput 156k tok/s, total time 812s, memory 43.34GiB(54.63%)
[2024-09-26_04:16:49]: iter 125: avg_loss 0.001, step_time 7157.0ms, mfu 139.7%, throughput 156k tok/s, total time 991s, memory 43.34GiB(54.63%)
[2024-09-26_04:19:48]: iter 150: avg_loss 0.001, step_time 7151.2ms, mfu 139.8%, throughput 156k tok/s, total time 1170s, memory 43.34GiB(54.63%)
[2024-09-26_04:22:47]: iter 175: avg_loss 0.001, step_time 7151.7ms, mfu 139.8%, throughput 156k tok/s, total time 1348s, memory 43.34GiB(54.63%)
[2024-09-26_04:25:46]: iter 200: avg_loss 0.001, step_time 7150.9ms, mfu 139.8%, throughput 156k tok/s, total time 1527s, memory 43.34GiB(54.63%)
[2024-09-26_04:28:44]: iter 225: avg_loss 0.001, step_time 7158.5ms, mfu 139.6%, throughput 156k tok/s, total time 1706s, memory 43.34GiB(54.63%)
[2024-09-26_04:31:43]: iter 250: avg_loss 0.001, step_time 7155.8ms, mfu 139.7%, throughput 156k tok/s, total time 1885s, memory 43.34GiB(54.63%)
[2024-09-26_04:34:42]: iter 275: avg_loss 0.001, step_time 7160.7ms, mfu 139.6%, throughput 156k tok/s, total time 2064s, memory 43.34GiB(54.63%)
[2024-09-26_04:37:41]: iter 300: avg_loss 0.001, step_time 7163.6ms, mfu 139.5%, throughput 156k tok/s, total time 2242s, memory 43.34GiB(54.63%)
[2024-09-26_04:40:40]: iter 325: avg_loss 0.001, step_time 7163.1ms, mfu 139.5%, throughput 156k tok/s, total time 2422s, memory 43.34GiB(54.63%)
[2024-09-26_04:43:39]: iter 350: avg_loss 0.001, step_time 7155.1ms, mfu 139.7%, throughput 156k tok/s, total time 2601s, memory 43.34GiB(54.63%)
[2024-09-26_04:46:38]: iter 375: avg_loss 0.001, step_time 7149.5ms, mfu 139.8%, throughput 156k tok/s, total time 2779s, memory 43.34GiB(54.63%)
[2024-09-26_04:49:37]: iter 400: avg_loss 0.001, step_time 7152.4ms, mfu 139.7%, throughput 156k tok/s, total time 2958s, memory 43.34GiB(54.63%)
[2024-09-26_04:52:36]: iter 425: avg_loss 0.001, step_time 7144.9ms, mfu 139.9%, throughput 156k tok/s, total time 3137s, memory 43.34GiB(54.63%)
[2024-09-26_04:55:34]: iter 450: avg_loss 0.001, step_time 7161.7ms, mfu 139.6%, throughput 156k tok/s, total time 3316s, memory 43.34GiB(54.63%)
[2024-09-26_04:58:33]: iter 475: avg_loss 0.001, step_time 7143.7ms, mfu 139.9%, throughput 156k tok/s, total time 3495s, memory 43.34GiB(54.63%)
[2024-09-26_05:01:32]: iter 500: avg_loss 0.001, step_time 7152.6ms, mfu 139.7%, throughput 156k tok/s, total time 3674s, memory 43.34GiB(54.63%)
[2024-09-26_05:04:31]: iter 525: avg_loss 0.001, step_time 7144.4ms, mfu 139.9%, throughput 156k tok/s, total time 3852s, memory 43.34GiB(54.63%)
[2024-09-26_05:07:30]: iter 550: avg_loss 0.001, step_time 7134.6ms, mfu 140.1%, throughput 156k tok/s, total time 4031s, memory 43.34GiB(54.63%)
[2024-09-26_05:10:29]: iter 575: avg_loss 0.001, step_time 7155.1ms, mfu 139.7%, throughput 156k tok/s, total time 4210s, memory 43.34GiB(54.63%)
[2024-09-26_05:13:27]: iter 600: avg_loss 0.001, step_time 7149.5ms, mfu 139.8%, throughput 156k tok/s, total time 4389s, memory 43.34GiB(54.63%)
[2024-09-26_05:16:27]: iter 625: avg_loss 0.001, step_time 7154.4ms, mfu 139.7%, throughput 156k tok/s, total time 4568s, memory 43.34GiB(54.63%)
[2024-09-26_05:19:26]: iter 650: avg_loss 0.001, step_time 7151.4ms, mfu 139.8%, throughput 156k tok/s, total time 4747s, memory 43.34GiB(54.63%)
[2024-09-26_05:22:25]: iter 675: avg_loss 0.001, step_time 7143.2ms, mfu 139.9%, throughput 156k tok/s, total time 4926s, memory 43.34GiB(54.63%)
[2024-09-26_05:25:23]: iter 700: avg_loss 0.001, step_time 7163.5ms, mfu 139.5%, throughput 156k tok/s, total time 5105s, memory 43.34GiB(54.63%)
[2024-09-26_05:28:22]: iter 725: avg_loss 0.001, step_time 7148.8ms, mfu 139.8%, throughput 156k tok/s, total time 5284s, memory 43.34GiB(54.63%)
[2024-09-26_05:31:21]: iter 750: avg_loss 0.001, step_time 7158.8ms, mfu 139.6%, throughput 156k tok/s, total time 5463s, memory 43.34GiB(54.63%)
[2024-09-26_05:34:20]: iter 775: avg_loss 0.001, step_time 7138.1ms, mfu 140.0%, throughput 156k tok/s, total time 5641s, memory 43.34GiB(54.63%)
[2024-09-26_05:37:19]: iter 800: avg_loss 0.001, step_time 7157.0ms, mfu 139.7%, throughput 156k tok/s, total time 5820s, memory 43.34GiB(54.63%)
[2024-09-26_05:40:18]: iter 825: avg_loss 0.001, step_time 7153.8ms, mfu 139.7%, throughput 156k tok/s, total time 5999s, memory 43.34GiB(54.63%)
[2024-09-26_05:43:16]: iter 850: avg_loss 0.001, step_time 7128.9ms, mfu 140.2%, throughput 156k tok/s, total time 6178s, memory 43.34GiB(54.63%)
[2024-09-26_05:46:15]: iter 875: avg_loss 0.001, step_time 7149.2ms, mfu 139.8%, throughput 156k tok/s, total time 6357s, memory 43.34GiB(54.63%)
[2024-09-26_05:49:14]: iter 900: avg_loss 0.001, step_time 7152.0ms, mfu 139.8%, throughput 156k tok/s, total time 6536s, memory 43.34GiB(54.63%)
[2024-09-26_05:52:14]: iter 925: avg_loss 0.001, step_time 7148.0ms, mfu 139.8%, throughput 156k tok/s, total time 6715s, memory 43.34GiB(54.63%)
[2024-09-26_05:55:12]: iter 950: avg_loss 0.001, step_time 7137.9ms, mfu 140.0%, throughput 156k tok/s, total time 6894s, memory 43.34GiB(54.63%)
[2024-09-26_05:58:11]: iter 975: avg_loss 0.001, step_time 7155.4ms, mfu 139.7%, throughput 156k tok/s, total time 7073s, memory 43.34GiB(54.63%)
[2024-09-26_06:01:10]: iter 1000: avg_loss 0.000, step_time 7154.9ms, mfu 139.7%, throughput 156k tok/s, total time 7252s, memory 43.34GiB(54.63%)
[2024-09-26_06:04:09]: iter 1025: avg_loss 0.001, step_time 7157.0ms, mfu 139.7%, throughput 156k tok/s, total time 7430s, memory 43.34GiB(54.63%)
[2024-09-26_06:07:08]: iter 1050: avg_loss 0.001, step_time 7167.1ms, mfu 139.5%, throughput 155k tok/s, total time 7609s, memory 43.34GiB(54.63%)
[2024-09-26_06:10:07]: iter 1075: avg_loss 0.001, step_time 7164.1ms, mfu 139.5%, throughput 156k tok/s, total time 7788s, memory 43.34GiB(54.63%)
[2024-09-26_06:13:06]: iter 1100: avg_loss 0.001, step_time 7143.4ms, mfu 139.9%, throughput 156k tok/s, total time 7967s, memory 43.34GiB(54.63%)
[2024-09-26_06:16:04]: iter 1125: avg_loss 0.001, step_time 7142.4ms, mfu 139.9%, throughput 156k tok/s, total time 8146s, memory 43.34GiB(54.63%)
[2024-09-26_06:19:03]: iter 1150: avg_loss 0.001, step_time 7170.3ms, mfu 139.4%, throughput 155k tok/s, total time 8325s, memory 43.34GiB(54.63%)
[2024-09-26_06:22:02]: iter 1175: avg_loss 0.001, step_time 7155.1ms, mfu 139.7%, throughput 156k tok/s, total time 8503s, memory 43.34GiB(54.63%)
[2024-09-26_06:25:01]: iter 1200: avg_loss 0.001, step_time 7153.2ms, mfu 139.7%, throughput 156k tok/s, total time 8682s, memory 43.34GiB(54.63%)
[2024-09-26_06:28:00]: iter 1225: avg_loss 0.001, step_time 7163.5ms, mfu 139.5%, throughput 156k tok/s, total time 8862s, memory 43.34GiB(54.63%)
[2024-09-26_06:30:59]: iter 1250: avg_loss 0.001, step_time 7148.8ms, mfu 139.8%, throughput 156k tok/s, total time 9041s, memory 43.34GiB(54.63%)
[2024-09-26_06:33:58]: iter 1275: avg_loss 0.001, step_time 7158.5ms, mfu 139.6%, throughput 156k tok/s, total time 9220s, memory 43.34GiB(54.63%)
[2024-09-26_06:36:57]: iter 1300: avg_loss 0.001, step_time 7161.5ms, mfu 139.6%, throughput 156k tok/s, total time 9398s, memory 43.34GiB(54.63%)
[2024-09-26_06:39:56]: iter 1325: avg_loss 0.001, step_time 7152.4ms, mfu 139.7%, throughput 156k tok/s, total time 9577s, memory 43.34GiB(54.63%)
[2024-09-26_06:42:55]: iter 1350: avg_loss 0.001, step_time 7147.1ms, mfu 139.8%, throughput 156k tok/s, total time 9756s, memory 43.34GiB(54.63%)
[2024-09-26_06:45:54]: iter 1375: avg_loss 0.001, step_time 7144.4ms, mfu 139.9%, throughput 156k tok/s, total time 9935s, memory 43.34GiB(54.63%)
[2024-09-26_06:48:52]: iter 1400: avg_loss 0.001, step_time 7145.5ms, mfu 139.9%, throughput 156k tok/s, total time 10114s, memory 43.34GiB(54.63%)
[2024-09-26_06:51:51]: iter 1425: avg_loss 0.001, step_time 7162.2ms, mfu 139.6%, throughput 156k tok/s, total time 10293s, memory 43.34GiB(54.63%)
[2024-09-26_06:54:50]: iter 1450: avg_loss 0.001, step_time 7158.9ms, mfu 139.6%, throughput 156k tok/s, total time 10472s, memory 43.34GiB(54.63%)
[2024-09-26_06:57:49]: iter 1475: avg_loss 0.001, step_time 7149.7ms, mfu 139.8%, throughput 156k tok/s, total time 10650s, memory 43.34GiB(54.63%)
[2024-09-26_07:00:48]: iter 1500: avg_loss 0.001, step_time 7134.0ms, mfu 140.1%, throughput 156k tok/s, total time 10829s, memory 43.34GiB(54.63%)
[2024-09-26_07:03:47]: iter 1525: avg_loss 0.001, step_time 7147.6ms, mfu 139.8%, throughput 156k tok/s, total time 11009s, memory 43.34GiB(54.63%)
[2024-09-26_07:06:46]: iter 1550: avg_loss 0.001, step_time 7141.0ms, mfu 140.0%, throughput 156k tok/s, total time 11187s, memory 43.34GiB(54.63%)
[2024-09-26_07:09:45]: iter 1575: avg_loss 0.001, step_time 7149.4ms, mfu 139.8%, throughput 156k tok/s, total time 11366s, memory 43.34GiB(54.63%)
[2024-09-26_07:12:31]: Eval -- [79686, 79687, 12268, 12269] Policy positive loss: tensor([3.3343, 2.6733]), torch.float32, Policy negative loss: tensor([3.7896, 2.4423]), torch.float32
[2024-09-26_07:12:31]: Eval -- Reference positive loss: tensor([3.3144, 2.6654]), torch.float32, Reference negative loss: tensor([3.7540, 2.4350]), torch.float32
[2024-09-26_07:12:31]: Eval -- Preference loss: tensor([8.8922e-05, 6.5819e-04]), torch.float32, Chosen rewards: tensor([-0.3988, -0.1584]), torch.float32, Rejected rewards: tensor([-0.7102, -0.1453]), torch.float32
[2024-09-26_07:12:35]: Eval -- [32278, 32279, 3774, 3775] Policy positive loss: tensor([3.1879, 2.7963]), torch.float32, Policy negative loss: tensor([3.4049, 3.3445]), torch.float32
[2024-09-26_07:12:35]: Eval -- Reference positive loss: tensor([3.1556, 2.7691]), torch.float32, Reference negative loss: tensor([3.3827, 3.3167]), torch.float32
[2024-09-26_07:12:35]: Eval -- Preference loss: tensor([0.0012, 0.0006]), torch.float32, Chosen rewards: tensor([-0.6447, -0.5431]), torch.float32, Rejected rewards: tensor([-0.4447, -0.5573]), torch.float32
[2024-09-26_07:12:38]: Eval -- [35222, 35223, 52820, 52821] Policy positive loss: tensor([2.8754, 2.7794]), torch.float32, Policy negative loss: tensor([3.0726, 2.6797]), torch.float32
[2024-09-26_07:12:38]: Eval -- Reference positive loss: tensor([2.8610, 2.7690]), torch.float32, Reference negative loss: tensor([3.0435, 2.6627]), torch.float32
[2024-09-26_07:12:38]: Eval -- Preference loss: tensor([0.0001, 0.0003]), torch.float32, Chosen rewards: tensor([-0.2874, -0.2076]), torch.float32, Rejected rewards: tensor([-0.5802, -0.3398]), torch.float32
[2024-09-26_07:12:42]: Eval -- [112166, 112167, 172506, 172507] Policy positive loss: tensor([2.3319, 2.6225]), torch.float32, Policy negative loss: tensor([2.5108, 2.9720]), torch.float32
[2024-09-26_07:12:42]: Eval -- Reference positive loss: tensor([2.3075, 2.6042]), torch.float32, Reference negative loss: tensor([2.4884, 2.9424]), torch.float32
[2024-09-26_07:12:42]: Eval -- Preference loss: tensor([0.0007, 0.0002]), torch.float32, Chosen rewards: tensor([-0.4884, -0.3660]), torch.float32, Rejected rewards: tensor([-0.4480, -0.5922]), torch.float32
[2024-09-26_07:12:46]: Eval -- [108892, 108893, 144530, 144531] Policy positive loss: tensor([3.0518, 3.2312]), torch.float32, Policy negative loss: tensor([2.7906, 3.0833]), torch.float32
[2024-09-26_07:12:46]: Eval -- Reference positive loss: tensor([3.0434, 3.2095]), torch.float32, Reference negative loss: tensor([2.7678, 3.0665]), torch.float32
[2024-09-26_07:12:46]: Eval -- Preference loss: tensor([0.0001, 0.0009]), torch.float32, Chosen rewards: tensor([-0.1683, -0.4338]), torch.float32, Rejected rewards: tensor([-0.4567, -0.3354]), torch.float32
[2024-09-26_07:12:50]: Eval -- [88086, 88087, 33916, 33917] Policy positive loss: tensor([3.0563, 3.1720]), torch.float32, Policy negative loss: tensor([2.0320, 3.1725]), torch.float32
[2024-09-26_07:12:50]: Eval -- Reference positive loss: tensor([3.0461, 3.1550]), torch.float32, Reference negative loss: tensor([2.0132, 3.1535]), torch.float32
[2024-09-26_07:12:50]: Eval -- Preference loss: tensor([0.0003, 0.0005]), torch.float32, Chosen rewards: tensor([-0.2032, -0.3398]), torch.float32, Rejected rewards: tensor([-0.3753, -0.3792]), torch.float32
[2024-09-26_07:12:53]: Eval -- [192786, 192787, 176874, 176875] Policy positive loss: tensor([3.1489, 2.6342]), torch.float32, Policy negative loss: tensor([3.2538, 2.3100]), torch.float32
[2024-09-26_07:12:53]: Eval -- Reference positive loss: tensor([3.1401, 2.6166]), torch.float32, Reference negative loss: tensor([3.2320, 2.2898]), torch.float32
[2024-09-26_07:12:53]: Eval -- Preference loss: tensor([0.0001, 0.0005]), torch.float32, Chosen rewards: tensor([-0.1759, -0.3518]), torch.float32, Rejected rewards: tensor([-0.4371, -0.4043]), torch.float32
[2024-09-26_07:12:57]: Eval -- [11202, 11203, 129308, 129309] Policy positive loss: tensor([3.2319, 3.8755]), torch.float32, Policy negative loss: tensor([3.0769, 3.4912]), torch.float32
[2024-09-26_07:12:57]: Eval -- Reference positive loss: tensor([3.2066, 3.8395]), torch.float32, Reference negative loss: tensor([3.0537, 3.4724]), torch.float32
[2024-09-26_07:12:57]: Eval -- Preference loss: tensor([0.0007, 0.0018]), torch.float32, Chosen rewards: tensor([-0.5059, -0.7212]), torch.float32, Rejected rewards: tensor([-0.4644, -0.3770]), torch.float32
[2024-09-26_07:13:01]: Eval -- [165114, 165115, 151232, 151233] Policy positive loss: tensor([3.0956, 2.4944]), torch.float32, Policy negative loss: tensor([3.2109, 3.1651]), torch.float32
[2024-09-26_07:13:01]: Eval -- Reference positive loss: tensor([3.0658, 2.4783]), torch.float32, Reference negative loss: tensor([3.1770, 3.1469]), torch.float32
[2024-09-26_07:13:01]: Eval -- Preference loss: tensor([0.0004, 0.0005]), torch.float32, Chosen rewards: tensor([-0.5944, -0.3234]), torch.float32, Rejected rewards: tensor([-0.6785, -0.3639]), torch.float32
[2024-09-26_07:13:04]: Eval -- [200768, 200769, 43092, 43093] Policy positive loss: tensor([2.3678, 3.5455]), torch.float32, Policy negative loss: tensor([2.9427, 3.4637]), torch.float32
[2024-09-26_07:13:04]: Eval -- Reference positive loss: tensor([2.3530, 3.5159]), torch.float32, Reference negative loss: tensor([2.9175, 3.4229]), torch.float32
[2024-09-26_07:13:04]: Eval -- Preference loss: tensor([0.0002, 0.0002]), torch.float32, Chosen rewards: tensor([-0.2961, -0.5922]), torch.float32, Rejected rewards: tensor([-0.5048, -0.8162]), torch.float32
[2024-09-26_07:13:08]: Eval -- [108056, 108057, 155098, 155099] Policy positive loss: tensor([3.5293, 3.1277]), torch.float32, Policy negative loss: tensor([2.6861, 3.7963]), torch.float32
[2024-09-26_07:13:08]: Eval -- Reference positive loss: tensor([3.5152, 3.0987]), torch.float32, Reference negative loss: tensor([2.6732, 3.7564]), torch.float32
[2024-09-26_07:13:08]: Eval -- Preference loss: tensor([0.0007, 0.0002]), torch.float32, Chosen rewards: tensor([-0.2830, -0.5791]), torch.float32, Rejected rewards: tensor([-0.2590, -0.7976]), torch.float32
[2024-09-26_07:13:12]: Eval -- [163216, 163217, 198280, 198281] Policy positive loss: tensor([3.0876, 3.2772]), torch.float32, Policy negative loss: tensor([3.4292, 3.3386]), torch.float32
[2024-09-26_07:13:12]: Eval -- Reference positive loss: tensor([3.0719, 3.2615]), torch.float32, Reference negative loss: tensor([3.4041, 3.3197]), torch.float32
[2024-09-26_07:13:12]: Eval -- Preference loss: tensor([0.0002, 0.0005]), torch.float32, Chosen rewards: tensor([-0.3136, -0.3125]), torch.float32, Rejected rewards: tensor([-0.5015, -0.3770]), torch.float32
[2024-09-26_07:13:15]: Eval -- [514, 515, 1542, 1543] Policy positive loss: tensor([3.3834, 2.2313]), torch.float32, Policy negative loss: tensor([3.4960, 2.3687]), torch.float32
[2024-09-26_07:13:15]: Eval -- Reference positive loss: tensor([3.3592, 2.2171]), torch.float32, Reference negative loss: tensor([3.4767, 2.3590]), torch.float32
[2024-09-26_07:13:15]: Eval -- Preference loss: tensor([0.0009, 0.0009]), torch.float32, Chosen rewards: tensor([-0.4841, -0.2852]), torch.float32, Rejected rewards: tensor([-0.3857, -0.1934]), torch.float32
[2024-09-26_07:13:19]: Eval -- [668, 669, 1916, 1917] Policy positive loss: tensor([2.6758, 3.5597]), torch.float32, Policy negative loss: tensor([2.4148, 3.2593]), torch.float32
[2024-09-26_07:13:19]: Eval -- Reference positive loss: tensor([2.6576, 3.5367]), torch.float32, Reference negative loss: tensor([2.3973, 3.2356]), torch.float32
[2024-09-26_07:13:19]: Eval -- Preference loss: tensor([0.0007, 0.0006]), torch.float32, Chosen rewards: tensor([-0.3649, -0.4600]), torch.float32, Rejected rewards: tensor([-0.3507, -0.4753]), torch.float32
[2024-09-26_07:13:23]: Eval -- [132, 133, 2024, 2025] Policy positive loss: tensor([2.7430, 2.8628]), torch.float32, Policy negative loss: tensor([2.5343, 2.7958]), torch.float32
[2024-09-26_07:13:23]: Eval -- Reference positive loss: tensor([2.7172, 2.8547]), torch.float32, Reference negative loss: tensor([2.5091, 2.7859]), torch.float32
[2024-09-26_07:13:23]: Eval -- Preference loss: tensor([0.0007, 0.0005]), torch.float32, Chosen rewards: tensor([-0.5168, -0.1617]), torch.float32, Rejected rewards: tensor([-0.5037, -0.1978]), torch.float32
[2024-09-26_07:13:26]: Eval -- [532, 533, 1376, 1377] Policy positive loss: tensor([2.8628, 3.1939]), torch.float32, Policy negative loss: tensor([3.3075, 2.9834]), torch.float32
[2024-09-26_07:13:26]: Eval -- Reference positive loss: tensor([2.8363, 3.1721]), torch.float32, Reference negative loss: tensor([3.2801, 2.9525]), torch.float32
[2024-09-26_07:13:26]: Eval -- Preference loss: tensor([0.0006, 0.0003]), torch.float32, Chosen rewards: tensor([-0.5299, -0.4360]), torch.float32, Rejected rewards: tensor([-0.5474, -0.6174]), torch.float32
[2024-09-26_07:13:30]: Eval -- [704, 705, 956, 957] Policy positive loss: tensor([3.4029, 2.5836]), torch.float32, Policy negative loss: tensor([3.3531, 2.5953]), torch.float32
[2024-09-26_07:13:30]: Eval -- Reference positive loss: tensor([3.3841, 2.5628]), torch.float32, Reference negative loss: tensor([3.3399, 2.5694]), torch.float32
[2024-09-26_07:13:30]: Eval -- Preference loss: tensor([0.0009, 0.0004]), torch.float32, Chosen rewards: tensor([-0.3748, -0.4152]), torch.float32, Rejected rewards: tensor([-0.2633, -0.5168]), torch.float32
[2024-09-26_07:13:34]: Eval -- [14, 15, 1024, 1025] Policy positive loss: tensor([2.6412, 3.2131]), torch.float32, Policy negative loss: tensor([2.7878, 3.2594]), torch.float32
[2024-09-26_07:13:34]: Eval -- Reference positive loss: tensor([2.6272, 3.1954]), torch.float32, Reference negative loss: tensor([2.7741, 3.2438]), torch.float32
[2024-09-26_07:13:34]: Eval -- Preference loss: tensor([0.0006, 0.0007]), torch.float32, Chosen rewards: tensor([-0.2786, -0.3529]), torch.float32, Rejected rewards: tensor([-0.2742, -0.3125]), torch.float32
[2024-09-26_07:13:37]: Eval -- [1962, 1963, 474, 475] Policy positive loss: tensor([2.9747, 2.4624]), torch.float32, Policy negative loss: tensor([3.1153, 3.7108]), torch.float32
[2024-09-26_07:13:37]: Eval -- Reference positive loss: tensor([2.9493, 2.4361]), torch.float32, Reference negative loss: tensor([3.0851, 3.6802]), torch.float32
[2024-09-26_07:13:37]: Eval -- Preference loss: tensor([0.0004, 0.0004]), torch.float32, Chosen rewards: tensor([-0.5070, -0.5256]), torch.float32, Rejected rewards: tensor([-0.6042, -0.6119]), torch.float32
[2024-09-26_07:13:41]: Eval -- [286, 287, 1828, 1829] Policy positive loss: tensor([3.0252, 3.0909]), torch.float32, Policy negative loss: tensor([2.9197, 2.9112]), torch.float32
[2024-09-26_07:13:41]: Eval -- Reference positive loss: tensor([3.0074, 3.0603]), torch.float32, Reference negative loss: tensor([2.8832, 2.8856]), torch.float32
[2024-09-26_07:13:41]: Eval -- Preference loss: tensor([3.8523e-05, 8.9829e-04]), torch.float32, Chosen rewards: tensor([-0.3551, -0.6119]), torch.float32, Rejected rewards: tensor([-0.7310, -0.5125]), torch.float32
[2024-09-26_07:13:45]: Eval -- [1382, 1383, 576, 577] Policy positive loss: tensor([3.0688, 2.9803]), torch.float32, Policy negative loss: tensor([3.1974, 2.6388]), torch.float32
[2024-09-26_07:13:45]: Eval -- Reference positive loss: tensor([3.0442, 2.9639]), torch.float32, Reference negative loss: tensor([3.1843, 2.6253]), torch.float32
[2024-09-26_07:13:45]: Eval -- Preference loss: tensor([0.0013, 0.0008]), torch.float32, Chosen rewards: tensor([-0.4928, -0.3267]), torch.float32, Rejected rewards: tensor([-0.2633, -0.2710]), torch.float32
[2024-09-26_07:13:48]: Eval -- [1792, 1793, 1366, 1367] Policy positive loss: tensor([3.2597, 3.3611]), torch.float32, Policy negative loss: tensor([3.2666, 3.6665]), torch.float32
[2024-09-26_07:13:48]: Eval -- Reference positive loss: tensor([3.2589, 3.3345]), torch.float32, Reference negative loss: tensor([3.2543, 3.6360]), torch.float32
[2024-09-26_07:13:48]: Eval -- Preference loss: tensor([0.0002, 0.0004]), torch.float32, Chosen rewards: tensor([-0.0153, -0.5321]), torch.float32, Rejected rewards: tensor([-0.2469, -0.6108]), torch.float32
[2024-09-26_07:13:52]: Eval -- [1348, 1349, 1186, 1187] Policy positive loss: tensor([2.9386, 3.7401]), torch.float32, Policy negative loss: tensor([3.2186, 3.5944]), torch.float32
[2024-09-26_07:13:52]: Eval -- Reference positive loss: tensor([2.9161, 3.7147]), torch.float32, Reference negative loss: tensor([3.1933, 3.5694]), torch.float32
[2024-09-26_07:13:52]: Eval -- Preference loss: tensor([0.0005, 0.0006]), torch.float32, Chosen rewards: tensor([-0.4491, -0.5081]), torch.float32, Rejected rewards: tensor([-0.5048, -0.4983]), torch.float32
[2024-09-26_07:13:56]: Eval -- [1716, 1717, 682, 683] Policy positive loss: tensor([2.7437, 3.3404]), torch.float32, Policy negative loss: tensor([3.4881, 3.3950]), torch.float32
[2024-09-26_07:13:56]: Eval -- Reference positive loss: tensor([2.7253, 3.3272]), torch.float32, Reference negative loss: tensor([3.4542, 3.3773]), torch.float32
[2024-09-26_07:13:56]: Eval -- Preference loss: tensor([8.8917e-05, 4.2106e-04]), torch.float32, Chosen rewards: tensor([-0.3682, -0.2633]), torch.float32, Rejected rewards: tensor([-0.6796, -0.3529]), torch.float32
[2024-09-26_07:13:58]: loss estimation took 88.2 seconds. (0.8% of loop)
[2024-09-26_07:13:58]: step 1599: train loss 3.7216, val loss 3.7123
[2024-09-26_07:15:25]: saving checkpoint to /app/suno/checkpoints/2024-09-26_03-57-23
[2024-09-26_07:16:12]: saving took 134.5 seconds. (1.2% of loop)
[2024-09-26_07:16:24]: iter 1599: avg_loss 0.000, step_time 235008.0ms, mfu 4.3%, throughput 5k tok/s, total time 11766s, memory 43.83GiB(55.25%)
[2024-09-26_07:16:24]: done.
[2024-09-26_07:16:24]: all seen idxs: 204928