working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT working from /home/tony/Work/glockenspiel/sunoGPT Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/v0_1 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 1e-06 Overriding: min_lr = 1e-07 Overriding: grad_clip = 0.1 Overriding: warmup_iters = 200 Overriding: max_iters = 2000 Overriding: eval_interval = 500 Overriding: eval_iters = 25 Overriding: step_save_iters = 500 Overriding: step_save_infer = False Overriding: block_size = 4288 Overriding: t_text = 1152 Overriding: t_memmap = 3008 Overriding: t_audio = 3136 Overriding: coarse_vocab_size = 2112 Overriding: coarse_codebook_size = 2048 Overriding: coarse_pad_token = 2048 Overriding: coarse_infer_token = 2049 Overriding: n_layer = 32 Overriding: n_head = 32 Overriding: d_head = 128 Overriding: n_kv_head = 4 Overriding: gradient_accumulation_steps = 1 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-01-20_22-01-25/last_ckpt.pt Overriding: preload_strict = False Overriding: wandb_log = True Overriding: wandb_project = chirp-v3-dpo Overriding: wandb_run_name = v3_dpo_t0_1_exp1_2k [2024-02-02_15:26:00]: ddp init, rank 2, local_rank 2 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 3, local_rank 3 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 4, local_rank 4 [2024-02-02_15:26:00]: ddp init, rank 6, local_rank 6 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: ddp init, rank 36, local_rank 4 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 34, local_rank 2 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 38, local_rank 6 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 37, local_rank 5 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 32, local_rank 0 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 35, local_rank 3 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 39, local_rank 7 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: ddp init, rank 33, local_rank 1 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: ddp init, rank 5, local_rank 5 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 7, local_rank 7 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 0, local_rank 0 loss discounts for codebooks: [2024-02-02_15:26:00]: ddp init, rank 1, local_rank 1 [0.377 0.094 0.087 0.079 0.071 0.063 0.056 0.048 0.04 0.033 0.025 0.017 0.009] [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: 5,414 lines of val loaded. [2024-02-02_15:26:00]: ddp init, rank 47, local_rank 7 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 43, local_rank 3 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:00]: ddp init, rank 42, local_rank 2 [2024-02-02_15:26:00]: loading data... [2024-02-02_15:26:01]: ddp init, rank 41, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 45, local_rank 5 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 44, local_rank 4 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 40, local_rank 0 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 46, local_rank 6 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 10, local_rank 2 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 15, local_rank 7 [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 21, local_rank 5 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 13, local_rank 5 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 61, local_rank 5 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 58, local_rank 2 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 9, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 8, local_rank 0 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 14, local_rank 6 [2024-02-02_15:26:01]: ddp init, rank 12, local_rank 4 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 25, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 56, local_rank 0 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 11, local_rank 3 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 51, local_rank 3 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 23, local_rank 7 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 63, local_rank 7 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 57, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 24, local_rank 0 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 59, local_rank 3 [2024-02-02_15:26:01]: ddp init, rank 17, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 16, local_rank 0 [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 62, local_rank 6 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 60, local_rank 4 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 31, local_rank 7 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 20, local_rank 4 [2024-02-02_15:26:01]: ddp init, rank 27, local_rank 3 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 28, local_rank 4 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 22, local_rank 6 [2024-02-02_15:26:01]: ddp init, rank 19, local_rank 3 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 18, local_rank 2 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 29, local_rank 5 [2024-02-02_15:26:01]: ddp init, rank 26, local_rank 2 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 48, local_rank 0 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 53, local_rank 5 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 49, local_rank 1 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 30, local_rank 6 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 52, local_rank 4 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 54, local_rank 6 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: ddp init, rank 55, local_rank 7 [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: ddp init, rank 50, local_rank 2 [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: loading data... [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:01]: 5,414 lines of val loaded. [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:03]: 265,210 lines of train loaded. [2024-02-02_15:26:03]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded.[2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:04]: 265,210 lines of train loaded. [2024-02-02_15:26:04]: Initializing a new model from scratch [2024-02-02_15:26:30]: logging checkpoint here: /app/suno/checkpoints/2024-02-02_15-26-29 [2024-02-02_15:26:30]: loading data... [2024-02-02_15:26:30]: 5,414 lines of val loaded. [2024-02-02_15:26:32]: 265,210 lines of train loaded. [2024-02-02_15:26:32]: train data weights: 50.0% perference_0 50.0% perference_1 [2024-02-02_15:26:32]: Initializing a new model from scratch number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint [2024-02-02_15:28:24]: not compiling model. [2024-02-02_15:28:24]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:24]: preloading checkpoint [2024-02-02_15:28:24]: not compiling model. [2024-02-02_15:28:24]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:25]: preloading checkpoint [2024-02-02_15:28:25]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:26]: not compiling model. [2024-02-02_15:28:26]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:26]: preloading checkpoint [2024-02-02_15:28:26]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. number of parameters: 5869M number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:27]: preloading checkpoint [2024-02-02_15:28:27]: not compiling model. [2024-02-02_15:28:27]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:28]: preloading checkpoint [2024-02-02_15:28:28]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:30]: preloading checkpoint [2024-02-02_15:28:30]: not compiling model. number of parameters: 5869M [2024-02-02_15:28:54]: preloading checkpoint [2024-02-02_15:28:54]: not compiling model. [2024-02-02_15:28:54]: wrapping model in FSDP .... [2024-02-02_15:28:54]: loading state dict into model ... on rank 0 NCCL version 2.18.5+cuda11.8 applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,860,096 parameters num non-decayed parameter tensors: 68, with 4,096 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,860,096 parameters num non-decayed parameter tensors: 68, with 4,096 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters applying fsdp activation checkpointing... using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters applying fsdp activation checkpointing... applying fsdp activation checkpointing... using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters applying fsdp activation checkpointing... num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,733,120 parameters num non-decayed parameter tensors: 68, with 131,072 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,733,120 parameters num non-decayed parameter tensors: 68, with 131,072 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing...applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,856,000 parameters num non-decayed parameter tensors: 68, with 8,192 parameters using fused Optimizer: False using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False using fused Optimizer: False applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False applying fsdp activation checkpointing... applying fsdp activation checkpointing... num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False num decayed parameter tensors: 220, with 93,864,192 parameters num non-decayed parameter tensors: 68, with 0 parameters using fused Optimizer: False [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training...[2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:20]: training... [2024-02-02_15:29:45]: loss estimation took 24.4 seconds. (100.0% of loop) [2024-02-02_15:29:45]: step 0: train loss 2.3708, val loss 2.3371 [2024-02-02_15:29:51]: iter 0: avg_loss 0.010, step_time 30921.6ms, mfu 0.0%, throughput 0k tok/s, total time 31s [2024-02-02_15:31:29]: iter 25: avg_loss 0.010, step_time 3836.2ms, mfu 30.1%, throughput 143k tok/s, total time 128s [2024-02-02_15:33:06]: iter 50: avg_loss 0.011, step_time 3795.7ms, mfu 30.4%, throughput 145k tok/s, total time 226s [2024-02-02_15:34:44]: iter 75: avg_loss 0.010, step_time 3916.2ms, mfu 29.5%, throughput 140k tok/s, total time 323s [2024-02-02_15:36:22]: iter 100: avg_loss 0.010, step_time 3966.0ms, mfu 29.1%, throughput 138k tok/s, total time 421s [2024-02-02_15:38:00]: iter 125: avg_loss 0.010, step_time 3887.8ms, mfu 29.7%, throughput 141k tok/s, total time 519s [2024-02-02_15:39:37]: iter 150: avg_loss 0.010, step_time 3865.3ms, mfu 29.8%, throughput 142k tok/s, total time 617s [2024-02-02_15:41:15]: iter 175: avg_loss 0.010, step_time 3849.9ms, mfu 30.0%, throughput 143k tok/s, total time 715s [2024-02-02_15:42:54]: iter 200: avg_loss 0.009, step_time 3926.2ms, mfu 29.4%, throughput 140k tok/s, total time 813s [2024-02-02_15:44:33]: iter 225: avg_loss 0.009, step_time 3878.1ms, mfu 29.7%, throughput 142k tok/s, total time 912s [2024-02-02_15:46:11]: iter 250: avg_loss 0.010, step_time 4051.0ms, mfu 28.5%, throughput 135k tok/s, total time 1010s [2024-02-02_15:47:48]: iter 275: avg_loss 0.008, step_time 3850.8ms, mfu 30.0%, throughput 143k tok/s, total time 1108s [2024-02-02_15:49:27]: iter 300: avg_loss 0.010, step_time 3861.0ms, mfu 29.9%, throughput 142k tok/s, total time 1206s [2024-02-02_15:51:05]: iter 325: avg_loss 0.009, step_time 3830.1ms, mfu 30.1%, throughput 143k tok/s, total time 1304s [2024-02-02_15:52:43]: iter 350: avg_loss 0.008, step_time 3901.4ms, mfu 29.6%, throughput 141k tok/s, total time 1402s [2024-02-02_15:54:22]: iter 375: avg_loss 0.009, step_time 3919.9ms, mfu 29.4%, throughput 140k tok/s, total time 1501s [2024-02-02_15:56:01]: iter 400: avg_loss 0.009, step_time 3904.1ms, mfu 29.5%, throughput 141k tok/s, total time 1600s [2024-02-02_15:57:38]: iter 425: avg_loss 0.008, step_time 3835.0ms, mfu 30.1%, throughput 143k tok/s, total time 1698s [2024-02-02_15:59:17]: iter 450: avg_loss 0.009, step_time 3931.7ms, mfu 29.3%, throughput 140k tok/s, total time 1797s [2024-02-02_16:00:56]: iter 475: avg_loss 0.008, step_time 4013.0ms, mfu 28.7%, throughput 137k tok/s, total time 1895s [2024-02-02_16:02:46]: loss estimation took 15.9 seconds. (0.8% of loop) [2024-02-02_16:02:46]: step 500: train loss 2.3693, val loss 2.3742 [2024-02-02_16:03:54]: saving checkpoint to /app/suno/checkpoints/2024-02-02_15-26-29 [2024-02-02_16:11:31]: saving took 457.6 seconds. [2024-02-02_16:11:35]: iter 500: avg_loss 0.008, step_time 544433.7ms, mfu 0.2%, throughput 1k tok/s, total time 2534s [2024-02-02_16:13:15]: iter 525: avg_loss 0.008, step_time 3827.8ms, mfu 30.1%, throughput 143k tok/s, total time 2634s [2024-02-02_16:14:54]: iter 550: avg_loss 0.010, step_time 3853.1ms, mfu 29.9%, throughput 142k tok/s, total time 2733s [2024-02-02_16:16:33]: iter 575: avg_loss 0.010, step_time 4221.8ms, mfu 27.3%, throughput 130k tok/s, total time 2832s [2024-02-02_16:18:12]: iter 600: avg_loss 0.008, step_time 3926.5ms, mfu 29.4%, throughput 140k tok/s, total time 2932s [2024-02-02_16:19:52]: iter 625: avg_loss 0.009, step_time 4303.6ms, mfu 26.8%, throughput 128k tok/s, total time 3032s [2024-02-02_16:21:33]: iter 650: avg_loss 0.010, step_time 3888.0ms, mfu 29.7%, throughput 141k tok/s, total time 3132s [2024-02-02_16:23:12]: iter 675: avg_loss 0.008, step_time 3935.7ms, mfu 29.3%, throughput 139k tok/s, total time 3231s [2024-02-02_16:24:51]: iter 700: avg_loss 0.008, step_time 3911.9ms, mfu 29.5%, throughput 140k tok/s, total time 3330s [2024-02-02_16:26:29]: iter 725: avg_loss 0.007, step_time 3888.6ms, mfu 29.7%, throughput 141k tok/s, total time 3428s [2024-02-02_16:28:08]: iter 750: avg_loss 0.009, step_time 3949.5ms, mfu 29.2%, throughput 139k tok/s, total time 3527s [2024-02-02_16:29:48]: iter 775: avg_loss 0.008, step_time 3984.6ms, mfu 28.9%, throughput 138k tok/s, total time 3627s [2024-02-02_16:31:27]: iter 800: avg_loss 0.010, step_time 4001.8ms, mfu 28.8%, throughput 137k tok/s, total time 3727s [2024-02-02_16:33:06]: iter 825: avg_loss 0.008, step_time 3964.3ms, mfu 29.1%, throughput 138k tok/s, total time 3826s [2024-02-02_16:34:47]: iter 850: avg_loss 0.008, step_time 4881.5ms, mfu 23.6%, throughput 112k tok/s, total time 3926s [2024-02-02_16:36:26]: iter 875: avg_loss 0.008, step_time 3982.0ms, mfu 29.0%, throughput 138k tok/s, total time 4026s [2024-02-02_16:38:06]: iter 900: avg_loss 0.008, step_time 3933.9ms, mfu 29.3%, throughput 140k tok/s, total time 4125s [2024-02-02_16:39:46]: iter 925: avg_loss 0.008, step_time 3936.4ms, mfu 29.3%, throughput 139k tok/s, total time 4225s [2024-02-02_16:41:26]: iter 950: avg_loss 0.006, step_time 3873.7ms, mfu 29.8%, throughput 142k tok/s, total time 4325s [2024-02-02_16:43:05]: iter 975: avg_loss 0.007, step_time 4070.2ms, mfu 28.3%, throughput 135k tok/s, total time 4425s [2024-02-02_16:44:57]: loss estimation took 15.9 seconds. (0.6% of loop) [2024-02-02_16:44:57]: step 1000: train loss 2.4376, val loss 2.4004 [2024-02-02_16:46:06]: saving checkpoint to /app/suno/checkpoints/2024-02-02_15-26-29 [2024-02-02_16:52:50]: saving took 406.8 seconds. [2024-02-02_16:52:54]: iter 1000: avg_loss 0.007, step_time 493067.6ms, mfu 0.2%, throughput 1k tok/s, total time 5013s [2024-02-02_16:54:34]: iter 1025: avg_loss 0.005, step_time 3947.5ms, mfu 29.2%, throughput 139k tok/s, total time 5113s [2024-02-02_16:56:13]: iter 1050: avg_loss 0.009, step_time 3953.2ms, mfu 29.2%, throughput 139k tok/s, total time 5212s [2024-02-02_16:57:52]: iter 1075: avg_loss 0.008, step_time 3933.4ms, mfu 29.3%, throughput 140k tok/s, total time 5312s [2024-02-02_16:59:32]: iter 1100: avg_loss 0.010, step_time 3916.9ms, mfu 29.4%, throughput 140k tok/s, total time 5411s [2024-02-02_17:01:12]: iter 1125: avg_loss 0.008, step_time 3863.9ms, mfu 29.9%, throughput 142k tok/s, total time 5511s [2024-02-02_17:02:50]: iter 1150: avg_loss 0.006, step_time 3887.1ms, mfu 29.7%, throughput 141k tok/s, total time 5610s [2024-02-02_17:04:30]: iter 1175: avg_loss 0.008, step_time 3964.9ms, mfu 29.1%, throughput 138k tok/s, total time 5709s [2024-02-02_17:06:10]: iter 1200: avg_loss 0.008, step_time 3965.5ms, mfu 29.1%, throughput 138k tok/s, total time 5809s [2024-02-02_17:07:49]: iter 1225: avg_loss 0.006, step_time 4023.2ms, mfu 28.7%, throughput 136k tok/s, total time 5908s [2024-02-02_17:09:29]: iter 1250: avg_loss 0.009, step_time 3965.9ms, mfu 29.1%, throughput 138k tok/s, total time 6008s [2024-02-02_17:11:09]: iter 1275: avg_loss 0.007, step_time 3924.0ms, mfu 29.4%, throughput 140k tok/s, total time 6108s [2024-02-02_17:12:48]: iter 1300: avg_loss 0.006, step_time 3895.1ms, mfu 29.6%, throughput 141k tok/s, total time 6208s [2024-02-02_17:14:30]: iter 1325: avg_loss 0.004, step_time 4219.0ms, mfu 27.3%, throughput 130k tok/s, total time 6309s [2024-02-02_17:16:08]: iter 1350: avg_loss 0.006, step_time 3920.3ms, mfu 29.4%, throughput 140k tok/s, total time 6408s [2024-02-02_17:17:48]: iter 1375: avg_loss 0.007, step_time 4134.7ms, mfu 27.9%, throughput 133k tok/s, total time 6508s [2024-02-02_17:19:28]: iter 1400: avg_loss 0.006, step_time 4123.4ms, mfu 28.0%, throughput 133k tok/s, total time 6607s [2024-02-02_17:21:07]: iter 1425: avg_loss 0.007, step_time 4111.6ms, mfu 28.1%, throughput 133k tok/s, total time 6707s [2024-02-02_17:22:47]: iter 1450: avg_loss 0.006, step_time 3895.4ms, mfu 29.6%, throughput 141k tok/s, total time 6806s [2024-02-02_17:24:26]: iter 1475: avg_loss 0.005, step_time 4433.8ms, mfu 26.0%, throughput 124k tok/s, total time 6906s [2024-02-02_17:26:18]: loss estimation took 15.9 seconds. (0.6% of loop) [2024-02-02_17:26:18]: step 1500: train loss 2.4797, val loss 2.4267 [2024-02-02_17:27:26]: saving checkpoint to /app/suno/checkpoints/2024-02-02_15-26-29 [2024-02-02_17:34:07]: saving took 402.5 seconds. [2024-02-02_17:34:10]: iter 1500: avg_loss 0.007, step_time 488292.1ms, mfu 0.2%, throughput 1k tok/s, total time 7490s [2024-02-02_17:35:51]: iter 1525: avg_loss 0.007, step_time 3951.4ms, mfu 29.2%, throughput 139k tok/s, total time 7590s [2024-02-02_17:37:31]: iter 1550: avg_loss 0.010, step_time 3890.2ms, mfu 29.6%, throughput 141k tok/s, total time 7691s [2024-02-02_17:39:11]: iter 1575: avg_loss 0.007, step_time 3967.5ms, mfu 29.1%, throughput 138k tok/s, total time 7791s [2024-02-02_17:40:52]: iter 1600: avg_loss 0.007, step_time 3999.8ms, mfu 28.8%, throughput 137k tok/s, total time 7891s [2024-02-02_17:42:33]: iter 1625: avg_loss 0.006, step_time 3948.1ms, mfu 29.2%, throughput 139k tok/s, total time 7992s [2024-02-02_17:44:12]: iter 1650: avg_loss 0.009, step_time 4082.9ms, mfu 28.2%, throughput 134k tok/s, total time 8091s [2024-02-02_17:45:52]: iter 1675: avg_loss 0.008, step_time 4118.9ms, mfu 28.0%, throughput 133k tok/s, total time 8192s [2024-02-02_17:47:33]: iter 1700: avg_loss 0.008, step_time 4037.4ms, mfu 28.6%, throughput 136k tok/s, total time 8292s [2024-02-02_17:49:13]: iter 1725: avg_loss 0.006, step_time 4036.6ms, mfu 28.6%, throughput 136k tok/s, total time 8393s [2024-02-02_17:50:54]: iter 1750: avg_loss 0.006, step_time 4301.9ms, mfu 26.8%, throughput 128k tok/s, total time 8493s [2024-02-02_17:52:34]: iter 1775: avg_loss 0.007, step_time 4041.6ms, mfu 28.5%, throughput 136k tok/s, total time 8594s [2024-02-02_17:54:15]: iter 1800: avg_loss 0.006, step_time 3877.0ms, mfu 29.7%, throughput 142k tok/s, total time 8695s [2024-02-02_17:55:55]: iter 1825: avg_loss 0.005, step_time 3954.7ms, mfu 29.2%, throughput 139k tok/s, total time 8794s [2024-02-02_17:57:35]: iter 1850: avg_loss 0.008, step_time 3887.8ms, mfu 29.7%, throughput 141k tok/s, total time 8895s [2024-02-02_17:59:17]: iter 1875: avg_loss 0.009, step_time 3936.2ms, mfu 29.3%, throughput 139k tok/s, total time 8996s [2024-02-02_18:00:56]: iter 1900: avg_loss 0.005, step_time 3978.1ms, mfu 29.0%, throughput 138k tok/s, total time 9096s [2024-02-02_18:02:37]: iter 1925: avg_loss 0.006, step_time 3980.8ms, mfu 29.0%, throughput 138k tok/s, total time 9197s [2024-02-02_18:04:18]: iter 1950: avg_loss 0.008, step_time 4110.1ms, mfu 28.1%, throughput 134k tok/s, total time 9297s [2024-02-02_18:06:00]: iter 1975: avg_loss 0.006, step_time 4018.8ms, mfu 28.7%, throughput 137k tok/s, total time 9400s [2024-02-02_18:07:48]: loss estimation took 15.9 seconds. (0.6% of loop) [2024-02-02_18:07:48]: step 1999: train loss 2.4714, val loss 2.5084 [2024-02-02_18:08:57]: saving checkpoint to /app/suno/checkpoints/2024-02-02_15-26-29 [2024-02-02_18:13:35]: saving took 280.0 seconds. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.[2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: iter 1999: avg_loss 0.005, step_time 366252.4ms, mfu 0.3%, throughput 1k tok/s, total time 9858s [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done. [2024-02-02_18:13:39]: done.